force the DMG file extension in git-annex.rb

For non-Mavericks installs.  Also add needed nested_container
stanza.  Fixes #3650.
Merge pull request #3651 from rolandwalker/git_annex_nested_dmg

force the DMG file extension in git-annex.rb