pack-objects: name pack files after trailer hash

Our current scheme for naming packfiles is to calculate the
sha1 hash of the sorted list of objects contained in the
packfile. This gives us a unique name, so we are reasonably
sure that two packs with the same name will contain the same
objects.

It does not, however, tell us that two such packs have the
exact same bytes. This makes things awkward if we repack the
same set of objects. Due to run-to-run variations, the bytes
may not be identical (e.g., changed zlib or git versions,
different source object reuse due to new packs in the
repository, or even different deltas due to races during a
multi-threaded delta search).

In theory, this could be helpful to a program that cares
that the packfile contains a certain set of objects, but
does not care about the particular representation. In
practice, no part of git makes use of that, and in many
cases it is potentially harmful. For example, if a dumb http
client fetches the .idx file, it must be sure to get the
exact .pack that matches it. Similarly, a partial transfer
of a .pack file cannot be safely resumed, as the actual
bytes may have changed.  This could also affect a local
client which opened the .idx and .pack files, closes the
.pack file (due to memory or file descriptor limits), and
then re-opens a changed packfile.

In all of these cases, git can detect the problem, as we
have the sha1 of the bytes themselves in the pack trailer
(which we verify on transfer), and the .idx file references
the trailer from the matching packfile. But it would be
simpler and more efficient to actually get the correct
bytes, rather than noticing the problem and having to
restart the operation.

This patch simply uses the pack trailer sha1 as the pack
name. It should be similarly unique, but covers the exact
representation of the objects. Other parts of git should not
care, as the pack name is returned by pack-objects and is
essentially opaque.

One test needs to be updated, because it actually corrupts a
pack and expects that re-packing the corrupted bytes will
use the same name. It won't anymore, but we can easily just
use the name that pack-objects hands back.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Merge branch 'mh/fetch-tags-in-addition-to-normal-refs' into jch

The "--tags" option to "git fetch" used to be literally a synonym to
a "refs/tags/*:refs/tags/*" refspec, which meant that (1) as an
explicit refspec given from the command line, it silenced the lazy
"git fetch" default that is configured, and (2) also as an explicit
refspec given from the command line, it interacted with "--prune"
to remove any tag that the remote we are fetching from does not
have.

This demotes it to an option; with it, we fetch all tags in
addition to what would be fetched without the option, and it does
not interact with the decision "--prune" makes to see what
remote-tracking refs the local has are missing the remote
counterpart.

* mh/fetch-tags-in-addition-to-normal-refs: (23 commits)
  fetch: improve the error messages emitted for conflicting refspecs
  handle_duplicate(): mark error message for translation
  ref_remote_duplicates(): extract a function handle_duplicate()
  ref_remove_duplicates(): simplify loop logic
  t5536: new test of refspec conflicts when fetching
  ref_remove_duplicates(): avoid redundant bisection
  git-fetch.txt: improve description of tag auto-following
  fetch-options.txt: simplify ifdef/ifndef/endif usage
  fetch, remote: properly convey --no-prune options to subprocesses
  builtin/remote.c:update(): use struct argv_array
  builtin/remote.c: reorder function definitions
  query_refspecs(): move some constants out of the loop
  fetch --prune: prune only based on explicit refspecs
  fetch --tags: fetch tags *in addition to* other stuff
  fetch: only opportunistically update references based on command line
  get_expanded_map(): avoid memory leak
  get_expanded_map(): add docstring
  builtin/fetch.c: reorder function definitions
  get_ref_map(): rename local variables
  api-remote.txt: correct section "struct refspec"
  ...
Merge branch 'tr/config-multivalue-lift-max' into jch

* tr/config-multivalue-lift-max:
  config: arbitrary number of matches for --unset and --replace-all
Merge branch 'tr/doc-git-cherry' into jch

* tr/doc-git-cherry:
  Documentation: revamp git-cherry(1)
Merge branch 'cl/p4-use-diff-tree' into jch

* cl/p4-use-diff-tree:
  git p4: Use git diff-tree instead of format-patch
Merge branch 'tr/commit-slab-cleanup' into jch

* tr/commit-slab-cleanup:
  commit-slab: sizeof() the right type in xrealloc
  commit-slab: declare functions "static inline"
  commit-slab: document clear_$slabname()
Merge branch 'jn/scripts-updates' into jch

* jn/scripts-updates:
  remove #!interpreter line from shell libraries
  test: replace shebangs with descriptions in shell libraries
  test: make FILEMODE a lazy prereq
  contrib: remove git-p4import
  mark contributed hooks executable
  mark perl test scripts executable
  mark Windows build scripts executable
Merge branch 'rs/doc-submitting-patches' into jch

* rs/doc-submitting-patches:
  SubmittingPatches: document how to handle multiple patches
Merge branch 'kb/fast-hashmap' into jch

Improvements to our hash table to get it to meet the needs of the
msysgit fscache project, with some nice performance improvements.

The preparatory clean-up to submodule from Jens is at the bottom. I
also squashed in a fix-up by Karsten found at $gmane/236468 (please
double-check the result).

* kb/fast-hashmap:
  read-cache.c: fix memory leaks caused by removed cache entries
  builtin/update-index.c: cleanup update_one
  fix 'git update-index --verbose --again' output
  remove old hash.[ch] implementation
  name-hash.c: remove cache entries instead of marking them CE_UNHASHED
  name-hash.c: use new hash map implementation for cache entries
  name-hash.c: remove unreferenced directory entries
  name-hash.c: use new hash map implementation for directories
  diffcore-rename.c: use new hash map implementation
  diffcore-rename.c: simplify finding exact renames
  diffcore-rename.c: move code around to prepare for the next patch
  buitin/describe.c: use new hash map implementation
  add a hashtable implementation that supports O(1) removal
  submodule: don't access the .gitmodules cache entry after removing it
Merge branch 'cn/thin-push-capability' into jch

Allow receive-pack to insist on receiving a fat pack from "git
push" clients.

* cn/thin-push-capability:
  send-pack: don't send a thin pack to a server which doesn't support it
Merge branch 'jk/remove-deprecated' into jch

* jk/remove-deprecated:
  stop installing git-tar-tree link
  peek-remote: remove deprecated alias of ls-remote
  lost-found: remove deprecated command
  tar-tree: remove deprecated command
  repo-config: remove deprecated alias for "git config"
### 2.0
Merge branch 'jc/push-2.0-default-to-simple' into jch

* jc/push-2.0-default-to-simple:
  push: switch default from "matching" to "simple"

# Conflicts:
# 	Documentation/config.txt
# 	Documentation/git-push.txt
# 	builtin/push.c
Merge branch 'jn/add-2.0-u-A-sans-pathspec' into jch

* jn/add-2.0-u-A-sans-pathspec:
  git add: -u/-A now affects the entire working tree

# Conflicts:
# 	builtin/add.c
Merge branch 'jc/add-2.0-ignore-removal' into jch

Updated endgame for "git add <pathspec>" that defaults to "--all"
aka "--no-ignore-removal".

* jc/add-2.0-ignore-removal:
  git add <pathspec>... defaults to "-A"

# Conflicts:
# 	builtin/add.c
Merge branch 'jc/core-checkstat-2.0' into jch

* jc/core-checkstat-2.0:
  core.statinfo: remove as promised in Git 2.0
Merge branch 'gj/push-more-verbose-advice' into jch

Explain 'simple' and 'matching' in "git push" advice message; the
topmost patch is a rebase of jc/push-2.0-default-to-simple on top
of it.

* gj/push-more-verbose-advice:
  push: switch default from "matching" to "simple"
Merge branch 'jc/hold-diff-remove-q-synonym-for-no-deletion' into jch

* jc/hold-diff-remove-q-synonym-for-no-deletion:
  diff: remove "diff-files -q" in a version of Git in a distant future

# Conflicts:
# 	diff.c
### match next
Merge branch 'tr/send-email-ssl' into jch

* tr/send-email-ssl:
  send-email: set SSL options through IO::Socket::SSL::set_client_defaults
  send-email: --smtp-ssl-cert-path takes an argument
  send-email: pass Debug to Net::SMTP::SSL::new