cat-file: refactor error handling of batch_objects

This just pulls the return value for the function out of the
inner loop, so we can break out of the loop rather than do
an early return. This will make it easier to put any cleanup
for the function in one place.
cat-file: fix a minor memory leak in batch_objects

We should always have been freeing our strbuf, but doing so
consistently was annoying until the refactoring in the
previous patch.
refs: pass flags through do_for_each_ref traversal

The do_for_each_ref iteration code does not currently accept
any option flags. It provides a void pointer for the
callback's use, but we cannot store any flags there that
would be used for the traversal itself, since the iteration
code doesn't know anything about the pointed-to type.  This
means we cannot provide any options that impact the
iteration itself (e.g., controlling the recursion depth).

This patch adds a flag parameter that gets passed through
all of the do_for_each_{ref,entry} functions, which will let
future patches implement some flags.

We also pass the flags along to the each_ref_entry_fn
callback. This lets us drop the special "flags" field
stuffed into the "ref_entry_cb". Instead we have a
single namespace for traversal flags, both for the internal
traversal code (the "do_for_each_entry" functions) and for the
internal do_for_each_ref that backs the for_each_ref
functions.
refs: teach do_for_each_* to avoid recursion

The normal do_for_each_* functions descend into sub-directories
of the ref namespace, triggering the callback for each ref
they find. However, in some cases we may want to just
iterate over a single level in the tree. With this patch,
you can do it by passing the DO_FOR_EACH_ONELEVEL flag.
get_sha1: speed up ambiguous 40-hex test

Since 798c35f (get_sha1: warn about full or short object
names that look like refs, 2013-05-29), a 40-hex sha1 causes
us to call dwim_ref on the result, on the off chance that we
have a matching ref. This can cause a noticeable slow-down
when there are a large number of objects.  E.g., on
linux.git:

  [baseline timing]
  $ best-of-five git rev-list --all --pretty=raw
  real    0m3.996s
  user    0m3.900s
  sys     0m0.100s

  [same thing, but calling get_sha1 on each commit from stdin]
  $ git rev-list --all >commits
  $ best-of-five -i commits git rev-list --stdin --pretty=raw
  real    0m7.862s
  user    0m6.108s
  sys     0m1.760s

The problem is that each call to dwim_ref individually stats
the possible refs in refs/heads, refs/tags, etc. In the
common case, there are no refs that look like sha1s at all.
We can therefore do the same check much faster by loading
all ambiguous-looking candidates once, and then checking our
index for each object.

This is technically more racy (somebody might create such a
ref after we build our index), but that's OK, as it's just a
warning (and we provide no guarantees about whether a
simultaneous process ran before or after the ref was created
anyway).

Here is the time after this patch, which implements the
strategy described above:

  $ best-of-five -i commits git rev-list --stdin --pretty=raw
  real    0m4.966s
  user    0m4.776s
  sys     0m0.192s

We still pay some price to read the commits from stdin, but
notice the system time is much lower, as we are avoiding
hundreds of thousands of stat() calls.
wip
get_sha1: drop object/refname ambiguity flag

Now that our object/refname ambiguity test is much faster
(thanks to the previous commit), there is no reason for code
like "cat-file --batch-check" to turn it off. Here are
before and after timings with this patch (on git.git):

  $ git rev-list --objects --all | cut -d' ' -f1 >objects

  [with flag]
  $ best-of-five -i objects ./git cat-file --batch-check
  real    0m0.392s
  user    0m0.368s
  sys     0m0.024s

  [without flag, without speedup; i.e., pre-25fba78]
  $ best-of-five -i objects ./git cat-file --batch-check
  real    0m1.652s
  user    0m0.904s
  sys     0m0.748s

  [without flag, with speedup]
  $ best-of-five -i objects ./git cat-file --batch-check
  real    0m0.388s
  user    0m0.356s
  sys     0m0.028s

So the new implementation does just as well as we did with
the flag turning the whole thing off (better actually, but
that is within the noise).