csum-file: make sha1write const-correct

We do not modify the buffer we are asked to write; mark it
with const so that callers with const buffers do not get
unnecessary complaints from the compiler.
strbuf: add string-chomping functions

Sometimes it is handy to cut a trailing string off the end
of a strbuf (e.g., a file extension). These helper functions
make it a one-liner.
introduce pack metadata cache files

The on-disk packfile format is nicely compact, but it does
not always provide the fastest format for looking up
information. This patch introduces the concept of
"metapacks", optional metadata files which can live
alongside packs and represent their data in different ways.
This can allow space-time tradeoffs in accessing certain
object data.

Such space-time tradeoffs have traditionally gone into the
.idx file (e.g., the fact that we can quickly find an
object's offset in the packfile is due to the index). In
theory, cached data could also go into the .idx file.
However, keeping it in a separate file makes backwards
compatibility much simpler. Older versions of git can simply
ignore the extra files and use the existing methods for
accessing object data. This also makes metapacks optional,
so you can easily tune the space-time tradeoff on a per-repo
basis.

TODO: document on-disk format in Documentation/technical
TODO: document api
introduce a commit metapack

When we are doing a commit traversal that does not need to
look at the commit messages themselves (e.g., rev-list,
merge-base, etc), we spend a lot of time accessing,
decompressing, and parsing the commit objects just to find
the parent and timestamp information. We can make a
space-time tradeoff by caching that information on disk in a
compact, uncompressed format.

TODO: document on-disk format in Documentation/technical
TODO: document API
add git-metapack command

This is a plumbing command for generating metapack files.
Right now it understands only the "commits" metapack (and
there is not yet a reader). Eventually we may want to build
this metapack automatically when we generate a new pack.
Let's be conservative for now, though, and let the idea
prove itself in practice before turning it on for everyone.

The commits metapack generated by this command is 84 bytes
per commit; for linux-2.6.git, this is about 31M.

TODO: documentation
commit: look up commit info in metapack

Now that we have the plumbing in place to generate and read
commit metapacks, we can hook them up to parse_commit to
fill in the traversal information much more quickly.

We only do so if save_commit_buffer is turned off;
otherwise, the callers will expect to be able to read
commit->buffer after parse_commit returns (and since our
cache obviously does not have that information, we must
leave it NULL). As callers learn to handle a NULL
commit->buffer, we can eventually relax this (while it might
seem like a useless no-op to use the cache if we are going
to load the commit anyway, many callers may first filter
based on the traversal, and end up loading the commit
message for only a subset of the commits).

With this patch (and having run "git metapack --all
--commits"), my best-of-five warm-cache "git rev-list
--count --all" traversal of linux-2.6.git drops from 4.219s
to 0.659s.

Similarly, cold-cache drops from 13.696s to 4.763s due to
the compactness of the cache data (but you are penalized, of
course, if you then want to actually look at the commit
messages, since you have not warmed them into the cache).
introduce a commit metapack
warning fixups for duys patch
metapack: increase the size of the "id" header

As I add more types, I find 4 characters is not very unique.
Let's make it 8, and pad out any shorter ones with NULs.
metapack: drop metapack_writer_foreach

It's so simple, it's not a big deal to inline it at the
caller. And it means the caller does not have to deal with a
callback.

While we're at it, convert the commit metapack generator to
use an array rather than a linked list, which is more
efficient (and plug a memory leak by actually freeing it),
and to split its get_commits parts out into more readable,
reusable functions.
wip for tree metapacks

I wanted to see how this performed for a reachability
traversal when used in conjunction with the commit
metapacks. But what it gives you is basically a free tree
diff between a commit and its parent. We should be able to
do the same thing with the tree-delta approach, and it will
be faster and take less space.