dir.c: optionally compute sha-1 of a .gitignore file

This is not used anywhere yet. But the goal is to compare quickly if a
.gitignore file has changed when one has SHA-1 of the old .gitignore.
dir.c: support caching in read_directory()

read_directory() plays a big part in the slowness of "git status".
This patch adds an option to cache the results so that after the first
slow read_directory(), the following calls could be fast.

The big comment block in dir.h lists all inputs that read_directory()
needs to produce the same output. We can cheaply validate all those
inputs for any directory _if_ your file system updates directory mtime
when the directory listing is changed.

If a file is added or removed from the index, the containing directory
is invalidated (but not its subdirs). If directory's mtime is changed,
the same happens. If a .gitignore is updated, the containing directory
and all subdirs are invalidated recursively. If dir_struct#flags or
other conditions change, the cache is ignored.

If a directory is invalidated, we opendir/readdir/closedir and run the
exclude machinery on that directory listing as usual. If untracked
cache is also enabled, we'll update the cache along the way. If a
directory is validated, we simply pull the untracked listing out from
the cache. The cache also records the list of direct subdirs that we
have to recurse in. Fully excluded directories are seen as "untracked
files".

In the best case when no dirs are invalidated, read_directory()
becomes a series of stat(dir), open(.gitignore), fstat, read, close
and optionally hash_sha1_file. For comparison, standard
read_directory() is a sequence of opendir, readdir, open(gitignore),
fstat, read, close, (expensive) last_exclude_matching and closedir.

Two hash_sha1_file() are required for $GIT_DIR/info/exclude and
core.excludesfile unless their stat data matches. hash_sha1_file() is
only needed when .gitignore is modified. We could store stat data for
.gitignore so we don't have to rehash them if their content is
different from index, but I think .gitignore files are rarely
modified, so not worth extra cache data (and hashing penalty because
untracked cache will be stored in $GIT_DIR/index)

We already try not to open(gitignore) if we know it does not exist, so
open/fstat/read/close sequence does not apply to every directory. The
sequence could be reduced further, as noted in prep_exclude(). So in
theory, the entire best-case read_directory sequence could be reduced
to a series of stat() and nothing else.

This is not a silver bullet approach. When you compile a C file, for
example, the old .o file is removed and a new one with the same name
created, effectively invalidating the containing directory's cache
(but not its subdirectories). If your build process touches every
directory, this cache adds extra overhead for nothing. Editors may use
the same strategy for saving files. And of course you're out of luck
running your repo on an unsupported filesytem and/or operating system.

FIXME update cache_tree_invalidate_path()
test code
read-cache: be specific what part of the index has changed

With this knowledge, we could only update updated extensions, or the
main cache... This patch does none of that. Just mark what parts have
changed.
read-cache: save index SHA-1 after reading
Save untracked cache as an extension
status: support untracked cache
status: save untracked cache outside of the index

Logically the untracked cache is part of the index. Physically it may
reside in $GIT_DIR/index.untracked, which strictly follows index format,
except that it only contains the untracked cache. $GIT_DIR/index still
has an empty untracked cache, to note that we may need to look for
$GIT_DIR/index.untracked.

The reason is for large repositories, writing cache takes a signficant
time. Take gentoo-x86.git (117k files, 25k dirs, 10MB index) for
example, reading the index (v4) takes 131ms, writing takes 200ms. On
this repository, read_directory() takes 450ms, which is reduced to
under 10ms with untracked cache. So we could have saved 400ms, but the
writing part cuts back 200ms, half of the gain.

PS. the idea could be applied to index refresh as well.
update-index: manually enable or disable untracked cache