read-cache: be specific what part of the index has changed
cache-tree: mark istate->cache_changed on cache tree invalidation
cache-tree: mark istate->cache_changed on cache tree update
cache-tree: mark istate->cache_changed on prime_cache_tree()
entry.c: update cache_changed if refresh_cache is set in checkout_entry()

Other fill_stat_cache_info() is on new entries, which should set
CE_ENTRY_ADDED in cache_changed, so we're safe.
read-cache: new API write_locked_index instead of write_index/write_cache
read-cache: relocate and unexport commit_locked_index()

This function is now only used by write_locked_index(). Move it to
read-cache.c (because read-cache.c will need to be aware of
alternate_index_output later) and unexport it.
read-cache: store in-memory flags in the first 12 bits of ce_flags

We're running out of room for in-memory flags. But since b60e188
(Strip namelen out of ce_flags into a ce_namelen field - 2012-07-11),
we copy the namelen (first 12 bits) to ce_namelen field. So those bits
are free to use. Just make sure we do not accidentally write any
in-memory flags back.
read-cache: save index SHA-1 after reading

Also update SHA-1 after writing. If we do not do that, the second
read_index() will see "initialized" variable already set and not read
.git/index again, which is fine, except istate->sha1 now has a stale
value.
read-cache: add support for split-index mode

This split-index mode stores index info in a pair $GIT_DIR/index and
$GIT_DIR/index.ext. The intent is "index" is large and nearly static
while "index.ext" is smaller and updated often. This mode only makes
sense for _large_ index where a write could take a few hundred
miliseconds to complete. With this mode, we write small index most of
the time. Read time is another matter, to be dealt separately.

A new non-optional extension "link" is introduced which contains an
SHA-1 for validation. The rules for split-index mode are:

 - If "link" extension is present in $GIT_DIR/index, we're in
   split-index mode and index.ext must be read if present.

 - "index.ext" must contain "link" extension, which contains the SHA-1
   signature of "index". If this SHA-1 does not match the main index's,
   index.ext is ignored.

 - If "index.ext" exists, it contains all extensions. "index" can
   contain both cached entries and extensions. However in split index
   mode, extensions in "index" are ignored ("link" extension is an
   exception).

 - $GIT_DIR/index.lock is always held no matter which index is
   updated. $GIT_DIR/index.ext.lock is also held if we update
   "index.ext". index.lock is removed last.

 - We either update index.ext only, or index only when lock is being
   held (with an exception, see below). If we allow updating both
   files and the second commit fails, there's no rolling back.

 - If "index" is updated, it must absorb all contents of "index.ext"
   to itself because after the update, the its signature will change
   and invalidate "index.ext".

 - If split-index mode has not been enabled (i.e. all changes are
   contained in $GIT_DIR/index), then the transition to split-index
   could result in file updates in the same lock, because we can still
   roll back if something fails:

   + $GIT_DIR/index and $GIT_DIR/index.ext are locked in the same
     order

   + new index content is prepared in index.lock, new index.ext
     content is prepared in index.ext.lock. The new index.ext contains
     SHA-1 signature of the new index of course.

   + index.ext and index are committed in the same order. If we fail
     to commit "index", we leave garbage in index.ext behind, but that
     should not hurt correctness of the old index.

Note that $GIT_DIR/index.ext does not play nice with prepare_index()
(or "git commit -a" or "git commit <file>"). write_locked_index()
forces always updating $GIT_DIR/index in this case. Maybe in future we
could introduce GIT_INDEX_EXT_FILE to support this case.

TODO: do we ever need to always keep some extensions in the main
index? Those must be large, and change little over time to justify. We
could do that by declaring those extension names in "link" extension.
read-cache: new extension "CEUP" for fast refresh and file update

When this extension is enabled and you refresh the index, or update
SHA-1 of an entry in index (but not add or delete), the changes will
be written in $GIT_DIR/index.2nd, leaving the main (potentially large)
$GIT_DIR/index untouched. This extension is automatically enabled when
split-index is used.

This extension only exists in $GIT_DIR/index.ext. If $GIT_DIR/index is
updated, all changes in the extension become part of the cached
entries before written down.
1
read-cache: ignore name hash when adding entries on case-sensitive fs

Name hash carries a big initialization cost and it's mostly used on
case-insensitive filesystems. On case-sensitive filesystems, name hash
is never initialized. When we add something to the index, we pay the
extra cycles.

Of course name hash can benefit when adding a huge numbers of files,
but that's unlikely the common case. And in that case, indexing may
cost high enough that whether name hash is used does not make a
difference.
2
update-index: new options to enable/disable split index

Enabling split-index mode is special, it does not follow the normal
split-index update rule that either one can be updated at a
time. Instead it will write both indexes in locked state, then commit
index.ext then index. The reason for this is we'll have the main index
with dead extensions that take part in .git/index rehash.

On gentoo-x86.git (117k files, 25k dirs), "git status" after touching
one file, without split-index

    89.153ms gitmodules_config:201 if (read_cache() < 0) die("index file
     0.003ms cmd_status:1296 read_cache_preload(&s.pathspec);
   122.678ms cmd_status:1299 refresh_index(&the_index, REFRESH_QUIET
   104.844ms cmd_status:1303 update_index_if_able(&the_index, &index_lock)
     3.355ms wt_status_collect:616 wt_status_collect_changes_worktree(s)
     4.430ms wt_status_collect:621 wt_status_collect_changes_index(s)
   350.394ms wt_status_collect:622 wt_status_collect_untracked(s)

and with split-index

   108.693ms gitmodules_config:201 if (read_cache() < 0) die("index file
    22.229ms cmd_status:1296 read_cache_preload(&s.pathspec);
   126.101ms cmd_status:1299 refresh_index(&the_index, REFRESH_QUIET
    19.500ms cmd_status:1303 update_index_if_able(&the_index, &index_lock)
     3.127ms wt_status_collect:616 wt_status_collect_changes_worktree(s)
     5.236ms wt_status_collect:621 wt_status_collect_changes_index(s)
   364.873ms wt_status_collect:622 wt_status_collect_untracked(s)

The important line is update_index_if_able. Other lines are shown for
a complete "git status" timing picture. write time is reduced by
80%. read time is increased by 20%. We reduce about 60ms.

Index read time could be reduced by keeping the index in memory, in a
daemon so we don't have to rehash the entire index every time we read
it.
1
Add TIME() macro for measuring how long a block is executed

Convenient macro to measure how long a code block takes. Support
nested timing. _Not_ meant to be actually used in official code.
TIME()