sequencer: do not update/refresh index if the lock cannot be held
read-cache: be strict about "changed" in remove_marked_cache_entries()

remove_marked_cache_entries() deletes entries marked with
CE_REMOVE. But if there is no such entry, do not mark the index as
"changed" because that could trigger an index update unnecessarily.
read-cache: be specific what part of the index has changed
cache-tree: mark istate->cache_changed on cache tree invalidation
cache-tree: mark istate->cache_changed on cache tree update
cache-tree: mark istate->cache_changed on prime_cache_tree()
entry.c: update cache_changed if refresh_cache is set in checkout_entry()

Other fill_stat_cache_info() is on new entries, which should set
CE_ENTRY_ADDED in cache_changed, so we're safe.
read-cache: new API write_locked_index instead of write_index/write_cache
read-cache: relocate and unexport commit_locked_index()

This function is now only used by write_locked_index(). Move it to
read-cache.c (because read-cache.c will need to be aware of
alternate_index_output later) and unexport it.
read-cache: store in-memory flags in the first 12 bits of ce_flags

We're running out of room for in-memory flags. But since b60e188
(Strip namelen out of ce_flags into a ce_namelen field - 2012-07-11),
we copy the namelen (first 12 bits) to ce_namelen field. So those bits
are free to use. Just make sure we do not accidentally write any
in-memory flags back.
read-cache: save index SHA-1 after reading

Also update SHA-1 after writing. If we do not do that, the second
read_index() will see "initialized" variable already set and not read
.git/index again, which is fine, except istate->sha1 now has a stale
value.
read-cache: add support for split-index mode

This split-index mode stores index info in a pair $GIT_DIR/index and
$GIT_DIR/index.ext. The intent is "index" is large and nearly static
while "index.ext" is smaller and updated often. This mode only makes
sense for _large_ index where a write could take a few hundred
miliseconds to complete. With this mode, we write small index most of
the time. Read time is another matter, to be dealt separately.

A new non-optional extension "link" is introduced which contains an
SHA-1 for validation. The rules for split-index mode are:

 - If "link" extension is present in $GIT_DIR/index, we're in
   split-index mode and index.ext must be read if present.

 - "index.ext" must contain "link" extension, which contains the SHA-1
   signature of "index". If this SHA-1 does not match the main index's,
   index.ext is ignored.

 - If "index.ext" exists, it contains all extensions. "index" can
   contain both cached entries and extensions. However in split index
   mode, extensions in "index" are ignored ("link" extension is an
   exception).

 - $GIT_DIR/index.lock is always held no matter which index is
   updated. $GIT_DIR/index.ext.lock is also held if we update
   "index.ext". index.lock is removed last.

 - We either update index.ext only, or index only when lock is being
   held (with an exception, see below). If we allow updating both
   files and the second commit fails, there's no rolling back.

 - If "index" is updated, it must absorb all contents of "index.ext"
   to itself because after the update, the its signature will change
   and invalidate "index.ext".

 - If split-index mode has not been enabled (i.e. all changes are
   contained in $GIT_DIR/index), then the transition to split-index
   could result in file updates in the same lock, because we can still
   roll back if something fails:

   + $GIT_DIR/index and $GIT_DIR/index.ext are locked in the same
     order

   + new index content is prepared in index.lock, new index.ext
     content is prepared in index.ext.lock. The new index.ext contains
     SHA-1 signature of the new index of course.

   + index.ext and index are committed in the same order. If we fail
     to commit "index", we leave garbage in index.ext behind, but that
     should not hurt correctness of the old index.

Note that $GIT_DIR/index.ext does not play nice with prepare_index()
(or "git commit -a" or "git commit <file>"). write_locked_index()
forces always updating $GIT_DIR/index in this case. Maybe in future we
could introduce GIT_INDEX_EXT_FILE to support this case.

TODO: do we ever need to always keep some extensions in the main
index? Those must be large, and change little over time to justify. We
could do that by declaring those extension names in "link" extension.
update-index: new options to enable/disable split index

Enabling split-index mode is special, it does not follow the normal
split-index update rule that either one can be updated at a
time. Instead it will write both indexes in locked state, then commit
index.ext then index. The reason for this is we'll have the main index
with dead extensions that take part in .git/index rehash.

On gentoo-x86.git (117k files, 25k dirs), "git status" after touching
one file, without split-index

    89.153ms gitmodules_config:201 if (read_cache() < 0) die("index file
     0.003ms cmd_status:1296 read_cache_preload(&s.pathspec);
   122.678ms cmd_status:1299 refresh_index(&the_index, REFRESH_QUIET
   104.844ms cmd_status:1303 update_index_if_able(&the_index, &index_lock)
     3.355ms wt_status_collect:616 wt_status_collect_changes_worktree(s)
     4.430ms wt_status_collect:621 wt_status_collect_changes_index(s)
   350.394ms wt_status_collect:622 wt_status_collect_untracked(s)

and with split-index

   108.693ms gitmodules_config:201 if (read_cache() < 0) die("index file
    22.229ms cmd_status:1296 read_cache_preload(&s.pathspec);
   126.101ms cmd_status:1299 refresh_index(&the_index, REFRESH_QUIET
    19.500ms cmd_status:1303 update_index_if_able(&the_index, &index_lock)
     3.127ms wt_status_collect:616 wt_status_collect_changes_worktree(s)
     5.236ms wt_status_collect:621 wt_status_collect_changes_index(s)
   364.873ms wt_status_collect:622 wt_status_collect_untracked(s)

The important line is update_index_if_able. Other lines are shown for
a complete "git status" timing picture. write time is reduced by
80%. read time is increased by 20%. We reduce about 60ms.

Index read time could be reduced by keeping the index in memory, in a
daemon so we don't have to rehash the entire index every time we read
it.
read-cache: new extension "dirc" for recording/replaying index changes

This extension is to be used in split-index mode only. The extension
is stored in "index.ext" and works like a patch to "index". It records
what entry has been added, changed or removed. At load time, those
changes are applied to the in-memory index. Like a patch, it's
supposed to be much smaller to carry around than the final output,
which means faster update time. This patch is to introduce the
extension. The real meat will be added in later patches.
split-index: save the main index listing after reading

We need to keep this to identify updated or removed entries. The
position of each index entry is also saved in the entry itself, so we
know its position in the original listing. We need this quickly locate
the entries at load time without looking them up by name.
read-cache: save deleted entries in dirc extension

References to deleted entries are in dirc->original. We just need to
not free them, and mark them deleted. And mark that entry removal
should not lead to merging index.ext back to index.
read-cache: mark new entries for dirc extension

CE_ADDED is not a good fit because it does not differentiate replaced
and added entries. Moreover it's used as a temporary flag in
unpack_trees(). Make sure entry addition does not lead to merging
index.ext back to index.
read-cache: mark replaced entries for dirc extension
read-cache: export create_from_disk() for use in dirc extension

While at there, fix constness of "ondisk" and keep the type
ondisk_cache_entry to read-cache.c only.
read-cache: add index_entry_to_strbuf()