file-watcher: remove socket on exit
file-watcher: add --detach

In daemon mode, stdout and stderr are saved in $WATCHER/log.
read-cache: save trailing sha-1

This will be used as signature to know if the index has changed.
read-cache: new flag CE_WATCHED to mark what file is watched

This bit is basically "dynamic CE_VALID". It marks entries that are
being watched by the incoming file-watcher. When an index is loaded,
file watcher is contacted and the list of updated paths is retrieved.

These paths will have CE_WATCHED cleared and lstat() will be called on
them. Those that have CE_WATCHED and not in the list will have
CE_VALID turn on to skip lstat(). The setting is temporarily, CE_VALID
is not saved to disk if CE_WATCHED is also set.

We keep the CE_WATCHED in a new extension, separated from the entries
to save some space because extended ce_flags adds 2 bytes per entry
and this flag would be present in the majority of entries. When stored
as bitmap, this extension could compress very well with ewah algorithm.
read-cache: basic file-watcher hand shaking

read_cache() connects to the file watcher, specified by
filewatcher.path config, and performs basic hand shaking. CE_WATCHED
is cleared if git and file watcher have different views on the index
state.

All send/receive calls must be complete within a limited time to avoid
a buggy file-watcher hang "git status" forever. And the whole point of
doing this is speed. If file watcher can't respond fast enough, for
whatever reason, then it should not be used.

The timeout is currently set to unlimited for development purposes
though.
read-cache: ask file watcher to watch files

We want to watch files that are never changed because lstat() on those
files is a wasted effort. So we sort unwatched files by date and start
adding them to the file watcher until it barfs (e.g. hits inotify
limit).

Note that we still do lstat() on these new watched files because they
could have changed before the file watcher could watch them. Watched
files may only skip lstat() at the next run.
read-cache: put some limits on file watching

watch_entries() is a lot of computation and could trigger a lot more
lookups in file-watcher. Normally after the first set of watches are
in place, we do not need to update often. Moreover if the number of
entries is small, the overhead of file watcher may actually slow git
down.

This patch only allows to update watches if the number of watchable
files is over a limit (and there are new files added if this is not
the first time). Measurements on Core i5-2520M and Linux 3.7.6, about
920 lstat() take 1ms. Somewhere between 2^16 and 2^17 lstat calls that
it starts to take longer than 100ms. 2^16 is chosen at the minimum
limit to start using file watcher.

Recently updated files are not considered watchable because they are
likely to be updated again soon, not worth the ping-pong game with
file watcher. Recent limit is ignored if there are no watched files
(e.g. fresh clone). The default limit 30min is just a random value.
read-cache: get changed file list from file watcher

When some paths are watched, they are added to the "watched" list in
file watcher. When a path in this list is updated, the path is moved
to "changed" list and no longer watched.

After loading the index, we could ask for this "changed"
list. CE_WATCHED entries that are not in the list are not changed and
will have CE_VALID set. CE_WATCHED in the list will have CE_WATCHED
removed.

The "changed" list remains until told to delete something and that
only happens after the index is written down (i.e. new CE_WATCHED set
saved).
file-watcher: support inotify

"git diff" on webkit:

        no file watcher  1st run   subsequent runs
real        0m1.361s    0m1.445s      0m0.691s
user        0m0.889s    0m0.940s      0m0.649s
sys         0m0.469s    0m0.495s      0m0.040s
file-watcher: tests

test-file-watcher is a simplied chat program to talk to file
watcher. Typically you would write something like this

cat >expect <<EOF
<hello
>hello
<index foo bar
>ok
EOF
test-file-watcher . <expect >actual

and test-file-watcher will execute the commands and get responses. If
all go well, actual should be the same as expect. '<' and '>' denote
send and receive packets respectively. '<<' and '>>' assume the
packets are in form: "something SP PKT-LINE PKT-LINE..." and it will
encode or decode the inner PKT-LINE for you.

$GIT_TEST_WATCHER enables a few more commands for testing
purposes. The most important one is 'test-mode' where inotify is taken
out and the tests can feed inotify events to file watcher via
test-file-watcher.