read-cache: ask file watcher to watch files

We want to watch files that are never changed because lstat() on those
files is a wasted effort. So we sort unwatched files by date and start
adding them to the file watcher until it barfs (e.g. hits inotify
limit). Recently updated entries are also excluded from watch list.
CE_VALID is used in combination with CE_WATCHED. Those entries that
have CE_VALID already set will never be watched.

We send as many paths as possible in one packet in pkt-line format to
reduce roundtrips. For small projects like git, all entries can be
packed in one packet. For large projects like webkit (182k entries) it
takes two packets. We may do prefix compression as well to send more
in fewer packets..

The file watcher replies how many entries it can watch (because at
least inotify has system limits).

Note that we still do lstat() on these new watched files because they
could have changed before the file watcher could watch them. Watched
files may only skip lstat() at the next git run.
read-cache: put some limits on file watching

watch_entries() is a lot of computation and could trigger a lot more
lookups in file-watcher. Normally after the first set of watches are
in place, we do not need to update often. Moreover if the number of
entries is small, the overhead of file watcher may actually slow git
down.

This patch only allows to update watches if the number of watchable
files is over a limit (and there are new files added if this is not
the first time). Measurements on Core i5-2520M and Linux 3.7.6, about
920 lstat() take 1ms. Somewhere between 2^16 and 2^17 lstat calls that
it starts to take longer than 100ms. 2^16 is chosen at the minimum
limit to start using file watcher.

Recently updated files are not considered watchable because they are
likely to be updated again soon, not worth the ping-pong game with
file watcher. The default limit 30min is just a random value.
read-cache: get modified file list from file watcher

A new command is added to file watcher to send back the list of
updated files to git. These entries will have CE_WATCHED removed. The
remaining CE_WATCHED entries will have CE_VALID set (i.e. no changes
and no lstat either).

The file watcher does not cache stat info and send back to git. Its
main purpose is to reduce lstat on most untouched files, not to
completely eliminate lstat.

The file watcher keeps reporting the same "updated" list until it
receives "forget" commands, which should only be issued after the
updated index is written down. This ensures that if git crashes half
way before it could update the index (or multiple processes is reading
the same index), "updated" info is not lost.

After the index is updated (e.g. in this case because of toggling
CE_WATCHED bits), git sends the new index signature to the file
watcher.
read-cache: add config to start file watcher automatically
read-cache: add GIT_TEST_FORCE_WATCHER for testing

This can be used to force watcher on when running the test
suite.

git-file-watcher processes are not automatically cleaned up after each
test. So after running the test suite you'll be left with plenty
git-file-watcher processes that should all end after about a minute.
file-watcher: add --shutdown and --log options
file-watcher: automatically quit

If $GIT_DIR/index.watcher or $GIT_DIR/index is gone, exit. We could
watch this path too, but we'll waste precious resources (at least with
inotify). And with inotify, it seems to miss the case when $GIT_DIR is
moved. Just check if the socket path still exists every minute.

As the last resort, if we do not receive any commands in the last 6
hours, exit. The code is structured this way because later on inotify
is also polled. On an busy watched directory, the timeout may never
happen for us to kil the watcher, even if index.watcher is already
gone.

For mass cleanup, "killall -USR1 git-file-watcher" asks every watcher
process to question the purpose of its existence.
file-watcher: support inotify

"git diff" on webkit:

        no file watcher  1st run   subsequent runs
real        0m1.361s    0m1.445s      0m0.691s
user        0m0.889s    0m0.940s      0m0.649s
sys         0m0.469s    0m0.495s      0m0.040s
file-watcher: exit when cwd is gone

t7406-submodule-update somehow triggers this, the worktree is probably
moved or something.
pkt-line.c: increase buffer size to 8192

test "very long name in the index handled sanely" in t0000 stretches
it a bit with a pathname of 4096 char length. Because we pass paths to
the file watcher using pkt-line protocol, it dies with "protocol
error: impossibly long line". Increase the buffer size a bit.
t1301: exclude sockets from file permission check

$GIT_DIR/index.watcher violates the rules with perm srwxrwxr-x.