read-cache: ask file watcher to watch files

We want to watch files that are never changed because lstat() on those
files is a wasted effort. So we sort unwatched files by date and start
adding them to the file watcher until it barfs (e.g. hits inotify
limit).

Note that we still do lstat() on these new watched files because they
could have changed before the file watcher could watch them. Watched
files may only skip lstat() at the next run.

In the future, file watcher should figure out what paths are
watchable, what not (e.g. network filesystems) and reject them. For
now it's the user resposibility to set (or unset) filewatcher.path
properly.

WIP
---

On webkit.git, normal "status -uno" takes 0m1.138s, first time with
file watcher takes 0m1.806s and about 0.6s is spent in
watch_entries(), too long. We send all paths in 214 packets and need
that many context switches (too many!). The path list is roughly 13M
sent over unix socket. gzip'd it's about 985K.
read-cache: put some limits on file watching

watch_entries() is a lot of computation and could trigger a lot more
lookups in file-watcher. Normally after the first set of watches are
in place, we do not need to update often. Moreover if the number of
entries is small, the overhead of file watcher may actually slow git
down.

This patch only allows to update watches if the number of watchable
files is over a limit (and there are new files added if this is not
the first time). Measurements on Core i5-2520M and Linux 3.7.6, about
920 lstat() take 1ms. Somewhere between 2^16 and 2^17 lstat calls that
it starts to take longer than 100ms. 2^16 is chosen at the minimum
limit to start using file watcher.

Recently updated files are not considered watchable because they are
likely to be updated again soon, not worth the ping-pong game with
file watcher. Recent limit is ignored if there are no watched files
(e.g. fresh clone). The default limit 10min is just a random value.
read-cache: get changed file list from file watcher

When some paths are watched, they are added to the "watched" list in
file watcher. When a path in this list is updated, the path is moved
to "changed" list and no longer watched.

After loading the index, we could ask for this "changed"
list. CE_WATCHED entries that are not in the list are not changed and
will have CE_VALID set. CE_WATCHED in the list will have CE_WATCHED
removed.

The "changed" list remains until told to delete something and that
only happens after the index is written down (i.e. new CE_WATCHED set
saved).
git-compat-util.h: add inotify stubs on non-Linux platforms

This is to avoid spreading #ifdef HAVE_INOTIFY around and keep most
code compiled even if it's never active.
file-watcher: inotify support, watching part

"struct dir" manages inotify file descriptor and forms a tree. "struct
file" manages a file. When a file is watched, all dirs up to the file
is watched. Any changes on a directory impacts all subdirs and files.

The way data structure is made might be inotify-specific. I haven't
thought of how other file notification mechanisms may be
implemented. So there may be some refactoring later when a new OS is
supported.
file-watcher: inotify support, notification part
file-watcher: tests for the daemon

test-file-watcher is a simplied chat program to talk to file
watcher. Typically you would write something like this

  cat >expect <<EOF
  # send "hello". Oh and this is a comment!
  <hello
  # Wait for reply and print to stdout.
  # test-file-watcher does not care about anything after '>'
  >hello
  <index foo bar
  >ok
  EOF
  test-file-watcher . <expect >actual

and test-file-watcher will execute the commands and get responses. If
all go well, "actual" should be the same as "expect". '<' and '>'
denote send and receive packets respectively. '<<' and '>>' assume the
packets are in form: "something SP PKT-LINE PKT-LINE..." and it will
encode or decode the inner PKT-LINE for you.

$GIT_TEST_WATCHER enables a few more commands for testing purposes.
The most important one is 'test-mode' where system inotify is taken
out and inotify events could be injected via test-file-watcher.

There are two debug commands in file-watcher that's not used by the
test suite, but would help debugging: setenv and log. They can be used
to turn on GIT_TRACE_PACKET then any "log" command will show, which
functions as barrier between events file watcher.

GIT_TRACE_WATCHER can also be enabled (dynamically or at startup) to
track inotify events.
file-watcher: debugging support on the client side

This is to force running the test suite with file-watcher with

git-file-watcher /tmp/

then open another terminal and run

GIT_TEST_WATCHER=2 GIT_TEST_WATCHER_PATH=/tmp/ make test

TIME_WAIT set set to unlimited by GIT_TEST_WATCHER=2 so the test suite
could hang up indefinitely due to a file-watcher bug..
Wrap CE_VALID test with ce_valid()

The next patch wants to ignore CE_VALID under some condition but not
really clears it. Centralizing its access makes such a change easier.

Not all "ce_flags & CE_VALID" is converted though. The tests that
really mean _bit_ CE_VALID remains so. The tests that mean "ignore
worktree" are converted.
read-cache: new variable to verify file-watcher results

If GIT_TEST_WATCHED is set to a non-zero value, Git still uses file
watcher if configured. But it does lstat() anyway and notifies the
user if a file is changed but the file watcher said otherwise.

Note that there is a race condition. Changed paths are retrieved at
time X, then refresh and validation at time Y. Even if X and Y are
very close, an update can happen between X and Y, causing a false
report.

If GIT_TEST_WATCHED is set greater than 1, git will abort instead of
just warn and move on. The test suite passed with GIT_TEST_WATCHED=2.