Testing: add --pause-upgrade-retry argument

Similar to the --pause-upgrade-loop argument, which loops indefinitely,
but only applies when a test fails, and retries the failed test after
upgrading (until it succeeds.)

The new argument implies --pause-on-failure.
Add proper, documented, read-only API

This commit does not add a complete API, only the beginnings of one.  It
also doesn't add any uses of the API.  Largely, the added API duplicates
older internal (more or less messy and undocumented) APIs.  The long-term
plan is to remove the duplication by using the new API where possible
(where a read-only view is sufficient) and by restructuring other code.

API design:

- There's a special "session" object of the class api.critic.Critic,
  typically called 'critic'.  It is used throughout the API as a reference
  to all global/per-session stuff, like the database connection.  (In this
  preliminary API version, it contains only a reference to the database
  connection.)  Most API objects have a data attribute 'critic' that
  returns this object, and most functions used to create API objects take
  the object as their first parameter.

- Most API objects are split into two, one "public" that defines the
  interface and one internal that contains the actual implementations.
  The interface definition contains class definitions, method signatures,
  documentation, and typically various assertions checking that the
  provided arguments are sane (which is also a form of documentation.)

- All data attributes on API objects are properties with a getter and no
  setter (thus read-only.)  The getter fetches the value from the
  implementation part of the object, possibly by calling a method that
  calculates the value.

- Data is exposed as data attributes whenever possible, even if the value
  is "complex" and is calculated upon reading.  If reading is expensive,
  the value will typically be calculated on demand and cached.

- Data is exposed via methods only when is makes sense to provide one or
  more arguments to the method.  (It's obviously impossible to provide
  arguments when accessing a data attribute, hence needing a method.)
Fix the C/C++ lexer's handling of byte order marks

By not supporting byte order marks, it ended up (as a fallback) treating
each character in the (UTF-8 encoded) byte order mark as an operator,
wrapping it in <b class="op">X</b>.  This of course meant the result was
invalid UTF-8, which would cause crashes when fetched via /fetchlines,
and incorrect rendering otherwise.
Fix number of replies for draft comments on /showreview

The internal number includes the initial comment, while the displayed
number should only include additional comments (i.e. replies.)  The
internal number also doesn't include any draft comments, not even those
by the user loading the page.  When combined, this leads to an internal
number of comments of zero for draft comments, which was adjusted to
minus one (to avoid counting the initial comment as a reply.)

To avoid displaying "-1 reply", display any internal number less than two
as "No replies".
Fix "Work log" batch summary for reviewed/unreviewed lines

Instead of saying "reviewed 10 lines" when the batch actually marked 10
lines worth of changes as reviewed, we said so when the batch changed
the reviewed state of 10 lines worth of changes that are currently
reviewed.  In other words, any set of batches that changed the reviewed
state of the same changes would always say the same thing, "reviewed
N lines" or "unreviewed N lines", depending on the current state of the
changes.

Fix this by looking at the reviewfilechanges.to(_state) column instead
of the reviewfiles.state column when categorizing work done by a batch.
Make sure a new repository's path is at least one character long

Otherwise, forgetting to input a path will create a repository with the
path /var/git/.git (typically) which is unhelpful.
Make extension and extension.{manifest,extension,resource} independent

That is, make it possible to import without an installed system.
Meaningless in itself, but convenient to have the possibility.
Redesign service manager start-up synchronization

When starting the service manager via the SysV init script (or possible
other future mechanism) we wish to wait until all services have actually
been started before "returning."  (This primarily to avoid races when
testing.)

This was previously done by waiting for each configured service's pid file
to be created.  That approach does not work if some configured service
decides to stop immediately after having determined that it need not run.
It also doesn't handle gracefully the case of some configured service
crashing immediately.  In both of these cases, the waiting mechanism would
simply wait until its timeout was reached, and not say anything or fail.

The new mechanism is that before starting some service, we create a
temporary file named "<pidfile_path>.starting", which the service deletes
after having performed its start-up work (for instance started listening
to its command socket) and before it goes into "normal operation" mode.

To implement this, add a new method BackgroundProcess.start() which should
be used to start a service rather than calling run() directly.  The start()
method calls the startup() method, then deletes the temporary file, then
calls run() and finally calls shutdown() just before exiting.
Add support for FilterHook extension roles

A FilterHook extension role allows an extension to implement additional
filter types (in addition to the native filter types "reviewer", "watcher"
and "ignored".)

When a user has installed the extension and has added such a filter, and
the filter matches changes in commits added to a review (including when
the review is first created) the extension role's function is called
(asynchronously) with the arguments "filter data", the Review object, a
CommitSet representing the added commits, and a list of File objects
representing the set of matched files that were touched by those commits.

The "filter data" is a string value that the user sets as part of the
filter (in the "Add filter" dialog) if the extension has requested it.
The extension requests it by defining the "DataDescription" setting in the
role's section in the MANIFEST file.  (If no such description is given,
the user is not presented with an input field for the "filter data" value,
and the value passed to the role's function is always null.)
Detect errors in the unprivileged part of extend.py

If run as root, extend.py starts a child process, in which privileges are
dropped, to run the --fetch and --build stages.  Failures were not
propagated from this child process, meaning the overall process continued
anyway, most likely doing the wrong thing.
Use the correct SHA-1 when constructing v8-jsshell cache file path

The current sub-module SHA-1 checked out in the repository in which the
tests were run was used, but this is wrong if --commit is used to override
the tested commit, and also wrong if extend() runs before upgrade().

Keep track of which commit is current installed as 'self.current_commit',
and use that when constructing the cache file path.
Check if extend.py exists in the correct commit in check_extend()

Add 'self.tested_commit' which is the same as 'self.upgrade_commit' if
we're upgrading, and otherwise 'self.install_commit', and thus the commit
we're ultimately testing against.

Use this instead of 'self.install_commit' as the commit to check for
extend.py existence, unless 'pre_upgrade' is true.
Run "git submodule update --recursive" before upgrade.py

Necessary when upgrading past a commit that changes a sub-module pointer,
since upgrade.py otherwise complains about the repository having local
modifications.
Disable v8-jsshell caching if upgrading the v8-jsshell sub-module

If the v8-jsshell sub-module is modified in an upgrade, upgrade.py runs
extend.py, and "extend.py --fetch" fails if we used a cached v8-jsshell,
since the sub-module directory isn't empty then (we copy the cached
executable into it.)

It's meaningful to test the (automatic) incremental recompilation of
v8-jsshell that happens as part of the upgrade, so simply disable the
caching in this case.

This means that the testing of such commits becomes quite expensive, but
this should be sufficiently offset by the fact that sub-module updates
are rare.
Propagate --headless argument from upgrade.py to extend.py
Pass --libcurl-flavor=gnutls argument to extend.py when testing

This allows better testing coverage (the default is to not install a
libcurl flavor, meaning the v8-jsshell executable has no URL loading
support.)

Also, the libcurl-gnutls-dev package is typically installed in the VMs
used for testing when running with the "with-prereqs" snapshot.
Redirect "git cat-file --batch" process's stderr

Not doing so leads to a situation where Communicate.run() (communicate.py)
ends up waiting for the stdout/stderr pipes to EOF indefinitely, since the
Git process keeps the stderr pipe (which it then inherited from the parent
extension process) open.  This combined with an unclean extension process
shutdown where the Git process was not terminated would lead to a sort of
deadlock, where Communicate.run() waited until it timed out.

Nothing it is typically written to stderr; "git cat-file --batch" writes
normal error messages ("missing object" and such) to stdout.
Fix leaking "git cat-file --batch" processes

When an extension process shuts down "uncleanly" (i.e. without killing its
"git cat-file --batch" processes first,) those Git processes where left
alive indefinitely.  The reason for that was that a reference to the
writing end of the pipe used as Git's stdin was inherited into the child
process and thus never fully closed.

Avoid this by setting the FD_CLOEXEC flag on the file descriptor.

Also update the v8-jsshell submodule pointer to the commit that adds the
IO.File.setCloseOnExec() method that we use to do that.
Improve KeyboardInterrupt handling in testing framework

Testing should be aborted.  But in practice (with recent changes) only
the current top-level test group was aborted.

Also, the "Testing aborted." error message was logged immediately, and
then followed by various clean-up logging, and the test summary (which
included the "error" that testing was aborted in the "Errors logged"
counter.)  Re-arrange so that there is no test summary, and so that the
"Testing aborted." error message is the last thing logged, for clarity.
Remove bzip2 compression of database dump

When we make a backup database dump as part of upgrade (default when the
database schema will be updated) we compress the dump using bzip2.  This
is pretty silly, since we tell pg_dump to use its "custom" format, which
is already compressed.  The additional compression achieved by bzip2 is
marginal at best, and mostly just a waste of CPU time.