Arcyd: refactor, extract abdi_processrepos

Move the logic of processing repositories out of abdcmd_processrepos and
into a new module so that it can be shared with the new 'arcyd start'
command.

Test Plan:
$ ./precommit.sh
Arcyd: fix repo paths for dashboard

Fix the parsing of the repo configs during 'arcyd start' to match the
expected input for 'arcyd process-repos'.

Fix the paths to the '.ok' and '.try' files to match the expected by the
reporter and dashboard code. In later work we'll want to make the
reporter and dashboard re-use the same code so the paths are identical.

Test Plan:
$ ./precommit.sh
$ ./testbed/arcyd/acceptance_tests.sh
$ ./testbed/arcyd/test_shell.sh

Conflicts:
	py/abd/abdcmd_start.py
Arcyd: move root config file to 'configfile'

Change the path of the root config file to 'configfile' to make way for
a 'config' directory.  Update abdcmd_start to get the name of the config
file from abdt_fs instead of hard-coding it.

Test Plan:
$ ./precommit.sh
Arcyd: move config files to new 'config' dir

Tidy the root directory by making a new directory hierarchy for
configuration files, separating 'phabricator' from 'repository'.

Simplify abdcmd_start by moving the config file discovery logic out to
abdt_fs.

Test Plan:
$ ./precommit.sh
$ ./testbed/arcyd/acceptance_tests.sh
$ ./testbed/arcyd/test_shell.sh
Arcyd: remove tests for refs/config/arcyd

We're not using the refs/config arcyd feature at the moment and testing
it is going to get in the way of the new CLI, which doesn't support it
yet.

Test Plan:
$ ./testbed/arcyd/acceptance_tests.sh
Arcyd: BREAK, remove 'arcyd process-repos'

Remove the 'arcyd process-repos' command now that 'arcyd start' has as
much feature parity as we want.  Replace all usage of 'process-repos' in
test scripts with the new CLI commands.

Maintaining both will be more expensive going forward, so we'll remove
it now.

Existing instances will need to be re-created using the new CLI commands
built for that purpose: init, add-phabricator, add-repo, start, stop

Test Plan:
$ ./precommit.sh
$ ./testbed/arcyd/loop_test.sh
$ ./testbed/arcyd/test_shell.sh
$ ./testbed/arcyd/acceptance_tests.sh
Arcyd: move 'review_url_format' to phab config

Reduce repetition in configuring repositories by moving the
'--review-url-format' argument from 'add repo' to the 'add-phabricator'
command.

Fix up existing references to these commands in tests.

Test Plan:
$ ./precommit.sh
$ ./testbed/arcyd/loop_test.sh
$ ./testbed/arcyd/test_shell.sh
$ ./testbed/arcyd/acceptance_tests.sh
Arcyd: BREAK, introduce 'arcyd add-repohost'

Add a new kind of config entity, the 'repohost', which allows
repositories to share configuration of:

- repo url
- repo snoop url
- branch url

This reduces error-prone repetition of tricky strings and makes it much
easier to change configurations later en-masse if necessary.

Make --repo-url a required argument of repositories, so it's possible
to introduce a '--repo-url-format' argument that the full repo url
can be derived from.

e.g.
  --repo-url myorg/myrepo
  --repo-url-format 'https://github.com/{}.git'
will expand to:
  "https://github.com/myorg/myrepo.git"

Pass the new --repo-url argument through the --branch-url-format
string as well so that can be re-used.

e.g.
  --branch-url-format
  'http://my.git/gitweb?p={repo_url}.git;a=log;h=refs/heads/{branch}'
would expand to
  'http://my.git/gitweb?p=myrepo.git;a=log;h=refs/heads/mybranch'

Remove --repo-snoop-url in favour of --repo-snoop-url-format, so that
can be re-used as above.

Convert tests to use new 'arcyd add-repohost' command.

Test Plan:
$ ./precommit.sh
$ ./testbed/arcyd/loop_test.sh
$ ./testbed/arcyd/test_shell.sh
$ ./testbed/arcyd/acceptance_tests.sh
Arcyd: refactor, remove per-repo '--sleep-secs'

Remove the '--sleep-secs' arg from the per-repository argument list, it
is not connected to anything.

Test Plan:
$ ./precommit.sh
$ ./testbed/arcyd/loop_test.sh
$ ./testbed/arcyd/test_shell.sh
$ ./testbed/arcyd/acceptance_tests.sh
Arcyd: BREAK, move '--arcyd-email' arg to init

Move the '--arcyd-email' argument to the 'arcyd init' command instead
of the 'arcyd add-repo' command.

This reduces redundant typing on the far more commonly called
'arcyd add-repo' command.

Add a new attribute to ArcydReporter so it can carry the email address
where it needs to go.  In later work it should actually use that address
itself.

Adjust existing tests to use the new interface.

Test Plan:
$ ./precommit.sh
$ ./testbed/arcyd/loop_test.sh
$ ./testbed/arcyd/test_shell.sh
$ ./testbed/arcyd/acceptance_tests.sh
Arcyd: allow multiple repo admin emails

Allow multiple repo admin email addresses and add support for them to be
specified on any combination of:

- the phabricator instance config
- the repohost config
- the repo config

Raise if a repository has no admin email addresses during setup and
arcyd startup.

Change the name of the admin email argument to '--admin-emails', a quirk
of argparse means that the '--admin-email' argument is still supported.

Adjust test_shell.sh to use multiple emails from multiple sources.

Test Plan:
$ ./precommit.sh
$ ./testbed/arcyd/loop_test.sh
$ ./testbed/arcyd/acceptance_tests.sh

try different combinations of emails in
$ ./testbed/arcyd/test_shell.sh
Arcyd: update usage example with new CLI

Update the 'arcyd --help' usage example to reference the new CLI, this
should hopefully make it easier to get started.

Test Plan:
run through usage exmaple in a temp dir to verify
Arcyd: BREAK, 'arcyd add-repo' has positional args

Make the add-repo command easier to use by changing phabricator,
repohost and url parameters to be positional, so that they don't have to
be prefixed with lengthy identifiers, e.g. '--phabricator-name'.

Add sensible defaults for 'repo_name' and 'repo_description' based on
the positional arguments.

This means that instead of an invocation like:

    $arcyd add-repo \
        --name localhost_fs_myrepo \
        --repo-desc 'localhost :: fs :: myrepo' \
        --phabricator-name localhost \
        --repohost-name fs \
        --repo-url origin

you can use a single line, like:

    $ arcyd add-repo localhost fs myrepo

which seems considerably easier.

Test Plan:
$ ./precommit.sh
$ ./testbed/arcyd/loop_test.sh
$ ./testbed/arcyd/test_shell.sh
$ ./testbed/arcyd/acceptance_tests.sh
Arcyd: move signal handler setup to arcyd start

To better ensure graceful exit, register the sigterm handler at the
start of Arcyd's start routing instead of during abdi_processrepos.

Note that arcyd instaweb doesn't seem to exit if it's processing when
sigterm is received - something is catching the exception.  If it wasn't
for that then we'd probably want to cover everything from Arcyd's main()
routine.

Test Plan:
$ ./precommit.sh
Arcyd: BREAK, daemonize 'arcyd start' by default

Users will generally want Arcyd to be a long-running process that isn't
tied to any particular terminal or session.  To that end, make
'arcyd start' launch as a daemon by default, with stdout and stderr
redirected to ./var/log in the arcyd dir.

Provide a '--foreground' option for preventing the daemonization, useful
during tests and if a third-party solution for daemonization is to be
used instead.

Test Plan:

Update existing tests to make use of new functionality.

$ ./precommit.sh
$ ./testbed/arcyd/test_shell.sh
$ ./testbed/arcyd/exercise_arcyd.sh
$ ./testbed/arcyd/acceptance_tests.sh
Add abdi_repo to manage arcyd's repositories

Simplify abdcmd_addrepo by moving it's cloning logic out to a 'setup'
context manager that will automatically clean up the repository if any
of the setup steps fails.

Test Plan:
$ ./precommit.sh
Arcyd: add 'fsck' command for fixing up instances

Introduce a command for checking the files in an Arcyd instance for
consistency and automatically fix some issues.

This is particularly useful for restoring the repositories under var/
to an Arcyd instance after cloning it from an Arcyd repository.
This is generally how you would restore an Arcyd instance from a
backup, since Arcyd stores all config in a local Git repo.

Propagate return values from Arcyd subcommands to the command-line so
'arcyd fsck' can exit with other than 0.

Test Plan:
$ ./precommit.sh

remove repos from /var and restore in
$ ./testbed/arcyd/test_shell.sh