clean up names in plugins.py

Summary:
- `plugin_roots` is confusing, because we use "root" for all sorts of
  stuff. Rename this broadly to `plugin_paths` a la `PATH`/`PYTHONPATH`.
- `_plugin_root_path` -> `_find_plugin_dir`
- `_plugin_manifest` -> `_find_plugin_files`
- `_plugin_invocation` -> `_get_plugin_definition`
- `PluginInvocation` and `PluginMetadata` have been combined into a
  single struct, `PluginDefinition`.

This cleanup is in preparation for the introduction of `PluginContext`.
(Name not final...) The number of arguments passed down to the plugins
functions is growing, and we need to collect all of this data into a
reasonable structure. We also want something that's easy to mock for
testing.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D66
create PluginContext

Summary:
This collects several plugin arguments into a single structure, which
can either be created by the `Runtime` or mocked in tests. Right now
this isn't a very big improvement, but we've got a bunch more parameters
coming in the near future (locks, output contexts) which will be *much*
easier to add to this struct than to tack onto our plugin function
signatures.

As far as the name goes, I'm not wedded to `PluginContext` (or
`Runtime`, for that matter). Suggestions please.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D67
move the semaphore into plugins.py

Summary:
Moving locks as close as possible to the resource they guard is a good
thing. In this case, the "resource" is job parallelism, and we can take
advantage of the new `PluginContext` to move the parallelism semaphore
all the way down to where we spawn the jobs.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D68
rename --parallel to --jobs

Summary: This is the familiar `-j` flag from Make.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D69
take locks on the cache key instead of the RemoteModule instance

Summary:
This catches the case where two modules have identical fields, so that we don't
double fetch. In retrospect, our module locks obviously should've been using
the same key as the resource they were guarding (the module cache).

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D70
make the plugin cache parallelism-safe

Summary:
We need to protect plugin caches from being operated on by more than one
job at a time. Previously we had just The Git Cache and The Hg Cache,
etc. In that world, no two git modules could safely be fetched at the
same time. To protect the caches without killing parallelism, this diff
creates more granular plugin cache dirs.

We can't key plugin cache dirs off of all module fields, because then
we'd lose the plugin cache every time any module field (like rev)
changed, and the cache would be pointless. Instead, we need the plugin
definition to tell us what fields should be included in this key. This
diff introduces the "cache fields" section to `plugin.yaml` (which
already listed "required fields" and "optional fields"). Peru will use
those fields plus the module type to generate the plugin cache key, so
the new value of `PERU_PLUGIN_CACHE` will look something like
`.peru/cache/plugins/git/[SHA1]`. (The `git/` component isn't strictly
necessary, because the key includes the type, but it seems nice to
have.) Peru now takes a new lock on the same key, to make sure no two
jobs operate on the same directory at once. So two git modules that
share the same url, for example, will share the same cache dir, but
they won't be fetched in parallel.

Plugins that don't specify "cache fields" in their `plugins.yaml` file
will now get `/dev/null` as their `PERU_PLUGIN_CACHE`. Since there's no
good default for that field (because including every module field make the
plugin cache useless), it's better to make this field mandatory for
plugins that want to do caching.

Unfortunately, the git plugin's submodule fetching doesn't fit well into
this model. Before, the git plugin knew where other git repos were
supposed to live in the cache, and it could clone submodules there. Now
it doesn't know, and even if it could guess, it wouldn't be
parallel-safe to touch those cache dirs. So the git plugin still needs
to use urlencoded subdirs, and all those submodule clones are still done
in series. Things are slightly worse than before, because two git
modules with the same submodule dependencies will have to make separate
clones. But this is kind of an edge case, so I don't think we need to
worry about making it Super Crazy Fast as long a it's correct.

Test Plan:
There's a lot of untested code in this and other recent
diffs. But with this in, our parallelism model will finally be solid,
and we can start introducing test hooks to check e.g. how many jobs
end up running at once. Future diffs.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D71
add excludes arg to read_dir()

Summary:
Our integration tests had some awful hacks to get `read_dir()` to work
in the presence of `peru.yaml` and `.peru`. Add a proper `excludes` list and
delete the hacks.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D72
refactor shared logic out of fetch and reup

Summary:
`plugin_fetch` and `plugin_get_reup_fields` had a ton of duplicated
code. Now that they manage locks as well, it's especially important to
deduplicate that. Factor out a new shared function, `_plugin_job`.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D73
fix the git plugin for empty commits

Summary:
We were hitting a checkout error when we tried to fetch an empty commit
with the git plugin:

error: pathspec '.' did not match any file(s) known to git.

Feels like a bug, but this is an edge case so who knows. Using plumbing
commands fixes the issue.

Mercurial doesn't really seem to support creating empty commits, so I
haven't tested our hg plugin yet.

Test Plan: new unit test

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D75
move run_peru_command to shared

Summary:
We're going to need this function for test_parallelism.py, and when we
split up test_integration.py. There's no real use case for
`capture_stdout=False`, so remove that as an option.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D76
split up test_integration.py into test_sync.py and test_reup.py

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D77
refactor contents checking into assert_contents()

Summary:
Many of our tests boil down to checking the contents of a directory
against an expected dict. We have read_dir() to help with this, but we
were still duplicating a lot of logic. Create `assert_contents` to
handle this kind of test, with an error message that's more helpfully
formatted than before.

Test Plan:
`assert_contents` makes sure that the test isn't failing just because
you excluded the wrong things. I checked that by hand.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D78
test_parallelism.py

Summary:
Finally write some tests that check how much parallelism is actually
going on. I initially wrote D74 to help, but that was way more
complicated than it needed to be. Instead, just count to a global
variable in `plugins.py` and write tests against that variable.

Several cases get exercised here:
- Parallelism works at all.
- `--jobs 1` turns off parallelism.
- Identical modules don't get fetched in parallel.
- Modules with the same plugin cache dir don't either.

For that last case, create the `noop_cache` plugin, which doesn't
actually do anything besides defining `cache fields` in its
`plugin.yaml`.

Test Plan:
For each of our locks (the module cache lock, the plugin cache lock, and
the parallelism semaphore), delete it, and confirm that a different test
fails each time. Also, set `DEFAULT_PARALLEL_FETCH_LIMIT` to 1 and
confirm that the first test fails.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D79
refactor svn logic into SvnRepo

Summary: This is going to be shared with the reup test.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D85
svn reup test

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D86
move clean tmp dir test logic to shared

Summary:
We like to confirm that no temp files are left over after our
integration tests. This refactor lets us share do this in the reup tests
too.

Test Plan:
Make sure we can fail both the reup and sync tests by deleting the
cleanup logic.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D87
make sure plugin env vars are always masked

Summary:
We had code for this before, but it wasn't tested. It turns out we were
setting the literal field names ('url') instead of the formatted field
names ('PERU_MODULE_URL'). We didn't notice this before, because we
were still using `os.environ.get` for our optional fields, which doesn't
fail on a missing variable. Get rid of those calls.

Test Plan:
Now that we're using `os.environ[]` everywhere, we should notice if this
breaks.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D88
[RFC] move reup output to PERU_REUP_OUTPUT

Summary:
Previously we've allowed plugin fetches to print directly to stdout, but
used the stdout of reup to get the actual reup data. This is kind of
inconsistent, and with the upcoming fancy console display, we're going
to want reup to print progress to stdout the same way fetch does. We'll
need a different channel for reup's actual data output.

Create the `PERU_REUP_OUTPUT` variable, which is defined during reup to
point to a temporary file where the plugin should write its
YAML-formatted result. Conceptually this is similar to the
`PERU_FETCH_DEST` variable already used for output during a fetch.

For now, discard the stdout from reup but allow the stderr to print to
the terminal. In the future, we'll do the same thing with stdout here
that we do in a fetch.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D89
don't catch PrintableError when verbose

Summary: Closes issue #35.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D84
delete compat.indent()

Summary: That was a python3.2 compatibility function. Now we're only >=3.3.

Test Plan: Ran `test.sh` inside a 3.3 virtualenv.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D83