get rid of the toplevel 'plugins:' field

Summary:
This feature didn't turn out the way we liked. It would be neat if
modules could fetch their own plugin definitions, but that would require some
kind of early fetch, so we went with a static version that requires checking in
plugins. But even that was going to cause a lot of complexity when we enable
recursive projects, which would involve unpacking the calling tree every time a
submodule was fetched.

Instead, we're going to fall back on the simpler design, where plugins have to
be installed on the user's system before they're used, just like peru itself.
Eventually we'll add plenty more levers for this, like default plugin dirs for
the system and for the user, and some env vars to extend that.

Test Plan:
Tests related to the plugins path have been deleted. We'll need to rewrite them
when we have those levers. Arguably we should have some more tests that
directly exercise the lower level methods of plugins.py.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D63
remove the 'build' command

Summary:
`peru build` was an interesting idea, but the more we think about what it will
need to support, the more it looks like it'll require a lot of complexity
without a lot of benefit. Peru doesn't get in the way of source control because
it doesn't have to know anything about it. That's a good design decision.
`peru build` feels like the opposite, like peru needs to know all about your
build system. That looks more and more like a mistake.

`peru sync` has the nice property that it involves very little state. You're
either synced, or you aren't. You don't need to worry about *which*
dependencies you've synced, etc. Build commands, on the other hand, tend to
have a lot of state. You might want to build particular targets (`make debug`
vs `make release`) or pass any number of options to the build system
(`make -j 8`). We really don't want to wrap all this functionality, because
we'll inevitably do a crap job of it and just end up making it harder to use
your tools.

This diff gets rid of the build command entirely. For projects that want to
sync and build, the preferred approach will be to have the build scripts call
peru, rather than having peru call the build scripts. The 'build' field at the
top level will no longer have any meaning until we introduce recursive
projects. (The same was always true of 'export' and 'files', so if anything
this is more consistent.)

We'll need to think a little more about how things should work in recursive
projects. For example, if a build tool in a module calls peru again, should
that Just Work? Or should peru be aware that it's running recursively and
behave differently? Thoughts for the future.

Test Plan:
The build command integration test included some important cases of
rule chaining in local modules. Move those to a new overrides test.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D64
prefer "module" over "plugin" when describing reup

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D80
check the version of python in bin/peru

Summary:
This happens before we import the peru libs and hit a bunch of sytax
errors on older Pythons. We'll need to keep `bin/peru` backwards compatible,
but that's fine since there should never be any significant logic in there.

Test Plan:
Try to run peru from a python3.2 and python2.7. Get a clean error in
both cases.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D65
clean up names in plugins.py

Summary:
- `plugin_roots` is confusing, because we use "root" for all sorts of
  stuff. Rename this broadly to `plugin_paths` a la `PATH`/`PYTHONPATH`.
- `_plugin_root_path` -> `_find_plugin_dir`
- `_plugin_manifest` -> `_find_plugin_files`
- `_plugin_invocation` -> `_get_plugin_definition`
- `PluginInvocation` and `PluginMetadata` have been combined into a
  single struct, `PluginDefinition`.

This cleanup is in preparation for the introduction of `PluginContext`.
(Name not final...) The number of arguments passed down to the plugins
functions is growing, and we need to collect all of this data into a
reasonable structure. We also want something that's easy to mock for
testing.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D66
create PluginContext

Summary:
This collects several plugin arguments into a single structure, which
can either be created by the `Runtime` or mocked in tests. Right now
this isn't a very big improvement, but we've got a bunch more parameters
coming in the near future (locks, output contexts) which will be *much*
easier to add to this struct than to tack onto our plugin function
signatures.

As far as the name goes, I'm not wedded to `PluginContext` (or
`Runtime`, for that matter). Suggestions please.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D67
move the semaphore into plugins.py

Summary:
Moving locks as close as possible to the resource they guard is a good
thing. In this case, the "resource" is job parallelism, and we can take
advantage of the new `PluginContext` to move the parallelism semaphore
all the way down to where we spawn the jobs.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D68
rename --parallel to --jobs

Summary: This is the familiar `-j` flag from Make.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D69
take locks on the cache key instead of the RemoteModule instance

Summary:
This catches the case where two modules have identical fields, so that we don't
double fetch. In retrospect, our module locks obviously should've been using
the same key as the resource they were guarding (the module cache).

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D70
make the plugin cache parallelism-safe

Summary:
We need to protect plugin caches from being operated on by more than one
job at a time. Previously we had just The Git Cache and The Hg Cache,
etc. In that world, no two git modules could safely be fetched at the
same time. To protect the caches without killing parallelism, this diff
creates more granular plugin cache dirs.

We can't key plugin cache dirs off of all module fields, because then
we'd lose the plugin cache every time any module field (like rev)
changed, and the cache would be pointless. Instead, we need the plugin
definition to tell us what fields should be included in this key. This
diff introduces the "cache fields" section to `plugin.yaml` (which
already listed "required fields" and "optional fields"). Peru will use
those fields plus the module type to generate the plugin cache key, so
the new value of `PERU_PLUGIN_CACHE` will look something like
`.peru/cache/plugins/git/[SHA1]`. (The `git/` component isn't strictly
necessary, because the key includes the type, but it seems nice to
have.) Peru now takes a new lock on the same key, to make sure no two
jobs operate on the same directory at once. So two git modules that
share the same url, for example, will share the same cache dir, but
they won't be fetched in parallel.

Plugins that don't specify "cache fields" in their `plugins.yaml` file
will now get `/dev/null` as their `PERU_PLUGIN_CACHE`. Since there's no
good default for that field (because including every module field make the
plugin cache useless), it's better to make this field mandatory for
plugins that want to do caching.

Unfortunately, the git plugin's submodule fetching doesn't fit well into
this model. Before, the git plugin knew where other git repos were
supposed to live in the cache, and it could clone submodules there. Now
it doesn't know, and even if it could guess, it wouldn't be
parallel-safe to touch those cache dirs. So the git plugin still needs
to use urlencoded subdirs, and all those submodule clones are still done
in series. Things are slightly worse than before, because two git
modules with the same submodule dependencies will have to make separate
clones. But this is kind of an edge case, so I don't think we need to
worry about making it Super Crazy Fast as long a it's correct.

Test Plan:
There's a lot of untested code in this and other recent
diffs. But with this in, our parallelism model will finally be solid,
and we can start introducing test hooks to check e.g. how many jobs
end up running at once. Future diffs.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D71
add excludes arg to read_dir()

Summary:
Our integration tests had some awful hacks to get `read_dir()` to work
in the presence of `peru.yaml` and `.peru`. Add a proper `excludes` list and
delete the hacks.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D72
refactor shared logic out of fetch and reup

Summary:
`plugin_fetch` and `plugin_get_reup_fields` had a ton of duplicated
code. Now that they manage locks as well, it's especially important to
deduplicate that. Factor out a new shared function, `_plugin_job`.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D73
fix the git plugin for empty commits

Summary:
We were hitting a checkout error when we tried to fetch an empty commit
with the git plugin:

error: pathspec '.' did not match any file(s) known to git.

Feels like a bug, but this is an edge case so who knows. Using plumbing
commands fixes the issue.

Test Plan: new unit test

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D75
move run_peru_command to shared

Summary:
We're going to need this function for test_parallelism.py, and when we
split up test_integration.py. There's no real use case for
`capture_stdout=False`, so remove that as an option.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D76
split up test_integration.py into test_sync.py and test_reup.py

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D77
refactor contents checking into assert_contents()

Summary:
Many of our tests boil down to checking the contents of a directory
against an expected dict. We have read_dir() to help with this, but we
were still duplicating a lot of logic. Create `assert_contents` to
handle this kind of test, with an error message that's more helpfully
formatted than before.

Test Plan:
`assert_contents` makes sure that the test isn't failing just because
you excluded the wrong things. I checked that by hand.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D78
test_parallelism.py

Summary:
Finally write some tests that check how much parallelism is actually
going on. I initially wrote D74 to help, but that was way more
complicated than it needed to be. Instead, just count to a global
variable in `plugins.py` and write tests against that variable.

Several cases get exercised here:
- Parallelism works at all.
- `--jobs 1` turns off parallelism.
- Identical modules don't get fetched in parallel.
- Modules with the same plugin cache dir don't either.

For that last case, create the `cp_cache` plugin, which is just like
`cp` except that it also defines `cache fields` in its `plugin.yaml`.

Test Plan:
For each of our locks (the module cache lock, the plugin cache lock, and
the parallelism semaphore), delete it, and confirm that a different test
fails each time. Also, set `DEFAULT_PARALLEL_FETCH_LIMIT` to 1 and
confirm that the first test fails.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D79
automatically sync after a reup

Summary:
In my experience I almost always follow `peru reup` with `peru sync`, so
let's make that the default. Add a `--nosync` flag to restore the old
reup-only behavior. Also add the `--force` flag to reup, to match the
options to sync.

Test Plan:
New integration test. Also clean up the existing reup tests a bit. I
also made sync respect the `--quiet` flag. That particular logic won't
last long (because fancy output is coming), but it keeps our test output
clean.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D81
SimpleJobDisplay kind of working
BaseJobDisplay