pass data to plugins via environment variables

Summary: Pass parameters to plugins via environment variables.

Reviewers: jacko

Reviewed By: jacko

Differential Revision: https://phabricator.buildinspace.com/D48
keep git submodules from getting stale

Summary:
Previously we were never running `git fetch` in repos that had been
downloaded as submodules. So eventually when the main repo moved forward, we
would get "not a tree" errors in the submodule. Unify the `checkout_tree` logic
in the git plugin, so that it always runs a fetch if it needs to.

Fixes https://github.com/buildinspace/peru/issues/21.

Test Plan: Add a test that fails before and passes now.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D43
implement Cache.read_file()

Summary:
As we move towards allowing remote repos to define their own
`peru.yaml`, we'll need to be able to read file contents out of trees
that we have in cache, without unpacking the whole tree. This diff
creates that method.

I notice that the `_git` method in Cache is getting a little crufty. But
my ideal fix for that is replacing the whole thing with libgit2, so I'm
not worrying about it too much for now.

Test Plan: New unit test.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D46
delete an unused flag

Summary:
I forgot to remove `--delete` when we changed the syntax for
`override`.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D47
update comments in compat.py

Summary:
If we ever remove support for 3.2, we'll probably want to delete
`compat.indent`.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D50
use the pipe character for targets with named rules

Summary:
When we add recursive modules, we're probably going to want to use the
dot character `.` for namespacing. (Colon is right out, because YAML uses
that.) We need something else to represent chaining named rules with a module.
@sean suggested pipe `|`. I like that one a lot, because it evokes shell
pipelining, instead of implying that one rule is a member of another or
something.

Old syntax:
```
imports:
    foo:bar: ./path/
```

New syntax:
```
imports:
    foo|bar: ./path/
```

Test Plan: Update integration tests.

Reviewers: sean

Subscribers: sean

Differential Revision: https://phabricator.buildinspace.com/D49
make sure export_tree() recreates deleted files

Summary:
In D39, we relaxed `export_tree` to allow deleted files, but we forgot
to handle the case where those files are both deleted and unmodified between
trees. In that case, `read-tree -mu` leaves them missing, instead of recreating
them. Add an extra `checkout-index --all` to make sure these files get
recreated.

Test Plan:
Add a new test for this behavior, and confirm that it was failing
before this diff. Also add a contents check where one was missing.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D51
rename export -> copy

Summary:
Export kind of makes sense (it's an SVN command that does a similar
thing) but it's not worth the confusion with our "export" field in modules,
which is a more important concept. "copy" seems like a reasonable name that's
less confusing.

Test Plan: Update integration test.

Reviewers: sean

Reviewed By: sean

Differential Revision: https://phabricator.buildinspace.com/D53
make sure the local build tests exercise the export logic

Summary:
Export doesn't really do much in the toplevel module, *except* that if
you chain rules together, subsequent rules should use the export dir of their
predecessor as their cwd. We were doing this correctly, but it wasn't being
tested.

Test Plan:
Confirm that this test case breaks if I remove the code from
LocalModule that respects export.

Reviewers: sean

Reviewed By: sean

Differential Revision: https://phabricator.buildinspace.com/D54
make validate_third_party.sh compatible with OSX

Summary:
The `--tmp-dir` flag isn't supported. Stop using `/tmp/perutest`
entirely, because we want to create that with broad write permissions in
`tests/shared.py`, and it's best not to duplicate that logic.

Test Plan:
Run it on a Mac. It would be great to have CI that supports OSX. I
think it's possible to hack that in with Travis, but we'd need to do some
whacky stuff with branches?

Reviewers: sean

Reviewed By: sean

Differential Revision: https://phabricator.buildinspace.com/D52
add excludes to Cache.import_tree()

Summary:
When we import files from local overrides, it's very important
that we don't include the `.peru` dir, because a) that could conflict
with the toplevel one and b) the user certainly doesn't want that
anyway. We've been getting that wrong until now, because we've never
tested an overridden module with imports (which would create the
`lastimports` file, otherwise `.peru` is empty).

To fix this, we need a mechanism to exclude specific paths from the
import. (We have `files` to whitelist paths, but that doesn't work for
us here.) This diff adds that new parameter. We'll fix local modules in
a subsequent diff.

We could consider exposing `excludes` to users as a rule field. I could
see legitimate use cases for that. @sean what do you think?

Test Plan:
Turns out `files` was untested at the Cache level. Add a ton
of unit tests for `files` and `excludes`.

Reviewers: sean

Subscribers: sean

Differential Revision: https://phabricator.buildinspace.com/D55
don't import .peru from local overrides

Summary:
We need to create `.peru` in local overrides, because it keeps
track of lastimports. But we don't want to import that dir into cache.
Take advantage of the new `excludes` param on `Cache.import_tree`.

Test Plan: New integration test that fails before this diff.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D56
remove unneeded lines from the git plugin

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D62
prepare remote module caching for parallel fetching

Summary:
When peru starts to do parallel fetching, it's going to be
important that a remote module can fetch itself and its dependencies at
the same time. Previously we had functions in `resolver` that fetched and
merged imports, and the remote module would fetch itself before calling
those. This diff refactors all of that.

- Create `merge.py` to provide `merge_import_trees()`. See the large
  comment in that function for more about why its structured the way it
  is. `resolver.py` is no longer aware of anything about imports; its
  only job is now to resolve targets into trees (and occasionally
  module/rule objects).
- Refactor `LocalModule` to call `merge_import_trees`. Clean up some of
  the lastimports code while we're at it.
- Make `RemoteModule` responsible for fetching its own dependencies, so
  that we can parallelize this soon. `RemoteModule.get_tree` no longer
  caches the final unified tree; it only caches the result of the fetch.
  The rest is left to `merge_import_trees`. This has the nice side
  effect that changing a module's imports won't cause it to refetch.

Test Plan: Relying on existing integration test.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D57
support lists of imports as well as maps

Summary:
The easiest way to represent imports is as a map, but sometimes you
want to import the same module more than once. (We particularly want this in
testing.) This diff adds support for lists of imports as well as maps. Those
look like this:

```
imports:
    - foo: path/
    - bar: path/
```

We also create the `Imports` data structure so that code outside the parser
doesn't need to worry about this distinction.

Closes https://github.com/buildinspace/peru/issues/38.

Test Plan: New unit tests.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D60
add the 3.3-compatible asyncio library to third-party

Summary: We will use this to fetch remote modules in parallel.

Test Plan: `./scripts/validate_third_party.sh`

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D61
parallel plugin fetching

Summary:
Use Python 3.4's `asyncio` to do plugin fetching in parallel. Right
now we don't try to parallelize any build or cache operations; those are still
done in series to keep things simple. We also don't do anything fancy with the
console; fetches will still print as they go, though reups have been modified a
little to make sure their outputs don't interleave.

This revision drops support for Python 3.2. We use the backport of `asyncio`
to maintain support for 3.3.

Test Plan:
All existing tests are passing. Not yet sure how best to test
the parallelism.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D58
add the --parallel option to limit simultaneous fetches

Summary:
Set the default limit to 10. Use an `asyncio` semaphore to limit the
parallelism. Docopt is wonderfully flexible with flag arguments, so the user
can write any one of `-p N`, `-pN`, `--parallel N`, or `--parallel=N`.

Test Plan:
Brief testing on my apartment wifi suggests that 10 is just as fast
as 15 for small GitHub repos. A Google search confirmed that 10 is pretty.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D59
get rid of the toplevel 'plugins:' field

Summary:
This feature didn't turn out the way we liked. It would be neat if
modules could fetch their own plugin definitions, but that would require some
kind of early fetch, so we went with a static version that requires checking in
plugins. But even that was going to cause a lot of complexity when we enable
recursive projects, which would involve unpacking the calling tree every time a
submodule was fetched.

Instead, we're going to fall back on the simpler design, where plugins have to
be installed on the user's system before they're used, just like peru itself.
Eventually we'll add plenty more levers for this, like default plugin dirs for
the system and for the user, and some env vars to extend that.

Test Plan:
Tests related to the plugins path have been deleted. We'll need to rewrite them
when we have those levers. Arguably we should have some more tests that
directly exercise the lower level methods of plugins.py.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D63
remove the 'build' command

Summary:
`peru build` was an interesting idea, but the more we think about what it will
need to support, the more it looks like it'll require a lot of complexity
without a lot of benefit. Peru doesn't get in the way of source control because
it doesn't have to know anything about it. That's a good design decision.
`peru build` feels like the opposite, like peru needs to know all about your
build system. That looks more and more like a mistake.

`peru sync` has the nice property that it involves very little state. You're
either synced, or you aren't. You don't need to worry about *which*
dependencies you've synced, etc. Build commands, on the other hand, tend to
have a lot of state. You might want to build particular targets (`make debug`
vs `make release`) or pass any number of options to the build system
(`make -j 8`). We really don't want to wrap all this functionality, because
we'll inevitably do a crap job of it and just end up making it harder to use
your tools.

This diff gets rid of the build command entirely. For projects that want to
sync and build, the preferred approach will be to have the build scripts call
peru, rather than having peru call the build scripts. The 'build' field at the
top level will no longer have any meaning until we introduce recursive
projects. (The same was always true of 'export' and 'files', so if anything
this is more consistent.)

We'll need to think a little more about how things should work in recursive
projects. For example, if a build tool in a module calls peru again, should
that Just Work? Or should peru be aware that it's running recursively and
behave differently? Thoughts for the future.

Test Plan:
The build command integration test included some important cases of
rule chaining in local modules. Move those to a new overrides test.

Reviewers: sean

Differential Revision: https://phabricator.buildinspace.com/D64