jvmdoc_gen: In the event of an exception, call get() on the remaining futures.

jvmdoc_gen: In the event of an exception, we want to call get() on the remaining futures because otherwise we get errors on exit when multiprocessing tries to kill them because what is dead may never die.

Ideally, we would kill them instead of waiting for them, but I don't see how to do that.

Testing Done:
./pants goal test tests/python/pants_test/tasks:jvmdoc_gen

It would be insanely difficult to write a real test for this, because the failure happens in an atexit handler.

Reviewed at https://rbcommons.com/s/twitter/r/1202/
Adds the ability to pass a filter method for ZIP extraction

One piece of the external_archive change I am working on.  I briefly looked at adding this to TAR archives and it should be possible, but I don't need it now so I didn't add it.

Testing Done:
Added new unit test.
CI running @ https://travis-ci.org/pantsbuild/pants/builds/38752080

Bugs closed: 710

Reviewed at https://rbcommons.com/s/twitter/r/1199/
Updated instructions on how to fetch code from a forked repo, how to kick of a travis-ci build from a PR

Testing Done:
ran build-support/bin/publish_docs.py and looked at the output and clicked on the links.

Bugs closed: 583, 713

Reviewed at https://rbcommons.com/s/twitter/r/1205/
Add a flexible directory re-mapper for the bundle.

Testing Done:
PANTS_DEV=1 ./pants goal test tests/python/pants_test:all
./build-support/bin/ci.sh

Bugs closed: 701

Reviewed at https://rbcommons.com/s/twitter/r/1181/
Do not crash on unicode filenames

Introduce a new function, safe_walk, which wraps os.path.walk, but
ensures that the returned values are unicode objects.  This isn't
strictly safe, in that it is possible that some paths will not be
decodeable, but that case is rare, and the only alternative is to
somehow avoid all interaction between paths and unicode objects, which
seems especially tough in the presence of unicode_literals. See e.g.
https://mail.python.org/pipermail/python-dev/2008-December/083856.html

Testing Done:
Ran new test.

Reviewed at https://rbcommons.com/s/twitter/r/1193/
Revert "Do not crash on unicode filenames"

This reverts commit 8e5131d5ec8d8be91972997620c3b8b73c6b568b.

I pushed a bogus version of this, which somehow lost the kwargs again.
Do not crash on unicode filenames

Introduce a new function, safe_walk, which wraps os.path.walk, but
ensures that the returned values are unicode objects.  This isn't
strictly safe, in that it is possible that some paths will not be
decodeable, but that case is rare, and the only alternative is to
somehow avoid all interaction between paths and unicode objects, which
seems especially tough in the presence of unicode_literals. See e.g.
https://mail.python.org/pipermail/python-dev/2008-December/083856.html

Testing Done:
Ran new test.

Reviewed at https://rbcommons.com/s/twitter/r/1193/
Remove spurious decodes (follow-up to RB 1193)

Testing Done:
Ran test_archive

Reviewed at https://rbcommons.com/s/twitter/r/1209/
Add the ability for JVM targets to refuse to allow themselves to be
cached in the artifact cache.

Sometimes, we'll later generate resources that ought to be added to an
artifact (e.g. args-apt in commons).

In the long run, the artifact cache will be able to cache resources
and this will be moot.

Testing Done:
Ran new tests.

Reviewed at https://rbcommons.com/s/twitter/r/1206/
Migrate NailgunKillall and Provides to new options.

Also fix a few bugs in Provides that cropped up while testing.

Testing Done:
Tested various flag combos.

CI baking.

Reviewed at https://rbcommons.com/s/twitter/r/1212/
Dogfood bintray for pants support binaries.

Testing Done:
Forced a local re-bootstrap of thrift and protoc:
```console
$ rm -rf ~/.pants.d
$ pants.dev goal clean-all gen testprojects/::
*** Running pants in dev mode from /home/jsirois/dev/3rdparty/jsirois-pants3/src/python/pants/bin/pants_exe.py ***

15:30:59 00:00 [main]
               (To run a reporting server: ./pants goal server)
15:30:59 00:00   [bootstrap]
15:31:00 00:01   [setup]
15:31:00 00:01     [parse] INFO] Attempting to fetch protoc binary from: https://dl.bintray.com/pantsbuild/bin/build-support/bin/protobuf/linux/x86_64/2.4.1/protoc ...
 INFO] Fetched protoc binary from: https://dl.bintray.com/pantsbuild/bin/build-support/bin/protobuf/linux/x86_64/2.4.1/protoc .
 INFO] Attempting to fetch thrift binary from: https://dl.bintray.com/pantsbuild/bin/build-support/bin/thrift/linux/x86_64/0.5.0-finagle/thrift ...
 INFO] Fetched thrift binary from: https://dl.bintray.com/pantsbuild/bin/build-support/bin/thrift/linux/x86_64/0.5.0-finagle/thrift .

               Executing tasks in goals: clean-all -> bootstrap -> imports -> gen
...
$ ~/.pants.d/bin/protobuf/linux/x86_64/2.4.1/protoc --version
libprotoc 2.4.1
$ ~/.pants.d/bin/thrift/linux/x86_64/0.5.0-finagle/thrift -version
Thrift version 0.5.0
```

CI went green here: https://travis-ci.org/pantsbuild/pants/builds/38865985

Bugs closed: 715

Reviewed at https://rbcommons.com/s/twitter/r/1208/
move codegen tasks to new options

Move codegen tasks to new options

Testing Done:
I considered adding integration tests, but for these options, but the expected behavior changes in many cases consist of side effects that are non-trivial to verify (other than thrift-linter).

For "thrift-linter", I ran the "thrift-linter" goal on the various thrift-linter targets in testprojects/. I verified that I could override the "bad-thrift-strict" target using "--no-thrift-linter-strict" and get it to succeed (and vice versa with the "bad-thrift-default" and "bad-thrift-non-strict" targets.

For "gen.scrooge", I ran the "gen" goal on tasks under examples/ and testprojects/. BTW, I could not get any output from the compiler to show up ever. I verified things were working by using the "-ldebug" flag to pants and see that it was indeed adding "--verbose" to the compiler invocation.

For "gen.protoc" and "gen.thrift", I ran the "gen" goal on tasks under testprojects/. I verified that the expected flags were passed to the thrift and proto compilers using the "-ldebug" flag.

Aside: for the "gen.protoc" target, I saw a strange result when indicating "--gen-protoc-lang=python" for a target where Java was actually needed:
  Exception message: 'PythonLibrary' object has no attribute 'jar_dependencies'
But this is unrelated to my change and happens with the code on master, too:
  $ devpants goal gen testprojects/src/protobuf/com/pants/testproject/proto-ordering:protos --gen-protoc-lang=python

Reviewed at https://rbcommons.com/s/twitter/r/1203/
Validate that registered args begin with dashes, as appropriate.

Otherwise we get a cryptic error further along.

Testing Done:
CI baking. Verified that bad registration triggers these validation checks.

Reviewed at https://rbcommons.com/s/twitter/r/1211/
Migrate more tasks, including all android tasks, to new options.

Renamed --vm-options to --jvm-options, for uniformity.

Testing Done:
Manual testing.

CI baking.

Reviewed at https://rbcommons.com/s/twitter/r/1214/
Fix reading of scalac plugin info from config.

The relevant keys are now:

[compile.scala]
plugins=['foo', 'bar']
plugin-args= {
  'foo': ['bar']
}

Testing Done:
Manually verified that the args are picked up.

Reviewed at https://rbcommons.com/s/twitter/r/1217/
Configurable Compression Level for Cache Artifacts

Allow changing (or disabling) the compression level of tarball artifacts via pants.ini

Also, require passing compression level in most init/callsites rather than allowing various default args to proliferate.

Default compression level in python gzip appears to be 9, so that is the default on TarballArtifact (replacing True).
The default when reading the config is currently 5, as we suspect that we're CPU rather than IO bound, but this change will also make it easier experiment with different values to find confirm that / find the optimal value.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/38756549
https://travis-ci.org/pantsbuild/pants/builds/38964649

Reviewed at https://rbcommons.com/s/twitter/r/1194/