Add support for running a command before tests.

This depends on https://rbcommons.com/s/twitter/r/1177/

Testing Done:
Ran new test.

Reviewed at https://rbcommons.com/s/twitter/r/1179/
Fix problem parsing stuff like options: --compile-java-args='-target 7 -source 7' in pants.rc files

Testing Done:
Ran new tests

Reviewed at https://rbcommons.com/s/twitter/r/1192/
Migrate all JvmTasks to use the new options system.

The common --jvm-options, --args, --debug and --confs flags now
live in the base class, but this required migrating all the
relevant tasks at once, hence the scope of this change.

This also gets rid of manual flag-to-config fallback, as the new
options system does this automatically. However this required some
changes to section and key names in pants.ini. I'm documenting
these changes in a spreadsheet, for smooth config migration in
pants install bases.

This required breaking a couple of inconsistently named flags, e.g.,
--benchmark-caliper-args is now just --benchmark-args.

Testing Done:
CI passed.

Manually tested that new flags and config settings work.

Reviewed at https://rbcommons.com/s/twitter/r/1186/
A simple script to detect unmigrated config keys.

Future changes that modify config keys should add the migration
to this script.

Reviewed at https://rbcommons.com/s/twitter/r/1195/
Provides documentation for some of the properties and methods in target.py

The type annotations help when using an IDE like IntelliJ

Testing Done:
There should be no code changes.  running CI

Bugs closed: 707

Reviewed at https://rbcommons.com/s/twitter/r/1196/
Print a diagnostic if a BUILD file references a source file that does not exist

Testing Done:
Edited a BUILD file, added a reference to a source file that does not exist.

```
diff --git a/src/python/pants/backend/core/BUILD b/src/python/pants/backend/core/BUILD
index 2da9682..8b1e827 100644
--- a/src/python/pants/backend/core/BUILD
+++ b/src/python/pants/backend/core/BUILD
@@ -4,7 +4,7 @@

 python_library(
   name = 'plugin',
-  sources = ['register.py'],
+  sources = ['register.py', 'does-not-exist.py'],
   dependencies = [
     'src/python/pants/backend/core/targets:all',
     'src/python/pants/backend/core/tasks:all',

```

Before this change, when running

```
PANTS_DEV=1 goal test tests/python/pants_test:all
```

the build would end with an error but not give any further information:

```
Exception message: [Errno 2] No such file or directory
```

After this change, you now get a diagnostic message that gives you a hint why the test failed:

```
 ERROR] Failed to copy src/python/pants/backend/core/does-not-exist.py for library PythonLibrary(BuildFileAddress(/Users/zundel/Src/pants/src/python/pants/backend/core/BUILD, plugin))

               FAILURE

Exception message: [Errno 2] No such file or directory
```

Reviewed at https://rbcommons.com/s/twitter/r/1198/
Fix all uncompressed artifacts stored as just '.tar'

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/38738752

Reviewed at https://rbcommons.com/s/twitter/r/1197/
jvmdoc_gen: In the event of an exception, call get() on the remaining futures.

jvmdoc_gen: In the event of an exception, we want to call get() on the remaining futures because otherwise we get errors on exit when multiprocessing tries to kill them because what is dead may never die.

Ideally, we would kill them instead of waiting for them, but I don't see how to do that.

Testing Done:
./pants goal test tests/python/pants_test/tasks:jvmdoc_gen

It would be insanely difficult to write a real test for this, because the failure happens in an atexit handler.

Reviewed at https://rbcommons.com/s/twitter/r/1202/
Adds the ability to pass a filter method for ZIP extraction

One piece of the external_archive change I am working on.  I briefly looked at adding this to TAR archives and it should be possible, but I don't need it now so I didn't add it.

Testing Done:
Added new unit test.
CI running @ https://travis-ci.org/pantsbuild/pants/builds/38752080

Bugs closed: 710

Reviewed at https://rbcommons.com/s/twitter/r/1199/
Updated instructions on how to fetch code from a forked repo, how to kick of a travis-ci build from a PR

Testing Done:
ran build-support/bin/publish_docs.py and looked at the output and clicked on the links.

Bugs closed: 583, 713

Reviewed at https://rbcommons.com/s/twitter/r/1205/
Add a flexible directory re-mapper for the bundle.

Testing Done:
PANTS_DEV=1 ./pants goal test tests/python/pants_test:all
./build-support/bin/ci.sh

Bugs closed: 701

Reviewed at https://rbcommons.com/s/twitter/r/1181/
Do not crash on unicode filenames

Introduce a new function, safe_walk, which wraps os.path.walk, but
ensures that the returned values are unicode objects.  This isn't
strictly safe, in that it is possible that some paths will not be
decodeable, but that case is rare, and the only alternative is to
somehow avoid all interaction between paths and unicode objects, which
seems especially tough in the presence of unicode_literals. See e.g.
https://mail.python.org/pipermail/python-dev/2008-December/083856.html

Testing Done:
Ran new test.

Reviewed at https://rbcommons.com/s/twitter/r/1193/
Revert "Do not crash on unicode filenames"

This reverts commit 8e5131d5ec8d8be91972997620c3b8b73c6b568b.

I pushed a bogus version of this, which somehow lost the kwargs again.
Do not crash on unicode filenames

Introduce a new function, safe_walk, which wraps os.path.walk, but
ensures that the returned values are unicode objects.  This isn't
strictly safe, in that it is possible that some paths will not be
decodeable, but that case is rare, and the only alternative is to
somehow avoid all interaction between paths and unicode objects, which
seems especially tough in the presence of unicode_literals. See e.g.
https://mail.python.org/pipermail/python-dev/2008-December/083856.html

Testing Done:
Ran new test.

Reviewed at https://rbcommons.com/s/twitter/r/1193/
Remove spurious decodes (follow-up to RB 1193)

Testing Done:
Ran test_archive

Reviewed at https://rbcommons.com/s/twitter/r/1209/
Add the ability for JVM targets to refuse to allow themselves to be
cached in the artifact cache.

Sometimes, we'll later generate resources that ought to be added to an
artifact (e.g. args-apt in commons).

In the long run, the artifact cache will be able to cache resources
and this will be moot.

Testing Done:
Ran new tests.

Reviewed at https://rbcommons.com/s/twitter/r/1206/
Migrate NailgunKillall and Provides to new options.

Also fix a few bugs in Provides that cropped up while testing.

Testing Done:
Tested various flag combos.

CI baking.

Reviewed at https://rbcommons.com/s/twitter/r/1212/
Dogfood bintray for pants support binaries.

Testing Done:
Forced a local re-bootstrap of thrift and protoc:
```console
$ rm -rf ~/.pants.d
$ pants.dev goal clean-all gen testprojects/::
*** Running pants in dev mode from /home/jsirois/dev/3rdparty/jsirois-pants3/src/python/pants/bin/pants_exe.py ***

15:30:59 00:00 [main]
               (To run a reporting server: ./pants goal server)
15:30:59 00:00   [bootstrap]
15:31:00 00:01   [setup]
15:31:00 00:01     [parse] INFO] Attempting to fetch protoc binary from: https://dl.bintray.com/pantsbuild/bin/build-support/bin/protobuf/linux/x86_64/2.4.1/protoc ...
 INFO] Fetched protoc binary from: https://dl.bintray.com/pantsbuild/bin/build-support/bin/protobuf/linux/x86_64/2.4.1/protoc .
 INFO] Attempting to fetch thrift binary from: https://dl.bintray.com/pantsbuild/bin/build-support/bin/thrift/linux/x86_64/0.5.0-finagle/thrift ...
 INFO] Fetched thrift binary from: https://dl.bintray.com/pantsbuild/bin/build-support/bin/thrift/linux/x86_64/0.5.0-finagle/thrift .

               Executing tasks in goals: clean-all -> bootstrap -> imports -> gen
...
$ ~/.pants.d/bin/protobuf/linux/x86_64/2.4.1/protoc --version
libprotoc 2.4.1
$ ~/.pants.d/bin/thrift/linux/x86_64/0.5.0-finagle/thrift -version
Thrift version 0.5.0
```

CI went green here: https://travis-ci.org/pantsbuild/pants/builds/38865985

Bugs closed: 715

Reviewed at https://rbcommons.com/s/twitter/r/1208/
move codegen tasks to new options

Move codegen tasks to new options

Testing Done:
I considered adding integration tests, but for these options, but the expected behavior changes in many cases consist of side effects that are non-trivial to verify (other than thrift-linter).

For "thrift-linter", I ran the "thrift-linter" goal on the various thrift-linter targets in testprojects/. I verified that I could override the "bad-thrift-strict" target using "--no-thrift-linter-strict" and get it to succeed (and vice versa with the "bad-thrift-default" and "bad-thrift-non-strict" targets.

For "gen.scrooge", I ran the "gen" goal on tasks under examples/ and testprojects/. BTW, I could not get any output from the compiler to show up ever. I verified things were working by using the "-ldebug" flag to pants and see that it was indeed adding "--verbose" to the compiler invocation.

For "gen.protoc" and "gen.thrift", I ran the "gen" goal on tasks under testprojects/. I verified that the expected flags were passed to the thrift and proto compilers using the "-ldebug" flag.

Aside: for the "gen.protoc" target, I saw a strange result when indicating "--gen-protoc-lang=python" for a target where Java was actually needed:
  Exception message: 'PythonLibrary' object has no attribute 'jar_dependencies'
But this is unrelated to my change and happens with the code on master, too:
  $ devpants goal gen testprojects/src/protobuf/com/pants/testproject/proto-ordering:protos --gen-protoc-lang=python

Reviewed at https://rbcommons.com/s/twitter/r/1203/
Validate that registered args begin with dashes, as appropriate.

Otherwise we get a cryptic error further along.

Testing Done:
CI baking. Verified that bad registration triggers these validation checks.

Reviewed at https://rbcommons.com/s/twitter/r/1211/