Multiprocessing Cache Check and Write

This is a slight refactor of a change we've been using internally at foursquare for a few months.

Avoiding recompression during downloads (https://rbcommons.com/s/twitter/r/1233/) reduces the amount of this work we do, while this change is about doing it faster. Most cache work is CPU-bound gzipping of the tarballs, so moving cache fetch/decompression and compression/upload to into subprocesses improves multicore utilization.

This is done using a multiprocessing.Pool, passing ArtifactCaches and the arguments to their methods to subprocesses (which is why they were made pickle-friendly in previous changes).

Additionally, we discovered (through a very painful debugging process) that python's zlib bindings (in 2.x) use a global static lock which, if in use by any thread when a subprocess is created via fork(), can be inherited by children in a locked state, meaning the child will deadlock if it later tries to use zlib.

This is why the pools are created early in run_tracker, rather than as-needed, and is discussed in the comment on SubprocPool in worker_pool.py.

Testing Done:
./pants goal test tests/python/pants_test/cache::
https://travis-ci.org/pantsbuild/pants/builds/39635717

Tested the debug message with https://gist.github.com/dt/a5d72f580b3fb9a60466 but not checking that in since wall-clock sleeps in automated tests, for just a log message, don't seem worth it.

Bugs closed: 754

Reviewed at https://rbcommons.com/s/twitter/r/1265/
copy changes and typo fixes for docs

I did a pass through the docs, running example commands and comparing command line args with help, then updating the corresponding files.

I found some of the targets referenced in the text had changed, so I updated the commands associated. Where the contents of the files had changed, I updated the text to reflect the new code.
Pin pants support down to python2.7 - dropping 2.6.

Testing the python3.3+ waters will happen seperately - this just drops support for running pants against 2.6 as agreed.

Testing Done:
CI away here: https://travis-ci.org/pantsbuild/pants/builds/40110691

Bugs closed: 722, 767

Reviewed at https://rbcommons.com/s/twitter/r/1278/
.rst files -> .md files.

re-play https://rbcommons.com/s/twitter/r/1235/ , "convert more .rst files to .md", but less crash-y. Also pull in some excellent feedback and update with doc changes that went into the .rst.

The crashy change converted many rst docs to Pants-enhanced markdown (yay) with BUILD targets (yay) with circular deps which 'goal markdown' is OK with (hmm) but which 'goal test' balks at, breaking CI (d'oh). So I reverted that change.

*This* change is like *that* change, but

* Uses the new `links` BUILD param instead of `dependencies`. Yay for no CI breaks.
* Pulls in an additition the excellent AReitz made to JVMProjects.rst meanwhile.
* Includes feedback from same excellent AReitz.
* pulls in .rst changes that came in at https://rbcommons.com/s/twitter/r/1279/

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/40211471

http://pantsbuild.github.io/sitegen-test/

Reviewed at https://rbcommons.com/s/twitter/r/1263/
Split up the travis run into a unit-test shard and multiple IT shards.

Also move tests/python/pants_test/tasks/jvm_compile/java:test_java_compile_integration
out of the unit test suite.

Testing Done:
CI went green here: https://travis-ci.org/pantsbuild/pants/builds/40216412

Bugs closed: 773

Reviewed at https://rbcommons.com/s/twitter/r/1290/
what_changed: the build graph is sorted twice even though the first iteration does not require the targets to be sorted. Removing that sort reduces the call time by about a third in a non scientific test.

Testing Done:
ci.sh passes locally

Reviewed at https://rbcommons.com/s/twitter/r/1291/
Switch all remaining accesses of 'context.options' to new options.

Killed IvyUtils's dependency on options. It only needed it for
two options that no one is using in practice and this made things
much much simpler.

Also made some methods in IvyUtil classmethods, so that call sites
don't need to instantiate it and pass it config.

Testing Done:
All task unittests passed.

CI passed.

Reviewed at https://rbcommons.com/s/twitter/r/1289/
setup_repo: now only 98% impossible

This page was full of doom and gloom: Want to start using Pants? Might as well just give up.

Walk back some of that. Point out some files that you need to copy+adjust.

This probably still doesn't capture everything folks need, but it could be a step in the right direction.

rm the howto_customize page. For a while, I thought such a page would be useful... but its material all falls into setup_repo and howto_plugin.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/40251344

Bugs closed: 632, 726, 776

Reviewed at https://rbcommons.com/s/twitter/r/1295/
Remove a bunch of setup_parser plumbing from goal/task.

reflect.py relied on the old options system via setup_parser, so
it's been broken for a while, ever since options migration began.
Fixed that by making it read from the new options.

Testing Done:
Manually ran the doc generator and inspected the resulting docs.

CI passed.

Reviewed at https://rbcommons.com/s/twitter/r/1297/
Remove some stray calls to setup_parser.

One is in help.py which is completely unused, so now deleted.

The other was in the new help printer.

Testing Done:
Verified that --help works.

CI passed.

Reviewed at https://rbcommons.com/s/twitter/r/1300/
docs: mention mvn2pants

while I was in there: setup_repo had a paragraph about top-dir BUILD files for config. LOL, nowadays, that config has moved into plugins. So pull that out, and re-jigger examples of BUILD.foo files.

Reviewed at https://rbcommons.com/s/twitter/r/1301/
Remove a superfluous test.

It only tests that Goal.parse_args works, but that function is
about to be killed itself.

Reviewed at https://rbcommons.com/s/twitter/r/1304/
Move cache logging to debug

Can get a tad noisy with several tasks all using setting up multiple caches.

Testing Done: https://travis-ci.org/pantsbuild/pants/builds/40392152

Bugs closed: 783

Reviewed at https://rbcommons.com/s/twitter/r/1310/