Singleton for requests session

Instead of instance-level, use a module level singleton so that in multiproc code,
subprocesses each have their own, reusable session.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/39542444 passed

Bugs closed: 751

Reviewed at https://rbcommons.com/s/twitter/r/1264/
sitegen: get page-level table-of-contents working again

workaround for: modifying the AST to add the here-links had the side effect
that the find-every-header function no longer finds headings. So reorder
operations: find every header before adding the here-links

While adding an import to test_sitegen.py , noticed it was missing the
usual "from __future__ import", so added that.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/39653188

patched this and https://rbcommons.com/s/twitter/r/1182/ into a branch and test-generated the doc site: http://pantsbuild.github.io/sitegen-test/JVMProjects.html

Reviewed at https://rbcommons.com/s/twitter/r/1191/
Pull cache test patterns into reusable contextmanagers

Pull cache testing helper setup/teardown patterns into reusable contextmanagers

Testing Done:
./pants goal test tests/python/pants_test/cache:: passes.

https://travis-ci.org/pantsbuild/pants/builds/39634283 (in progress)

Bugs closed: 757

Reviewed at https://rbcommons.com/s/twitter/r/1267/
Add passthru arg support in the new options system.

Anything after a double-dash is captured as-is. Future changes will
plumb them through to tasks that want them.

This is useful e.g., for passing args through to a tool invoked by a task.

Testing Done:
Ran option tests. CI passed.

Reviewed at https://rbcommons.com/s/twitter/r/1270/
Refactor protobuf parsing logic out of protobuf_gen

This is in preparation of adding support for the Wire generator which also uses .proto files as sources.

Testing Done:
CI is green: https://travis-ci.org/pantsbuild/pants/builds/39766434

Bugs closed: 762

Reviewed at https://rbcommons.com/s/twitter/r/1273/
Correctly parse named_is_latest entries from the pushdb

The named_is_latest entry was being implicitly str()d on the way into the database, but was defaulting to a boolean on the way out. Strings are "truthy", so the "False" string would be treated as true.

Reviewed at https://rbcommons.com/s/twitter/r/1245/
Migrate jar_publish to new options.

Testing Done:
CI passed. Ran jar_publish's unit and integration tests. The integration tests exercise most of the flags.

It's a bit hard to test manually because parts of jar_publish are broken for other reasons. But we're pretty confident about migration changes at this point.

Reviewed at https://rbcommons.com/s/twitter/r/1276/
Migrate dependencies, depmap, detect_duplicates and check_published_deps to new options

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/39655628

Reviewed at https://rbcommons.com/s/twitter/r/1269/
Deeper selection of internal targets during publishing

The current behavior of selecting directly declared Jarable targets broke ages ago during Patrick's target refactor. In order to appropriately walk through intermediate jar_library/target declarations, we need to recurse.

It's common practice internally to have a top-level alias (a 'target' declaration) to point deeper into the directory structure to an actual java_library or scala_library. This meant that publishing was 1) ignoring those deps during fingerprint calculation and 2) not including them via DependencyWriter in written poms.

Testing Done:
test publishes; will follow up to add unit tests here if things look sane.

Reviewed at https://rbcommons.com/s/twitter/r/1213/
Add Task.workdir docs.

Testing Done:
CI went green here: https://travis-ci.org/pantsbuild/pants/builds/39909933

Bugs closed: 766

Reviewed at https://rbcommons.com/s/twitter/r/1277/
Fix DetectDuplicate's handling of an 'append'-type flag.

Testing Done:
Ran the broken tests, verified that they succeed with this change.

Reviewed at https://rbcommons.com/s/twitter/r/1282/
Switch a bunch of migrated tasks to read option values the new way.

Note that this also allowed us to get rid of some explicit fallbacks
to the config file, as new options do this automatically, assuming
the right naming conventions.

Some configs were renamed and added to the config migration script,
to align them with these conventions.

Note that this slightly changes the semantics of one flag. Previously
--extension in MarkdownToHtml would overwrite the default values of
['.md', '.markdown']. Now it appends to those two defaults. So the
default extensions will always be processed, plus whatever is added.

Testing Done:
CI pending.

Reviewed at https://rbcommons.com/s/twitter/r/1281/
Remove more stray reads from old-style options.

Also migrated jar_create to new-style options.

Also fixed up some test infrastructure.

Testing Done:
CI passed.

Reviewed at https://rbcommons.com/s/twitter/r/1286/
Multiprocessing Cache Check and Write

This is a slight refactor of a change we've been using internally at foursquare for a few months.

Avoiding recompression during downloads (https://rbcommons.com/s/twitter/r/1233/) reduces the amount of this work we do, while this change is about doing it faster. Most cache work is CPU-bound gzipping of the tarballs, so moving cache fetch/decompression and compression/upload to into subprocesses improves multicore utilization.

This is done using a multiprocessing.Pool, passing ArtifactCaches and the arguments to their methods to subprocesses (whichi is why they were made pickle-friendly in previous changes).

Additionally, we discovered (through a very painful debugging process) that python's zlib bindings use a global static lock which, if in use by any thread when a subprocess is created via fork(), can be inherited my children in a locked state meaning, the child will deadlock if it later tries to use zlib. This is why the pools are created early in run_tracker rather than as-needed, and is discussed in the comment on SubprocPool in worker_pool.py.