RESTfulCache delegates tarfile handling to LocalArtifactCache

The motivation for this change is to allow artifacts downloaded by RESTfulCache to be directly added to the LocalArtifactCache without needing to decompress and recompress them as we do now.

At a high level, this is done by refactoring RESTfulCache such that it no longer handles creating or extracting tarfiles directly and instead delegates those operations to a LocalArtifactCache instance passed in its constructor, which may store them in the process.

This is a revision of a previous proposal (https://rbcommons.com/s/twitter/r/985/) along similar lines, though this iteration has RESTful artifact cache always passed a LocalArtifactCache at construction, to which it can always delegate all local file interaction (tarfile creation and extraction)

This simplifies RESTful cache, reducing it to only the HTTP-related functionality. This also eliminates duplication of tarfile creation and extraction logic, which now justs lives in the LocalArtifactCache.

TempLocalArtifactCache provides an alternate implementation of LocalArtifactCache, backed by tmpfiles instead of a cache_root. This can be passes to RESTfulCache to handle artifacts when running in a remote-only cache configuration.

Notes:

This change removes all usage of CombinedCache -- the only case where it was used,
composing an HTTP and local cache, is covered by the above changes.

It also removes the  "backfill" behavior from CombinedCache, which only made sense in the above setup: two remote caches should likely use rsyncd or whatever to share files rather than relying on clients building to shuttle artifacts between them and local caches can use symlinks or mounts.

Caches also now use stdlib logging instead of context.log since the latter
has references to the runtracker making it almost impossible to pickle.

Caches now use string.format instead of % for composing log messages.

Add more Sphinx style docstrings.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/39519073 passed

Bugs closed: 737

Reviewed at https://rbcommons.com/s/twitter/r/1233/
new param for page target, links. List of hyperlinked-to targets

List of hyperlinked-to targets. We use dependencies for this now.
But circular dependencies are an error while hyperlink-cycles should be OK.

So param page(links=[...]) feeds into traversable_specs (versus traversable_dependency_specs).

While I was looking at the tests, un-commented-in some commented-out assertions that had a done TODO.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/39520576

Bugs closed: 253, 742

Reviewed at https://rbcommons.com/s/twitter/r/1242/
Migrate some utility tasks to new-style options.

Reviewed at https://rbcommons.com/s/twitter/r/1248/
fix [run.junit] -> [test.junit]

Testing Done:
https://travis-ci.org/jinfeng/jinfeng-pants-fork/builds/39534255

Reviewed at https://rbcommons.com/s/twitter/r/1256/
fix the error msg in favor of the new style arg name

Reviewed at https://rbcommons.com/s/twitter/r/1258/
WikiArtifact instances also have provides; limit ivy to jvm

This should never get hit unless you accidentally do goal compile on a
wiki -- but maybe you do compile ...:: and hit it.

Testing Done:
Ran tests.

Reviewed at https://rbcommons.com/s/twitter/r/1259/
Refactor artifact cache exceptions.

Presently, the artifact cache will treat any exception during the
insertion process as a non-fatal error.  But some errors should be
fatal -- any sort of misconfiguration, or an attempt to cache
nonexistent files, or an exception thrown while moving files around.
Also remove ArtifactCache.CacheError, as it was not clear when
to use ArtifactCache.CacheError and when to use ArtifactCacheError.

Reviewed at https://rbcommons.com/s/twitter/r/1261/
included code snippets: don't just slap in a pre. syntax highlighting

The pygments code-syntax-highlighting code has some built-in functionality to guess at a code snippet's language based on filename and content. We hadn't hooked it up. This hooks it up.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/39542386

patched this and https://rbcommons.com/s/twitter/r/1191/ into a branch and test-generated the doc site: http://pantsbuild.github.io/sitegen-test/JVMProjects.html

Bugs closed: 693

Reviewed at https://rbcommons.com/s/twitter/r/1182/
Fix error message: add missing space

Reviewed at https://rbcommons.com/s/twitter/r/1266/
Singleton for requests session

Instead of instance-level, use a module level singleton so that in multiproc code,
subprocesses each have their own, reusable session.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/39542444 passed

Bugs closed: 751

Reviewed at https://rbcommons.com/s/twitter/r/1264/
sitegen: get page-level table-of-contents working again

workaround for: modifying the AST to add the here-links had the side effect
that the find-every-header function no longer finds headings. So reorder
operations: find every header before adding the here-links

While adding an import to test_sitegen.py , noticed it was missing the
usual "from __future__ import", so added that.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/39653188

patched this and https://rbcommons.com/s/twitter/r/1182/ into a branch and test-generated the doc site: http://pantsbuild.github.io/sitegen-test/JVMProjects.html

Reviewed at https://rbcommons.com/s/twitter/r/1191/
Pull cache test patterns into reusable contextmanagers

Pull cache testing helper setup/teardown patterns into reusable contextmanagers

Testing Done:
./pants goal test tests/python/pants_test/cache:: passes.

https://travis-ci.org/pantsbuild/pants/builds/39634283 (in progress)

Bugs closed: 757

Reviewed at https://rbcommons.com/s/twitter/r/1267/
Add passthru arg support in the new options system.

Anything after a double-dash is captured as-is. Future changes will
plumb them through to tasks that want them.

This is useful e.g., for passing args through to a tool invoked by a task.

Testing Done:
Ran option tests. CI passed.

Reviewed at https://rbcommons.com/s/twitter/r/1270/
Refactor protobuf parsing logic out of protobuf_gen

This is in preparation of adding support for the Wire generator which also uses .proto files as sources.

Testing Done:
CI is green: https://travis-ci.org/pantsbuild/pants/builds/39766434

Bugs closed: 762

Reviewed at https://rbcommons.com/s/twitter/r/1273/
Correctly parse named_is_latest entries from the pushdb

The named_is_latest entry was being implicitly str()d on the way into the database, but was defaulting to a boolean on the way out. Strings are "truthy", so the "False" string would be treated as true.

Reviewed at https://rbcommons.com/s/twitter/r/1245/
Migrate jar_publish to new options.

Testing Done:
CI passed. Ran jar_publish's unit and integration tests. The integration tests exercise most of the flags.

It's a bit hard to test manually because parts of jar_publish are broken for other reasons. But we're pretty confident about migration changes at this point.

Reviewed at https://rbcommons.com/s/twitter/r/1276/
Migrate dependencies, depmap, detect_duplicates and check_published_deps to new options

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/39655628

Reviewed at https://rbcommons.com/s/twitter/r/1269/
Deeper selection of internal targets during publishing

The current behavior of selecting directly declared Jarable targets broke ages ago during Patrick's target refactor. In order to appropriately walk through intermediate jar_library/target declarations, we need to recurse.

It's common practice internally to have a top-level alias (a 'target' declaration) to point deeper into the directory structure to an actual java_library or scala_library. This meant that publishing was 1) ignoring those deps during fingerprint calculation and 2) not including them via DependencyWriter in written poms.

Testing Done:
test publishes; will follow up to add unit tests here if things look sane.

Reviewed at https://rbcommons.com/s/twitter/r/1213/
Add Task.workdir docs.

Testing Done:
CI went green here: https://travis-ci.org/pantsbuild/pants/builds/39909933

Bugs closed: 766

Reviewed at https://rbcommons.com/s/twitter/r/1277/
Fix DetectDuplicate's handling of an 'append'-type flag.

Testing Done:
Ran the broken tests, verified that they succeed with this change.

Reviewed at https://rbcommons.com/s/twitter/r/1282/