Refactor artifact cache exceptions.

Presently, the artifact cache will treat any exception during the
insertion process as a non-fatal error.  But some errors should be
fatal -- any sort of misconfiguration, or an attempt to cache
nonexistent files, or an exception thrown while moving files around.
Also remove ArtifactCache.CacheError, as it was not clear when
to use ArtifactCache.CacheError and when to use ArtifactCacheError.

Reviewed at https://rbcommons.com/s/twitter/r/1261/
included code snippets: don't just slap in a pre. syntax highlighting

The pygments code-syntax-highlighting code has some built-in functionality to guess at a code snippet's language based on filename and content. We hadn't hooked it up. This hooks it up.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/39542386

patched this and https://rbcommons.com/s/twitter/r/1191/ into a branch and test-generated the doc site: http://pantsbuild.github.io/sitegen-test/JVMProjects.html

Bugs closed: 693

Reviewed at https://rbcommons.com/s/twitter/r/1182/
Fix error message: add missing space

Reviewed at https://rbcommons.com/s/twitter/r/1266/
Singleton for requests session

Instead of instance-level, use a module level singleton so that in multiproc code,
subprocesses each have their own, reusable session.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/39542444 passed

Bugs closed: 751

Reviewed at https://rbcommons.com/s/twitter/r/1264/
sitegen: get page-level table-of-contents working again

workaround for: modifying the AST to add the here-links had the side effect
that the find-every-header function no longer finds headings. So reorder
operations: find every header before adding the here-links

While adding an import to test_sitegen.py , noticed it was missing the
usual "from __future__ import", so added that.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/39653188

patched this and https://rbcommons.com/s/twitter/r/1182/ into a branch and test-generated the doc site: http://pantsbuild.github.io/sitegen-test/JVMProjects.html

Reviewed at https://rbcommons.com/s/twitter/r/1191/
Pull cache test patterns into reusable contextmanagers

Pull cache testing helper setup/teardown patterns into reusable contextmanagers

Testing Done:
./pants goal test tests/python/pants_test/cache:: passes.

https://travis-ci.org/pantsbuild/pants/builds/39634283 (in progress)

Bugs closed: 757

Reviewed at https://rbcommons.com/s/twitter/r/1267/
Add passthru arg support in the new options system.

Anything after a double-dash is captured as-is. Future changes will
plumb them through to tasks that want them.

This is useful e.g., for passing args through to a tool invoked by a task.

Testing Done:
Ran option tests. CI passed.

Reviewed at https://rbcommons.com/s/twitter/r/1270/
Refactor protobuf parsing logic out of protobuf_gen

This is in preparation of adding support for the Wire generator which also uses .proto files as sources.

Testing Done:
CI is green: https://travis-ci.org/pantsbuild/pants/builds/39766434

Bugs closed: 762

Reviewed at https://rbcommons.com/s/twitter/r/1273/
Correctly parse named_is_latest entries from the pushdb

The named_is_latest entry was being implicitly str()d on the way into the database, but was defaulting to a boolean on the way out. Strings are "truthy", so the "False" string would be treated as true.

Reviewed at https://rbcommons.com/s/twitter/r/1245/
Migrate jar_publish to new options.

Testing Done:
CI passed. Ran jar_publish's unit and integration tests. The integration tests exercise most of the flags.

It's a bit hard to test manually because parts of jar_publish are broken for other reasons. But we're pretty confident about migration changes at this point.

Reviewed at https://rbcommons.com/s/twitter/r/1276/
Migrate dependencies, depmap, detect_duplicates and check_published_deps to new options

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/39655628

Reviewed at https://rbcommons.com/s/twitter/r/1269/
Deeper selection of internal targets during publishing

The current behavior of selecting directly declared Jarable targets broke ages ago during Patrick's target refactor. In order to appropriately walk through intermediate jar_library/target declarations, we need to recurse.

It's common practice internally to have a top-level alias (a 'target' declaration) to point deeper into the directory structure to an actual java_library or scala_library. This meant that publishing was 1) ignoring those deps during fingerprint calculation and 2) not including them via DependencyWriter in written poms.

Testing Done:
test publishes; will follow up to add unit tests here if things look sane.

Reviewed at https://rbcommons.com/s/twitter/r/1213/
Add Task.workdir docs.

Testing Done:
CI went green here: https://travis-ci.org/pantsbuild/pants/builds/39909933

Bugs closed: 766

Reviewed at https://rbcommons.com/s/twitter/r/1277/
Fix DetectDuplicate's handling of an 'append'-type flag.

Testing Done:
Ran the broken tests, verified that they succeed with this change.

Reviewed at https://rbcommons.com/s/twitter/r/1282/
Switch a bunch of migrated tasks to read option values the new way.

Note that this also allowed us to get rid of some explicit fallbacks
to the config file, as new options do this automatically, assuming
the right naming conventions.

Some configs were renamed and added to the config migration script,
to align them with these conventions.

Note that this slightly changes the semantics of one flag. Previously
--extension in MarkdownToHtml would overwrite the default values of
['.md', '.markdown']. Now it appends to those two defaults. So the
default extensions will always be processed, plus whatever is added.

Testing Done:
CI pending.

Reviewed at https://rbcommons.com/s/twitter/r/1281/
Remove more stray reads from old-style options.

Also migrated jar_create to new-style options.

Also fixed up some test infrastructure.

Testing Done:
CI passed.

Reviewed at https://rbcommons.com/s/twitter/r/1286/
fix typos

...that the world sees in the BUILD Dictionary, whoops

Reviewed at https://rbcommons.com/s/twitter/r/1284/
Fix name of non-fatal cache exception. Resolves:
```WARNING:pants.cache.restful_artifact_cache:
Error while reading from remote artifact cache: global name 'NonfatalCacheError' is not defined```

Testing Done:
./pants goal test tests/python/pants_test/cache::, but there's no actual coverage on these errors (yet).

Bugs closed: 764

Reviewed at https://rbcommons.com/s/twitter/r/1274/
Multiprocessing Cache Check and Write

This is a slight refactor of a change we've been using internally at foursquare for a few months.

Avoiding recompression during downloads (https://rbcommons.com/s/twitter/r/1233/) reduces the amount of this work we do, while this change is about doing it faster. Most cache work is CPU-bound gzipping of the tarballs, so moving cache fetch/decompression and compression/upload to into subprocesses improves multicore utilization.

This is done using a multiprocessing.Pool, passing ArtifactCaches and the arguments to their methods to subprocesses (which is why they were made pickle-friendly in previous changes).

Additionally, we discovered (through a very painful debugging process) that python's zlib bindings (in 2.x) use a global static lock which, if in use by any thread when a subprocess is created via fork(), can be inherited by children in a locked state, meaning the child will deadlock if it later tries to use zlib.

This is why the pools are created early in run_tracker, rather than as-needed, and is discussed in the comment on SubprocPool in worker_pool.py.

Testing Done:
./pants goal test tests/python/pants_test/cache::
https://travis-ci.org/pantsbuild/pants/builds/39635717

Tested the debug message with https://gist.github.com/dt/a5d72f580b3fb9a60466 but not checking that in since wall-clock sleeps in automated tests, for just a log message, don't seem worth it.

Bugs closed: 754

Reviewed at https://rbcommons.com/s/twitter/r/1265/
copy changes and typo fixes for docs

I did a pass through the docs, running example commands and comparing command line args with help, then updating the corresponding files.

I found some of the targets referenced in the text had changed, so I updated the commands associated. Where the contents of the files had changed, I updated the text to reflect the new code.