sitegen: get page-level table-of-contents working again

workaround for: modifying the AST to add the here-links had the side effect
that the find-every-header function no longer finds headings. So reorder
operations: find every header before adding the here-links

While adding an import to test_sitegen.py , noticed it was missing the
usual "from __future__ import", so added that.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/39653188

patched this and https://rbcommons.com/s/twitter/r/1182/ into a branch and test-generated the doc site: http://pantsbuild.github.io/sitegen-test/JVMProjects.html

Reviewed at https://rbcommons.com/s/twitter/r/1191/
Pull cache test patterns into reusable contextmanagers

Pull cache testing helper setup/teardown patterns into reusable contextmanagers

Testing Done:
./pants goal test tests/python/pants_test/cache:: passes.

https://travis-ci.org/pantsbuild/pants/builds/39634283 (in progress)

Bugs closed: 757

Reviewed at https://rbcommons.com/s/twitter/r/1267/
Add passthru arg support in the new options system.

Anything after a double-dash is captured as-is. Future changes will
plumb them through to tasks that want them.

This is useful e.g., for passing args through to a tool invoked by a task.

Testing Done:
Ran option tests. CI passed.

Reviewed at https://rbcommons.com/s/twitter/r/1270/
Refactor protobuf parsing logic out of protobuf_gen

This is in preparation of adding support for the Wire generator which also uses .proto files as sources.

Testing Done:
CI is green: https://travis-ci.org/pantsbuild/pants/builds/39766434

Bugs closed: 762

Reviewed at https://rbcommons.com/s/twitter/r/1273/
Correctly parse named_is_latest entries from the pushdb

The named_is_latest entry was being implicitly str()d on the way into the database, but was defaulting to a boolean on the way out. Strings are "truthy", so the "False" string would be treated as true.

Reviewed at https://rbcommons.com/s/twitter/r/1245/
Migrate jar_publish to new options.

Testing Done:
CI passed. Ran jar_publish's unit and integration tests. The integration tests exercise most of the flags.

It's a bit hard to test manually because parts of jar_publish are broken for other reasons. But we're pretty confident about migration changes at this point.

Reviewed at https://rbcommons.com/s/twitter/r/1276/
Migrate dependencies, depmap, detect_duplicates and check_published_deps to new options

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/39655628

Reviewed at https://rbcommons.com/s/twitter/r/1269/
Deeper selection of internal targets during publishing

The current behavior of selecting directly declared Jarable targets broke ages ago during Patrick's target refactor. In order to appropriately walk through intermediate jar_library/target declarations, we need to recurse.

It's common practice internally to have a top-level alias (a 'target' declaration) to point deeper into the directory structure to an actual java_library or scala_library. This meant that publishing was 1) ignoring those deps during fingerprint calculation and 2) not including them via DependencyWriter in written poms.

Testing Done:
test publishes; will follow up to add unit tests here if things look sane.

Reviewed at https://rbcommons.com/s/twitter/r/1213/
Add Task.workdir docs.

Testing Done:
CI went green here: https://travis-ci.org/pantsbuild/pants/builds/39909933

Bugs closed: 766

Reviewed at https://rbcommons.com/s/twitter/r/1277/
Fix DetectDuplicate's handling of an 'append'-type flag.

Testing Done:
Ran the broken tests, verified that they succeed with this change.

Reviewed at https://rbcommons.com/s/twitter/r/1282/
Switch a bunch of migrated tasks to read option values the new way.

Note that this also allowed us to get rid of some explicit fallbacks
to the config file, as new options do this automatically, assuming
the right naming conventions.

Some configs were renamed and added to the config migration script,
to align them with these conventions.

Note that this slightly changes the semantics of one flag. Previously
--extension in MarkdownToHtml would overwrite the default values of
['.md', '.markdown']. Now it appends to those two defaults. So the
default extensions will always be processed, plus whatever is added.

Testing Done:
CI pending.

Reviewed at https://rbcommons.com/s/twitter/r/1281/
Remove more stray reads from old-style options.

Also migrated jar_create to new-style options.

Also fixed up some test infrastructure.

Testing Done:
CI passed.

Reviewed at https://rbcommons.com/s/twitter/r/1286/
fix typos

...that the world sees in the BUILD Dictionary, whoops

Reviewed at https://rbcommons.com/s/twitter/r/1284/
Fix name of non-fatal cache exception. Resolves:
```WARNING:pants.cache.restful_artifact_cache:
Error while reading from remote artifact cache: global name 'NonfatalCacheError' is not defined```

Testing Done:
./pants goal test tests/python/pants_test/cache::, but there's no actual coverage on these errors (yet).

Bugs closed: 764

Reviewed at https://rbcommons.com/s/twitter/r/1274/