Migrate dependencies, depmap, detect_duplicates and check_published_deps to new options

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/39655628

Reviewed at https://rbcommons.com/s/twitter/r/1269/
Deeper selection of internal targets during publishing

The current behavior of selecting directly declared Jarable targets broke ages ago during Patrick's target refactor. In order to appropriately walk through intermediate jar_library/target declarations, we need to recurse.

It's common practice internally to have a top-level alias (a 'target' declaration) to point deeper into the directory structure to an actual java_library or scala_library. This meant that publishing was 1) ignoring those deps during fingerprint calculation and 2) not including them via DependencyWriter in written poms.

Testing Done:
test publishes; will follow up to add unit tests here if things look sane.

Reviewed at https://rbcommons.com/s/twitter/r/1213/
Add Task.workdir docs.

Testing Done:
CI went green here: https://travis-ci.org/pantsbuild/pants/builds/39909933

Bugs closed: 766

Reviewed at https://rbcommons.com/s/twitter/r/1277/
Fix DetectDuplicate's handling of an 'append'-type flag.

Testing Done:
Ran the broken tests, verified that they succeed with this change.

Reviewed at https://rbcommons.com/s/twitter/r/1282/
Switch a bunch of migrated tasks to read option values the new way.

Note that this also allowed us to get rid of some explicit fallbacks
to the config file, as new options do this automatically, assuming
the right naming conventions.

Some configs were renamed and added to the config migration script,
to align them with these conventions.

Note that this slightly changes the semantics of one flag. Previously
--extension in MarkdownToHtml would overwrite the default values of
['.md', '.markdown']. Now it appends to those two defaults. So the
default extensions will always be processed, plus whatever is added.

Testing Done:
CI pending.

Reviewed at https://rbcommons.com/s/twitter/r/1281/
Remove more stray reads from old-style options.

Also migrated jar_create to new-style options.

Also fixed up some test infrastructure.

Testing Done:
CI passed.

Reviewed at https://rbcommons.com/s/twitter/r/1286/
fix typos

...that the world sees in the BUILD Dictionary, whoops

Reviewed at https://rbcommons.com/s/twitter/r/1284/
Fix name of non-fatal cache exception. Resolves:
```WARNING:pants.cache.restful_artifact_cache:
Error while reading from remote artifact cache: global name 'NonfatalCacheError' is not defined```

Testing Done:
./pants goal test tests/python/pants_test/cache::, but there's no actual coverage on these errors (yet).

Bugs closed: 764

Reviewed at https://rbcommons.com/s/twitter/r/1274/
Multiprocessing Cache Check and Write

This is a slight refactor of a change we've been using internally at foursquare for a few months.

Avoiding recompression during downloads (https://rbcommons.com/s/twitter/r/1233/) reduces the amount of this work we do, while this change is about doing it faster. Most cache work is CPU-bound gzipping of the tarballs, so moving cache fetch/decompression and compression/upload to into subprocesses improves multicore utilization.

This is done using a multiprocessing.Pool, passing ArtifactCaches and the arguments to their methods to subprocesses (which is why they were made pickle-friendly in previous changes).

Additionally, we discovered (through a very painful debugging process) that python's zlib bindings (in 2.x) use a global static lock which, if in use by any thread when a subprocess is created via fork(), can be inherited by children in a locked state, meaning the child will deadlock if it later tries to use zlib.

This is why the pools are created early in run_tracker, rather than as-needed, and is discussed in the comment on SubprocPool in worker_pool.py.

Testing Done:
./pants goal test tests/python/pants_test/cache::
https://travis-ci.org/pantsbuild/pants/builds/39635717

Tested the debug message with https://gist.github.com/dt/a5d72f580b3fb9a60466 but not checking that in since wall-clock sleeps in automated tests, for just a log message, don't seem worth it.

Bugs closed: 754

Reviewed at https://rbcommons.com/s/twitter/r/1265/
copy changes and typo fixes for docs

I did a pass through the docs, running example commands and comparing command line args with help, then updating the corresponding files.

I found some of the targets referenced in the text had changed, so I updated the commands associated. Where the contents of the files had changed, I updated the text to reflect the new code.
Pin pants support down to python2.7 - dropping 2.6.

Testing the python3.3+ waters will happen seperately - this just drops support for running pants against 2.6 as agreed.

Testing Done:
CI away here: https://travis-ci.org/pantsbuild/pants/builds/40110691

Bugs closed: 722, 767

Reviewed at https://rbcommons.com/s/twitter/r/1278/
.rst files -> .md files.

re-play https://rbcommons.com/s/twitter/r/1235/ , "convert more .rst files to .md", but less crash-y. Also pull in some excellent feedback and update with doc changes that went into the .rst.

The crashy change converted many rst docs to Pants-enhanced markdown (yay) with BUILD targets (yay) with circular deps which 'goal markdown' is OK with (hmm) but which 'goal test' balks at, breaking CI (d'oh). So I reverted that change.

*This* change is like *that* change, but

* Uses the new `links` BUILD param instead of `dependencies`. Yay for no CI breaks.
* Pulls in an additition the excellent AReitz made to JVMProjects.rst meanwhile.
* Includes feedback from same excellent AReitz.
* pulls in .rst changes that came in at https://rbcommons.com/s/twitter/r/1279/

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/40211471

http://pantsbuild.github.io/sitegen-test/

Reviewed at https://rbcommons.com/s/twitter/r/1263/
Split up the travis run into a unit-test shard and multiple IT shards.

Also move tests/python/pants_test/tasks/jvm_compile/java:test_java_compile_integration
out of the unit test suite.

Testing Done:
CI went green here: https://travis-ci.org/pantsbuild/pants/builds/40216412

Bugs closed: 773

Reviewed at https://rbcommons.com/s/twitter/r/1290/