new param for page target, links. List of hyperlinked-to targets

List of hyperlinked-to targets. We use dependencies for this now.
But circular dependencies are an error while hyperlink-cycles should be OK.

So param page(links=[...]) feeds into traversable_specs (versus traversable_dependency_specs).

While I was looking at the tests, un-commented-in some commented-out assertions that had a done TODO.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/39520576

Bugs closed: 253, 742

Reviewed at https://rbcommons.com/s/twitter/r/1242/
Migrate some utility tasks to new-style options.

Reviewed at https://rbcommons.com/s/twitter/r/1248/
fix [run.junit] -> [test.junit]

Testing Done:
https://travis-ci.org/jinfeng/jinfeng-pants-fork/builds/39534255

Reviewed at https://rbcommons.com/s/twitter/r/1256/
fix the error msg in favor of the new style arg name

Reviewed at https://rbcommons.com/s/twitter/r/1258/
WikiArtifact instances also have provides; limit ivy to jvm

This should never get hit unless you accidentally do goal compile on a
wiki -- but maybe you do compile ...:: and hit it.

Testing Done:
Ran tests.

Reviewed at https://rbcommons.com/s/twitter/r/1259/
Refactor artifact cache exceptions.

Presently, the artifact cache will treat any exception during the
insertion process as a non-fatal error.  But some errors should be
fatal -- any sort of misconfiguration, or an attempt to cache
nonexistent files, or an exception thrown while moving files around.
Also remove ArtifactCache.CacheError, as it was not clear when
to use ArtifactCache.CacheError and when to use ArtifactCacheError.

Reviewed at https://rbcommons.com/s/twitter/r/1261/
included code snippets: don't just slap in a pre. syntax highlighting

The pygments code-syntax-highlighting code has some built-in functionality to guess at a code snippet's language based on filename and content. We hadn't hooked it up. This hooks it up.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/39542386

patched this and https://rbcommons.com/s/twitter/r/1191/ into a branch and test-generated the doc site: http://pantsbuild.github.io/sitegen-test/JVMProjects.html

Bugs closed: 693

Reviewed at https://rbcommons.com/s/twitter/r/1182/
Fix error message: add missing space

Reviewed at https://rbcommons.com/s/twitter/r/1266/
Pull cache test patterns into reusable contextmanagers
Multiprocessing Cache Check and Write

This is a slight refactor of a change we've been using internally at foursquare for a few months.

Avoiding recompression during downloads (https://rbcommons.com/s/twitter/r/1233/) reduces the amount of this work we do, while this change is about doing it faster. Most cache work is CPU-bound gzipping of the tarballs, so moving cache fetch/decompression and compression/upload to into subprocesses improves multicore utilization.

This is done using a multiprocessing.Pool, passing ArtifactCaches and the arguments to their methods to subprocesses (whichi is why they were made pickle-friendly in previous changes).

Additionally, we discovered (through a very painful debugging process) that python's zlib bindings use a global static lock which, if in use by any thread when a subprocess is created via fork(), can be inherited my children in a locked state meaning, the child will deadlock if it later tries to use zlib. This is why the pools are created early in run_tracker rather than as-needed, and is discussed in the comment on SubprocPool in worker_pool.py.