un-deprecate jar_library

http://pantsbuild.github.io/build_dictionary.html#jar-library says jar_library is just an old name for "dependencies". But everything old is new again.

Reviewed at https://rbcommons.com/s/twitter/r/409/
Switch pants to use the java jar-tool utility to create jars.

It turns out the java jar utility and the backing java APIs hack the zip
format to support > 2^16 jar entries in zip32 jars (pre java 7).  The
easiest way to access this functionality is to delegate to the JDK jar
APIs.

This change maintains the in-use open_jar context manager but extends
its use to all pants jar creators and updaters as well as expanding its
functionality to cover explicit support for main and classpath manfiest
entries and merging jars.

Since the new jarring uses java APIs, duplicates are explicity
forbidden and so JarRules are exposed to control handling of dups via
a new JvmBinary.deploy_jar_rules attribute and the ability to set a
global site-wide set of default jar rules.

A little test infra housekeeping - aso add a JvmToolTaskTestBase for
task tests that need jvm tool bootstrapping infra to function (most
non-console Tasks).

Testing Done:
All tests pass including JarTaskTest and JarCreateTest:
$ PANTS_DEV=1 ./pants tests/python/pants_test/:all

Manual tests using the pingpong including binary, bundle and bundle --bundle-deployjar.
Fatjar timings are ~4 times faster than old pants.

Bugs closed: 144

Reviewed at https://rbcommons.com/s/twitter/r/395/
Group JarCreate jarring actions in a jar-create MULTITOOL workunit.

Testing Done:
Before:
=======
$ PANTS_DEV=1 ./pants goal bundle --bundle-deployjar src/java/com/pants/examples/pingpong/::
...
08:04:20 00:08   [jar]
08:04:20 00:08     [jar]
08:04:20 00:08       [bootstrap-jar_tool]
08:04:22 00:10       [jar-tool]
08:04:22 00:10       [jar-tool]
08:04:22 00:10   [bundle]
08:04:22 00:10     [bundle]
                   creating dist/pingpong-bundle
...

After (note the dot per collapsed jar-tool workunit):
======
$ PANTS_DEV=1 ./pants goal bundle --bundle-deployjar src/java/com/pants/examples/pingpong/::
...
08:04:38 00:01   [jar]
08:04:38 00:01     [jar]
08:04:38 00:01       [jar-create]..
08:04:38 00:01   [bundle]
08:04:38 00:01     [bundle]
                   creating dist/pingpong-bundle

Bugs closed: 164

Reviewed at https://rbcommons.com/s/twitter/r/416/
Add an option to open the pants server ui.

This also fixes a bug in rcfile augmentation of args.

Testing Done:
$ tail -4 ~/.pants.rc
[server]
options:
  --server-open

$ PANTS_DEV=1 ./pants goal server
*** Running pants in dev mode from /home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/src/python/pants/bin/pants_exe.py ***
(using pantsrc expansion: pants goal server --server-open)
Server already running at http://localhost:50579
Created new window in existing browser session.

$ PANTS_DEV=1 ./pants goal killserver server
*** Running pants in dev mode from /home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/src/python/pants/bin/pants_exe.py ***
(using pantsrc expansion: pants goal killserver server --server-open)
Killed server with pid 26443 at http://localhost:50579
Launching server with pid 27492 at http://localhost:36291
Created new window in existing browser session.

Bugs closed: 165

Reviewed at https://rbcommons.com/s/twitter/r/418/
doc: comment out most of "how to install" page, which turned out to be a troll

comment out System-wide Installation, Virtualenv-based Installation sections.

replaced "pants" with "pantsbuild.pants" in the commented-out text.

Bring back the "Pants is not an install-able" blurb we had before. Slightly updated; the old text assumed Pants lived in twitter/commons.

Bugs closed: 114, 158

Reviewed at https://rbcommons.com/s/twitter/r/414/
added 'jvm_arg' option for jar-tool config in pants.ini

This does not solve issue 169, but it makes is less painful and easier to configure/diagnose.

Testing Done:
Added this to pants.ini:

[jar-tool]
jvm_args:['-Xmx257M']

I used '257' because its an unlikely value.

PANTS_DEV=1 ./pants goal ng-killall
PANTS_DEV=1 ./pants goal bundle <target>

ps auxww | grep 257M shows the NGServer process.

Bugs closed: 169

Reviewed at https://rbcommons.com/s/twitter/r/421/
Improve detect-duplicates to detect dups in internal dependencies
Clean up some nits.
doc: 3rdparty_jvm dependencies(dependencies=) -> jar_library(jars=)

Caught some of the more obvious places where we were saying deps(deps=[...]) instead of jar_library(jars=[...])

Bugs closed: 141

Reviewed at https://rbcommons.com/s/twitter/r/410/
Upgrade com.twitter.common#jar-tool to 0.1.5

This picks up a fix for memory leaks under nailgun's
repeated invocations of main.

Testing Done:
$ PANTS_DEV=1 ./pants goal jar src/{java,scala}:: -ldebug
...
21:01:06 00:00   [jar]
21:01:06 00:00     [jar]
21:01:06 00:00       [jar-create].DEBUG] Executing: /opt/jdk1.7.0/bin/java -Divy.cache.dir=/home/jsirois/.ivy2/twitter-commons -Xmx1g -XX:MaxPermSize=256m -Dsun.io.useCanonCaches=false -cp /home/jsirois/.ivy2/twitter-commons/org.apache.ivy/ivy/jars/ivy-2.3.0.jar org.apache.ivy.Main -settings /home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/build-support/ivy/ivysettings.xml -ivy /home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/.pants.d/ivy/4147dbb500c708344acc417e09c84556032b610c.da39a3ee5e6b#4c4fdc2a0cf8/ivy.xml -confs default -cachepath /home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/.pants.d/ivy/4147dbb500c708344acc417e09c84556032b610c.da39a3ee5e6b#4c4fdc2a0cf8/classpath.raw.tmp
.DEBUG] No ng server found with fingerprint 7ae73b6e87959b5c95b27fd71e95487367a07863, spawning...
DEBUG] Executing: /opt/jdk1.7.0/bin/java -Xmx64M -Dpants.buildroot=/home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants -Dpants.nailgun.owner=/home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/.pants.d/ng/jar-tool -Dpants.nailgun.fingerprint=7ae73b6e87959b5c95b27fd71e95487367a07863 -cp /home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/.pants.d/ivy/jars/com.martiansoftware/nailgun-server/jars/nailgun-server-0.9.1.jar:/home/jsirois/.ivy2/twitter-commons/com.twitter.common/jar-tool/jars/jar-tool-0.1.5.jar:/home/jsirois/.ivy2/twitter-commons/com.twitter.common/args/jars/args-0.2.13.jar:/home/jsirois/.ivy2/twitter-commons/com.twitter.common/args-apt/jars/args-apt-0.1.17.jar:/home/jsirois/.ivy2/twitter-commons/com.twitter.common/args-core/jars/args-core-0.1.18.jar:/home/jsirois/.ivy2/twitter-commons/com.google.guava/guava/bundles/guava-16.0.1.jar:/home/jsirois/.ivy2/twitter-commons/com.google.code.findbugs/jsr305/jars/jsr305-2.0.1.jar:/home/jsirois/.ivy2/twitter-commons/commons-lang/commons-lang/jars/commons-lang-2.6.jar:/home/jsirois/.ivy2/twitter-commons/com.twitter.common/base/jars/base-0.0.95.jar:/home/jsirois/.ivy2/twitter-commons/com.twitter.common/quantity/jars/quantity-0.0.79.jar:/home/jsirois/.ivy2/twitter-commons/com.twitter.common/collections/jars/collections-0.0.82.jar:/home/jsirois/.ivy2/twitter-commons/com.twitter.common/util-system-mocks/jars/util-system-mocks-0.0.81.jar:/home/jsirois/.ivy2/twitter-commons/com.twitter.common/net-util/jars/net-util-0.0.84.jar:/home/jsirois/.ivy2/twitter-commons/commons-codec/commons-codec/jars/commons-codec-1.6.jar:/home/jsirois/.ivy2/twitter-commons/com.twitter.common/io/jars/io-0.0.56.jar:/home/jsirois/.ivy2/twitter-commons/commons-io/commons-io/jars/commons-io-2.1.jar:/home/jsirois/.ivy2/twitter-commons/com.twitter.common/logging/jars/logging-0.0.62.jar:/home/jsirois/.ivy2/twitter-commons/joda-time/joda-time/jars/joda-time-2.3.jar:/home/jsirois/.ivy2/twitter-commons/org.joda/joda-convert/jars/joda-convert-1.5.jar com.martiansoftware.nailgun.NGServer :0
DEBUG] Spawned ng server with fingerprint 7ae73b6e87959b5c95b27fd71e95487367a07863 @ 27168
DEBUG] Detected ng server up on port 59408
DEBUG] Found ng server with fingerprint 7ae73b6e87959b5c95b27fd71e95487367a07863 @ pid:27168 port:59408
DEBUG] Connected to ng server with fingerprint 7ae73b6e87959b5c95b27fd71e95487367a07863 pid: 27168 @ port: 59408
...

Bugs closed: 169

Reviewed at https://rbcommons.com/s/twitter/r/433/
doc: don't HTML-escape already-escaped text

It's a little silly that http://pantsbuild.github.io/goals_reference.html says "-&gt;" when it means "->"

Testing Done:
./build-support/bin/publish_docs.sh -o

Temporarily tossed in a <b>YARGLE</b> to make sure it was still escaped and I wasn't turning the goals_reference page into some javascript sploit thing.

Reviewed at https://rbcommons.com/s/twitter/r/427/
Emit full path for docs produced by sphinx.

RB_ID=https://rbcommons.com/s/twitter/r/432/
doc: when your diamond dependencies are more like a triangle

Until we finally consolidate the world's source code into one monorepo, we may occasionally encounter this situation.

Reviewed at https://rbcommons.com/s/twitter/r/430/
A mechanical refactor of tasks/targets into pants.backend.<backend>

* Modularize registration of BUILD parse context into
per-backend register.py entry points.
* Remove all global state from BuildFileParser.  Even BUILD
file context is instance local now.
* Adjust tests to inject the minimum amount of BUILD context
into their local BuildFileParser.
* This closes most of pants/issues/5 and fixes a regression
from rbt 321 that slowed down test running significantly.

Testing Done:
CI passes

Reviewed at https://rbcommons.com/s/twitter/r/417/
Add a bunch of missing __init__.py's.

No idea how this always manages to slip by me.
More missing __init__.py's
Plumb FingerprintStrategy through invalidation logic

Implement a custom IvyResolveFingerprintStrategy to selectively
invalidate ivy resolution only on the contents of JarLibrary payloads
and the excludes + configurations in JvmTarget payloads.

Testing Done:
CI Passes

Bugs closed: https://github.com/pantsbuild/pants/issues/172

Reviewed at https://rbcommons.com/s/twitter/r/434/
Upgrade jmake to 1.3.8-2. This contains the following change: https://github.com/pantsbuild/jmake/commit/51a8f3f1ff700cde95ce9feacb57bedc155cbcd9

Reviewed at https://rbcommons.com/s/twitter/r/436/
docs: update docstring to point to actual create_arifact_cache

This was moved out of pants.cache with 27f01c43. An option is
maybe take the hardcoded path out, since they sometimes age poorly.

Testing Done:
Just a docstring change, but I ran the CI so this section wouldn't be blank. CI SUCCESS.

Reviewed at https://rbcommons.com/s/twitter/r/438/
Push group handling into GroupTask.

This eliminates knowledge of groups from the core Engine
implementations and isolates their handling to a specialized
GroupTask that delegates to GroupMembers each producing the
same product type.

This also eliminates the trace of groups in every Task's
execute signature which now no-longer takes targets.  Tasks
instead get the targets they need from the context uniformly.

Finally, a prepare lifecycle method is added to the base task
interface.  Today this is called lockstep after the contructor
by the core engine, but in the future it will be where things
like product requirements are registered.

Testing Done:
Unit:
=====
$ PANTS_DEV=1 ./pants tests/python/pants_test:all

Group in the field:
===================
$ PANTS_DEV=1 ./pants goal test tests/{java,scala}:: -ldebug

Grouped flags maintain their old spelling - they are displayed in 1 section though:
==========================================
$ PANTS_DEV=1 ./pants goal help compile
...
jvm options:
  --compile-apt-warnings, --no-compile-apt-warnings
                          [True] Compile with all configured warnings enabled.
  ...
  --compile-apt-args=JAVA_COMPILE_ARGS
                          Pass these extra args to javac.
  --compile-java-warnings, --no-compile-java-warnings
                          [True] Compile with all configured warnings enabled.
  ...
  --compile-java-args=JAVA_COMPILE_ARGS
                          Pass these extra args to javac.
  --compile-scala-warnings, --no-compile-scala-warnings
                          [True] Compile with all configured warnings enabled.
  ...
  --compile-scala-plugins=PLUGINS
                          Use these scalac plugins. Default is set in pants.ini.

Explain still ~works:
=====================
$ PANTS_DEV=1 ./pants goal test tests/{java,scala}:: --explain
*** Running pants in dev mode from /home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/src/python/pants/bin/pants_exe.py ***
Phase Execution Order:

bootstrap -> gen -> check-exclusives -> resolve -> compile -> resources -> test

Phase [Goal->Task] Order:

bootstrap [bootstrap-jvm-tools->BootstrapJvmTools]
gen [thrift->ApacheThriftGen, scrooge->ScroogeGen, protoc->ProtobufGen, antlr->AntlrGen]
check-exclusives [check-exclusives->CheckExclusives]
resolve [ivy->IvyResolve]
compile [jvm->SingletonGroupTask]
resources [prepare->PrepareResources]
test [junit->JUnitRun, specs->SpecsRun]

Although "compile [jvm->SingletonGroupTask]" doesn't say all it could, I'm willing to live with that for now if you all are.

Bugs closed: 167, 171

Reviewed at https://rbcommons.com/s/twitter/r/431/