Improve detect-duplicates to detect dups in internal dependencies
Clean up some nits.
doc: 3rdparty_jvm dependencies(dependencies=) -> jar_library(jars=)

Caught some of the more obvious places where we were saying deps(deps=[...]) instead of jar_library(jars=[...])

Bugs closed: 141

Reviewed at https://rbcommons.com/s/twitter/r/410/
Upgrade com.twitter.common#jar-tool to 0.1.5

This picks up a fix for memory leaks under nailgun's
repeated invocations of main.

Testing Done:
$ PANTS_DEV=1 ./pants goal jar src/{java,scala}:: -ldebug
...
21:01:06 00:00   [jar]
21:01:06 00:00     [jar]
21:01:06 00:00       [jar-create].DEBUG] Executing: /opt/jdk1.7.0/bin/java -Divy.cache.dir=/home/jsirois/.ivy2/twitter-commons -Xmx1g -XX:MaxPermSize=256m -Dsun.io.useCanonCaches=false -cp /home/jsirois/.ivy2/twitter-commons/org.apache.ivy/ivy/jars/ivy-2.3.0.jar org.apache.ivy.Main -settings /home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/build-support/ivy/ivysettings.xml -ivy /home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/.pants.d/ivy/4147dbb500c708344acc417e09c84556032b610c.da39a3ee5e6b#4c4fdc2a0cf8/ivy.xml -confs default -cachepath /home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/.pants.d/ivy/4147dbb500c708344acc417e09c84556032b610c.da39a3ee5e6b#4c4fdc2a0cf8/classpath.raw.tmp
.DEBUG] No ng server found with fingerprint 7ae73b6e87959b5c95b27fd71e95487367a07863, spawning...
DEBUG] Executing: /opt/jdk1.7.0/bin/java -Xmx64M -Dpants.buildroot=/home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants -Dpants.nailgun.owner=/home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/.pants.d/ng/jar-tool -Dpants.nailgun.fingerprint=7ae73b6e87959b5c95b27fd71e95487367a07863 -cp /home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/.pants.d/ivy/jars/com.martiansoftware/nailgun-server/jars/nailgun-server-0.9.1.jar:/home/jsirois/.ivy2/twitter-commons/com.twitter.common/jar-tool/jars/jar-tool-0.1.5.jar:/home/jsirois/.ivy2/twitter-commons/com.twitter.common/args/jars/args-0.2.13.jar:/home/jsirois/.ivy2/twitter-commons/com.twitter.common/args-apt/jars/args-apt-0.1.17.jar:/home/jsirois/.ivy2/twitter-commons/com.twitter.common/args-core/jars/args-core-0.1.18.jar:/home/jsirois/.ivy2/twitter-commons/com.google.guava/guava/bundles/guava-16.0.1.jar:/home/jsirois/.ivy2/twitter-commons/com.google.code.findbugs/jsr305/jars/jsr305-2.0.1.jar:/home/jsirois/.ivy2/twitter-commons/commons-lang/commons-lang/jars/commons-lang-2.6.jar:/home/jsirois/.ivy2/twitter-commons/com.twitter.common/base/jars/base-0.0.95.jar:/home/jsirois/.ivy2/twitter-commons/com.twitter.common/quantity/jars/quantity-0.0.79.jar:/home/jsirois/.ivy2/twitter-commons/com.twitter.common/collections/jars/collections-0.0.82.jar:/home/jsirois/.ivy2/twitter-commons/com.twitter.common/util-system-mocks/jars/util-system-mocks-0.0.81.jar:/home/jsirois/.ivy2/twitter-commons/com.twitter.common/net-util/jars/net-util-0.0.84.jar:/home/jsirois/.ivy2/twitter-commons/commons-codec/commons-codec/jars/commons-codec-1.6.jar:/home/jsirois/.ivy2/twitter-commons/com.twitter.common/io/jars/io-0.0.56.jar:/home/jsirois/.ivy2/twitter-commons/commons-io/commons-io/jars/commons-io-2.1.jar:/home/jsirois/.ivy2/twitter-commons/com.twitter.common/logging/jars/logging-0.0.62.jar:/home/jsirois/.ivy2/twitter-commons/joda-time/joda-time/jars/joda-time-2.3.jar:/home/jsirois/.ivy2/twitter-commons/org.joda/joda-convert/jars/joda-convert-1.5.jar com.martiansoftware.nailgun.NGServer :0
DEBUG] Spawned ng server with fingerprint 7ae73b6e87959b5c95b27fd71e95487367a07863 @ 27168
DEBUG] Detected ng server up on port 59408
DEBUG] Found ng server with fingerprint 7ae73b6e87959b5c95b27fd71e95487367a07863 @ pid:27168 port:59408
DEBUG] Connected to ng server with fingerprint 7ae73b6e87959b5c95b27fd71e95487367a07863 pid: 27168 @ port: 59408
...

Bugs closed: 169

Reviewed at https://rbcommons.com/s/twitter/r/433/
doc: don't HTML-escape already-escaped text

It's a little silly that http://pantsbuild.github.io/goals_reference.html says "-&gt;" when it means "->"

Testing Done:
./build-support/bin/publish_docs.sh -o

Temporarily tossed in a <b>YARGLE</b> to make sure it was still escaped and I wasn't turning the goals_reference page into some javascript sploit thing.

Reviewed at https://rbcommons.com/s/twitter/r/427/
Emit full path for docs produced by sphinx.

RB_ID=https://rbcommons.com/s/twitter/r/432/
doc: when your diamond dependencies are more like a triangle

Until we finally consolidate the world's source code into one monorepo, we may occasionally encounter this situation.

Reviewed at https://rbcommons.com/s/twitter/r/430/
A mechanical refactor of tasks/targets into pants.backend.<backend>

* Modularize registration of BUILD parse context into
per-backend register.py entry points.
* Remove all global state from BuildFileParser.  Even BUILD
file context is instance local now.
* Adjust tests to inject the minimum amount of BUILD context
into their local BuildFileParser.
* This closes most of pants/issues/5 and fixes a regression
from rbt 321 that slowed down test running significantly.

Testing Done:
CI passes

Reviewed at https://rbcommons.com/s/twitter/r/417/
Add a bunch of missing __init__.py's.

No idea how this always manages to slip by me.
More missing __init__.py's
Plumb FingerprintStrategy through invalidation logic

Implement a custom IvyResolveFingerprintStrategy to selectively
invalidate ivy resolution only on the contents of JarLibrary payloads
and the excludes + configurations in JvmTarget payloads.

Testing Done:
CI Passes

Bugs closed: https://github.com/pantsbuild/pants/issues/172

Reviewed at https://rbcommons.com/s/twitter/r/434/
Upgrade jmake to 1.3.8-2. This contains the following change: https://github.com/pantsbuild/jmake/commit/51a8f3f1ff700cde95ce9feacb57bedc155cbcd9

Reviewed at https://rbcommons.com/s/twitter/r/436/
docs: update docstring to point to actual create_arifact_cache

This was moved out of pants.cache with 27f01c43. An option is
maybe take the hardcoded path out, since they sometimes age poorly.

Testing Done:
Just a docstring change, but I ran the CI so this section wouldn't be blank. CI SUCCESS.

Reviewed at https://rbcommons.com/s/twitter/r/438/
Push group handling into GroupTask.

This eliminates knowledge of groups from the core Engine
implementations and isolates their handling to a specialized
GroupTask that delegates to GroupMembers each producing the
same product type.

This also eliminates the trace of groups in every Task's
execute signature which now no-longer takes targets.  Tasks
instead get the targets they need from the context uniformly.

Finally, a prepare lifecycle method is added to the base task
interface.  Today this is called lockstep after the contructor
by the core engine, but in the future it will be where things
like product requirements are registered.

Testing Done:
Unit:
=====
$ PANTS_DEV=1 ./pants tests/python/pants_test:all

Group in the field:
===================
$ PANTS_DEV=1 ./pants goal test tests/{java,scala}:: -ldebug

Grouped flags maintain their old spelling - they are displayed in 1 section though:
==========================================
$ PANTS_DEV=1 ./pants goal help compile
...
jvm options:
  --compile-apt-warnings, --no-compile-apt-warnings
                          [True] Compile with all configured warnings enabled.
  ...
  --compile-apt-args=JAVA_COMPILE_ARGS
                          Pass these extra args to javac.
  --compile-java-warnings, --no-compile-java-warnings
                          [True] Compile with all configured warnings enabled.
  ...
  --compile-java-args=JAVA_COMPILE_ARGS
                          Pass these extra args to javac.
  --compile-scala-warnings, --no-compile-scala-warnings
                          [True] Compile with all configured warnings enabled.
  ...
  --compile-scala-plugins=PLUGINS
                          Use these scalac plugins. Default is set in pants.ini.

Explain still ~works:
=====================
$ PANTS_DEV=1 ./pants goal test tests/{java,scala}:: --explain
*** Running pants in dev mode from /home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/src/python/pants/bin/pants_exe.py ***
Phase Execution Order:

bootstrap -> gen -> check-exclusives -> resolve -> compile -> resources -> test

Phase [Goal->Task] Order:

bootstrap [bootstrap-jvm-tools->BootstrapJvmTools]
gen [thrift->ApacheThriftGen, scrooge->ScroogeGen, protoc->ProtobufGen, antlr->AntlrGen]
check-exclusives [check-exclusives->CheckExclusives]
resolve [ivy->IvyResolve]
compile [jvm->SingletonGroupTask]
resources [prepare->PrepareResources]
test [junit->JUnitRun, specs->SpecsRun]

Although "compile [jvm->SingletonGroupTask]" doesn't say all it could, I'm willing to live with that for now if you all are.

Bugs closed: 167, 171

Reviewed at https://rbcommons.com/s/twitter/r/431/
Non-chunk-specific logic in jvm compilation now in prepare_execute.

- Fixes the global analysis splitting bug.
- Classpaths are now modified once, up front.

(Previously the classes/resources dirs were added to the classpath
multiple times, once per chunk. Also, there was some bogus logic
that updated the classpath at the end of execute(), implying that
the update was only for downstream tasks. However the update would
be visible in the next chunk, so that was misleading, and also
led to the first chunk being 'special'.)
A scala library with java sources should not trigger the warning 'Missing BUILD dependency' message
In order to avoid incorrect warnings include the java sources in the check
Also added a small project that should not trigger the warning anymore.

Reviewed at https://rbcommons.com/s/twitter/r/435/
Fix the py command and add py in the ci script

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/26624784

Local testing

tw-172-25-20-142 pants]$ PANTS_DEV=1 ./pants py --pex src/python/pants:pants
*** Running pants in dev mode from /Users/tdesai/projects/pants/src/python/pants/bin/pants_exe.py ***
Loading pants backends from source
/Users/tdesai/projects/pants src/python/pants:pants
WARNING: Target has multiple python_binary targets!
Wrote /Users/tdesai/projects/pants/dist/src.python.pants.pants.pex

Bugs closed: 179

Reviewed at https://rbcommons.com/s/twitter/r/444/
Fix duplicate detection when target has no sources.

See https://github.com/pantsbuild/pants/issues/176.
fix Python py verbose mode

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/26632712
[tdesai]PANTS_DEV=1 ./pants py --pex -v src/python/pants/bin:pants
Cleaning up...
Loading pants backends from source
Detected interpreter /usr/bin/python: CPython-2.6.7
Detected interpreter /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7: CPython-2.7.2
Detected interpreter /usr/local/bin/python2.7: CPython-2.7.5
/Users/tdesai/projects/pants src/python/pants/bin:pants
Build operating on targets: PythonBinary(BuildFileAddress(src/python/pants/bin/BUILD, pants))

Reviewed at https://rbcommons.com/s/twitter/r/445/
Add a hook to allow pants customizations to inject backends.

This may be provisional until the backend reorg finalizes, but
for now it assumes the additional backend packages injected
have been arranged to be on the PYTHONPATH by other means.

Testing Done:
$ ./build-support/bin/ci.sh

Bugs closed: 447

Reviewed at https://rbcommons.com/s/twitter/r/447/