Added some docs and a bugfix on debugging a JVM tool (like jar-tool or checkstyle) locally

Testing Done:
To test the new debugging tool, I edited pants.ini as follows:

[jar-tool]
    jvm_args: ['-Xmx300m', '-Xdebug', '-Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=%(debug_port)s']

Then attached to it with IntelliJ using a remote debug config for the twitter-commons repo

Bugs closed: 413

Reviewed at https://rbcommons.com/s/twitter/r/791/
Relativize classpath for non-ng java execution

-Factored the classpath relativization logic into it's own module and reworked it slightly.
-Made jmake relativize classpath in addition to the nailgun executor

The jvm_utils BUILD file has an empty dependencies key to follow the convention I found in another file.

Testing Done:
Ran a java/scala test target with nailgun enabled/disabled and made sure the classpath was minimized and the tests passed.

Reviewed at https://rbcommons.com/s/twitter/r/804/
Fixup JvmCompile to always deliver non-None products that were required by downstream.

This change augments GroupTask with a pre_execute complement to post_execute and
uses this lifecycle point to ensure JvmCompile delivers on its promise of supply product maps
regardless of the availability of chunks in the round.  Further adds tests for the new lifecycle
as well as coverage for pre-existing un-tested always-run semantics of construct, prepare and
post-execute.

Testing Done:
Travis CI went green here: https://travis-ci.org/pantsbuild/pants/builds/31199740

Also:
==
$ PANTS_DEV=1 ./pants goal test tests/python/pants_test/tasks/:group_task --test-pytest-options=-v
...
21:12:20 00:01   [test]
21:12:20 00:01     [pytest]
21:12:20 00:01       [run]
                     ============== test session starts ===============
                     platform linux2 -- Python 2.6.9 -- py-1.4.22 -- pytest-2.6.0 -- /usr/bin/python2.6
                     plugins: cov, timeout
                     collected 5 items

                     tests/python/pants_test/tasks/test_group_task.py@57::GroupIteratorSingleTest::test PASSED
                     tests/python/pants_test/tasks/test_group_task.py@73::GroupIteratorMultipleTest::test PASSED
                     tests/python/pants_test/tasks/test_group_task.py@179::GroupTaskTest::test_groups PASSED
                     tests/python/pants_test/tasks/test_group_task.py@232::EmptyGroupTaskTest::test_groups PASSED
                     tests/python/pants_test/tasks/test_group_task.py@263::TransitiveGroupTaskTest::test_transitive_groups PASSED

                     ============ 5 passed in 0.17 seconds ============

And tested this change live against an internal use-case that was failing before this as such:
==
...
  File "/Users/david/.pex/install/pantsbuild.pants-0.0.21-py2-none-any.whl.0ce746980853d744737145525a44d8f91b0c5d84/pantsbuild.pants-0.0.21-py2-none-any.whl/pants/engine/round_engine.py", line 42, in attempt
    task.execute()
  File "/Users/david/.pex/install/twitter.common.pants-0.4.0-py2-none-any.whl.e1f378c90f5282b48c1ebc94c79373050751b771/twitter.common.pants-0.4.0-py2-none-any.whl/twitter/common/pants/jvm/args/tasks/resource_mapper.py", line 99, in execute
    self._add_args_resources(target, resources_by_target, transitive=False)
  File "/Users/david/.pex/install/twitter.common.pants-0.4.0-py2-none-any.whl.e1f378c90f5282b48c1ebc94c79373050751b771/twitter.common.pants-0.4.0-py2-none-any.whl/twitter/common/pants/jvm/args/tasks/resource_mapper.py", line 118, in _add_args_resources
    collect_resource_dirs(target)
  File "/Users/david/.pex/install/twitter.common.pants-0.4.0-py2-none-any.whl.e1f378c90f5282b48c1ebc94c79373050751b771/twitter.common.pants-0.4.0-py2-none-any.whl/twitter/common/pants/jvm/args/tasks/resource_mapper.py", line 110, in collect_resource_dirs
    mapping = classes_by_target.get(tgt)
AttributeError: 'NoneType' object has no attribute 'get'

Notably, this is the t.c.pants args-apt plugin [1] and it requires 'classes_by_target' [2], but gets None [3] prior to this fix.
[1] https://pypi.python.org/pypi/twitter.common.pants/0.4.0
[2] https://github.com/twitter/commons/blob/master/pants-plugins/src/python/twitter/common/pants/jvm/args/tasks/resource_mapper.py#L89
[3] https://github.com/twitter/commons/blob/master/pants-plugins/src/python/twitter/common/pants/jvm/args/tasks/resource_mapper.py#L105

Bugs closed: 415

Reviewed at https://rbcommons.com/s/twitter/r/794/
Create a mixin for "quiet tasks".

This separates the two concerns formerly handled by ConsoleTask:
being quiet, and being a task whose primary job is to print to the console.

Now you mix in QuietTaskMixin for the first, and extend ConsoleTask
(which itself mixes in QuietTaskMixin) for the second.

This rids us of silliness such as the clean-all task having a flag
to specify its output separator.
`goal repl` consumes targets transitively

- Partition by selected language vs all other languages to allow for handling of 'dependencies'/'jar_library' and other inner nodes
- Filter the entire context, rather than just target roots
- Label JarLibrary as a JVM target (and fix related fallout)

Testing Done:
new unit test
manual validation of mixed/scala/python contexts:

-----
PANTS_DEV=1 ./pants goal repl src/scala/com/pants/testproject/unicode
...
scala> com.pants.testproject.unicode.shapeless.ShapelessExample.greek()
res1: String = shapeless success
-----
PANTS_DEV=1 ./pants goal repl src/python/example/hello/greet
...
>>> from example.hello.greet.greet import greet
>>> greet("yo")
u'\x1b[32mHello, yo!\x1b[0m'
-----
PANTS_DEV=1 ./pants goal repl src/scala/com/pants/testproject/unicode src/python/example/hello/greet
...
FAILURE: Mutually incompatible targets specified: PythonLibrary(BuildFileAddress(/Users/stuhood/src/pants/src/python/example/hello/greet/BUILD, greet)) vs ScalaLibrary(BuildFileAddress(/Users/stuhood/src/pants/src/scala/com/pants/testproject/unicode/shapeless/BUILD, shapeless))
-----

Reviewed at https://rbcommons.com/s/twitter/r/781/
Updated NailgunExecutor to take the java version into account when deciding if the server needs to be re-started.

Testing Done:
CI passed @ https://travis-ci.org/pantsbuild/pants/builds/31273593

setjdk 1.7
pants goal bundle src/java/com/pants/examples/hello/main
ps auxww | grep NGServer  (all are running java 1.7)

setjdk 1.8
ants goal bundle src/java/com/pants/examples/hello/main
ps auxww | grep NGServer  (get info messages about NG servers killed.  All are now running java 1.8)

Bugs closed: 423

Reviewed at https://rbcommons.com/s/twitter/r/802/
Updated scala-compile ref to scala-compiler in goal idea. Also updated build target reference strings to be absolute addresses

Updated scala-compile ref to scala-compiler. Also updated build target reference strings to be absolute addresses

Testing Done:
I'm not sure if I actually tickled the code that this affects, I just found this outdated reference while researching another problem.

This only changes some default strings, so I went into pants.ini and commented out the references to //:twitter-checkstyle and //:scala-compiler-2.9.3 then ran:

PANTS_DEV=1 ./pants goal idea src:: --exclude-target-regexp='.*android.*'

Things seemed to work.

ci.sh is running @ https://travis-ci.org/pantsbuild/pants/builds/31411849

Bugs closed: 431

Reviewed at https://rbcommons.com/s/twitter/r/816/
Convert options registered in the new style into old-style options.

Temporary code needed only during migration to the new options system.

Note: No __init__.py or BUILD files here. They are in another commit.
A command-line argument splitter.

Splits a Pants command line into a set of scoped flags and a set of targets.

Not wired up to anything yet.
A specialized container for option values.

This container supports key forwarding and ranked values. See docstrings
for details.
A custom argparse help formatter.

Squelches the extraneous parts of the argparse help message
(headers, usage etc.) since we format those ourselves. Leaves
just the per-flag help.

Note that subclassing the formatter is an official part of the
argparse API, so this is not underhanded or fragile.
Hook up the pants/option tests to pants_test:all.

Also fix a bug in a test.
more descriptive __name__ for curry_context'd things

If you ask a build_file_aliases curry_context-thingy for its __module__.__name__ , it says pants.base.build_file_aliases.<lambda>

That seems less useful than, say, pants.base.build_file_aliases.curry_context.pants.backend.python.python_requirements.python_requirements

Testing Done:
./build-support/bin/publish_docs.sh

https://travis-ci.org/pantsbuild/pants/builds/31350494

Bugs closed: 354

Reviewed at https://rbcommons.com/s/twitter/r/756/
docs: Close paren on TaskDeveloper code snippet

Someone might want to copypasta this and then would get an error.

I am sure I don't know anyone who would just go around
copypasta-ing the docs. But if they did, is all I am saying.

*whistling*

Testing Done:
Published the docs and gave an eyeball.

Reviewed at https://rbcommons.com/s/twitter/r/809/
Get rid of a spurious dep on t.c.lang.Compatibility.
Add missing dependency.
Add JarDep to aaptGen create_target method

This is two commits, one by me and another patch from jsirois.

This adds the android.jar to the created target and allows
the android target to properly compile. This outputs
java class files.

The java compile requires a java target. This also
finishes adding the android.jar. AndroidTargets will
now compile through pants. ivy requires a rev to resolve
jars and that fits so we can process each sdk group
together.

Reviewed at https://rbcommons.com/s/twitter/r/822/
Point to the new maven-central https endpoint.
This was brought to my attention by aurora threads and the announce was this weekend here: http://central.stage.sonatype.org/articles/2014/Aug/03/https-support-launching-now/

Also kill an unused resolver copied over from twitter/commons.

Testing Done:
$ rm -rf ~/.pants.d/tools/
$ PANTS_DEV=1 ./pants goal clean-all
$ PANTS_IVY_CACHE_DIR=/tmp/ivy2 PANTS_DEV=1 ./pants goal test {src,tests}/{scala,java}/::
...
               Executing goals in phases: bootstrap -> imports -> gen -> check-exclusives -> resolve -> compile -> resources -> test
13:10:18 00:17   [bootstrap]
13:10:18 00:17     [bootstrap-jvm-tools]
13:10:18 00:17   [imports]
13:10:18 00:17     [ivy-imports] INFO]
Downloading https://repo1.maven.org/maven2/org/apache/ivy/ivy/2.3.0/ivy-2.3.0.jar
 INFO] Starting new HTTPS connection (1): repo1.maven.org
100% .................................................. 1193 KB 6.273s
 INFO] sha1: c5ebf1c253ad4959a29f4acfe696ee48cdd9f473
...
13:20:11 10:10   [test]
13:20:11 10:10     [pytest]
13:20:11 10:10     [junit]
13:20:11 10:10       [bootstrap-junit]
13:20:40 10:39       [run]
                     ..........
                     Time: 0.177

                     OK (10 tests)

13:20:41 10:40     [specs]
               Waiting for background workers to finish.
               SUCCESS

$ find /tmp/ivy2 -name "ivydata-*.properties" | xargs grep location | cut -d= -f2 | cut -d/ -f-4 | sort | uniq -c
     15 http\://maven.twttr.com/com
      3 http\://maven.twttr.com/org
    225 https\://repo1.maven.org/maven2

Bugs closed: 438

Reviewed at https://rbcommons.com/s/twitter/r/824/
Add a new jvm specific fingerprinting for reintroducing the java and scala versions into the artifact hash. This is needed to avoid build cache artifacts mixing classes from different versions.

Reviewed here: https://rbcommons.com/s/twitter/r/808/
foo/README.md and bar/README.md don't clobber each other.

'./pants goal markdown ::' has a problem: every .../README.md file generates to the same path (.pants.d/markdown/markdown/html/README.html)

With this change, you instead get
.pants.d/markdown/markdown/html/_.readme.html
.pants.d/markdown/markdown/html/_src.java.com.pants.examples.hello.main.readme.html
.pants.d/markdown/markdown/html/_src.java.com.pants.testproject.page.readme.html
.pants.d/markdown/markdown/html/_src.python.pants.backend.python.readme.html
.pants.d/markdown/markdown/html/_src.python.pants.readme.html

Also in this change:
Turned some unnecessarily complex python into simple mustache templates.

Added src/java/com/pants/testproject/page to make sure I didn't break [[pants(...)]] "wikilinks".
Doesn't contain java, despite the dir.

Testing Done:
(got confluence goal temporarily working to make sure that markdown-page-moves didn't break confluence publish)

https://travis-ci.org/pantsbuild/pants/builds/31464526

Reviewed at https://rbcommons.com/s/twitter/r/750/