Author: Dan Davydov <davydovdan@gmail.com>
Date:   Fri Jul 18 10:39:00 2014 -0700

    Enable Nailgun Per Task

    For discussion around this patch please see: https://groups.google.com/forum/#!searchin/pants-devel/nailgun/pants-devel/wcvYfUvhPeg/uyFfy6_IqPEJ
    Note for antlr3 / antlr4 I made their ng-killall option under a new more generic [antlr] config section. Also note that any classes that extend NailgunTask must supply
    the config_section constructor parameter (otherwise when nailgun is attempted to be used it will fail), this is intentional
    (though this would obviously be better do to during compile time rather than run time).

    Also did a bit of style/duplicate code refactoring.

    Testing Done:
    Added print statements to nailgun/non-nailgun execution paths, and tested different combinations of pants configs:
    -No use_nailgun flags (should always use nailgun)
    -use_nailgun: False in [DEFAULT] (should never use nailgun)
    -use_nailgun: False in [DEFAULT] and use_nailgun: True in [scala-compile] (nailgun was only used for scala)

    Also
    ./build-support/ci.sh

    Reviewed at https://rbcommons.com/s/twitter/r/687/
Fix pants test libraries missing dependencies

Currently on master if you don't have PANTS_DEV=1, pants_test:all target fails.

Fixed following python test targets:
1. tests/python/pants_test/test_utf8_header.py F due to
BackendConfigurationError: Failed to load the pants.backend.core.register backend: No module named goal

2.tests/python/pants_test/base/test_build_configuration.py:16: in <module>
    from pants.base.build_configuration import BuildConfiguration
/var/folders/t_/ck6c3zvj2z5cg77dqx2kh8tr0000gp/T/tmpy8_YJ_/pants/base/build_configuration.py:12: in <module>
    from pants.base.build_file_aliases import BuildFileAliases
E   ImportError: No module named build_file_aliases

3. tests/python/pants_test/java/test_executor.py:18: in <module>
    from pants.java.executor import SubprocessExecutor
/var/folders/t_/ck6c3zvj2z5cg77dqx2kh8tr0000gp/T/tmp3lxmyq/pants/java/executor.py:18: in <module>
    from pants.base.build_environment import get_buildroot
E   ImportError: No module named build_environment

4. >   ???
E   ImportError: No module named authentication.netrc_util

Testing Done:
CI passes as usual.
https://travis-ci.org/pantsbuild/pants/builds/30308887

On my branch:- Following command succeeds.
 rm pants.pex && ./pants tests/python/pants_test:all

Bugs closed: 355

Reviewed at https://rbcommons.com/s/twitter/r/685/
rm doc-generation code that made mostly-unused docs

This code worked hard to generate some documentation pages.
The Google analytics numbers point out: these pages were largely unread.
Some (but not all) of this code broke in a refactor: it generated empty docs.
Yet nobody complained.

Rather than work hard to fix and maintain this code, I'm ripping it out.
Behold the fiery debate: https://groups.google.com/forum/#!topic/pants-devel/f7V4_6kLdZs

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/30318159

Reviewed at https://rbcommons.com/s/twitter/r/704/
This is a fix for issue #367

Objects of class BinaryNotFound can not be instantiated because the constructor does not qualify the class name with its parent when it invokes the constructor of the superclass.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/30476122

Reviewed at https://rbcommons.com/s/twitter/r/705/
Resurrected pants goal idea.

This mostly involved updating outdated references, and making utility functions to mimick functionality refactored long ago.

Source and Javadoc jars are not mapped in, and I'm not quite sure how to get that working again. I'd appreciate any input on that matter.

The code in idea_gen and ide_gen isn't very clean, because it wasn't very clean in the first place. It would be good to go back at some point and tidy it up.

Idea gen is also much more efficient than it used to be, thanks to the round manager: it now avoids compiling or jarring anything, which saves tons of time. It would still be nice to include a flag to optionally disable codegen, but I'm not sure how (un)trivial that is at this juncture.

Testing Done:
Added (passing) integration tests under tasks/test_idea_integration.py
Tests under tasks:integration pass.

Bugs closed: 349

Reviewed at https://rbcommons.com/s/twitter/r/695/
Upgrade to twitter.common.python 0.5.10.

This brings in the following bugfix:

"Update the mtime on retranslation of existing distributions.

1bff97e stopped existing distributions from being overwritten, to
prevent subtle errors. However without updating the mtime these
distributions will appear to be permanently expired wrt the ttl."
Made a small change to stop problems we were getting compiling our annotation processors.

This is related to the pants-devel discussion 'Nondeterministic Phase Execution Order'. Briefly:
when the resources phase happens to precede the compile phase, resources in .pants.d  pollute the
jmake classpath when compiling some of Square's internal annotation processors. This makes jmake
quite irate, so this code stops that from happening.

This fix is not terribly elegant, in that it doesn't address the underlying problem of resources
getting on the classpath in the first place. Perhaps this problem will go away when a formal
classpath object is created.

Testing Done:
Code which didn't compile now compiles.

Ran test tests/python/pants_test:: and nothing fails that doesn't also fail on the master branch.

Bugs closed: 373

Reviewed at https://rbcommons.com/s/twitter/r/708/
Use resources from junit_tests, in addition to from jvm_binary

https://rbcommons.com/s/twitter/r/711/
Fix transitive dependencies for GroupIterator/GroupTask

A Java target A has a transitive dep on a Scala target C via an
intermediate JarLibrary/Dependencies target B (ie, "un-flavored" from
GroupTask's perspective.) GroupIterator's call to sort_targets correctly
includes target B, which results in a topsort of 'C <- B <- A'. But
coalesce_targets drops target B, because it doesn't match the
discriminator ("is unflavored".) Because target B is ignored and
coalesce only looks at direct deps, C is allowed to be reordered before
A.

- Must not reorder past an intermediate (dropped) target in
  GroupIterator
  - Allow for more subclasses of GroupTaskTest
  - Add reproduction unit test

https://rbcommons.com/s/twitter/r/706/
junit_run needs to setup resources before executing.

RB_ID=https://rbcommons.com/s/twitter/r/709/
Create an internal version of just the parts of t.c.dirutil we need.

Switch a handful of uses over.

This is to reduce our deps on twitter.commons.
don't insist on @manual.builddict; rm it from many places

@manual.builddict is mostly redundant now. So clean clean clean.

Change the BUILD-Dictionary-building logic: document registered things; not just those registered things that were tagged @manual.builddict.

added some docstrings
lambda x : foo x loses foo's docstring, so add a helper func to carry that along

Motivation:

A couple of refactorings ago, we had lots of symbols def'd in BUILD files we didn't want folks to use. "Yeah, you _could_ use JvmTarget for that, but please use java_library instead." We used @manual.builddict to carefully mark those things we wanted to show up in the BUILD Dictionary.

In these enlightened days, we carefully craft our register.py files to specify the symbols we want. Thus, @manual.builddict is mostly a duplication of effort. (It's still useful for methods: most methods shouldn't be used in BUILD files. But a few of them are still around. So, even with this change, we mark those with @manual.builddict)

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/30317126

publish_docs.sh and eyeballing

Reviewed at https://rbcommons.com/s/twitter/r/703/
Copy a dirutil unittest from commons.
Make the dirutil test a proper unittest2 test.

Use unittest asserts instead of the assert keyword, use setUp/tearDown etc.

Also make sure we start with a clean slate, in case other tests use
the atexit functionality.
Remove all remaining deps on the trivial parts of t.c.dirutil.
Our own copy of contextutils.

Make its tests properly unittest-y (use test asserts etc.)

Make the timer test use a fake clock instead of relying on system clock behavior.

Change a handful of callsites to use the new location.
Fixup broken bundle alias.

Bundle was incorrectly saving mutable state at BUILD file scope instead of
per-use scope.  This change adds a factory method at the BUILD file scope
(parse_context), and collapses the __init__ and __call__ into one. This is both
clearer and now not broken.

Add a test to exercise this - creating >1 bundle in a BUILD file.

In the course of adding this test, find and fix a bug in BuildFileAliases.merge.
Consequently, add a comprehensive test for BuildFileAliases.

Testing Done:
$ PANTS_DEV=1 ./pants goal test tests/python/pants_test/base:build_file_aliases tests/python/pants_test/targets/:jvm_app --test-pytest-options=-v
*** Running pants in dev mode from /home/jsirois/dev/3rdparty/jsirois-pants/src/python/pants/bin/pants_exe.py ***
...
19:06:07 00:00   [test]
19:06:07 00:00     [pytest]
19:06:07 00:00       [run]
                     ============== test session starts ===============
                     platform linux2 -- Python 2.6.9 -- py-1.4.22 -- pytest-2.6.0 -- /usr/bin/python2.6
                     plugins: cov, timeout
                     collected 15 items

                     tests/python/pants_test/base/test_build_file_aliases.py@15::BuildFileAliasesTest::test_create PASSED
                     tests/python/pants_test/base/test_build_file_aliases.py@61::BuildFileAliasesTest::test_curry_context PASSED
                     tests/python/pants_test/base/test_build_file_aliases.py@72::BuildFileAliasesTest::test_merge PASSED
                     tests/python/pants_test/targets/test_jvm_app.py@36::BinaryTest::test_binary_via_binary PASSED
                     tests/python/pants_test/targets/test_jvm_app.py@49::BinaryTest::test_binary_via_dependencies PASSED
                     tests/python/pants_test/targets/test_jvm_app.py@62::BinaryTest::test_degenerate_binaries PASSED
                     tests/python/pants_test/targets/test_jvm_app.py@76::BinaryTest::test_no_binary PASSED
                     tests/python/pants_test/targets/test_jvm_app.py@115::BinaryTest::test_not_a_binary PASSED
                     tests/python/pants_test/targets/test_jvm_app.py@88::BinaryTest::test_too_many_binaries_mixed PASSED
                     tests/python/pants_test/targets/test_jvm_app.py@102::BinaryTest::test_too_many_binaries_via_deps PASSED
                     tests/python/pants_test/targets/test_jvm_app.py@201::BundleTest::test_bundle_add_add PASSED
                     tests/python/pants_test/targets/test_jvm_app.py@158::BundleTest::test_bundle_filemap_dest_byglobs PASSED
                     tests/python/pants_test/targets/test_jvm_app.py@139::BundleTest::test_bundle_filemap_dest_bypath PASSED
                     tests/python/pants_test/targets/test_jvm_app.py@180::BundleTest::test_bundle_filemap_dest_relative PASSED
                     tests/python/pants_test/targets/test_jvm_app.py@226::BundleTest::test_multiple_bundles PASSED

                     =========== 15 passed in 0.35 seconds ============

19:06:09 00:02     [junit]
19:06:09 00:02     [specs]
               SUCCESS

Bugs closed: 375, 377

Reviewed at https://rbcommons.com/s/twitter/r/722/
Fixup test fallout from https://rbcommons.com/s/twitter/r/722/

Aka - always run ci.sh or check out Travis.

Switch from manual alias_groups setup to letting the backends
do their thing for continued robustness at the test's current
abstraction level.

Testing Done:
I noticed travis-ci was broken for-real here: https://travis-ci.org/pantsbuild/pants/jobs/30611368
From my change here: https://rbcommons.com/s/twitter/r/722/

This time did the right thing and the following is green:
./build-support/bin/ci.sh

Bugs closed: 378

Reviewed at https://rbcommons.com/s/twitter/r/724/
Fix misleading package description.

Testing Done:
$ PANTS_DEV=1 ./pants setup_py src/python/pants:pants-packaged
*** Running pants in dev mode from /home/jsirois/dev/3rdparty/jsirois-pants/src/python/pants/bin/pants_exe.py ***
WARNING!  /home/jsirois/dev/3rdparty/jsirois-pants/dist/pantsbuild.pants.FT6ETM/src/pants/docs/conf.py is source but does not belong to a package!
WARNING!  /home/jsirois/dev/3rdparty/jsirois-pants/dist/pantsbuild.pants.FT6ETM/src/pants/docs/pagerank.py is source but does not belong to a package!
Running packager against /home/jsirois/dev/3rdparty/jsirois-pants/dist/pantsbuild.pants-0.0.19
Writing /home/jsirois/dev/3rdparty/jsirois-pants/dist/pantsbuild.pants-0.0.19.tar.gz
$ cd dist/
$ tar -xzf pantsbuild.pants-0.0.19.tar.gz
$ cd pantsbuild.pants-0.0.19/
$ python setup.py --description
A bottom-up build tool.

Bugs closed: 381

Reviewed at https://rbcommons.com/s/twitter/r/725/
Require information about jars in depmap with --depmap-project-info

Reviewed at https://rbcommons.com/s/twitter/r/721