Fix transitive dependencies for GroupIterator/GroupTask

A Java target A has a transitive dep on a Scala target C via an
intermediate JarLibrary/Dependencies target B (ie, "un-flavored" from
GroupTask's perspective.) GroupIterator's call to sort_targets correctly
includes target B, which results in a topsort of 'C <- B <- A'. But
coalesce_targets drops target B, because it doesn't match the
discriminator ("is unflavored".) Because target B is ignored and
coalesce only looks at direct deps, C is allowed to be reordered before
A.

- Must not reorder past an intermediate (dropped) target in
  GroupIterator
  - Allow for more subclasses of GroupTaskTest
  - Add reproduction unit test

https://rbcommons.com/s/twitter/r/706/
junit_run needs to setup resources before executing.

RB_ID=https://rbcommons.com/s/twitter/r/709/
Create an internal version of just the parts of t.c.dirutil we need.

Switch a handful of uses over.

This is to reduce our deps on twitter.commons.
don't insist on @manual.builddict; rm it from many places

@manual.builddict is mostly redundant now. So clean clean clean.

Change the BUILD-Dictionary-building logic: document registered things; not just those registered things that were tagged @manual.builddict.

added some docstrings
lambda x : foo x loses foo's docstring, so add a helper func to carry that along

Motivation:

A couple of refactorings ago, we had lots of symbols def'd in BUILD files we didn't want folks to use. "Yeah, you _could_ use JvmTarget for that, but please use java_library instead." We used @manual.builddict to carefully mark those things we wanted to show up in the BUILD Dictionary.

In these enlightened days, we carefully craft our register.py files to specify the symbols we want. Thus, @manual.builddict is mostly a duplication of effort. (It's still useful for methods: most methods shouldn't be used in BUILD files. But a few of them are still around. So, even with this change, we mark those with @manual.builddict)

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/30317126

publish_docs.sh and eyeballing

Reviewed at https://rbcommons.com/s/twitter/r/703/
Copy a dirutil unittest from commons.
Make the dirutil test a proper unittest2 test.

Use unittest asserts instead of the assert keyword, use setUp/tearDown etc.

Also make sure we start with a clean slate, in case other tests use
the atexit functionality.
Remove all remaining deps on the trivial parts of t.c.dirutil.
Our own copy of contextutils.

Make its tests properly unittest-y (use test asserts etc.)

Make the timer test use a fake clock instead of relying on system clock behavior.

Change a handful of callsites to use the new location.
Fixup broken bundle alias.

Bundle was incorrectly saving mutable state at BUILD file scope instead of
per-use scope.  This change adds a factory method at the BUILD file scope
(parse_context), and collapses the __init__ and __call__ into one. This is both
clearer and now not broken.

Add a test to exercise this - creating >1 bundle in a BUILD file.

In the course of adding this test, find and fix a bug in BuildFileAliases.merge.
Consequently, add a comprehensive test for BuildFileAliases.

Testing Done:
$ PANTS_DEV=1 ./pants goal test tests/python/pants_test/base:build_file_aliases tests/python/pants_test/targets/:jvm_app --test-pytest-options=-v
*** Running pants in dev mode from /home/jsirois/dev/3rdparty/jsirois-pants/src/python/pants/bin/pants_exe.py ***
...
19:06:07 00:00   [test]
19:06:07 00:00     [pytest]
19:06:07 00:00       [run]
                     ============== test session starts ===============
                     platform linux2 -- Python 2.6.9 -- py-1.4.22 -- pytest-2.6.0 -- /usr/bin/python2.6
                     plugins: cov, timeout
                     collected 15 items

                     tests/python/pants_test/base/test_build_file_aliases.py@15::BuildFileAliasesTest::test_create PASSED
                     tests/python/pants_test/base/test_build_file_aliases.py@61::BuildFileAliasesTest::test_curry_context PASSED
                     tests/python/pants_test/base/test_build_file_aliases.py@72::BuildFileAliasesTest::test_merge PASSED
                     tests/python/pants_test/targets/test_jvm_app.py@36::BinaryTest::test_binary_via_binary PASSED
                     tests/python/pants_test/targets/test_jvm_app.py@49::BinaryTest::test_binary_via_dependencies PASSED
                     tests/python/pants_test/targets/test_jvm_app.py@62::BinaryTest::test_degenerate_binaries PASSED
                     tests/python/pants_test/targets/test_jvm_app.py@76::BinaryTest::test_no_binary PASSED
                     tests/python/pants_test/targets/test_jvm_app.py@115::BinaryTest::test_not_a_binary PASSED
                     tests/python/pants_test/targets/test_jvm_app.py@88::BinaryTest::test_too_many_binaries_mixed PASSED
                     tests/python/pants_test/targets/test_jvm_app.py@102::BinaryTest::test_too_many_binaries_via_deps PASSED
                     tests/python/pants_test/targets/test_jvm_app.py@201::BundleTest::test_bundle_add_add PASSED
                     tests/python/pants_test/targets/test_jvm_app.py@158::BundleTest::test_bundle_filemap_dest_byglobs PASSED
                     tests/python/pants_test/targets/test_jvm_app.py@139::BundleTest::test_bundle_filemap_dest_bypath PASSED
                     tests/python/pants_test/targets/test_jvm_app.py@180::BundleTest::test_bundle_filemap_dest_relative PASSED
                     tests/python/pants_test/targets/test_jvm_app.py@226::BundleTest::test_multiple_bundles PASSED

                     =========== 15 passed in 0.35 seconds ============

19:06:09 00:02     [junit]
19:06:09 00:02     [specs]
               SUCCESS

Bugs closed: 375, 377

Reviewed at https://rbcommons.com/s/twitter/r/722/
Fixup test fallout from https://rbcommons.com/s/twitter/r/722/

Aka - always run ci.sh or check out Travis.

Switch from manual alias_groups setup to letting the backends
do their thing for continued robustness at the test's current
abstraction level.

Testing Done:
I noticed travis-ci was broken for-real here: https://travis-ci.org/pantsbuild/pants/jobs/30611368
From my change here: https://rbcommons.com/s/twitter/r/722/

This time did the right thing and the following is green:
./build-support/bin/ci.sh

Bugs closed: 378

Reviewed at https://rbcommons.com/s/twitter/r/724/
Fix misleading package description.

Testing Done:
$ PANTS_DEV=1 ./pants setup_py src/python/pants:pants-packaged
*** Running pants in dev mode from /home/jsirois/dev/3rdparty/jsirois-pants/src/python/pants/bin/pants_exe.py ***
WARNING!  /home/jsirois/dev/3rdparty/jsirois-pants/dist/pantsbuild.pants.FT6ETM/src/pants/docs/conf.py is source but does not belong to a package!
WARNING!  /home/jsirois/dev/3rdparty/jsirois-pants/dist/pantsbuild.pants.FT6ETM/src/pants/docs/pagerank.py is source but does not belong to a package!
Running packager against /home/jsirois/dev/3rdparty/jsirois-pants/dist/pantsbuild.pants-0.0.19
Writing /home/jsirois/dev/3rdparty/jsirois-pants/dist/pantsbuild.pants-0.0.19.tar.gz
$ cd dist/
$ tar -xzf pantsbuild.pants-0.0.19.tar.gz
$ cd pantsbuild.pants-0.0.19/
$ python setup.py --description
A bottom-up build tool.

Bugs closed: 381

Reviewed at https://rbcommons.com/s/twitter/r/725/
Require information about jars in depmap with --depmap-project-info

Reviewed at https://rbcommons.com/s/twitter/r/721
Expose types needed to specify jvm_binary.deploy_jar_rules.

This also fixes up the pydoc strings to get full BUILD dictionary
linkage from jvm_binary through jar_rules to Skip and Duplicate.

Testing Done:
$ ./build-support/bin/publish_docs.sh -opd jsirois

See here:
http://pantsbuild.github.io/jsirois/build_dictionary.html#jvm_binary
http://pantsbuild.github.io/jsirois/build_dictionary.html#bdict-jar-rules
http://pantsbuild.github.io/jsirois/build_dictionary.html#bdict-skip
http://pantsbuild.github.io/jsirois/build_dictionary.html#bdict-duplicate

Bugs closed: 383, 384

Reviewed at https://rbcommons.com/s/twitter/r/727/
This commit is comprised of the following 2 dependent RBs:

1.) Add AndroidTask and AaptGen task

This adds an AndroidTask base task and an AaptGen codegen task using the aapt tool.
The AndroidTask merely obtains an AndroidDistribution for subclasses to work with.

The AaptTask processes resource folders and outputs an R.java class. This will be
compiled along with the regular .java files in the android_compile phase coming up. The
aapt tool itself packs an enormous amount of action and this class will continue to
grow as Android support grows. I could see an AaptMixin or AaptBase class in the future.

The aapt_gen class allows a command-line override of the target_sdk and build-tools versions
set by the manifest parser in the AndroidTarget class.

Currently the SyntheticTarget output by the aaptGen class is a JavaLibrary. This will likely be
changed as I create a task and target to compile .java files for Android builds next.

Testing Done:
CI.sh passes locally.

Travis is currently failing elsewhere, but I have
manually tested all of the setup_parser flags, including the new
ignore-assets flag.

Reviewed at https://rbcommons.com/s/twitter/r/672/

2.) Refactor AndroidDistribution to remove SDK checks in __init__

This review is dependent on 672. I kept this
separate to make reviewing (moderately) palatable.

This is a refactor of AndroidDistribution. Instead of validating sdk_paths at instantiation, we
are now only validating SDKs when a tool is requested by a path. This makes better sense, in
that we would prefer to know about the tools available to us instead of the dirs that
traditionally contain them. It also moves action out of _init__. The AndroidTask instantiates
an AndroidDistribution but with unverified variables until the tools are needed.

I also added a command-line flag allowing users to specify a path of an Android SDK if they want.
This is an improvement over the earlier model that only cycled through common environmental aliases.

Users are likely to have just one SDK installed, with the issue being what tools are installed. This
checks the tools when the tasks that use them are invoked. If the user
has a custom SDK, they can now pass that at invocation.

This allows CI and Travis to pass (although Travis is failing on an unrelated issue currently, see below).

test_android_distribution was also refactored.

Testing Done:
CI.sh passes locally.
Travis passes  (with 672 patched in as well) [0].

[0] https://travis-ci.org/pantsbuild/pants/builds/30097096

Reviewed at https://rbcommons.com/s/twitter/r/676/
Update android target definitions

This is a follow-up based on the comments to merged CR #668.
This can be safely merged without relying on the other two
open reviews.

This gets rid of a couple of the "sensible defaults". I believe
in them as useful, but they have could proven confusing for anyone
with a non-traditional (or just maven-style) layout.

This requires all android targets to explicitly define a "manifest"
field. Android_resource targets must further define a
"resource_dir". I named it "resource_dir" instead of "resources"
to contrast with what pants expects elsewhere: the tool that
processes android resources expects a directory as input, not
a list of files. The aapt tool spiders down the dir, so the
user must specify which directory holds those resources.

Somewhere down the line, I would like to support common layouts
by default. But the way it was before this commit, it implied
that we supported things that we do not.

Testing Done:
local ci.sh passes, android targets can be processed.

Travis looks to be down. [0]

[0] https://travis-ci.org/pantsbuild/pants/builds/30210018

Reviewed at https://rbcommons.com/s/twitter/r/700/
Add support for the ragel state-machine generator.

Testing Done:
Ran the new unit tests.

Also converted wikid in birdcage to use this, and tested it.  That's on birdcage in dturner-ragel-oss-pants.

Bugs closed: https://github.com/pantsbuild/pants/issues/353

Reviewed at https://rbcommons.com/s/twitter/r/678/
expose ConfluencePublish to BUILD files; remove a lie from page.rst

http://pantsbuild.github.io/page.html#publishing says I can get my org set up with confluence-publishing via a few steps , including adding to a BUILD file...

class OurConfluence(ConfluencePublish):

That was a funny troll, but we should add ConfluencePublish to the register.py so folks really can use it from a BUILD file.

Also, tweak the page.html doc to "include" code that we know compiles OK.

Testing Done:
With this change and with other things hinted at in https://github.com/pantsbuild/pants/issues/385 and https://github.com/pantsbuild/pants/issues/387 , got things set up on my laptop so I could publish to Twitter's wiki. (thus reassuring myself that the change I _wanted_ to make doesn't break confluence publishing, yay)

Furthermore, added a test
https://travis-ci.org/pantsbuild/pants/builds/30708337

Bugs partially addressed: 385

Reviewed at https://rbcommons.com/s/twitter/r/728/
stop claiming Twitter is sole doc author. fix sphinx warnings

This change started out as fixing up a couple of (unused) Sphinx-doc config settings; If we ever exercised those settings, they woulda claimed Twitter was sole docs author. Not true nope nope nope.

While making sure that I didn't break Sphinx, I went on a warning-squishing rampage.

Testing Done:
./build-support/bin/publish_docs.sh

Reviewed at https://rbcommons.com/s/twitter/r/729/
Move all remaining uses of t.c.contextutil to pants.util.contextutil.
Get rid of the t.c.contextutil dep entirely.

- Cleaned up a stray use in code that got committed after my previous change.
- Also cleaned up style in that code. Will ding code-reviewer for not addressing.
- Fixed some missing deps.