get 'goal builddict' and 'tests/python/pants_test/tasks:builddict' working again

un-break/re-enable 'builddict' goal and its test

As part of this added a class function to BuildFileParser: report_registered_context()
Returns a dict of symbols defined in BUILD files. This dict is quite useful for code that spews docs about symbols defined in BUILD files.

somewhat sad, but tolerable: many target constructor sigs now look roughly like __init__(arg, kwargs):Super(...).__init__(args,kwargs) . Thus, several of our formerly-detailed function sig docs are now less informative "args, kwargs". Maybe if I dig around in pydoc code and/or other reflection-y stuff, there's some straightforward way to do something lovely that automagically accumulates a method signature up the inheritance tree. I won't try to figure that out today, though.

Testing Done:
a lot of looking at old-Pants output vs new-Pants output, reconciling differences

re-enabled a test. and got it working again.

./build-support/bin/ci.sh

Reviewed at https://rbcommons.com/s/twitter/r/362/
Make classpath components relative to the current directory to solve problem of very long classpaths in jobs running under mesos.
Add resources to examples

* Add a resources subdirectory, and add it as a dependency of the java/scala
  examples
* Confirm loading the resource from the classpath
    Fixed misleading typo in pants.ini
Set no_color=True when TERM=dumb

  reviewed at https://rbcommons.com/s/twitter/r/370/
docs: don't suggest './rbt status'. That fails. Instead, './rbt help'

For the background, see this mail thread
https://groups.google.com/forum/#!topic/pants-devel/oZvsn_LzDNU

Testing Done:
moved my cached cookie aside
ran ./rbt commands
This review posted OK.

Bugs closed: https://github.com/pantsbuild/pants/issues/136

Reviewed at https://rbcommons.com/s/twitter/r/384/
Updated cache_validation exception handler to catch all types of exceptions.

This change will catch and re-throw with the original stack trace, enhancing the exception message with the name of the pants target and location of the build file.

Was:
*** Running pants in dev mode from /Users/zundel/Src/pants/src/python/pants/bin/pants_exe.py ***

10:18:40 00:00 [main]
               See a report at: http://localhost:50651/run/pants_run_2014_05_22_10_18_40_62
10:18:40 00:00   [bootstrap]
10:18:40 00:00   [setup]
10:18:40 00:00     [parse]
10:18:40 00:00   [jvm-locate]
10:18:40 00:00   [jvm-locate]
10:18:40 00:00   [jvm-locate]
10:18:40 00:00   [jvm-locate]
10:18:40 00:00   [jvm-locate]
10:18:40 00:00   [jvm-locate]
10:18:40 00:00   [bootstrap]
10:18:40 00:00     [bootstrap-jvm-tools]
10:18:40 00:00   [gen]
10:18:40 00:00     [thrift]
10:18:40 00:00     [scrooge]
10:18:40 00:00     [protoc]

               FAILURE
Traceback (most recent call last):
  File "/Users/zundel/Src/pants/src/python/pants/bin/pants_exe.py", line 175, in <module>
    main()
  File "/Users/zundel/Src/pants/src/python/pants/bin/pants_exe.py", line 169, in main
    _run()
  File "/Users/zundel/Src/pants/src/python/pants/bin/pants_exe.py", line 150, in _run
    result = command.run(lock)
  File "/Users/zundel/Src/pants/src/python/pants/commands/goal.py", line 234, in run
    return engine.execute(context, self.phases)
  File "/Users/zundel/Src/pants/src/python/pants/engine/engine.py", line 49, in execute
    self.attempt(context, phases)
  File "/Users/zundel/Src/pants/src/python/pants/engine/group_engine.py", line 278, in attempt
    phase_executor.attempt(explain)
  File "/Users/zundel/Src/pants/src/python/pants/engine/group_engine.py", line 170, in attempt
    execute_task(goal, self._tasks_by_goal[goal], self._context.targets())
  File "/Users/zundel/Src/pants/src/python/pants/engine/group_engine.py", line 141, in execute_task
    task.execute(targets)
  File "/Users/zundel/Src/pants/src/python/pants/tasks/code_gen.py", line 97, in execute
    with self.invalidated(gentargets, invalidate_dependents=True) as invalidation_check:
  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/Users/zundel/Src/pants/src/python/pants/tasks/task.py", line 195, in invalidated
    invalidation_check = cache_manager.check(targets, partition_size_hint, colors)
  File "/Users/zundel/Src/pants/src/python/pants/base/cache_manager.py", line 217, in check
    all_vts = self._sort_and_validate_targets(targets)
  File "/Users/zundel/Src/pants/src/python/pants/base/cache_manager.py", line 242, in _sort_and_validate_targets
    cache_key = self._key_for(target, transitive=self._invalidate_dependents)
  File "/Users/zundel/Src/pants/src/python/pants/base/cache_manager.py", line 262, in _key_for
    return self._cache_key_generator.key_for_target(target, transitive=transitive)
  File "/Users/zundel/Src/pants/src/python/pants/base/build_invalidator.py", line 85, in key_for_target
    target_key = target.transitive_invalidation_hash()
  File "/Users/zundel/Src/pants/src/python/pants/base/target.py", line 179, in transitive_invalidation_hash
    target_hash = self.invalidation_hash()
  File "/Users/zundel/Src/pants/src/python/pants/base/target.py", line 162, in invalidation_hash
    self._cached_invalidation_hash = self.compute_invalidation_hash()
  File "/Users/zundel/Src/pants/src/python/pants/base/target.py", line 158, in compute_invalidation_hash
    return self.payload.invalidation_hash()
  File "/Users/zundel/Src/pants/src/python/pants/base/payload.py", line 112, in invalidation_hash
    sources_hash = hash_sources(get_buildroot(), self.sources_rel_path, self.sources)
  File "/Users/zundel/Src/pants/src/python/pants/base/payload.py", line 22, in hash_sources
    with open(os.path.join(root_path, rel_path, source), 'rb') as f:
  File "/Users/zundel/Development/java/build-support/pants_dev_deps.venv/bin/../lib/python2.7/posixpath.py", line 75, in join
    if b.startswith('/'):
AttributeError: 'Fileset' object has no attribute 'startswith'

Now:
  ...

  File "/Users/zundel/Src/pants/src/python/pants/base/target.py", line 158, in compute_invalidation_hash
    return self.payload.invalidation_hash()
  File "/Users/zundel/Src/pants/src/python/pants/base/payload.py", line 112, in invalidation_hash
    sources_hash = hash_sources(get_buildroot(), self.sources_rel_path, self.sources)
  File "/Users/zundel/Src/pants/src/python/pants/base/payload.py", line 22, in hash_sources
    with open(os.path.join(root_path, rel_path, source), 'rb') as f:
  File "/Users/zundel/Development/java/build-support/pants_dev_deps.venv/bin/../lib/python2.7/posixpath.py", line 75, in join
    if b.startswith('/'):
pants.base.cache_manager.CacheValidationError: Problem validating target squarepants.hack-protos.src.main.proto.proto in squarepants/hack-protos/src/main/proto: 'Fileset' object has no attribute 'startswith'

Testing Done:
This is just a diagnostic - I hope we cache problems like this explicitly before it comes down to an unexpected problem in validation.  I updated the exception handling to help me track down which BUILD is causing a problem after the refactor... I'll track down that issue separately.

Bugs closed: 145

Reviewed at https://rbcommons.com/s/twitter/r/396/
Only run tests once each...

This was due to the regular target dep-following logic being duplicated
by legacy python_test_suite logic.

This change gets rid of python_test_suite (or rather, makes it
an alias of the 'dependencies' target, to not break existing BUILD files).

We're now back to taking ~2m20s to run all the pants tests.
Add missing dep.
Fix some broken codegen target synthesis code.

Also got rid of bogus duplicate is_synthetic concept.
Quick fix for gen_java/gen_python in codegen.  I ran into issues with running codegen because the task was trying to access the build graph during its __init__.

Testing Done:
CI passes

Reviewed at https://rbcommons.com/s/twitter/r/400/
Add non-closure injection methods to BuildGraph

This came up for optimizing an internal implementation of WhatChanged.

Testing Done:
CI passes

Reviewed at https://rbcommons.com/s/twitter/r/399/
Name codegen synth targets by target id, instead of name.

Before, we were just getting lucky that we didn't have two colliding target names for our codegen targets, but their synthetic targets should be 1:1 with their (language, build file target) tuple.

Testing Done:
CI passes

Bugs closed: https://github.com/pantsbuild/pants/pull/148

Reviewed at https://rbcommons.com/s/twitter/r/398/
doc: pants.new.rc is old, pants.rc is new

Bugs closed: https://github.com/pantsbuild/pants/issues/137

Reviewed at https://rbcommons.com/s/twitter/r/401/
Fix publishing and also add integration tests

Fix publishing errors
1. Fix no config javadoc found for targets with no depdencies.
==== stderr ====
                     Exception in thread "main" java.text.ParseException: Problem occurred while parsing ivy file: Cannot add artifact 'example.jar(javadoc)' to configuration 'javadoc' of module com.twitter#example;0.0.1-SNAPSHOT because this configuration doesn't exist! in file:/Users/tdesai/projects/pants/.pants.d/publish/publish/com.twitter/example/ivy-0.0.1-SNAPSHOT.xml

2. Add Integration tests

Testing Done:
locally:
 ./pants tests/python/pants_test/tasks:jar_publish_integratio
tests/python/pants_test/tasks/test_jar_publish_integration.py ..

============================================================================= 2 passed in 23.32 seconds =============================================================================
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/private/var/folders/t_/ck6c3zvj2z5cg77dqx2kh8tr0000gp/T/tmpKYikI8/.deps/twitter.common.dirutil-0.3.0-py26-none-any.whl/twitter/common/dirutil/__init__.py", line 61, in _mkdtemp_atexit_cleaner
    for td in _MKDTEMP_DIRS.pop(os.getpid(), []):
AttributeError: 'NoneType' object has no attribute 'pop'
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/private/var/folders/t_/ck6c3zvj2z5cg77dqx2kh8tr0000gp/T/tmpKYikI8/.deps/twitter.common.dirutil-0.3.0-py26-none-any.whl/twitter/common/dirutil/__init__.py", line 61, in _mkdtemp_atexit_cleaner
    for td in _MKDTEMP_DIRS.pop(os.getpid(), []):
AttributeError: 'NoneType' object has no attribute 'pop'
tests.python.pants_test.tasks.jar_publish_integration                           .....   SUCCESS
[tw-172-25-20-142 pants]$ ./pants tests/python/pants_test/tasks:jar_publish_integration
Build operating on targets: OrderedSet([PythonTests(tests/python/pants_test/tasks/BUILD:jar_publish_integration)])

Bugs closed: 120

Reviewed at https://rbcommons.com/s/twitter/r/356/
dev docs: += some debugging advice from a mail thread

Reviewed at https://rbcommons.com/s/twitter/r/404/
Author: Tejal Desai <tejal29@gmail.com>
Date:   Fri May 23 13:17:18 2014 -0700

    Fix publishing and also add integration tests

    Fix publishing errors
    1. Fix no config javadoc found for targets with no depdencies.
    ==== stderr ====
                         Exception in thread "main" java.text.ParseException: Problem occurred while parsing ivy file: Cannot add artifact 'example.jar(javadoc)' to configuration

    2. Add Integration tests

    Testing Done:
    locally:
     ./pants tests/python/pants_test/tasks:jar_publish_integratio
    tests/python/pants_test/tasks/test_jar_publish_integration.py ..

    ============================================================================= 2 passed in 23.32 seconds =======================================================================
    Error in atexit._run_exitfuncs:
    Traceback (most recent call last):
      File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/atexit.py", line 24, in _run_exitfuncs
        func(*targs, **kargs)
      File "/private/var/folders/t_/ck6c3zvj2z5cg77dqx2kh8tr0000gp/T/tmpKYikI8/.deps/twitter.common.dirutil-0.3.0-py26-none-any.whl/twitter/common/dirutil/__init__.py", line 61, i
        for td in _MKDTEMP_DIRS.pop(os.getpid(), []):
    AttributeError: 'NoneType' object has no attribute 'pop'
    Error in sys.exitfunc:
    Traceback (most recent call last):
      File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/atexit.py", line 24, in _run_exitfuncs
        func(*targs, **kargs)
      File "/private/var/folders/t_/ck6c3zvj2z5cg77dqx2kh8tr0000gp/T/tmpKYikI8/.deps/twitter.common.dirutil-0.3.0-py26-none-any.whl/twitter/common/dirutil/__init__.py", line 61, i
        for td in _MKDTEMP_DIRS.pop(os.getpid(), []):
    AttributeError: 'NoneType' object has no attribute 'pop'
    tests.python.pants_test.tasks.jar_publish_integration                           .....   SUCCESS
    [tw-172-25-20-142 pants]$ ./pants tests/python/pants_test/tasks:jar_publish_integration
    Build operating on targets: OrderedSet([PythonTests(tests/python/pants_test/tasks/BUILD:jar_publish_integration)])

    Bugs closed: 120

    Reviewed at https://rbcommons.com/s/twitter/r/356/
A fast test-running mode for python tests.

Runs all tests in a single chroot instead of creating one per test target.

Benefits:
  1) The pants tests run in 12 seconds instead of 160 seconds.
  2) Test isolation can actually hide problems sometimes.

Drawbacks:
  Individual chroots are better for verifying that a test
  target's dependencies are self-contained, i.e., that they
  reflect all the actual deps. When running multiple tests
  in a single chroot, test B could provide a dep that test A
  relies on but doesn't declare, thus hiding a missing dep
  problem.

This drawback is minor - it only verifies that the test target's
deps are good, not the deps of the target being tested.
Squashed commit of the following:
    improve detect-duplicates to detect dups in internal dependencies
    This reverts commit 24cc65dbae0b29db4dc6c4f7ffb1b9cb8d41f87d.