Added project-info flag to depmap goal.

Prints source roots, dependencies, and paths to libraries for targets and their dependencies in json format.
It can be run with: ./pants goal depmap /path/to/target --depmap-project-info

Adding the -format flag causes the output to return a single line of json:
./pants goal depmap /path/to/target --depmap-project-info --depmap-format

This functionality will be used for project import in the IntelliJ Pants plugin.

Testing Done:
Added tests for the goal (test_depmap.py), they pass

ci.sh runs successfully

Reviewed at https://rbcommons.com/s/twitter/r/600/
Fixup clean-all, clean-all-async and invalidate dependencies.

In the wake of Goal.dependencies no longer being used for scheduling,
clean-all and invalidate no longer properly depend on ng-killall.

Testing Done:
Ad-hoc:
$ for goal in clean-all clean-all-async invalidate ng-killall; do PANTS_DEV=1 ./pants goal $goal -e 2>/dev/null | grep $goal | grep "\["; done
clean-all [ng-killall->NailgunKillall, invalidate->Invalidator, clean-all->Cleaner]
clean-all-async [ng-killall->NailgunKillall, invalidate->Invalidator, clean-all-async->AsyncCleaner]
invalidate [ng-killall->NailgunKillall, invalidate->Invalidator]
ng-killall [ng-killall->NailgunKillall]

Bugs closed: 319

Reviewed at https://rbcommons.com/s/twitter/r/638/
Fix missing java & scala product types for ScroogeGen.

CodeGen handles this, but ScroogeGen is not a subclass.

Testing Done:
None.  I noticed this missing when working through RoundEngine validation improvements.

Bugs closed: 320

Reviewed at https://rbcommons.com/s/twitter/r/639/
Kill unused antlr deps in 3rdparty/BUILD.

Add a TODO in PythonAntlrBuilder to merge with
the AntlrGen task - right now PythonAntlrBuilder
does ~out of band ivy resolution of antlr tools.

Testing Done:
$ PANTS_DEV=1 ./pants goal clean-all
$ PANTS_DEV=1 ./pants goal test {src,tests}/{java,scala,python}::

Bugs closed: 322

Reviewed at https://rbcommons.com/s/twitter/r/642/
Add a note that build_lint.py is broken

Testing Done:
None

Bugs closed: 323

Reviewed at https://rbcommons.com/s/twitter/r/643/
Fixup a hidden jar <-> publish dependency cycle.

This simplifies JarCreate letting it just create classfile jars and
moves the responsibility of creating source and doc jars to the
only user - JarPublish.

Testing Done:
./build-support/bin/ci.sh

travis-ci went green here: https://travis-ci.org/pantsbuild/pants/builds/29627541

Bugs closed: 335

Reviewed at https://rbcommons.com/s/twitter/r/660/
Fixup RoundManager scheduling for phases that depend on themselves.

Handle this case explicitly and add test coverage for the
RoundEngine.

This also adds proper cli phase de-duping and validation
that requested data products actually have a producer.

Testing Done:
$ PANTS_DEV=1 ./pants goal clean-all
$ PANTS_DEV=1 ./pants goal test {src,tests}/{java,scala,python}::

Bugs closed: 324

Reviewed at https://rbcommons.com/s/twitter/r/644/
Fix the '--bundle-dup-excludes' flag and add '--bundle-dup-max-dups'

Now the default output prints out how many more you get:

   Duplicate classes and/or resources detected in artifacts: (u'commons-beanutils-commons-beanutils-1.7.0.jar', u'commons-beanutils-commons-beanutils-core-1.8.0.jar')
                        org/apache/commons/beanutils/BeanAccessLanguageException.class
                        org/apache/commons/beanutils/converters/URLConverter.class
                        org/apache/commons/beanutils/locale/converters/IntegerLocaleConverter.class
                        org/apache/commons/beanutils/converters/CharacterArrayConverter.class
                        org/apache/commons/beanutils/MethodUtils.class
                        org/apache/commons/beanutils/converters/BooleanConverter.class
                        org/apache/commons/beanutils/converters/ClassConverter.class
                        org/apache/commons/beanutils/DynaBean.class
                        org/apache/commons/beanutils/converters/BooleanArrayConverter.class
                        org/apache/commons/beanutils/Converter.class
                        ... 123 more ...

Testing Done:
I've got a lot of duplicate classes on the classpath to sort out...

devpants goal bundle <target> --bundle-dup-max-dups=50 --bundle-dup-excludes=dependencies,license,notice,.DS_Store,notice.txt,cmdline.arg.info.txt.1,license.txt,org.apache.hadoop.fs.FileSystem,BCKEY.DSA,BCKEY.SF,ASL2.0,javax.servlet.ServletContainerInitializer,about.html

Bugs closed: 332

Reviewed at https://rbcommons.com/s/twitter/r/656/
Fix depmap to be able to see past a Dependencies target.  We use dependencies() liberally to create short aliases users can use to build common targets.

Testing Done:
Added some unit tests to test_depmap.py

Bugs closed: 242, 336

Reviewed at https://rbcommons.com/s/twitter/r/661/
Simplify python_requirements UI.

In order to simplify the python_requirements interface this change
consolidates BUILD file helper functions into a single macro interface
that allows for either macro types or macro functions.  Both get a
MacroContext injected that exposes controlled bits of the BUILD file
parse context for the macros to generate targets, objects and paths.

In the course of consolidating util types, the BuildFileParser is refactored
to extract the global plugin BuildConfiguration from the parser itself and
firm up the notion of BuildFileAliases.

As fallout many tests are adjusted to the new apis.

Testing Done:
$ ./build-support/bin/ci.sh

travis-ci went green too: https://travis-ci.org/pantsbuild/pants/builds/28304477

Bugs closed: 191, 270

Reviewed at https://rbcommons.com/s/twitter/r/587/
Improve backend plugin ergonomics.

This adds support for omitting a backend entrypoint where
no backend plugin is provided and also improves import
errors by wrapping and giving the full module path that was
not found.

Test coverage is also added.

Testing Done:
$ PANTS_DEV=1 ./pants goal test tests/python/pants_test/base:all

Bugs closed: 666

Reviewed at https://rbcommons.com/s/twitter/r/666/
Simplify the JvmTarget resources property.

Resources are just another form of dependency as demonstrated
by the traversable_dependency_specs implementation.  They all
get resolved as dependencies during BUILD parsing so we just need
to look at our direct dependencies that are Resources target to
implement the resources property.  This has the side-benefit that the
standard dependency injection primitives for stitching synthetic targets
into a BUILD graph can be used to stitch in new Resources dependencies.

Testing Done:
$ ./build-support/bin/ci.sh

CI went green here: https://travis-ci.org/pantsbuild/pants/builds/29898110

Bugs closed: 344, 346

Reviewed at https://rbcommons.com/s/twitter/r/673/
doc python_req things

give python_requirements_library a docstring

tag python_requirements with @manual so it shows up in BUILD dictionary

add example to python_requirements with a "this gives you something like..."

Bugs closed: 292

Reviewed at https://rbcommons.com/s/twitter/r/629/
Upgrade default setuptools to 5.4.1

Testing Done:
CI passed.  Looking at https://pypi.python.org/pypi/setuptools , there appear to be no expected breaking changes between our current setuptools (5.0) and this one.  We want this upgrade because of the bugfix in 5.1, and the other bug fixes since then seem good to pull in.

Reviewed at https://rbcommons.com/s/twitter/r/674/
Patch around unicode dup detection failure.

I'm not super happy with this solution and I think it deserves a closer look, but since the thread for this fizzled out, this is the patch we've been running internally to unblock ourselves.  Thread for reference: https://groups.google.com/forum/#!searchin/pants-devel/unicode$20decode/pants-devel/tx4NHhxX1fI/VRP0fzzG4C4J

Testing Done:
CI passed

Reviewed at https://rbcommons.com/s/twitter/r/615/
Add manifest parser to AndroidTarget.

This removes the need to enter target_sdk and package variables
for every BUILD file in the package. It was always on the TODO,
but as I am prepping the codegen tasks, it seemed time. This removes
several params from the targets, docstrings are updated accordingly.

Also added BadManifestError to catch malformed or incomplete manifests.
These errors would eventually cause problems in the compilation
process, but since we are parsing the manifest here, uncaught
exceptions would appear to be pants-related problems, as opposed
to an issue with the user's Android project. So I added an exception
to catch the issue.

Testing Done:
ci.sh passed locally.
Travis passed [0].

There will be some further integration tests of this functionality when I push up a
separate patch this weekend comprised of the aapt_gen class and its test coverage.

[0] https://travis-ci.org/pantsbuild/pants/builds/29720531

Reviewed at: https://rbcommons.com/s/twitter/r/668/
Fix relativity of antlr sources

This fixes an issue where CacheValidator's source hashing was failing on
the synthetic targets generated by AntlrGen, since the assumed source
paths mismatched the actual generated paths.

- Adds an integration test for antlr4 codegen
- Uses sources_relative_to_source_root for the assumed output sources,
  and fixes the synthetic source root for the antlr output. We can't
  control antlr's output relative paths very tightly, so we can't use
  the output directory directly as a sourceroot.
Lock globs into rootdir and below

This avoids the situation where a user adds a sources=globs('../*.scala'), causing it to be silently ignored but causes issues later.

Reviewed here: https://rbcommons.com/s/twitter/r/686/
Closes: https://github.com/pantsbuild/pants/issues/348
Pluralize product_type of Task to product_types
doc: Python doc, don't use twitter/commons as example

Back when Pants' source code lived in twitter/commons, it made sense to use example code from there. Nowadays, not so much. Use Pants' own src/python/example instead.

Instead of having sample code in the doc, "include" from src/python/example; that's easier to keep correct.

Testing Done:
build-support/bin/publish_docs.sh

Reviewed at https://rbcommons.com/s/twitter/r/654/