Adding os.path.realpath to help macos build pass tests.

Testing Done:
pants goal test tests/python/pants::

I get 3 errors when I do this locally due to the fact that scala no longer works on my box.

Bugs closed: 312

Reviewed at https://rbcommons.com/s/twitter/r/624/
docstrings (and more) update post-target-refactor

target docs were obsolete and/or just plain wrong:
  claimed sources= param was a list of strings
  claimed targets= param was a list of targets

target method Page.register_wiki
was tagged (with @manual.builddict) for inclusion in
the BUILD dictionary; this is now a troll, you can't call
target methods in BUILD files.

Testing Done:
./build-support/bin/ci.sh

./build-support/bin/publish_docs.sh -o
searched BUILD dictionary for old phrases "list of targets", "(list of files)", "base.Target"; confirmed they're gone

Bugs closed: 138, 140

Reviewed at https://rbcommons.com/s/twitter/r/391/
Kill Context.state - its no longer used nor should it be used again.

Testing Done:
CI ran green: https://travis-ci.org/pantsbuild/pants/builds/28965745

Bugs closed: 313

Reviewed at https://rbcommons.com/s/twitter/r/625/
Fixup RoundEngine scheduling.

Do a recursive walk to construct and prepare Tasks as before
but then do a tsort after the full walk when all the dependency
edges are known.

This also takes a pass over all jvm Tasks to ensure proper deps
are requested in prepare and in the cases where the deps are
improper TODOs are added calling out whats needed.

Testing Done:
CI passes locally and travis is away here: https://travis-ci.org/pantsbuild/pants/builds/28997865

I did these sanity checks for jvm goals:
$ function sanity() { for goal in "$@"; do echo "$goal:" && PANTS_DEV=1 ./pants goal --explain $goal 2>/dev/null | head -3 | tail -1 && echo; done; }
$ sanity gen resolve compile test run repl doc jar binary bundle
gen:
bootstrap -> gen

resolve:
bootstrap -> gen -> check-exclusives -> resolve

compile:
bootstrap -> gen -> check-exclusives -> resolve -> compile

test:
bootstrap -> gen -> check-exclusives -> resolve -> compile -> resources -> test

run:
bootstrap -> gen -> check-exclusives -> resolve -> compile -> resources -> run

repl:
bootstrap -> gen -> check-exclusives -> resolve -> compile -> resources -> repl

doc:
bootstrap -> gen -> check-exclusives -> resolve -> compile -> doc

jar:
bootstrap -> gen -> check-exclusives -> resolve -> compile -> resources -> jar

binary:
bootstrap -> gen -> check-exclusives -> resolve -> compile -> resources -> binary

bundle:
bootstrap -> gen -> check-exclusives -> resolve -> compile -> resources -> bundle

Also checking the un-constrained cli handling - note clean-all killserver server order is maintained and compile is de-duped:
$ PANTS_DEV=1 ./pants goal --explain clean-all killserver server test depmap compile
*** Running pants in dev mode from /home/jsirois/dev-jsirois-pants/src/python/pants/bin/pants_exe.py ***
(using pantsrc expansion: pants goal --explain clean-all killserver server test depmap compile --server-open)
Phase Execution Order:

clean-all -> killserver -> server -> bootstrap -> gen -> check-exclusives -> resources -> resolve -> compile -> test -> depmap

Bugs closed: 314, 316

Reviewed at https://rbcommons.com/s/twitter/r/627/
Adding ability to exclude a spec from a list of specs that might include expansion.  If you add a caret '^' prefix to a target address on the command line, it will be removed from the list of targets to be evaluated.

Testing Done:
pants goal test tests/python/pants_test::

Added unit tests

Also ran these 2 commands on the command line and compared the output:

./pants goal list tests/python/pants_test/::

./pants goal list tests/python/pants_test/:: ^tests/python/pants_test/base::

Bugs closed: 315

Reviewed at https://rbcommons.com/s/twitter/r/626/
Update setuptools to 5.0.2.  5.0 causes build failures for at least some Linux distributions, including Ubuntu 12.04

Testing Done:
./build-support/bin/ci.sh

This has some failures, but they're all also present on master.

Bugs closed: https://github.com/pantsbuild/pants/issues/317

Reviewed at https://rbcommons.com/s/twitter/r/630/
Interim fix for cyclic dependency
Prevent stat-uploading errors from polluting stdout

The error/warning messages that occur during stat-uploading are currently being printed to stdout, which breaks scripts that use pants output (e.g. "./pants goal list ::" chained with "./pants goal compile"). This RB is a fix for this issue.

Testing Done:
Local testing in a REPL

Reviewed at https://rbcommons.com/s/twitter/r/650/
This is done by way of a new 'imports' parameter added to the java_protobuf_library target, which accepts jar dependencies as inputs. The syntax for this is:

java_protobuf_library(name='foo',
  imports=[
    jar(org='foo.com', name='bar', rev='1.2.3'), jar(org='bar.com', name='foo', rev='3.2.1'),
    '3rdparty:some-protobuf-jar-library',
  ],
  ...,
)

An extra ivy step called 'ivy-imports' (rather than ivy-resolve) has been added to accomodate this. It occurs after bootstrap and before codegen.

Testing Done:
Added junit test which uses .proto which includes imports .proto files I jarred and stuck on the maven central nexus. ci.sh is green on my machine and on travis. Also tested internally in the square repo.

The proto files on the maven central nexus were bundled with maven and do not reside in the main pants repo.

Bugs Closed: 297 http://github.com/pantsbuild/pants/issues#issue/297

Reviewd at: https://rbcommons.com/s/twitter/r/592/
AndroidDistribution class to fetch and validate Android SDK

The Android SDK has two components that are modularly installable and used in Android BUILD files- the API level for the target,
and the version of the build-tools used to construct the target. When a user has an SDK in their path, there is no guarantee that either
the API nor the build-tools version they need have been installed into their sdk.

This class looks for an Android SDK on the machine's path (using common SDK aliases) and then checks to see if the
requested API (target_sdk) and build-tools (build_tools_version) are installed. Most lookups are cached to minimize
repeated actions.

How we verify these installations is through checking for executables/files that comprise them. The SDK already provides a command-line tool
to manage installation of these modules but the only way to check installed targets is through grepping its output.
Since that tool doesn't seem to do much more than check for the presence of files/executables it seemed better to mimic those
checks as opposed to hacking up an output parser. As the class comments say, we can add more checks if we prefer.

Testing Done:
ci.sh passes locally.

Reviewed at https://rbcommons.com/s/twitter/r/636/
Added project-info flag to depmap goal.

Prints source roots, dependencies, and paths to libraries for targets and their dependencies in json format.
It can be run with: ./pants goal depmap /path/to/target --depmap-project-info

Adding the -format flag causes the output to return a single line of json:
./pants goal depmap /path/to/target --depmap-project-info --depmap-format

This functionality will be used for project import in the IntelliJ Pants plugin.

Testing Done:
Added tests for the goal (test_depmap.py), they pass

ci.sh runs successfully

Reviewed at https://rbcommons.com/s/twitter/r/600/
Fixup clean-all, clean-all-async and invalidate dependencies.

In the wake of Goal.dependencies no longer being used for scheduling,
clean-all and invalidate no longer properly depend on ng-killall.

Testing Done:
Ad-hoc:
$ for goal in clean-all clean-all-async invalidate ng-killall; do PANTS_DEV=1 ./pants goal $goal -e 2>/dev/null | grep $goal | grep "\["; done
clean-all [ng-killall->NailgunKillall, invalidate->Invalidator, clean-all->Cleaner]
clean-all-async [ng-killall->NailgunKillall, invalidate->Invalidator, clean-all-async->AsyncCleaner]
invalidate [ng-killall->NailgunKillall, invalidate->Invalidator]
ng-killall [ng-killall->NailgunKillall]

Bugs closed: 319

Reviewed at https://rbcommons.com/s/twitter/r/638/
Fix missing java & scala product types for ScroogeGen.

CodeGen handles this, but ScroogeGen is not a subclass.

Testing Done:
None.  I noticed this missing when working through RoundEngine validation improvements.

Bugs closed: 320

Reviewed at https://rbcommons.com/s/twitter/r/639/
Kill unused antlr deps in 3rdparty/BUILD.

Add a TODO in PythonAntlrBuilder to merge with
the AntlrGen task - right now PythonAntlrBuilder
does ~out of band ivy resolution of antlr tools.

Testing Done:
$ PANTS_DEV=1 ./pants goal clean-all
$ PANTS_DEV=1 ./pants goal test {src,tests}/{java,scala,python}::

Bugs closed: 322

Reviewed at https://rbcommons.com/s/twitter/r/642/
Add a note that build_lint.py is broken

Testing Done:
None

Bugs closed: 323

Reviewed at https://rbcommons.com/s/twitter/r/643/