Fix antlr package derivation.

Also fixup jvm tool integration tests to respect
TMPDIR being on a different device than the build root.

Testing Done:
$ PANTS_DEV=1 ./pants goal list tests/python/pants_test/tasks: | grep antlr | PANTS_DEV=1 xargs ./pants goal test

CI went green here: https://travis-ci.org/pantsbuild/pants/builds/42654357

Bugs closed: 833

Reviewed at https://rbcommons.com/s/twitter/r/1410/
Remove support for __file__ in BUILDs.

NB: this is a BUILD API break.

Fixup BuildConfigurationTest to match removal of __file__ and fixup code formatting.

Testing Done:
CI went green here: https://travis-ci.org/pantsbuild/pants/builds/42913383

Bugs closed: 835

Reviewed at https://rbcommons.com/s/twitter/r/1419/
Fixup BuildGraph to handle mixes of synthetic and BUILD targets.

An upcoming change [1] will inject synthetic JarLibraries that jvm
tools will depend on.  This tickles a corner where the BuildGraph
can be asked to resolve a SyntheticAddress that has been injected.

Tighten up interfaces removing unused methods and prefixing private
helpers with _ to clarify APIs and tighten them.

Modify tests to validate the new translation of SyntheticAddress to
BuildFileAddress when lookups are made and resurrect tests in
BuildGraphTest.

This also fixes some minor fallout in production code and tests.

[1] https://rbcommons.com/s/twitter/r/1423/

Testing Done:
CI went green here: https://travis-ci.org/pantsbuild/pants/builds/42935970

Bugs closed: 837

Reviewed at https://rbcommons.com/s/twitter/r/1420/
Port pants to pex 0.8.x

SSIA.  The user facing parts of this change:
  - package resolution happens via requests now -- this is way more reliable
    and results in fewer Untranslateable exceptions
  - PEX_VERBOSE now controls all pex verbosity, including better messages around Untranslateable (yay!)
  - fixes a major pex regression with namespace packages introduced somewhere in 0.5.x

The pex 0.7.0 -> 0.8.0 changelog:

* *API change*: Decouple translation from package iteration.  This removes
  the Obtainer construct entirely, which likely means if you're using PEX as
  a library, you will need to change your code if you were doing anything
  nontrivial.  This adds a couple new options to ``resolve`` but simplifies
  the story around how to cache packages.
  [RB #785](https://rbcommons.com/s/twitter/r/785/)
* Refactor http handling in pex to allow for alternate http implementations.  Adds support
  for [requests](https://github.com/kennethreitz/requests),
  improving both performance and security.   For more information, read the commit notes at
  [91c7f32](https://github.com/pantsbuild/pex/commit/91c7f324085c18af714d35947b603a5f60aeb682).
  [RB #778](https://rbcommons.com/s/twitter/r/778/)
* Improvements to API documentation throughout.
* Renamed ``Tracer`` to ``TraceLogger`` to prevent nondeterministic isort ordering.
* Refactor tox.ini to increase the number of environment combinations and improve coverage.
* Adds HTTP retry support for the RequestsContext.
  [RB #1303](https://rbcommons.com/s/twitter/r/1303/)
* Make pex --version correct.
  [Issue #19](https://github.com/pantsbuild/pex/issues/19)
* Bug fix: Fix over-aggressive sys.modules scrubbing for namespace packages.  Under
  certain circumstances, namespace packages in site-packages could conflict with packages
  within a PEX, causing them to fail importing.
  [RB #1378](https://rbcommons.com/s/twitter/r/1378/)
* Bug fix: Replace uses of ``os.unsetenv(...)`` with ``del os.environ[...]``
  [Pull Request #11](https://github.com/pantsbuild/pex/pull/11)
* Bug fix: Scrub sys.path and sys.modules based upon both supplied path and
  realpath of files and directories.  Newer versions of virtualenv on Linux symlink site-packages
  which caused those packages to not be removed from sys.path correctly.
  [Issue #21](https://github.com/pantsbuild/pex/issues/21)
* Bug fix: The pex -s option was not correctly pulling in transitive dependencies.
  [Issue #22](https://github.com/pantsbuild/pex/issues/22)
* Bug fix: Adds ``content`` method to HTTP contexts that does HTML content decoding, fixing
  an encoding issue only experienced when using Python 3.
  [Issue #10](https://github.com/pantsbuild/pex/issues/10)

Testing Done:
build-support/bin/ci.sh

Reviewed at https://rbcommons.com/s/twitter/r/1421/
Fix error message in scrooge_gen

I hit this error within an error when I tried to run the gen goal.

Testing Done:
Built a pex, triggered the error.

Reviewed at https://rbcommons.com/s/twitter/r/1426/
fix untentionally -> unintentionally in build_files.md

Reviewed at https://rbcommons.com/s/twitter/r/1430/
Avoid throwing in ApacheThriftGen.__init__

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/43046341

Bugs closed: 842

Reviewed at https://rbcommons.com/s/twitter/r/1428/
Support option reads via indexing.

This will support tasks with table-driven logic.

Testing Done:
$ pants.dev goal test tests/python/pants_test/option

CI went green here: https://travis-ci.org/pantsbuild/pants/builds/43132907

Bugs closed: 846

Reviewed at https://rbcommons.com/s/twitter/r/1435/
Properly detect existence of a symlink.

os.path.exists() returns False for broken symlinks.

Reviewed at https://rbcommons.com/s/twitter/r/1437/
Document how to create a simple plugin

This adds a short tutorial on how to create a simple plugin.

Reviewed at https://rbcommons.com/s/twitter/r/1434/
Better way to avoid unlinking a non-existent file.

This solves a race condition too (although there is still another race),
and is more consistent with similar patterns elsewhere in the file.

Testing Done:
Relevant integration tests pass (after cleaning, so this triggers both a failed unlink and a successful one).

Manually tested the relevant code block.

Reviewed at https://rbcommons.com/s/twitter/r/1438/
Get rid of option_helpers.py.

It was redundant with global_options.py.

This allowed us to make doc gen for global options
more uniform with that for task-specific options.

Testing Done:
CI passed.

Manually generated docs and eyeballed them.

Reviewed at https://rbcommons.com/s/twitter/r/1439/
More intuitive env var names for global-scope options.

Previously for global option --bar-baz you had to set PANTS_DEFAULT_BAR_BAZ.
Now we allow PANTS_BAR_BAZ, eliding the explicit global scope 'DEFAULT'.

If the option name itself already starts with 'pants-', e.g., --pants-workdir,
we also allow PANTS_WORKDIR, instead of PANTS_PANTS_WORKDIR or
PANTS_DEFAULT_PANTS_WORKDIR.

Using env vars to set option values isn't necessarily common, but this will make
it more conveninent and intuitive. Also, it will allow the PANTS_CONFIG_OVERRIDE
mechanism to be retconned into a --config-override option.

Testing Done:
CI passes.

Reviewed at https://rbcommons.com/s/twitter/r/1441/
Eval list- and dict-typed option values, instead of JSON-parsing them.

This is to support list concatenation syntax in config files, e.g.,

dev_opts: %(basic_opts)s + %(big_mem_opts)s + ['-ldebug']

This is a legitimate use case that we actually have, but that I overlooked
when I made the JSON requirement in 8c3b5b33e8cabdf599ed1332067283ffc067a13a.

Testing Done:
Ran options unit tests.

CI baking.

Reviewed at https://rbcommons.com/s/twitter/r/1443/
rename dev_backend_loader to extension_loader

This is a subset of https://rbcommons.com/s/twitter/r/1429 but poor rename detection was making that diff much harder to read. Splitting this into its own change should make the added functionality and tests easier to identify in that change.

Testing Done:
./pants goal test tests/python/pants_test::
https://travis-ci.org/pantsbuild/pants/builds/43414631

Bugs closed: 849

Reviewed at https://rbcommons.com/s/twitter/r/1444/
Handy rbt status tip from RBCommons newsletter

Testing Done:
ran ./build-support/bin/publish_docs.sh -o and looked

Reviewed at https://rbcommons.com/s/twitter/r/1446/