Better way to avoid unlinking a non-existent file.

This solves a race condition too (although there is still another race),
and is more consistent with similar patterns elsewhere in the file.

Testing Done:
Relevant integration tests pass (after cleaning, so this triggers both a failed unlink and a successful one).

Manually tested the relevant code block.

Reviewed at https://rbcommons.com/s/twitter/r/1438/
Get rid of option_helpers.py.

It was redundant with global_options.py.

This allowed us to make doc gen for global options
more uniform with that for task-specific options.

Testing Done:
CI passed.

Manually generated docs and eyeballed them.

Reviewed at https://rbcommons.com/s/twitter/r/1439/
More intuitive env var names for global-scope options.

Previously for global option --bar-baz you had to set PANTS_DEFAULT_BAR_BAZ.
Now we allow PANTS_BAR_BAZ, eliding the explicit global scope 'DEFAULT'.

If the option name itself already starts with 'pants-', e.g., --pants-workdir,
we also allow PANTS_WORKDIR, instead of PANTS_PANTS_WORKDIR or
PANTS_DEFAULT_PANTS_WORKDIR.

Using env vars to set option values isn't necessarily common, but this will make
it more conveninent and intuitive. Also, it will allow the PANTS_CONFIG_OVERRIDE
mechanism to be retconned into a --config-override option.

Testing Done:
CI passes.

Reviewed at https://rbcommons.com/s/twitter/r/1441/
Eval list- and dict-typed option values, instead of JSON-parsing them.

This is to support list concatenation syntax in config files, e.g.,

dev_opts: %(basic_opts)s + %(big_mem_opts)s + ['-ldebug']

This is a legitimate use case that we actually have, but that I overlooked
when I made the JSON requirement in 8c3b5b33e8cabdf599ed1332067283ffc067a13a.

Testing Done:
Ran options unit tests.

CI baking.

Reviewed at https://rbcommons.com/s/twitter/r/1443/
rename dev_backend_loader to extension_loader

This is a subset of https://rbcommons.com/s/twitter/r/1429 but poor rename detection was making that diff much harder to read. Splitting this into its own change should make the added functionality and tests easier to identify in that change.

Testing Done:
./pants goal test tests/python/pants_test::
https://travis-ci.org/pantsbuild/pants/builds/43414631

Bugs closed: 849

Reviewed at https://rbcommons.com/s/twitter/r/1444/
Handy rbt status tip from RBCommons newsletter

Testing Done:
ran ./build-support/bin/publish_docs.sh -o and looked

Reviewed at https://rbcommons.com/s/twitter/r/1446/
Support loading config from multiple files.

This subsumes two disparate pieces of functionality:

A) PANTS_CONFIG_OVERRIDES.
B) rcfiles.

A) This env variable has been retconned int a --config-overrides
   bootstrap option.

B) Rcfiles used to contain extra command-line flags for various commands, which
   was weird. Since all flags are now options, and so can be set from config,
   the new rcfiles are just regular pants.ini-style config files.

After deploying pants with this commit, existing rcfiles will need to be converted
to pants.ini style. However currently rcfiles are broken anyway, so this isn't
an onerous requirement.

This commit also includes various changes needed to get things to work under
this new config scheme. Some of these are temporary hacks that will go away
when we get rid of Command, or when we get rid of all direct accesses to config.

One prominent change is a modification to config caching. Previously
configs were keyed by the file they were read from. But in practice
code always calls Config.from_cache() with no argument, which
defaults to the regular pants.ini. This leads to dangerous confusion
when config can be read from other files. So now there's just a single
global cached config, which must be set explicitly. Again, this will
go away once we convert all direct config accesses to options accesses,
and plumb options through to all the necessary places.

Testing Done:
CI passes.

Reviewed at https://rbcommons.com/s/twitter/r/1442/
doc: on skinny displays, don't show navigation UI by default.

also on skinny displays, show a button that toggles navigation UI [in]visible

Testing Done:
Looked at test-publish
http://pantsbuild.github.io/lahosken-okskinny/
with a few browsers, including my Android's built-in and Chrome on Android.

Reviewed at https://rbcommons.com/s/twitter/r/1440/
Support loading plugins via pkg_resources entry points

This provides an alternative way to find the methods `build_file_aliases` and `register_goals` that
are called as part of extension loading, specifically that for off-the-shelf plugins installed as
distrubutions, these methods are found using pkg_resources' entry points configuration.

Source backends are useful for quick and easy loading of custom tasks or aliases from sources that
live in a project. This change is not entended to replace source backends, but rather to make use
of richer metadata avilable on packaged distributions used for deploying off-the-shelf plugins.

Distributions (egg/sdist/etc) have version information and can define their entry points, which allows
cleaner separation of layout and implementation from core pants. Version requirements can be checked at
load and plugins can assert other plugins are loaded before them (since order matters eg "replace=True").

Adding such a plugin when using virtualenv pants would consist of adding it to requirements.txt used to
setup the virtualenv (or simply pip installing if using global site-packages), then adding it to
pants.ini's `plugins` list under the section `backends`.

For deployments running pants from a pex, the plugins could either be built into the distributed pex or
plugins could still be installed to a virualenv using a requirments.txt if that env were somehow then
added to the python path when invoking the pex.

This change does not include support for resolving, fetching or in any way installing plugins at
runtime -- requested plugins are loaded if they are found or an error is raised if not.  If or how how
to support loading plugins that are not previously installed is left for a separate discussion.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/43419387
Added tests with mocked out distributions to test_extension_loader.py.
Demo repo: https://github.com/foursquare/pants-plugins-demo

Bugs closed: 843

Reviewed at https://rbcommons.com/s/twitter/r/1429/
Remove the old setup_py command.

It has been replaced by the python_setup goal.

There are no longer any remaining commands in pants, other than GoalRunner.

Note that this requires 49bb9f43eeb1331342f35cef331265cf4b8199bb, so it may not work if you cherry-pick it.

Testing Done:
CI passes.

Reviewed at https://rbcommons.com/s/twitter/r/1448/
Move the spec exclude integration test from command to base.

It is base-level functionality that has nothing to do with command.

Testing Done:
Integration test passed in its new location.

CI baking.

Reviewed at https://rbcommons.com/s/twitter/r/1449/
migrate_config only complain about keys not from defaults

has_option seems to return true if a key is found from defaults even if it isn't overridden
in a given section (ours was complaining about 'confs' in a a mostly empty junit-run section).

This seems to fix it, though in the event a section was actually overriding a default,
just with the same value, it will ignore that. I couldn't figure out how to see if a section
actually defined a key without poking around in ConfigParser's private fields.

Reviewed at https://rbcommons.com/s/twitter/r/1451/
Gracefully handle another run creating latest link

Gracefully handle a race with another run creating or removing the latest link (which can happen with concurrent runs).

Testing Done:
Concurrent runs ~8µs apart.
https://travis-ci.org/pantsbuild/pants/builds/41989058

Bugs closed: 827

Reviewed at https://rbcommons.com/s/twitter/r/1396/