Change certain core configs into registered options.

These are options like pants_workdir, whose values can be interpolated
into pants.ini to create values for other options. So we treat them
specially: We refer to them as "bootstrap options" and we determine their
values in an early pass before getting the values for all the other options.

This allows, e.g., overriding --pants-workdir on the cmd line, like any other
option, but still having %(pants_workdir)s interpolations in config files
work as expected.

Testing Done:
CI passed.

Reviewed at https://rbcommons.com/s/twitter/r/1373/
Minor docs fix: ./pants.d -> .pants.d

Fix path in the http://pantsbuild.github.io/tshoot.html page: ./pants.d  -> .pants.d

Testing Done:
./build-support/bin/publish_docs.sh -d tmp -o; verified the fix.

Reviewed at https://rbcommons.com/s/twitter/r/1387/
Make Pants goal run for Python exit with error code 1 if the python
program exits non-zero (as pants goal run does for java).

Testing Done:
Ran new tests.

Reviewed at https://rbcommons.com/s/twitter/r/1374/
Fix .proto files with unicode characters in the comments. (Problem found in '_same_contents')

Testing Done:
Updated unit tests, added a testproject example.

I tried naming the .proto file with a utf-8 character and it failed miserably. Fortunately I don't have that case, so I didn't try to fix it.

Bugs closed: 792

Reviewed at https://rbcommons.com/s/twitter/r/1330/
Remove execute_on_subproc and background subproc pool

Turns out we didn't get the parallelism we wanted with threads making blocking calls to subprocesses.

Re-using the subproc_map that is used in cache reads, and sending the whole call as a single piece of Work,
gets parallel compression and uploads, though at the expense of slightly less granular reporting. It also
turns out that separate process pools for aren't that useful, and slows down startup/shutdown.
Make all access to Context's lock via helpers

`context._lock` should be an implementation detail with callers only concerned with acquire_lock and release_lock.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/41820893

Bugs closed: 818

Reviewed at https://rbcommons.com/s/twitter/r/1391/
Get rid of the old config registration mechanism.

It was hardly used, and is now superceded by the new options system.

Moved a bunch of accesses of pants_workdir from config to new options.
There are still several such accesses that will take more work, as
they are in code locations with no reference to an Options instance.

To do so, made bootstrap option values available during option registration.
This uses function properties, which actually seemed like a reasonable
solution here.  Yay python for making everything an object.

Testing Done:
CI passes.

Reviewed at https://rbcommons.com/s/twitter/r/1385/
Make interpreter_cache a property, acquire lock in accessor.

Intended to address https://github.com/pantsbuild/pants/issues/819

Testing Done:
Locally repoduced the mentioned issue, applied patch and reattempted repro. Runs printed lock acquisition message and did not reproduce the issue.

CI: https://travis-ci.org/pantsbuild/pants/builds/41899735

Bugs closed: 819, 820

Reviewed at https://rbcommons.com/s/twitter/r/1392/
remove outer (pants_exe) lock and serialized cmd

All two subcalsses of command (eg GoalRunner and SetupPy) do not override `serialized` to return `True` and thus do not require the outer lock.
Removing it simplifies pants_exe and reduces the number of places we depend on dirutil.Lock and shutil.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/41782185

Bugs closed: 817

Reviewed at https://rbcommons.com/s/twitter/r/1388/
Adds custom list and dict option types.

Values (in cmd-line args, env vars and config files) must be JSON lists/objects.

Currently our config reader 'parses' list and dict values using python 'eval', so we
change all of those in our pants.ini to be valid JSON. Fortunately they remain valid
eval-able python too. Eventually we'll get rid of that eval-ing code, but for now
we need to support both, during the transition.

Adds checks in migrate_config.py for list/dict values that aren't valid JSON.

Note that the list type is different from action='append'. The latter
appends cmd-line arg values to the default. The former replaces the
default.

Testing Done:
CI passes.

Reviewed at https://rbcommons.com/s/twitter/r/1389/
Subclasses of Exception have strange deserialization

We pass these back from subprocs so they need to ser/de correctly, but Exception has special
deserialization. UnreadableArtifact is basically just a sentinel value so subclassing Exception
just adds complexity.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/41981675

Bugs closed: 826

Reviewed at https://rbcommons.com/s/twitter/r/1395/
Replace twitter commons dirutil Lock with lockfile wrapper

Every now and then we see a pants process die without cleaning up its pidfile and subsequent runs in the same workspace hang forever.

This change replaces the twitter-commons dirutil.Lock with a thin wrapper around lockfile's PIDLockFile which prints the owner process commandline or breaks the lock if the owner process doesn't exist.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/41821386

Bugs closed: 816

Reviewed at https://rbcommons.com/s/twitter/r/1390/