Add a CI action to support CI against OSX.

This adds a sync action run on any push to pantsbuild/pants
master that replaces pantsbuild/pants-for-travis-osx-ci master
modifying the active .travis.yml to trigger an OSX CI run.

Testing Done:
Normal CI went green here: https://travis-ci.org/pantsbuild/pants/builds/35900516

OSX CI went green here: https://travis-ci.org/pantsbuild/pants-for-travis-osx-ci/builds/35899630
Via:
```console
$ set | grep GH_ | cut -d= -f1
GH_EMAIL
GH_TOKEN
GH_USER
$ ./build-support/bin/ci-sync.sh
[jsirois/pants-for-travis-osx-ci/sync 9a13170] Prepare pants OSX mirror for CI.
 Author: Pants CI <pants-ci@googlegroups.com>
 1 file changed, 13 insertions(+), 40 deletions(-)
 rewrite .travis.yml (87%)
Counting objects: 37, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (18/18), done.
Writing objects: 100% (23/23), 3.05 KiB | 0 bytes/s, done.
Total 23 (delta 14), reused 9 (delta 3)
To https://github.com/pantsbuild/pants-for-travis-osx-ci.git
 + 888c8bf...9a13170 HEAD -> master (forced update)
$ gitbinfo
* jsirois/pants-for-travis-osx-ci/sync 9a13170 [pants/master: ahead 2] Prepare pants OSX mirror for CI.
$ git slog -2
commit 9a13170039dfe93824fd9487f78f96c134e09d88
Author: Pants CI <pants-ci@googlegroups.com>
Date:   Sun Sep 21 20:22:52 2014 -0600

    Prepare pants OSX mirror for CI.

 .travis.yml | 35 ++++-------------------------------
 1 file changed, 4 insertions(+), 31 deletions(-)

commit f4cb72ba4cc48ec5f6a23c86903fe83b735a8f3f
Author: John Sirois <john.sirois@gmail.com>
Date:   Sun Sep 21 17:30:29 2014 -0600

    Add a CI action to support CI against OSX.
...
```

Bugs closed: 596

Reviewed at https://rbcommons.com/s/twitter/r/1058/
Sync the OSX CI eagerly and add diagnostics.

Move the OSX CI sync up-front to leverage parallelism
instead of waiting on the whole local build matrix to complete.

Also add diagnostics that record why the sync is skipped if
skipped in addition to making skipping & syncing decisions more
robust and explicit.

Testing Done:
CI away here already shows proper fork sync-skipping diagnostic: https://travis-ci.org/pantsbuild/pants/builds/35967008
This will have to land on pantsbuild/pants master to vet fully.

Bugs closed: 599

Reviewed at https://rbcommons.com/s/twitter/r/1064/
Fixup ci-sync.sh to cover committer ident.

Testing Done:
None - can only be tested on pantsbuild.pants master, but in response to this failure: https://travis-ci.org/pantsbuild/pants/jobs/35969933
CI did go green here though - so nothing horribly more broken: https://travis-ci.org/pantsbuild/pants/builds/35972662

Bugs closed: 600

Reviewed at https://rbcommons.com/s/twitter/r/1065/
Remove recommendation of manual edits to the pushdb

The manual-edits-to-the-pushdb approach gets a bit too down and dirty. Remove the recommendation in favor of resetting and using --publish-override.

Reviewed at https://rbcommons.com/s/twitter/r/1056/
Use `.travis.yml` notifications for `.travis.osx.yml`

This copies over the notifications settings of the main
`.travis.yml` to `.travis.osx.yml` so that we can maintain
them centrally.

The change also tweaks the commit message to aid in quick
linking / correspondence of a pantsbuild/pants-for-travis-osx-ci
commit to a pantsbuild/pants commit.

Testing Done:
Ad hoc:
```bash
$ venv_setup pyyaml PyYAML && source ~/.pythons/venvs/pyyaml/bin/activate
(pyyaml)$ python << EOF
> import yaml
> with open('.travis.yml') as fp:
>   config = yaml.safe_load(fp)
> with open('.travis.osx.yml') as fp:
>   config_new = yaml.safe_load(fp)
> config_new['notifications'] = config['notifications']
> with open('.travis.new.yml', 'w') as fp:
>   yaml.safe_dump(config_new, fp)
> EOF
(pyyaml)$ cat .travis.new.yml
language: objective-c
notifications:
  email: [john.sirois@gmail.com, lhosken@twitter.com, zundel@squareup.com, benjyw@gmail.com,
    areitz@twitter.com]
script: 'sw_vers

  python --version

  java -version

  ./build-support/bin/ci.sh -d

  '
```

Bugs closed: 601

Reviewed at https://rbcommons.com/s/twitter/r/1067/
remove some more redundant docstrings

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/35376388

Reviewed at https://rbcommons.com/s/twitter/r/1035/
new "sitegen" goal in internal backend

Defines a new "sitegen" goal. It's the start of a static site generator for Pants docs.

We don't want to ship this goal to all Pants users.
We do want to use it in the Pants repo.
So set up a new dir src/python/internal_backend/
It doesn't get pex'd up with src/python/pants code.
But can be pulled in by adding backends to pants.ini
(e.g., build-support/bin/publish_docs_new.sh creates pants.ini.sitegen by appending a backends to pants.ini)

So far this goal looks for a site description config'd in a json file,
and based on that copies some .html files
(fixing up links between them to be OK in spite of relative-path changes)
and wraps them in a potentially-pretty .html template and stylesheet.
That ain't everything it should do; but it's enough to get an idea of code structure.

To invoke this code,
./build-support/bin/publish_docs_new.sh

Testing Done:
added tests/python/internal_backend_test:: to ci.sh.
https://travis-ci.org/pantsbuild/pants/builds/35997435

Reviewed at https://rbcommons.com/s/twitter/r/1030/
Prepare the 0.0.24 release.

Bump version & complete the 0.0.24 CHANGELOG section.

Testing Done:
```console
$ ./build-support/bin/release.sh -n
[== Performing a dry run release - no artifacts will be uploaded. ==]
...
[== Building package pantsbuild.pants-0.0.24 with target '//src/python/pants:pants-packaged' ... ==]
...
[== Building package pantsbuild.pants.testinfra-0.0.24 with target '//src/python/pants:test_infra' ... ==]
...
[== Installing and testing package pantsbuild.pants-0.0.24 ... ==]
...
[== Installing and testing package pantsbuild.pants.testinfra-0.0.24 ... ==]
...
[== Dry run release succeeded. ==]
```

Bugs closed: 597, 606

Reviewed at https://rbcommons.com/s/twitter/r/1069/
Fixup backticks embedded in a here document.

These were intended to be in the context of a comment
but really were being evaluated.

Testing Done:
Saw the error before this fix here: https://travis-ci.org/pantsbuild/pants/jobs/36065429 - note `./build-support/bin/ci-sync.sh: line 51: .travis.yml: command not found`
```console
$ ./build-support/bin/ci-sync.sh
Syncing OSX CI to 0e3db04c45888b68689fb2fcbd1a4c3c658ef384 for CI build 2055.
Requirement already satisfied (use --upgrade to upgrade): PyYAML in /home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages
Cleaning up...
./build-support/bin/ci-sync.sh: line 51: .travis.yml: command not found
[detached HEAD 9f425f7] Sync of https://github.com/pantsbuild/pants/commit/0e3db04c45888b68689fb2fcbd1a4c3c658ef384
 1 file changed, 13 insertions(+), 41 deletions(-)
 rewrite .travis.yml (92%)
Counting objects: 1900, done.
Delta compression using up to 32 threads.
Compressing objects: 100% (1062/1062), done.
Writing objects: 100% (1900/1900), 2.27 MiB | 0 bytes/s, done.
Total 1900 (delta 608), reused 1896 (delta 607)
To https://github.com/pantsbuild/pants-for-travis-osx-ci.git
 + 2b75951...9f425f7 HEAD -> master (forced update)
```

Bugs closed: 605

Reviewed at https://rbcommons.com/s/twitter/r/1070/
Add support for publishing plugins to pants.

Reviewed at https://rbcommons.com/s/twitter/r/1021
Narrow setup 'Operating System' classfiers to known-good.

We know for a fact pants does not work out of the box on Windows
as reported here:
  https://groups.google.com/d/topic/pants-devel/N6CKjY5j2ps/discussion

For now, restrict the claimed OS support to those we know pants
works with / we have CI for.

Also add appropriate 'Topic' classifiers.

Testing Done:
$ pants.dev setup_py src/python/pants:pants-packaged
*** Running pants in dev mode from /home/jsirois/dev/3rdparty/jsirois-pants/src/python/pants/bin/pants_exe.py ***
WARNING!  /home/jsirois/dev/3rdparty/jsirois-pants/dist/pantsbuild.pants.m5cE9B/src/pants/docs/conf.py is source but does not belong to a package!
WARNING!  /home/jsirois/dev/3rdparty/jsirois-pants/dist/pantsbuild.pants.m5cE9B/src/pants/docs/pagerank.py is source but does not belong to a package!
Running packager against /home/jsirois/dev/3rdparty/jsirois-pants/dist/pantsbuild.pants-0.0.24
Writing /home/jsirois/dev/3rdparty/jsirois-pants/dist/pantsbuild.pants-0.0.24.tar.gz

$ tar -xzf dist/pantsbuild.pants-0.0.24.tar.gz -C dist/
$ tail dist/pantsbuild.pants-0.0.24/PKG-INFO
        * Initial published version of ``pantsbuild.pants``.

Platform: UNKNOWN
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python
Classifier: Topic :: Software Development :: Build Tools

$ pants.dev setup_py src/python/pants:test_infra
*** Running pants in dev mode from /home/jsirois/dev/3rdparty/jsirois-pants/src/python/pants/bin/pants_exe.py ***
Running packager against /home/jsirois/dev/3rdparty/jsirois-pants/dist/pantsbuild.pants.testinfra-0.0.24
Writing /home/jsirois/dev/3rdparty/jsirois-pants/dist/pantsbuild.pants.testinfra-0.0.24.tar.gz
$ tar -xzf dist/pantsbuild.pants.testinfra-0.0.24.tar.gz -C dist/
$ tail dist/pantsbuild.pants.testinfra-0.0.24/PKG-INFO

Platform: UNKNOWN
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python
Classifier: Topic :: Software Development :: Build Tools
Classifier: Topic :: Software Development :: Testing

Bugs closed: 593, 609

Reviewed at https://rbcommons.com/s/twitter/r/1075/
Make internal_backend plugins 1st class local pants plugins.

This moves wiring-up of the pants_internal backends to pants.ini
and make the (future) role of src/python/pants/bin:pants clear
and distinct from src/python/pants/bin:pants_local_binary.

It also fixes some broken docsite.json paths and adds some
TODOs to nuke 3rdparty/python/dev_only_requirements.txt.

Testing Done:
CI went green here: https://travis-ci.org/pantsbuild/pants/builds/36185854

# Publishing still works (publish_docs_new newly re-works)
$ ./build-support/bin/publish_docs.sh
$ ./build-support/bin/publish_docs_new.sh

# internal_backend tests still work
$ PANTS_DEV=1 ./pants goal test tests/python/internal_backend_test::
$ rm pants.pex && ./pants goal test tests/python/internal_backend_test::

# the local pants.pex contains the internal_backend
$ zipinfo pants.pex | grep internal_backend
-rw-r--r--  2.0 unx       55 b- defN 14-Sep-24 11:19 internal_backend/__init__.py
-rw-r--r--  2.0 unx        0 b- defN 14-Sep-23 14:29 internal_backend/sitegen/__init__.py
-rw-r--r--  2.0 unx      123 b- defN 14-Sep-24 11:18 internal_backend/sitegen/__init__.pyc
-rw-r--r--  2.0 unx      555 b- defN 14-Sep-23 14:29 internal_backend/sitegen/register.py
-rw-r--r--  2.0 unx      813 b- defN 14-Sep-24 11:18 internal_backend/sitegen/register.pyc
-rw-r--r--  2.0 unx       55 b- defN 14-Sep-24 11:19 internal_backend/sitegen/tasks/__init__.py
-rw-r--r--  2.0 unx     5749 b- defN 14-Sep-23 14:29 internal_backend/sitegen/tasks/sitegen.py
-rw-r--r--  2.0 unx     7291 b- defN 14-Sep-24 11:18 internal_backend/sitegen/tasks/sitegen.pyc

# The published sdists do not contain the internal_backend
$ ./pants setup_py //src/python/pants:pants-packaged //src/python/pants:test_infra
WARNING!  /home/jsirois/dev/3rdparty/jsirois-pants2/dist/pantsbuild.pants.47Hv89/src/pants/docs/conf.py is source but does not belong to a package!
WARNING!  /home/jsirois/dev/3rdparty/jsirois-pants2/dist/pantsbuild.pants.47Hv89/src/pants/docs/pagerank.py is source but does not belong to a package!
Running packager against /home/jsirois/dev/3rdparty/jsirois-pants2/dist/pantsbuild.pants-0.0.24
Writing /home/jsirois/dev/3rdparty/jsirois-pants2/dist/pantsbuild.pants-0.0.24.tar.gz
$ tar -tzf dist/pantsbuild.pants-0.0.24.tar.gz | grep internal_backend
$

$ ./pants setup_py //src/python/pants:test_infra
Running packager against /home/jsirois/dev/3rdparty/jsirois-pants2/dist/pantsbuild.pants.testinfra-0.0.24
Writing /home/jsirois/dev/3rdparty/jsirois-pants2/dist/pantsbuild.pants.testinfra-0.0.24.tar.gz
$ tar -tzf dist/pantsbuild.pants.testinfra-0.0.24.tar.gz | grep internal_backend
$

Bugs closed: 608

Reviewed at https://rbcommons.com/s/twitter/r/1073/
pants: Survive targets without derivations

Walks the full derivation chain for a target (which in the case of a particular target type at Twitter will be 3 entries long.)

Testing Done:
Rather than:

    Exception caught:
      File "src/python/pants/bin/pants_exe.py", line 205, in <module>
        main()
      File "src/python/pants/bin/pants_exe.py", line 200, in main
        _run()
      File "src/python/pants/bin/pants_exe.py", line 181, in _run
        result = command.run(lock)
      File "src/python/pants/commands/goal_runner.py", line 295, in run
        return engine.execute(context, self.goals)
      File "src/python/pants/engine/engine.py", line 48, in execute
        self.attempt(context, goals)
      File "src/python/pants/engine/round_engine.py", line 184, in attempt
        goal_executor.attempt(explain)
      File "src/python/pants/engine/round_engine.py", line 42, in attempt
        task.execute()
      File "src/python/pants/backend/jvm/tasks/jar_publish.py", line 523, in execute
        self.check_targets(exported_targets)
      File "src/python/pants/backend/jvm/tasks/jar_publish.py", line 760, in check_targets
        derived_target = derived_by_target[invalid_target]

    Exception message: ScalaLibrary(SyntheticAddress(.pants.d/gen/scrooge/scrooge/scala-finagle:.pants.d.gen.idl-extract.3rdparty.jvm.com.twitter.<snip>))

...you get:

    FAILURE: The following errors must be resolved to publish.
      Cannot publish BuildFileAddress(x/y/src/main/scala/BUILD, scala) due to:
        SyntheticAddress(.pants.d/gen/idl-extract/3rdparty.jvm.com.twitter.<snip>) - Does not provide a binary artifact.

Reviewed at https://rbcommons.com/s/twitter/r/1066/
Refactor Payload.

* Break Payload into a Payload/PayloadField abstraction, and fix the major
  design flaw preventing clean Target subclassing.
* Allow explicit freezing of Payloads and always freeze Payloads in the
  Target superclass `__init__`.
* Tighten up invalidation hashing (fingerprinting) and testing.
* Change the name of "invalidation hash" in general to "fingerprint", since
  the former is ambiguous and used to mean different things in several
  different places.
* Make Repository and Wiki plain objects rather than Target subclasses.
  Users who want to expose a custom Repository instance in all BUILD files now
  register this instance in their internal backend's register.py as an exposed
  object.  Then consumers of this Repository refer to this object directly.
  This allows the removal of the horrible hack in JvmTarget.provides.
  It gets us much closer to having truly immutable Payloads.

Testing Done:
ci.sh is green locally

CI is green: https://travis-ci.org/pantsbuild/pants/builds/36363381

Reviewed at https://rbcommons.com/s/twitter/r/1063/
Coalesce errors when parsing BUILDS in a spec

The idea is that when you submit a pants command, it will emit an error for ALL erroneous build files in a spec.

Note: if multiple specs are given on the command line, if a given one has an error, then only the first's will be printed, then it will error out.

I handled the errors as best as I could, becasue we need to know the flag before a lot of the options parsing happens.

Testing Done:
I tested manually and it works. I need to update the unit tests but wanted to validate my approach first.

Reviewed at https://rbcommons.com/s/twitter/r/1061/
Remove ROOT_DIR everywhere it is mentioned and replace it with build root.

Testing Done:
$ ack ROOT_DIR
(no results)

(since this is only docstring changes, no need for tests)

Reviewed at https://rbcommons.com/s/twitter/r/1084/
Fix order-dependent force handling: if a version is forced in one place, it is forced everywhere.  Two JarLibraries might compare equal even though their force setting is different; we want force to be dominant.

Reviewed at https://rbcommons.com/s/twitter/r/1085/
Support old-style flags even in the new flag system.

E.g., --compile-java-foo-bar anywhere on the cmd-line is the same
as ./pants compile.java --foo-bar.

This will make migration easier, as people are used to the old style,
and we may even want to keep this forever, if some users prefer
the verbose flags.

Testing Done:
Ran all the tests.

Reviewed at https://rbcommons.com/s/twitter/r/1093/
Tiny tweak to legacy option printing.

Also rewrote some comments for clarity.

Reviewed at https://rbcommons.com/s/twitter/r/1094/
Allow a task type know its own options scope.

E.g., JavaCompile should know that its options are under compile.java.

Also fix some stale comments from the goal->task terminology change.

Testing Done:
Ran all the tests. Tested that this works when registering options in the new style (will be sent for review in a future change).

Reviewed at https://rbcommons.com/s/twitter/r/1095/