doc: if rbt fails, don't give up and workaround. things to try

Reviewed at https://rbcommons.com/s/twitter/r/1048/
remove page target whose README.md does not exist

Reviewed at https://rbcommons.com/s/twitter/r/1046/
Correct old examples/ reference - use testprojects/ copy.

Testing Done:
CI went green here: https://travis-ci.org/pantsbuild/pants/builds/35687312

Bugs closed: 585

Reviewed at https://rbcommons.com/s/twitter/r/1049/
Improve python CI robustness.

Ensure all tests/python targets are selected when dividing up unit
and integration tests - no longer use aggregator targets as a guide.

This change leaves the aggregators in-place and just divorces CI from
them to ensure we always CI all test targets.

 build-support/bin/ci.sh | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)

Testing Done:
Locally:
$ ./build-support/bin/ci.sh

Travis went green here: https://travis-ci.org/pantsbuild/pants/builds/35691561

Bugs closed: 586

Reviewed at https://rbcommons.com/s/twitter/r/1050/
jvm_binary init handling of name param made more sense before kwargs idiom

...but now it's probably better not to mention it in the sig

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/35743270

Reviewed at https://rbcommons.com/s/twitter/r/1054/
Fix coverage tests on MacOS

Testing Done:
build-support/bin/ci.sh on my mac laptop.

ci green @ https://travis-ci.org/pantsbuild/pants/builds/35870205

Bugs closed: 595

Reviewed at https://rbcommons.com/s/twitter/r/1057/
Add a CI action to support CI against OSX.

This adds a sync action run on any push to pantsbuild/pants
master that replaces pantsbuild/pants-for-travis-osx-ci master
modifying the active .travis.yml to trigger an OSX CI run.

Testing Done:
Normal CI went green here: https://travis-ci.org/pantsbuild/pants/builds/35900516

OSX CI went green here: https://travis-ci.org/pantsbuild/pants-for-travis-osx-ci/builds/35899630
Via:
```console
$ set | grep GH_ | cut -d= -f1
GH_EMAIL
GH_TOKEN
GH_USER
$ ./build-support/bin/ci-sync.sh
[jsirois/pants-for-travis-osx-ci/sync 9a13170] Prepare pants OSX mirror for CI.
 Author: Pants CI <pants-ci@googlegroups.com>
 1 file changed, 13 insertions(+), 40 deletions(-)
 rewrite .travis.yml (87%)
Counting objects: 37, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (18/18), done.
Writing objects: 100% (23/23), 3.05 KiB | 0 bytes/s, done.
Total 23 (delta 14), reused 9 (delta 3)
To https://github.com/pantsbuild/pants-for-travis-osx-ci.git
 + 888c8bf...9a13170 HEAD -> master (forced update)
$ gitbinfo
* jsirois/pants-for-travis-osx-ci/sync 9a13170 [pants/master: ahead 2] Prepare pants OSX mirror for CI.
$ git slog -2
commit 9a13170039dfe93824fd9487f78f96c134e09d88
Author: Pants CI <pants-ci@googlegroups.com>
Date:   Sun Sep 21 20:22:52 2014 -0600

    Prepare pants OSX mirror for CI.

 .travis.yml | 35 ++++-------------------------------
 1 file changed, 4 insertions(+), 31 deletions(-)

commit f4cb72ba4cc48ec5f6a23c86903fe83b735a8f3f
Author: John Sirois <john.sirois@gmail.com>
Date:   Sun Sep 21 17:30:29 2014 -0600

    Add a CI action to support CI against OSX.
...
```

Bugs closed: 596

Reviewed at https://rbcommons.com/s/twitter/r/1058/
Sync the OSX CI eagerly and add diagnostics.

Move the OSX CI sync up-front to leverage parallelism
instead of waiting on the whole local build matrix to complete.

Also add diagnostics that record why the sync is skipped if
skipped in addition to making skipping & syncing decisions more
robust and explicit.

Testing Done:
CI away here already shows proper fork sync-skipping diagnostic: https://travis-ci.org/pantsbuild/pants/builds/35967008
This will have to land on pantsbuild/pants master to vet fully.

Bugs closed: 599

Reviewed at https://rbcommons.com/s/twitter/r/1064/
Fixup ci-sync.sh to cover committer ident.

Testing Done:
None - can only be tested on pantsbuild.pants master, but in response to this failure: https://travis-ci.org/pantsbuild/pants/jobs/35969933
CI did go green here though - so nothing horribly more broken: https://travis-ci.org/pantsbuild/pants/builds/35972662

Bugs closed: 600

Reviewed at https://rbcommons.com/s/twitter/r/1065/
Remove recommendation of manual edits to the pushdb

The manual-edits-to-the-pushdb approach gets a bit too down and dirty. Remove the recommendation in favor of resetting and using --publish-override.

Reviewed at https://rbcommons.com/s/twitter/r/1056/
Use `.travis.yml` notifications for `.travis.osx.yml`

This copies over the notifications settings of the main
`.travis.yml` to `.travis.osx.yml` so that we can maintain
them centrally.

The change also tweaks the commit message to aid in quick
linking / correspondence of a pantsbuild/pants-for-travis-osx-ci
commit to a pantsbuild/pants commit.

Testing Done:
Ad hoc:
```bash
$ venv_setup pyyaml PyYAML && source ~/.pythons/venvs/pyyaml/bin/activate
(pyyaml)$ python << EOF
> import yaml
> with open('.travis.yml') as fp:
>   config = yaml.safe_load(fp)
> with open('.travis.osx.yml') as fp:
>   config_new = yaml.safe_load(fp)
> config_new['notifications'] = config['notifications']
> with open('.travis.new.yml', 'w') as fp:
>   yaml.safe_dump(config_new, fp)
> EOF
(pyyaml)$ cat .travis.new.yml
language: objective-c
notifications:
  email: [john.sirois@gmail.com, lhosken@twitter.com, zundel@squareup.com, benjyw@gmail.com,
    areitz@twitter.com]
script: 'sw_vers

  python --version

  java -version

  ./build-support/bin/ci.sh -d

  '
```

Bugs closed: 601

Reviewed at https://rbcommons.com/s/twitter/r/1067/
remove some more redundant docstrings

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/35376388

Reviewed at https://rbcommons.com/s/twitter/r/1035/
new "sitegen" goal in internal backend

Defines a new "sitegen" goal. It's the start of a static site generator for Pants docs.

We don't want to ship this goal to all Pants users.
We do want to use it in the Pants repo.
So set up a new dir src/python/internal_backend/
It doesn't get pex'd up with src/python/pants code.
But can be pulled in by adding backends to pants.ini
(e.g., build-support/bin/publish_docs_new.sh creates pants.ini.sitegen by appending a backends to pants.ini)

So far this goal looks for a site description config'd in a json file,
and based on that copies some .html files
(fixing up links between them to be OK in spite of relative-path changes)
and wraps them in a potentially-pretty .html template and stylesheet.
That ain't everything it should do; but it's enough to get an idea of code structure.

To invoke this code,
./build-support/bin/publish_docs_new.sh

Testing Done:
added tests/python/internal_backend_test:: to ci.sh.
https://travis-ci.org/pantsbuild/pants/builds/35997435

Reviewed at https://rbcommons.com/s/twitter/r/1030/
Prepare the 0.0.24 release.

Bump version & complete the 0.0.24 CHANGELOG section.

Testing Done:
```console
$ ./build-support/bin/release.sh -n
[== Performing a dry run release - no artifacts will be uploaded. ==]
...
[== Building package pantsbuild.pants-0.0.24 with target '//src/python/pants:pants-packaged' ... ==]
...
[== Building package pantsbuild.pants.testinfra-0.0.24 with target '//src/python/pants:test_infra' ... ==]
...
[== Installing and testing package pantsbuild.pants-0.0.24 ... ==]
...
[== Installing and testing package pantsbuild.pants.testinfra-0.0.24 ... ==]
...
[== Dry run release succeeded. ==]
```

Bugs closed: 597, 606

Reviewed at https://rbcommons.com/s/twitter/r/1069/
Fixup backticks embedded in a here document.

These were intended to be in the context of a comment
but really were being evaluated.

Testing Done:
Saw the error before this fix here: https://travis-ci.org/pantsbuild/pants/jobs/36065429 - note `./build-support/bin/ci-sync.sh: line 51: .travis.yml: command not found`
```console
$ ./build-support/bin/ci-sync.sh
Syncing OSX CI to 0e3db04c45888b68689fb2fcbd1a4c3c658ef384 for CI build 2055.
Requirement already satisfied (use --upgrade to upgrade): PyYAML in /home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages
Cleaning up...
./build-support/bin/ci-sync.sh: line 51: .travis.yml: command not found
[detached HEAD 9f425f7] Sync of https://github.com/pantsbuild/pants/commit/0e3db04c45888b68689fb2fcbd1a4c3c658ef384
 1 file changed, 13 insertions(+), 41 deletions(-)
 rewrite .travis.yml (92%)
Counting objects: 1900, done.
Delta compression using up to 32 threads.
Compressing objects: 100% (1062/1062), done.
Writing objects: 100% (1900/1900), 2.27 MiB | 0 bytes/s, done.
Total 1900 (delta 608), reused 1896 (delta 607)
To https://github.com/pantsbuild/pants-for-travis-osx-ci.git
 + 2b75951...9f425f7 HEAD -> master (forced update)
```

Bugs closed: 605

Reviewed at https://rbcommons.com/s/twitter/r/1070/
Add support for publishing plugins to pants.

Reviewed at https://rbcommons.com/s/twitter/r/1021
Narrow setup 'Operating System' classfiers to known-good.

We know for a fact pants does not work out of the box on Windows
as reported here:
  https://groups.google.com/d/topic/pants-devel/N6CKjY5j2ps/discussion

For now, restrict the claimed OS support to those we know pants
works with / we have CI for.

Also add appropriate 'Topic' classifiers.

Testing Done:
$ pants.dev setup_py src/python/pants:pants-packaged
*** Running pants in dev mode from /home/jsirois/dev/3rdparty/jsirois-pants/src/python/pants/bin/pants_exe.py ***
WARNING!  /home/jsirois/dev/3rdparty/jsirois-pants/dist/pantsbuild.pants.m5cE9B/src/pants/docs/conf.py is source but does not belong to a package!
WARNING!  /home/jsirois/dev/3rdparty/jsirois-pants/dist/pantsbuild.pants.m5cE9B/src/pants/docs/pagerank.py is source but does not belong to a package!
Running packager against /home/jsirois/dev/3rdparty/jsirois-pants/dist/pantsbuild.pants-0.0.24
Writing /home/jsirois/dev/3rdparty/jsirois-pants/dist/pantsbuild.pants-0.0.24.tar.gz

$ tar -xzf dist/pantsbuild.pants-0.0.24.tar.gz -C dist/
$ tail dist/pantsbuild.pants-0.0.24/PKG-INFO
        * Initial published version of ``pantsbuild.pants``.

Platform: UNKNOWN
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python
Classifier: Topic :: Software Development :: Build Tools

$ pants.dev setup_py src/python/pants:test_infra
*** Running pants in dev mode from /home/jsirois/dev/3rdparty/jsirois-pants/src/python/pants/bin/pants_exe.py ***
Running packager against /home/jsirois/dev/3rdparty/jsirois-pants/dist/pantsbuild.pants.testinfra-0.0.24
Writing /home/jsirois/dev/3rdparty/jsirois-pants/dist/pantsbuild.pants.testinfra-0.0.24.tar.gz
$ tar -xzf dist/pantsbuild.pants.testinfra-0.0.24.tar.gz -C dist/
$ tail dist/pantsbuild.pants.testinfra-0.0.24/PKG-INFO

Platform: UNKNOWN
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python
Classifier: Topic :: Software Development :: Build Tools
Classifier: Topic :: Software Development :: Testing

Bugs closed: 593, 609

Reviewed at https://rbcommons.com/s/twitter/r/1075/
Make internal_backend plugins 1st class local pants plugins.

This moves wiring-up of the pants_internal backends to pants.ini
and make the (future) role of src/python/pants/bin:pants clear
and distinct from src/python/pants/bin:pants_local_binary.

It also fixes some broken docsite.json paths and adds some
TODOs to nuke 3rdparty/python/dev_only_requirements.txt.

Testing Done:
CI went green here: https://travis-ci.org/pantsbuild/pants/builds/36185854

# Publishing still works (publish_docs_new newly re-works)
$ ./build-support/bin/publish_docs.sh
$ ./build-support/bin/publish_docs_new.sh

# internal_backend tests still work
$ PANTS_DEV=1 ./pants goal test tests/python/internal_backend_test::
$ rm pants.pex && ./pants goal test tests/python/internal_backend_test::

# the local pants.pex contains the internal_backend
$ zipinfo pants.pex | grep internal_backend
-rw-r--r--  2.0 unx       55 b- defN 14-Sep-24 11:19 internal_backend/__init__.py
-rw-r--r--  2.0 unx        0 b- defN 14-Sep-23 14:29 internal_backend/sitegen/__init__.py
-rw-r--r--  2.0 unx      123 b- defN 14-Sep-24 11:18 internal_backend/sitegen/__init__.pyc
-rw-r--r--  2.0 unx      555 b- defN 14-Sep-23 14:29 internal_backend/sitegen/register.py
-rw-r--r--  2.0 unx      813 b- defN 14-Sep-24 11:18 internal_backend/sitegen/register.pyc
-rw-r--r--  2.0 unx       55 b- defN 14-Sep-24 11:19 internal_backend/sitegen/tasks/__init__.py
-rw-r--r--  2.0 unx     5749 b- defN 14-Sep-23 14:29 internal_backend/sitegen/tasks/sitegen.py
-rw-r--r--  2.0 unx     7291 b- defN 14-Sep-24 11:18 internal_backend/sitegen/tasks/sitegen.pyc

# The published sdists do not contain the internal_backend
$ ./pants setup_py //src/python/pants:pants-packaged //src/python/pants:test_infra
WARNING!  /home/jsirois/dev/3rdparty/jsirois-pants2/dist/pantsbuild.pants.47Hv89/src/pants/docs/conf.py is source but does not belong to a package!
WARNING!  /home/jsirois/dev/3rdparty/jsirois-pants2/dist/pantsbuild.pants.47Hv89/src/pants/docs/pagerank.py is source but does not belong to a package!
Running packager against /home/jsirois/dev/3rdparty/jsirois-pants2/dist/pantsbuild.pants-0.0.24
Writing /home/jsirois/dev/3rdparty/jsirois-pants2/dist/pantsbuild.pants-0.0.24.tar.gz
$ tar -tzf dist/pantsbuild.pants-0.0.24.tar.gz | grep internal_backend
$

$ ./pants setup_py //src/python/pants:test_infra
Running packager against /home/jsirois/dev/3rdparty/jsirois-pants2/dist/pantsbuild.pants.testinfra-0.0.24
Writing /home/jsirois/dev/3rdparty/jsirois-pants2/dist/pantsbuild.pants.testinfra-0.0.24.tar.gz
$ tar -tzf dist/pantsbuild.pants.testinfra-0.0.24.tar.gz | grep internal_backend
$

Bugs closed: 608

Reviewed at https://rbcommons.com/s/twitter/r/1073/
pants: Survive targets without derivations

Walks the full derivation chain for a target (which in the case of a particular target type at Twitter will be 3 entries long.)

Testing Done:
Rather than:

    Exception caught:
      File "src/python/pants/bin/pants_exe.py", line 205, in <module>
        main()
      File "src/python/pants/bin/pants_exe.py", line 200, in main
        _run()
      File "src/python/pants/bin/pants_exe.py", line 181, in _run
        result = command.run(lock)
      File "src/python/pants/commands/goal_runner.py", line 295, in run
        return engine.execute(context, self.goals)
      File "src/python/pants/engine/engine.py", line 48, in execute
        self.attempt(context, goals)
      File "src/python/pants/engine/round_engine.py", line 184, in attempt
        goal_executor.attempt(explain)
      File "src/python/pants/engine/round_engine.py", line 42, in attempt
        task.execute()
      File "src/python/pants/backend/jvm/tasks/jar_publish.py", line 523, in execute
        self.check_targets(exported_targets)
      File "src/python/pants/backend/jvm/tasks/jar_publish.py", line 760, in check_targets
        derived_target = derived_by_target[invalid_target]

    Exception message: ScalaLibrary(SyntheticAddress(.pants.d/gen/scrooge/scrooge/scala-finagle:.pants.d.gen.idl-extract.3rdparty.jvm.com.twitter.<snip>))

...you get:

    FAILURE: The following errors must be resolved to publish.
      Cannot publish BuildFileAddress(x/y/src/main/scala/BUILD, scala) due to:
        SyntheticAddress(.pants.d/gen/idl-extract/3rdparty.jvm.com.twitter.<snip>) - Does not provide a binary artifact.

Reviewed at https://rbcommons.com/s/twitter/r/1066/
Refactor Payload.

* Break Payload into a Payload/PayloadField abstraction, and fix the major
  design flaw preventing clean Target subclassing.
* Allow explicit freezing of Payloads and always freeze Payloads in the
  Target superclass `__init__`.
* Tighten up invalidation hashing (fingerprinting) and testing.
* Change the name of "invalidation hash" in general to "fingerprint", since
  the former is ambiguous and used to mean different things in several
  different places.
* Make Repository and Wiki plain objects rather than Target subclasses.
  Users who want to expose a custom Repository instance in all BUILD files now
  register this instance in their internal backend's register.py as an exposed
  object.  Then consumers of this Repository refer to this object directly.
  This allows the removal of the horrible hack in JvmTarget.provides.
  It gets us much closer to having truly immutable Payloads.

Testing Done:
ci.sh is green locally

CI is green: https://travis-ci.org/pantsbuild/pants/builds/36363381

Reviewed at https://rbcommons.com/s/twitter/r/1063/