Add support for publishing plugins to pants.

Reviewed at https://rbcommons.com/s/twitter/r/1021
Narrow setup 'Operating System' classfiers to known-good.

We know for a fact pants does not work out of the box on Windows
as reported here:
  https://groups.google.com/d/topic/pants-devel/N6CKjY5j2ps/discussion

For now, restrict the claimed OS support to those we know pants
works with / we have CI for.

Also add appropriate 'Topic' classifiers.

Testing Done:
$ pants.dev setup_py src/python/pants:pants-packaged
*** Running pants in dev mode from /home/jsirois/dev/3rdparty/jsirois-pants/src/python/pants/bin/pants_exe.py ***
WARNING!  /home/jsirois/dev/3rdparty/jsirois-pants/dist/pantsbuild.pants.m5cE9B/src/pants/docs/conf.py is source but does not belong to a package!
WARNING!  /home/jsirois/dev/3rdparty/jsirois-pants/dist/pantsbuild.pants.m5cE9B/src/pants/docs/pagerank.py is source but does not belong to a package!
Running packager against /home/jsirois/dev/3rdparty/jsirois-pants/dist/pantsbuild.pants-0.0.24
Writing /home/jsirois/dev/3rdparty/jsirois-pants/dist/pantsbuild.pants-0.0.24.tar.gz

$ tar -xzf dist/pantsbuild.pants-0.0.24.tar.gz -C dist/
$ tail dist/pantsbuild.pants-0.0.24/PKG-INFO
        * Initial published version of ``pantsbuild.pants``.

Platform: UNKNOWN
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python
Classifier: Topic :: Software Development :: Build Tools

$ pants.dev setup_py src/python/pants:test_infra
*** Running pants in dev mode from /home/jsirois/dev/3rdparty/jsirois-pants/src/python/pants/bin/pants_exe.py ***
Running packager against /home/jsirois/dev/3rdparty/jsirois-pants/dist/pantsbuild.pants.testinfra-0.0.24
Writing /home/jsirois/dev/3rdparty/jsirois-pants/dist/pantsbuild.pants.testinfra-0.0.24.tar.gz
$ tar -xzf dist/pantsbuild.pants.testinfra-0.0.24.tar.gz -C dist/
$ tail dist/pantsbuild.pants.testinfra-0.0.24/PKG-INFO

Platform: UNKNOWN
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python
Classifier: Topic :: Software Development :: Build Tools
Classifier: Topic :: Software Development :: Testing

Bugs closed: 593, 609

Reviewed at https://rbcommons.com/s/twitter/r/1075/
Make internal_backend plugins 1st class local pants plugins.

This moves wiring-up of the pants_internal backends to pants.ini
and make the (future) role of src/python/pants/bin:pants clear
and distinct from src/python/pants/bin:pants_local_binary.

It also fixes some broken docsite.json paths and adds some
TODOs to nuke 3rdparty/python/dev_only_requirements.txt.

Testing Done:
CI went green here: https://travis-ci.org/pantsbuild/pants/builds/36185854

# Publishing still works (publish_docs_new newly re-works)
$ ./build-support/bin/publish_docs.sh
$ ./build-support/bin/publish_docs_new.sh

# internal_backend tests still work
$ PANTS_DEV=1 ./pants goal test tests/python/internal_backend_test::
$ rm pants.pex && ./pants goal test tests/python/internal_backend_test::

# the local pants.pex contains the internal_backend
$ zipinfo pants.pex | grep internal_backend
-rw-r--r--  2.0 unx       55 b- defN 14-Sep-24 11:19 internal_backend/__init__.py
-rw-r--r--  2.0 unx        0 b- defN 14-Sep-23 14:29 internal_backend/sitegen/__init__.py
-rw-r--r--  2.0 unx      123 b- defN 14-Sep-24 11:18 internal_backend/sitegen/__init__.pyc
-rw-r--r--  2.0 unx      555 b- defN 14-Sep-23 14:29 internal_backend/sitegen/register.py
-rw-r--r--  2.0 unx      813 b- defN 14-Sep-24 11:18 internal_backend/sitegen/register.pyc
-rw-r--r--  2.0 unx       55 b- defN 14-Sep-24 11:19 internal_backend/sitegen/tasks/__init__.py
-rw-r--r--  2.0 unx     5749 b- defN 14-Sep-23 14:29 internal_backend/sitegen/tasks/sitegen.py
-rw-r--r--  2.0 unx     7291 b- defN 14-Sep-24 11:18 internal_backend/sitegen/tasks/sitegen.pyc

# The published sdists do not contain the internal_backend
$ ./pants setup_py //src/python/pants:pants-packaged //src/python/pants:test_infra
WARNING!  /home/jsirois/dev/3rdparty/jsirois-pants2/dist/pantsbuild.pants.47Hv89/src/pants/docs/conf.py is source but does not belong to a package!
WARNING!  /home/jsirois/dev/3rdparty/jsirois-pants2/dist/pantsbuild.pants.47Hv89/src/pants/docs/pagerank.py is source but does not belong to a package!
Running packager against /home/jsirois/dev/3rdparty/jsirois-pants2/dist/pantsbuild.pants-0.0.24
Writing /home/jsirois/dev/3rdparty/jsirois-pants2/dist/pantsbuild.pants-0.0.24.tar.gz
$ tar -tzf dist/pantsbuild.pants-0.0.24.tar.gz | grep internal_backend
$

$ ./pants setup_py //src/python/pants:test_infra
Running packager against /home/jsirois/dev/3rdparty/jsirois-pants2/dist/pantsbuild.pants.testinfra-0.0.24
Writing /home/jsirois/dev/3rdparty/jsirois-pants2/dist/pantsbuild.pants.testinfra-0.0.24.tar.gz
$ tar -tzf dist/pantsbuild.pants.testinfra-0.0.24.tar.gz | grep internal_backend
$

Bugs closed: 608

Reviewed at https://rbcommons.com/s/twitter/r/1073/
pants: Survive targets without derivations

Walks the full derivation chain for a target (which in the case of a particular target type at Twitter will be 3 entries long.)

Testing Done:
Rather than:

    Exception caught:
      File "src/python/pants/bin/pants_exe.py", line 205, in <module>
        main()
      File "src/python/pants/bin/pants_exe.py", line 200, in main
        _run()
      File "src/python/pants/bin/pants_exe.py", line 181, in _run
        result = command.run(lock)
      File "src/python/pants/commands/goal_runner.py", line 295, in run
        return engine.execute(context, self.goals)
      File "src/python/pants/engine/engine.py", line 48, in execute
        self.attempt(context, goals)
      File "src/python/pants/engine/round_engine.py", line 184, in attempt
        goal_executor.attempt(explain)
      File "src/python/pants/engine/round_engine.py", line 42, in attempt
        task.execute()
      File "src/python/pants/backend/jvm/tasks/jar_publish.py", line 523, in execute
        self.check_targets(exported_targets)
      File "src/python/pants/backend/jvm/tasks/jar_publish.py", line 760, in check_targets
        derived_target = derived_by_target[invalid_target]

    Exception message: ScalaLibrary(SyntheticAddress(.pants.d/gen/scrooge/scrooge/scala-finagle:.pants.d.gen.idl-extract.3rdparty.jvm.com.twitter.<snip>))

...you get:

    FAILURE: The following errors must be resolved to publish.
      Cannot publish BuildFileAddress(x/y/src/main/scala/BUILD, scala) due to:
        SyntheticAddress(.pants.d/gen/idl-extract/3rdparty.jvm.com.twitter.<snip>) - Does not provide a binary artifact.

Reviewed at https://rbcommons.com/s/twitter/r/1066/
Refactor Payload.

* Break Payload into a Payload/PayloadField abstraction, and fix the major
  design flaw preventing clean Target subclassing.
* Allow explicit freezing of Payloads and always freeze Payloads in the
  Target superclass `__init__`.
* Tighten up invalidation hashing (fingerprinting) and testing.
* Change the name of "invalidation hash" in general to "fingerprint", since
  the former is ambiguous and used to mean different things in several
  different places.
* Make Repository and Wiki plain objects rather than Target subclasses.
  Users who want to expose a custom Repository instance in all BUILD files now
  register this instance in their internal backend's register.py as an exposed
  object.  Then consumers of this Repository refer to this object directly.
  This allows the removal of the horrible hack in JvmTarget.provides.
  It gets us much closer to having truly immutable Payloads.

Testing Done:
ci.sh is green locally

CI is green: https://travis-ci.org/pantsbuild/pants/builds/36363381

Reviewed at https://rbcommons.com/s/twitter/r/1063/
Coalesce errors when parsing BUILDS in a spec

The idea is that when you submit a pants command, it will emit an error for ALL erroneous build files in a spec.

Note: if multiple specs are given on the command line, if a given one has an error, then only the first's will be printed, then it will error out.

I handled the errors as best as I could, becasue we need to know the flag before a lot of the options parsing happens.

Testing Done:
I tested manually and it works. I need to update the unit tests but wanted to validate my approach first.

Reviewed at https://rbcommons.com/s/twitter/r/1061/
Remove ROOT_DIR everywhere it is mentioned and replace it with build root.

Testing Done:
$ ack ROOT_DIR
(no results)

(since this is only docstring changes, no need for tests)

Reviewed at https://rbcommons.com/s/twitter/r/1084/
Fix order-dependent force handling: if a version is forced in one place, it is forced everywhere.  Two JarLibraries might compare equal even though their force setting is different; we want force to be dominant.

Reviewed at https://rbcommons.com/s/twitter/r/1085/