doc doc.py

a page about 'page' and 'goal markdown'

gave src/java/examples/hello a README.md and a corresponding BUILD target

tweaked a couple of doc pages that "included" hello/main/BUILD for a code example: tweaked them so that they wouldn't mention the page target; would be strange in that context.

Bugs closed: 257

Reviewed at https://rbcommons.com/s/twitter/r/582/
Fix missing search&replace of .buildfile -> .build_file

Find cases of .buildfile and replace with .build_file.  Running python tests with coverage was broken.

Testing Done:
Ran python tests with PANTS_PY_COVERAGE=1

Reviewed at https://rbcommons.com/s/twitter/r/593/
rm java/examples/pingpong and many of its 3rdparty deps

rm {src|tests}/java/com/pants/examples/pingpong

rm -r 3rdparty/jvm/com/

3rdparty/BUILD -= ['guice', 'easymock']

3rdparty/BUILD -= ['mockito'] <- pingpong didn't use this; 'git grep' says nothing else did either

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/28428093

Reviewed at https://rbcommons.com/s/twitter/r/596/
Payloads with sources now give more useful errors for nesting

Because the creation of the Payloads happens later in the process
(rather than during the build file parsing), we still seem to lose track
of which file the error occurred in... is there a solution for that?

- Adds validation to SourcesMixin and switches it to inheritance with
  SourcesPayload
- Adds a Payloads test file

Reviewed on https://rbcommons.com/s/twitter/r/586/
doc: kick jar_library out of the list of Python-related BUILD file things

back when this was "dependencies", it made sense to list it with Pythonic things. Now, not so much.

Reviewed at https://rbcommons.com/s/twitter/r/598/
dev docs: talk about Travis-CI in one place, not two. Pull requests trigger CI

Mateo was surprised to find out a pull request gets Travis-CI's attention. I'd like to tease him for that, but, uhm, I didn't know it either. So document it. Once it's documented, I can tease the *next* nerd to hit this.

While I was in there, saw an old claim that pants_tests:all ran all the tests. But now our integration tests are separate. So tweaked that wording.

We had two places that talked about running all the tests (ci.sh/Travis). I was too lazy to add this pull-request blurb to both of 'em, so I consolidated. Now one of them is a hyperlink to the other.

Bugs closed: 267

Reviewed at https://rbcommons.com/s/twitter/r/597/
Add support for explicit absolute target address sepcs.

This provides a clear  way to refer to root level targets
from deeper in the BUILD tree.

Testing Done:
$ PANTS_DEV=1 ./pants goal test tests::

travis-ci went green here: https://travis-ci.org/pantsbuild/pants/builds/28347405

Bugs closed: 269, 272, 278

Reviewed at https://rbcommons.com/s/twitter/r/590/
Fixup ScalaLibrary dep injection - do this at BuildGraph population time.

This is effectively a restoration of 24681429b by Ryan Williams.

Previously ScalaLibrary runtime deps were injected in the classpath by
ScalaCompile at execute time.  This let downstream goals like test and
run that just use an aggregate classpath work, but it failed for any goal
needing correct deps at a per-target level like binary, bundle and publish.

This change introduces a scala TargetPlatform class as a very rough start
to centralizing the scala version to target and uses this in ScalaLibrary and
ZincUtils.

Testing Done:

Before:
=======
$ PANTS_DEV=1 ./pants goal bundle src/scala/com/pants/example/hello/exe
$ java -jar dist/exe-bundle/exe.jar
Exception in thread "main" java.lang.NoClassDefFoundError: scala/ScalaObject
...
$ tree dist/exe-bundle/
dist/exe-bundle/
├── exe.jar
└── libs

1 directory, 1 file

After:
======
$ PANTS_DEV=1 ./pants goal bundle src/scala/com/pants/example/hello/exe
$ java -jar dist/exe-bundle/exe.jar
$ java -jar dist/exe-bundle/exe.jar
Num args passed: 0. Stand by for welcome...
Hello, Resource World!
$ tree dist/exe-bundle/
dist/exe-bundle/
├── exe.jar
└── libs
    └── org.scala-lang-scala-library-2.9.3.jar -> /home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/.pants.d/ivy/mapped-jars/src.scala.com.pants.example.hello.exe.exe/org.scala-lang/scala-library/default/org.scala-lang-scala-library-2.9.3.jar

1 directory, 2 files

Bugs closed: 208, 256

Reviewed at https://rbcommons.com/s/twitter/r/581/
Link the payload test into all.

Testing Done:
$ PANTS_DEV=1 ./pants goal test tests/python/pants_test/base:all
09:01:48 00:00   [test]
09:01:48 00:00     [pytest]
09:01:48 00:00       [run]
                     ============== test session starts ===============
                     platform linux3 -- Python 2.6.8 -- py-1.4.20 -- pytest-2.5.2
                     plugins: cov, timeout
                     collected 65 items

                     tests/python/pants_test/base/test_double_dag.py ....
                     tests/python/pants_test/base/test_abbreviate_target_ids.py ....
                     tests/python/pants_test/base/test_cmd_line_spec_parser.py .....
                     tests/python/pants_test/base/test_generator.py ....
                     tests/python/pants_test/base/test_build_invalidator.py .
                     tests/python/pants_test/base/test_payload.py ...
                     tests/python/pants_test/base/test_build_file_parser.py ............
                     tests/python/pants_test/base/test_run_info.py ..
                     tests/python/pants_test/base/test_address.py ........
                     tests/python/pants_test/base/test_hash_utils.py ..
                     tests/python/pants_test/base/test_build_root.py ......
                     tests/python/pants_test/base/test_revision.py ......
                     tests/python/pants_test/base/test_build_file.py ........

                     =========== 65 passed in 0.45 seconds ============

09:02:21 00:33     [junit]
09:02:21 00:33     [specs]
               SUCCESS

Also:
$ comm -3 <(grep -A17 "name = 'base'" tests/python/pants_test/base/BUILD | grep pants | cut -d: -f2 | cut -d\' -f1 | sort) <(./pants goal list tests/python/pants_test/base: | cut -d: -f2 | sort)
	all
	base
	context_utils

The all and base targets are aggregators and context_utils is a python_library so linkage in base is now full.

Bugs closed: 285

Reviewed at https://rbcommons.com/s/twitter/r/602/
Add missed __init__.py
doc: add a cross reference from Bundle comment to JVM page

If you're trying to remember Pants after a year away from it, this would help. Based on a true story by Jeff Hodges.

Reviewed at https://rbcommons.com/s/twitter/r/601/
Use an absolute path for sourcing set_archflags.sh.

This was breaking wrapping usages from foreign repos.

Testing Done:
The error manifested like so in a Twitter internal repo, note the 1st line:
jsirois@gill ~/dev/twitter/pants_internal (jsirois/oss_pants/upgrade_scala_library_dep_injection_fix) $ PANTS_DEV=1 ./pants
/home/jsirois/.pants.dev/os_pants/build-support/pants_venv: line 5: build-support/set_archflags.sh: No such file or directory
*** Running pants in dev mode from /home/jsirois/.pants.dev/os_pants/src/python/pants/bin/pants_exe.py ***
***   with extra requirements: /home/jsirois/dev/twitter/pants_internal/3rdparty/python/requirements.txt ***
***   with extra sources /home/jsirois/dev/twitter/pants_internal/src/python ***

OSS pants bootstrapping still works.
$ git clean -fdx build-support
$ PANTS_DEV=1 ./pants
$ PANTS_DEV=1 ./pants
*** Running pants in dev mode from /home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/src/python/pants/bin/pants_exe.py ***
...

Bugs closed: 291

Reviewed at https://rbcommons.com/s/twitter/r/604/
Clean up some python cruft under goal.

We don't use the python_root target paradigm, nor the
setup_python_environment task.
Refactored binary_util methods to be wrapped inside BinaryUtil object.

BinaryUtil can be initialized with bootstrap_dir, baseurls, and timeout; missing parameters are loaded from config.
See John's comment on https://rbcommons.com/s/twitter/r/556 .

Testing Done:
Existing test cases didn't break.

Reviewed at https://rbcommons.com/s/twitter/r/605/
use JvmTask.classpath to get classpath in JvmdocGen

Testing Done:
same (modulo version skew) change hacked into my twitter internal pants works for me

./build-support/bin/ci.sh underway

Reviewed at https://rbcommons.com/s/twitter/r/599/
Kill support for legacy scrooge.

This change punts on cleaning up the ScroogeGen task
which now can be simplified a good deal.  The intent is
just to remove the possibility of folks binding to this capability
going forward.

Testing Done:
$ PANTS_DEV=1 ./pants goal test {src,tests}/{java,scala}::
$ PANTS_DEV=1 ./pants goal test tests/python/pants_test/:all
$ PANTS_DEV=1 ./pants goal test tests/python/pants_test/:integration

Bugs closed: 288, 289

Reviewed at https://rbcommons.com/s/twitter/r/603/
Raise an exception when the package is missing a template

pkgutils.get_data returns None when it fails to find the requested file. Prior to this change, we would just pass the None on to pystache to render and it would give a confusing message about not being able to find "none_type.mustache". This change checks for the None and raises an Error with the name of the missing template.

I didn't see an obvious existing Error type to use, so I created a new one. Let me know if there's an existing Error I should be using instead.

Testing Done:
ran unittests.

Reviewed at https://rbcommons.com/s/twitter/r/608/
Update docstring of AndroidBinary target

tested at  https://travis-ci.org/pantsbuild/pants/builds/28688902

reviewed at https://rbcommons.com/s/twitter/r/610/
Improve inheritance and kwargs passing between android_binary and target.

Remove unused params from AndroidBinary __init__, as referencing them
would prevent those kwargs from being passed in the super() call.

Referenced from lahosken @ https://rbcommons.com/s/twitter/r/610

This patch is dependent on that CR, review #610

Testing Done:
ci.sh passed.
Travis passed: https://travis-ci.org/pantsbuild/pants/builds/28694692

Reviewed at https://rbcommons.com/s/twitter/r/614/
start doc'ing py goals. set up python_old for old UI, start refactoring python-readme.

Python supports 'goal test' and 'goal binary' now. python-readme should reflect that.
This change doesn't 100% fix that; but it's a step in the right direction.

Twitter's biggest python-Pants codebase is on an old Pants.
(also, a couple of open-source Pants-built repos)
This makes me hesitate to blow away the old doc just yet.

This change...

  copies python-readme to python_old.

  adds paragraph at top of python-readme "Are your Pants old? Then see python_old"

  rm'd some historical python-readme parts now less-obsoletely doc'd on other pages

  mv'd some PEX-design-background material onto a new page pex_design.
  (It might make more sense for me to mv this to twitter/commons?
   I dunno. If you tell me to move it to twitter/commons, I will.)

  mv'd some python-readme parts to other pages

  Added a couple of TODO github issue links for biggish problems noticed
  along the way.

I'm not trying to get python-readme perfect in this pass.
This is mostly moving material around.

That said, a couple of examples now use 'goal'. YAY

Testing Done:
./build-support/bin/publish_docs.sh # stuff looks OK

./build-support/bin/ci.sh # docstring changes apparently nonfatal

Bugs closed: 258

Reviewed at https://rbcommons.com/s/twitter/r/594/