Enable passthru args for python run.

Also enabled an explicit flag for this, which we shockingly didn't have before somehow.

Made some changes to example code to allow manual testing of this. Try:

./pants goal run.py --args="benjy"  examples/src/python/example/hello/main/ -- zundel jsirois

Changed the python run task's registration name from 'python-run' to 'py',  so that the cmd-line
scope is run.py instead of run.python_run.

Also added some missing __init__.py files.

Reviewed at https://rbcommons.com/s/twitter/r/1321/
Allow more succinct invocations with passthru args.

If passthru args "belong" to a goal scope then all tasks in that goal get them.

E.g., you can do:

pants goal run <python-target> -- passthru args

and the run.py task will get the passthru args. So will run.jvm, but that's OK
because it's a no-op for <python-target>.

This obviously is only good when in practice only one passthru-arg-supporting
task in that goal is not a no-op. However that is the common case.

Reviewed at https://rbcommons.com/s/twitter/r/1323/
Get rid of the 'build' and 'py' commands.

Only one more non-goal command to go!

Testing Done:
CI passed.

Reviewed at https://rbcommons.com/s/twitter/r/1327/
Don't highlight 'goal eclipse' in the first Pants Tutorial

In hindsight, the ide-gen goals are not so exemplary.

Bugs closed: 708

Reviewed at https://rbcommons.com/s/twitter/r/1315/
Add java_sources to the depmap-info.

More context:
Ity added this diff long long ago https://rbcommons.com/s/twitter/r/733 to include java_sources in self.context which had some performance issues.

I went ahead and removed it https://rbcommons.com/s/twitter/r/1130.
We relied on java_sources to be already present in self.targets for depmap goal.

This diff fixes the issue where for a scala target owning java_sources, the java_target is included in depmap output under targets found.

Without this patch, the java_sources is mising from targets but is present in targets['examples/src/scala/com/pants/example/scala_with_java_sources:scala_with_java_source']['targets']

[tw-172-25-146-131 pants]$ ./pants goal depmap --depmap-project-info examples/src/scala/com/pants/example/scala_with_java_sources:
*** Running pants in dev mode from /Users/tdesai/projects/pants/src/python/pants/bin/pants_exe.py ***
{
    "libraries": {},
    "targets": {
        "examples/src/scala/com/pants/example/scala_with_java_sources:scala_with_java_sources": {
            "libraries": [
                "org.scala-lang:scala-library:2.9.3"
            ],
            "is_code_gen": false,
            "targets": [
                ":scala-library-2.9.3",
                "examples/src/java/com/pants/examples/java_sources:java_sources"
            ],
            "roots": [
                {
                    "source_root":   "/Users/tdesai/projects/pants/examples/src/scala/com/pants/example/scala_with_java_sources",
                    "package_prefix": "com.pants.example.scala_with_java_sources"
                }
            ],
            "target_type": "SOURCE"
        },

        "examples/src/java/com/pants/examples/java_sources:java_sources": {
            "libraries": [],
            "is_code_gen": false,
            "targets": [],
            "roots": [
                {
                    "source_root": "/Users/tdesai/projects/pants/examples/src/java/com/pants/examples/java_sources",
                    "package_prefix": "com.pants.examples.java_sources"
                }
            ],
            "target_type": "SOURCE"
        },
        ":scala-library-2.9.3": {
            "libraries": [],
            "is_code_gen": false,
            "targets": [],
            "roots": [],
            "target_type": "SOURCE"
        }
    }
}

Testing Done:
yes.

./pants tests/python/pants_test/tasks:depmap_integration

and
./pants tests/python/pants_test/tasks:depmap

pass

CI:
https://travis-ci.org/pantsbuild/pants/builds/40829185

Reviewed at https://rbcommons.com/s/twitter/r/1324/
publish docs the new way

mv publish_docs_new.sh publish_docs.sh ; rm src/python/pants/docs/*.rst , other Sphinx things

the parts I only halfway-understood, listed in order of rising scariness:

Did some clean-up in our scripts:
rm'd sphinx things from 3rdparty/python/dev_only_requirements.txt . which was all of it so...
rm'd 3rdparty/python/dev_only_requirements.txt
rm'd DEV_REQUIREMENTS from pants_venv
rm'd reference to DEV_REQUIREMENTS from ./pants , use REQUIREMENTS instead.

There might have been other cleanup possible. Backed off from some possible
changes that got into gnarly territory.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/40938680

./build-support/bin/publish_docs.sh -opd sitegen-test

for the ./pants change:
export WRAPPER_REQUIREMENTS="tmp_requirements.txt" # file with one line: scrapy
in some temporarily-added pants code, from scrapy import Spider, Item, Field
worked OK. (and correctly errored out in another shell that didn't have WRAPPER_REQUIREMENTS set)

Reviewed at https://rbcommons.com/s/twitter/r/1292/
Remove legacy flag support from new options.

We no longer need it now that we're on the new parser.

Removed a couple of legacy= kwargs, to prove that things still work.
Future changes will clean up the remaining kwargs from option registrations.

Testing Done:
Integration and unit tests passed.

Full CI baking.

Reviewed at https://rbcommons.com/s/twitter/r/1328/
Turn dependency dupes into errors

We see a bunch of dupes pop up in dependency lists in our internal repos. Often they go unnoticed by whoever should be fixing them. They also make each parsing of the build graph quite noisy for the rest of our users.

This branch turns those warnings into errors. That ought to ensure they're caught early by whoever added the duplicate dependency and they won't hit master.

Testing Done:
ci.sh passes
tested against a dependency list with dupes

Reviewed at https://rbcommons.com/s/twitter/r/1332/
Clean pants script which used "build"

Clean pants script which referred to "build"

[tw-172-25-146-131 pants]$ UNSET PANTS_DEV
[tw-172-25-146-131 pants]$ build-support/python/clean.sh && ./pants
Building pants.pex to /Users/tdesai/projects/pants/pants.pex...
Pants build missing a command: None

*** Pants bootstrapping failed.
[tw-172-25-146-131 pants]$

Testing Done:
yes
./pants now works.

I can add this in CI if required.

Reviewed at https://rbcommons.com/s/twitter/r/1335/
update Pants+IntelliJ docs to say the Plugin's the new hotness, link to plugin's README

Reviewed at https://rbcommons.com/s/twitter/r/891/
Fix a nasty bug when injecting target closures in BuildGraph

Just because a target has been injected doesn't mean that its
closure (and therefore its dependency edges) have been injected.
The current code misses the case where a target was injected
(but not its closure) and then has its closure injected later.

Testing Done:
Added a test that identifies the bug, which now passes (doesn't on master):
tests/python/pants_test/graph test_inject_then_inject_closure.

CI is green: https://travis-ci.org/pantsbuild/pants/builds/41270498

Reviewed at https://rbcommons.com/s/twitter/r/1337/
Fix java_sources missing dep detection.

This seems to just be an oversight in the original implementation.
We noticed incorrect missing dependency errors (we have it set to
error rather than warn) when we started using java_sources in our codegen.
This resolved our issue internally.

The general idea is that a ScalaLibrary should be claiming its
java_sources sources for the purpose of mapping generated classfiles
back to the sources zinc used to generate them.

Testing Done:
CI is green: https://travis-ci.org/pantsbuild/pants/builds/41043159

Reviewed at https://rbcommons.com/s/twitter/r/1336/
Kill all nailgun tasks after an integration test run to keep from leaking NGServer instances.

Before this change, Travis-CI builds are failing with: Cannot create GC thread. Out of system resources.
For example:  https://travis-ci.org/pantsbuild/pants/jobs/40841557

Testing Done:
CI running at: https://travis-ci.org/pantsbuild/pants/builds/41295684

Manual test: while running devpants goal test ./tests/python/pants_test:integration

before:
ps auxww | grep NGServer | wc -l
143

after:
ps auxww | grep NGServer | wc -l
13

Bugs closed: 799

Reviewed at https://rbcommons.com/s/twitter/r/1345/
Remove legacy option dests from all non-jvm-y tasks.

Testing Done:
CI passed.

Reviewed at https://rbcommons.com/s/twitter/r/1341/
Remove legacy dests from jvm task options.

Testing Done:
CI passed.

Reviewed at https://rbcommons.com/s/twitter/r/1343/
Remove support for legacy flag dests in new options.

We no longer use it anywhere.

Testing Done:
CI passes.

Reviewed at https://rbcommons.com/s/twitter/r/1351/
doc site cross references: call them pantsmark, pantsref instead of xmark, xref

<a xmark="foo"> looks like it could be some obscure HTML thing. Switch to <a pantsmark="foo"> which is more obviously a pants thing.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/41306457

Reviewed at https://rbcommons.com/s/twitter/r/1348/
Use awk instead of sed to filter ever Nth line out of the integration test output The sed -n "K~N" syntax is not compatible with sed on MacOS.

Testing Done:
compared the list of integration tests listed when modifying the -i N:M parameters to the script.
Here is a good CI run https://travis-ci.org/pantsbuild/pants/builds/41359083

Bugs closed: 800

Reviewed at https://rbcommons.com/s/twitter/r/1350/
fix string formatting so 'test -h' does not crash

Testing Done:
PANTS_DEV=1 ./pants goal test -h

doesn't crash, unlike

./pants goal test -h

https://travis-ci.org/pantsbuild/pants/builds/41402745

Bugs closed: 805

Reviewed at https://rbcommons.com/s/twitter/r/1353/
Add dummy readme in examples/src/../scala/hello for Intellij plugin integration tests.

We added capability to the pants plugin to index non target owning files e.g. deploy scripts or READMEs
here https://rbcommons.com/s/twitter/r/1338/

We added the for the plugins.
Integration tests for Intllij plugin

Testing Done:
yes.
https://github.com/pantsbuild/pants/commit/d0fc5d05e8fae14311bab4cb7e664fa23fb02a8c

Reviewed at https://rbcommons.com/s/twitter/r/1356/