make it clear: you can only download those pexes because you previously uploaded 'em.

they don't appear by magic.

Reviewed at https://rbcommons.com/s/twitter/r/1296/
Ensure that the new options system reports goals in the right order.

We sometimes rely on this, e.g., when specifying "clean-all compile".

Testing Done:
option unittests pass. CI passed when run with https://rbcommons.com/s/twitter/r/1309/

Reviewed at https://rbcommons.com/s/twitter/r/1308/
Switch option parsing for goals to the new parser.

Relies on https://rbcommons.com/s/twitter/r/1308/.

Testing Done:
CI passed when run with https://rbcommons.com/s/twitter/r/1308/.

Reviewed at https://rbcommons.com/s/twitter/r/1309/
Add support for the Wire protobuf library.  Adds the java_wire_library() target.

https://github.com/square/wire

Testing Done:
Added an example, plus new unit and integration tests.
CI is running @ https://travis-ci.org/pantsbuild/pants/jobs/39988400

Bugs closed: 765

Reviewed at https://rbcommons.com/s/twitter/r/1275/
Remove some remaining old options references.

Testing Done:
CI passed. Manual testing of cmd-line flags.

Reviewed at https://rbcommons.com/s/twitter/r/1317/
docs: confluence-publish's wiki info lives in a plugin now.

also, give a sample ConfluencePublish goal.

Bugs closed: 723

Reviewed at https://rbcommons.com/s/twitter/r/1299/
doc: update import path for PEX

"I don't see any code still importing twitter.common.python pex (except for examples in rst files,which should probably be fixed)"

Bugs closed: 428

Reviewed at https://rbcommons.com/s/twitter/r/1298/
switch src/* usages of Config.load to use Config.from_cache instead

Now all usages of Config.load in src use a cache if the config already has been parsed.

This doesn't change test usages, but since tests use temp directories, tests with different ini files don't interfer

Testing Done:
ran ci.sh locally

travis ci in progress: https://travis-ci.org/pantsbuild/pants/builds/40687043

Bugs closed: 788

Reviewed at https://rbcommons.com/s/twitter/r/1319/
- Create VIRTUALENV_PACKAGE_LOCATION env var so we can customize virtualenv package download
- Refactor to have a common.sh containing the set_archflags and logging functionalities.
- Make the virtual env script a bit more robust against curl/tar/activate errors

Testing Done:
manually tested:
1) not specifying VIRTUALENV_PACKAGE_LOCATION,
2) specifying a valid custom VIRTUALENV_PACKAGE_LOCATION location, and
3) specifying an invalid VIRTUALENV_PACKAGE_LOCATION location.

Travis: https://travis-ci.org/jinfeng/jinfeng-pants-fork/builds/40690744

Reviewed at https://rbcommons.com/s/twitter/r/1316/
Skip compressing/writing artifacts that are already in the cache.

Compressing things is expensive, so no reason clients should do it if they don't have to and writing the same thing over and over wastes IO capacity on the cache server.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/39729173 passed

Bugs closed: 760

Reviewed at https://rbcommons.com/s/twitter/r/1272/
Support passthru args in tasks.

Tasks must declare that they take them.

The last task explicitly mentioned on the cmd-line gets them.

Implemented this for pytest.

Testing Done:
Verified that this does what you expect:

./pants goal test.pytest compile tests/python/pants_test/option/ -- -s

CI passes.

Reviewed at https://rbcommons.com/s/twitter/r/1320/
update the "Pants Essentials" tech talk slides

Nothing changed much; everything changed some

Testing Done:
eyeball

Reviewed at https://rbcommons.com/s/twitter/r/1314/
Merge branch 'master' into scala_howto
Enable passthru args for python run.

Also enabled an explicit flag for this, which we shockingly didn't have before somehow.

Made some changes to example code to allow manual testing of this. Try:

./pants goal run.py --args="benjy"  examples/src/python/example/hello/main/ -- zundel jsirois

Changed the python run task's registration name from 'python-run' to 'py',  so that the cmd-line
scope is run.py instead of run.python_run.

Also added some missing __init__.py files.

Reviewed at https://rbcommons.com/s/twitter/r/1321/
Allow more succinct invocations with passthru args.

If passthru args "belong" to a goal scope then all tasks in that goal get them.

E.g., you can do:

pants goal run <python-target> -- passthru args

and the run.py task will get the passthru args. So will run.jvm, but that's OK
because it's a no-op for <python-target>.

This obviously is only good when in practice only one passthru-arg-supporting
task in that goal is not a no-op. However that is the common case.

Reviewed at https://rbcommons.com/s/twitter/r/1323/
Get rid of the 'build' and 'py' commands.

Only one more non-goal command to go!

Testing Done:
CI passed.

Reviewed at https://rbcommons.com/s/twitter/r/1327/
Don't highlight 'goal eclipse' in the first Pants Tutorial

In hindsight, the ide-gen goals are not so exemplary.

Bugs closed: 708

Reviewed at https://rbcommons.com/s/twitter/r/1315/
Add java_sources to the depmap-info.

More context:
Ity added this diff long long ago https://rbcommons.com/s/twitter/r/733 to include java_sources in self.context which had some performance issues.

I went ahead and removed it https://rbcommons.com/s/twitter/r/1130.
We relied on java_sources to be already present in self.targets for depmap goal.

This diff fixes the issue where for a scala target owning java_sources, the java_target is included in depmap output under targets found.

Without this patch, the java_sources is mising from targets but is present in targets['examples/src/scala/com/pants/example/scala_with_java_sources:scala_with_java_source']['targets']

[tw-172-25-146-131 pants]$ ./pants goal depmap --depmap-project-info examples/src/scala/com/pants/example/scala_with_java_sources:
*** Running pants in dev mode from /Users/tdesai/projects/pants/src/python/pants/bin/pants_exe.py ***
{
    "libraries": {},
    "targets": {
        "examples/src/scala/com/pants/example/scala_with_java_sources:scala_with_java_sources": {
            "libraries": [
                "org.scala-lang:scala-library:2.9.3"
            ],
            "is_code_gen": false,
            "targets": [
                ":scala-library-2.9.3",
                "examples/src/java/com/pants/examples/java_sources:java_sources"
            ],
            "roots": [
                {
                    "source_root":   "/Users/tdesai/projects/pants/examples/src/scala/com/pants/example/scala_with_java_sources",
                    "package_prefix": "com.pants.example.scala_with_java_sources"
                }
            ],
            "target_type": "SOURCE"
        },

        "examples/src/java/com/pants/examples/java_sources:java_sources": {
            "libraries": [],
            "is_code_gen": false,
            "targets": [],
            "roots": [
                {
                    "source_root": "/Users/tdesai/projects/pants/examples/src/java/com/pants/examples/java_sources",
                    "package_prefix": "com.pants.examples.java_sources"
                }
            ],
            "target_type": "SOURCE"
        },
        ":scala-library-2.9.3": {
            "libraries": [],
            "is_code_gen": false,
            "targets": [],
            "roots": [],
            "target_type": "SOURCE"
        }
    }
}

Testing Done:
yes.

./pants tests/python/pants_test/tasks:depmap_integration

and
./pants tests/python/pants_test/tasks:depmap

pass

CI:
https://travis-ci.org/pantsbuild/pants/builds/40829185

Reviewed at https://rbcommons.com/s/twitter/r/1324/
publish docs the new way

mv publish_docs_new.sh publish_docs.sh ; rm src/python/pants/docs/*.rst , other Sphinx things

the parts I only halfway-understood, listed in order of rising scariness:

Did some clean-up in our scripts:
rm'd sphinx things from 3rdparty/python/dev_only_requirements.txt . which was all of it so...
rm'd 3rdparty/python/dev_only_requirements.txt
rm'd DEV_REQUIREMENTS from pants_venv
rm'd reference to DEV_REQUIREMENTS from ./pants , use REQUIREMENTS instead.

There might have been other cleanup possible. Backed off from some possible
changes that got into gnarly territory.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/40938680

./build-support/bin/publish_docs.sh -opd sitegen-test

for the ./pants change:
export WRAPPER_REQUIREMENTS="tmp_requirements.txt" # file with one line: scrapy
in some temporarily-added pants code, from scrapy import Spider, Item, Field
worked OK. (and correctly errored out in another shell that didn't have WRAPPER_REQUIREMENTS set)

Reviewed at https://rbcommons.com/s/twitter/r/1292/
Remove legacy flag support from new options.

We no longer need it now that we're on the new parser.

Removed a couple of legacy= kwargs, to prove that things still work.
Future changes will clean up the remaining kwargs from option registrations.

Testing Done:
Integration and unit tests passed.

Full CI baking.

Reviewed at https://rbcommons.com/s/twitter/r/1328/