.rst files -> .md files.

re-play https://rbcommons.com/s/twitter/r/1235/ , "convert more .rst files to .md", but less crash-y. Also pull in some excellent feedback and update with doc changes that went into the .rst.

The crashy change converted many rst docs to Pants-enhanced markdown (yay) with BUILD targets (yay) with circular deps which 'goal markdown' is OK with (hmm) but which 'goal test' balks at, breaking CI (d'oh). So I reverted that change.

*This* change is like *that* change, but

* Uses the new `links` BUILD param instead of `dependencies`. Yay for no CI breaks.
* Pulls in an additition the excellent AReitz made to JVMProjects.rst meanwhile.
* Includes feedback from same excellent AReitz.
* pulls in .rst changes that came in at https://rbcommons.com/s/twitter/r/1279/

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/40211471

http://pantsbuild.github.io/sitegen-test/

Reviewed at https://rbcommons.com/s/twitter/r/1263/
Split up the travis run into a unit-test shard and multiple IT shards.

Also move tests/python/pants_test/tasks/jvm_compile/java:test_java_compile_integration
out of the unit test suite.

Testing Done:
CI went green here: https://travis-ci.org/pantsbuild/pants/builds/40216412

Bugs closed: 773

Reviewed at https://rbcommons.com/s/twitter/r/1290/
what_changed: the build graph is sorted twice even though the first iteration does not require the targets to be sorted. Removing that sort reduces the call time by about a third in a non scientific test.

Testing Done:
ci.sh passes locally

Reviewed at https://rbcommons.com/s/twitter/r/1291/
Switch all remaining accesses of 'context.options' to new options.

Killed IvyUtils's dependency on options. It only needed it for
two options that no one is using in practice and this made things
much much simpler.

Also made some methods in IvyUtil classmethods, so that call sites
don't need to instantiate it and pass it config.

Testing Done:
All task unittests passed.

CI passed.

Reviewed at https://rbcommons.com/s/twitter/r/1289/
setup_repo: now only 98% impossible

This page was full of doom and gloom: Want to start using Pants? Might as well just give up.

Walk back some of that. Point out some files that you need to copy+adjust.

This probably still doesn't capture everything folks need, but it could be a step in the right direction.

rm the howto_customize page. For a while, I thought such a page would be useful... but its material all falls into setup_repo and howto_plugin.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/40251344

Bugs closed: 632, 726, 776

Reviewed at https://rbcommons.com/s/twitter/r/1295/
Remove a bunch of setup_parser plumbing from goal/task.

reflect.py relied on the old options system via setup_parser, so
it's been broken for a while, ever since options migration began.
Fixed that by making it read from the new options.

Testing Done:
Manually ran the doc generator and inspected the resulting docs.

CI passed.

Reviewed at https://rbcommons.com/s/twitter/r/1297/
Remove some stray calls to setup_parser.

One is in help.py which is completely unused, so now deleted.

The other was in the new help printer.

Testing Done:
Verified that --help works.

CI passed.

Reviewed at https://rbcommons.com/s/twitter/r/1300/
docs: mention mvn2pants

while I was in there: setup_repo had a paragraph about top-dir BUILD files for config. LOL, nowadays, that config has moved into plugins. So pull that out, and re-jigger examples of BUILD.foo files.

Reviewed at https://rbcommons.com/s/twitter/r/1301/
Remove a superfluous test.

It only tests that Goal.parse_args works, but that function is
about to be killed itself.

Reviewed at https://rbcommons.com/s/twitter/r/1304/
Move cache logging to debug

Can get a tad noisy with several tasks all using setting up multiple caches.

Testing Done: https://travis-ci.org/pantsbuild/pants/builds/40392152

Bugs closed: 783

Reviewed at https://rbcommons.com/s/twitter/r/1310/
make it clear: you can only download those pexes because you previously uploaded 'em.

they don't appear by magic.

Reviewed at https://rbcommons.com/s/twitter/r/1296/
Ensure that the new options system reports goals in the right order.

We sometimes rely on this, e.g., when specifying "clean-all compile".

Testing Done:
option unittests pass. CI passed when run with https://rbcommons.com/s/twitter/r/1309/

Reviewed at https://rbcommons.com/s/twitter/r/1308/
Switch option parsing for goals to the new parser.

Relies on https://rbcommons.com/s/twitter/r/1308/.

Testing Done:
CI passed when run with https://rbcommons.com/s/twitter/r/1308/.

Reviewed at https://rbcommons.com/s/twitter/r/1309/
Add support for the Wire protobuf library.  Adds the java_wire_library() target.

https://github.com/square/wire

Testing Done:
Added an example, plus new unit and integration tests.
CI is running @ https://travis-ci.org/pantsbuild/pants/jobs/39988400

Bugs closed: 765

Reviewed at https://rbcommons.com/s/twitter/r/1275/
Remove some remaining old options references.

Testing Done:
CI passed. Manual testing of cmd-line flags.

Reviewed at https://rbcommons.com/s/twitter/r/1317/
docs: confluence-publish's wiki info lives in a plugin now.

also, give a sample ConfluencePublish goal.

Bugs closed: 723

Reviewed at https://rbcommons.com/s/twitter/r/1299/
doc: update import path for PEX

"I don't see any code still importing twitter.common.python pex (except for examples in rst files,which should probably be fixed)"

Bugs closed: 428

Reviewed at https://rbcommons.com/s/twitter/r/1298/
switch src/* usages of Config.load to use Config.from_cache instead

Now all usages of Config.load in src use a cache if the config already has been parsed.

This doesn't change test usages, but since tests use temp directories, tests with different ini files don't interfer

Testing Done:
ran ci.sh locally

travis ci in progress: https://travis-ci.org/pantsbuild/pants/builds/40687043

Bugs closed: 788

Reviewed at https://rbcommons.com/s/twitter/r/1319/
- Create VIRTUALENV_PACKAGE_LOCATION env var so we can customize virtualenv package download
- Refactor to have a common.sh containing the set_archflags and logging functionalities.
- Make the virtual env script a bit more robust against curl/tar/activate errors

Testing Done:
manually tested:
1) not specifying VIRTUALENV_PACKAGE_LOCATION,
2) specifying a valid custom VIRTUALENV_PACKAGE_LOCATION location, and
3) specifying an invalid VIRTUALENV_PACKAGE_LOCATION location.

Travis: https://travis-ci.org/jinfeng/jinfeng-pants-fork/builds/40690744

Reviewed at https://rbcommons.com/s/twitter/r/1316/
Skip compressing/writing artifacts that are already in the cache.

Compressing things is expensive, so no reason clients should do it if they don't have to and writing the same thing over and over wastes IO capacity on the cache server.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/39729173 passed

Bugs closed: 760

Reviewed at https://rbcommons.com/s/twitter/r/1272/