Switch from os.rename to shutil.move to support cross-fs renames when needed.

In each of these cases dst is a file or a new dir.  In these 2 cases the ouput of os.rename and shutil.move are identical (they are only different when dst is an existing dir).

Testing Done:
CI went green here: https://travis-ci.org/pantsbuild/pants/builds/38112860

Bugs closed: 671

Reviewed at https://rbcommons.com/s/twitter/r/1157/
A base class for tests that test a single task type.

Subclasses advertise the task type they care about in setupClass().

Deals with option registration and scoping for the task under test.

Also addresses some small code review comments from a previous commit.

Testing Done:
ci.sh passed.

Reviewed at https://rbcommons.com/s/twitter/r/1168/
Fix a test.

Should have been in the previous commit.
Move workdirs creation from __init__ to pre_execute in jvm_compile & Remove QuietTaskMixin from several tasks.

- jvm_compile creates a bunch of dirs in its __init__ method which gets wiped out by clean-all. Instead, move the dirs creation to pre_execute().
- Another issue is clean-all/clean-all-async/invalidate/ng-killall are with QuietTaskMixin, and once there is one task is QuietTaskMixin, console output will be all suppressed by reporter. This is a fairly common use of combining these goals with others like compile/publish/test/etc.

This is because we only have one reporter for all tasks. A long term solution could be that we have two reporters (or more), one for quiet or one verbose. And tasks can each use its own reporter (not sure if there is any output lineage problem if combined).

Or we can simply make these 3 tasks non quiet. Rest of the ConsoleTask(QuietTaskMixin) derived goals/tasks, their usage is generally standalone. Thus no need for changes.

Testing Done:
PANTS_DEV=1 ./pants goal test tests::

Bugs closed: 325

Reviewed at https://rbcommons.com/s/twitter/r/1173/
Add PantsRunIntegrationTest into pantsbuild.pants.testinfra package.

Testing Done:
ran `build-support/bin/release.sh -n` and verified the inclusion.

Bugs closed: 696

Reviewed at https://rbcommons.com/s/twitter/r/1185/
doc: dust off the Task Developer's Guide :-) rm the "pagerank" example

Our page about how to develop a Task described "science-era" Pants.

Removed a lot of what's there; it talked about APIs that have since become new and improved.

Removed the "pagerank" example. :-( When I started on this change, I thought I was gonna update this example and maybe move it to examples/src/... so it would get exercised and not drift back into b0rken-ness . But it's kind of a weird example--it doesn't really care about products, it doesn't do "the usual" things with deps. So I find myself deleting it... but with some misgivings, because it's cool. If someone wants to do the work to modernize it and move it to examples, I'll happily code review :-)

Replaced copy-pasted code snippets with "includes" from live tasks.
Maybe this won't drift out of date so easily from now on.
(Didn't add includes for the JVM section, just a TODO. But did remove now-obsolete sample code.)

rm'd some redundant-and-obsolete info from the Internals page, replacing w/link to Task Developer's guide. (Thanks to Mateo for spotting this!) The basic concepts were OK, but the details, yikes. Internals page had info about GroupTask, which Task Developers page lacked... so move it over.

Bugs closed: 433

Reviewed at https://rbcommons.com/s/twitter/r/1074/
Use a Session to enable connection pooling

Requests and urllib3 automatically reuse connections if using a Session, which otherwise has all the same methods as the main Requests API, making this a drop-in replacement: http://docs.python-requests.org/en/latest/user/advanced/

In my tests, this change cut 830 HEAD requests to Foursquare's internal artifact cache from 22s to 11s.

Testing Done:
Using 830 cache keys copied from a real build, this change cut the time to do HEAD requests for each from 22s to 11s.

https://travis-ci.org/pantsbuild/pants/builds/38515947

Reviewed at https://rbcommons.com/s/twitter/r/981/
Move requests-logging setting to global scope

This should happen at import, since the __init__ may have been run in a different proc, and the cache pickled, since then.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/38522554

Reviewed at https://rbcommons.com/s/twitter/r/1022/
The traversal order used is preorder, not inorder; support postorder traversal

1. The term "inorder traversal" does not make sense outside of the context of binary trees.  Instead, use the term "preorder", which is actually what is done.

2. Also allow for postorder traversal (and test that it works).

Postorder stuff is needed for https://rbcommons.com/s/twitter/r/1179/

Testing Done:
./pants goal test tests/python/pants_test/graph:graph

Reviewed at https://rbcommons.com/s/twitter/r/1177/
Handle list-typed options (action='append') correctly.

Ensure that we read them from config in the right way, and apply type
conversion to the list members.

Added test cases for this.

Testing Done:
Ran the options unittests before and after the fix, to verify
that the test discriminates this.

CI passed.

Reviewed at https://rbcommons.com/s/twitter/r/1183/
convert and move about 1/2 of our docs from .rst to .md

This change starts converting our docs from sphinx-powered-rst to Pants-enhanced-md, moving files to some different directories:

src/docs/*.md  <= language-agnostic Pants-user docs

examples/src/java/com/pants/examples/*.md  <= Pants-with-Java docs

examples/src/python/example/*.md  <= Pants-with-py docs

src/python/pants/docs  <= good place for pants-dev docs

Keeping foolang docs with examples/src/foolang feels right; if we change the examples, we probably want to change the docs, too. Less-certain about where language-agnostic docs go.

Testing Done:
eyeball-compare pages gen'd from .rst, .md

Reviewed at https://rbcommons.com/s/twitter/r/1176/
'goal sitegen' tweaks inspired by eyeballing more Pants doc pages

Don't insert pilcrow anchor in the heading; instead, lay it nearby with a table. Otherwise, it looks funny if the line breaks right there.

Add a "Generated by publish_docs dist/markdown/html/README.html 2014-10-17T11\:16:07.253750" so that folks who are looking at a doc from two years ago have an inkling of the danger. And/or if they want to fix a mistake, they have a clue where the content came from.

Widened margin around the page-level table of contents.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/38290410

Reviewed at https://rbcommons.com/s/twitter/r/1178/
Make CacheKey hash and pickle friendly

Don't need payloads on cache_keys, so removing them makes them pickle-friendly.

Testing Done:
tests/python/pants_test/cache::

Passed: https://travis-ci.org/pantsbuild/pants/builds/38538962

Reviewed at https://rbcommons.com/s/twitter/r/986/
Add test for CombinedArtifactCache's behavior

Makes sure that remote + local correctly "backfills"
local when fetching from remote.

This is just documenting/asserting current behavior.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/38635762

Reviewed at https://rbcommons.com/s/twitter/r/982/
Add support for running a command before tests.

This depends on https://rbcommons.com/s/twitter/r/1177/

Testing Done:
Ran new test.

Reviewed at https://rbcommons.com/s/twitter/r/1179/
Fix problem parsing stuff like options: --compile-java-args='-target 7 -source 7' in pants.rc files

Testing Done:
Ran new tests

Reviewed at https://rbcommons.com/s/twitter/r/1192/
Migrate all JvmTasks to use the new options system.

The common --jvm-options, --args, --debug and --confs flags now
live in the base class, but this required migrating all the
relevant tasks at once, hence the scope of this change.

This also gets rid of manual flag-to-config fallback, as the new
options system does this automatically. However this required some
changes to section and key names in pants.ini. I'm documenting
these changes in a spreadsheet, for smooth config migration in
pants install bases.

This required breaking a couple of inconsistently named flags, e.g.,
--benchmark-caliper-args is now just --benchmark-args.

Testing Done:
CI passed.

Manually tested that new flags and config settings work.

Reviewed at https://rbcommons.com/s/twitter/r/1186/
A simple script to detect unmigrated config keys.

Future changes that modify config keys should add the migration
to this script.

Reviewed at https://rbcommons.com/s/twitter/r/1195/
Provides documentation for some of the properties and methods in target.py

The type annotations help when using an IDE like IntelliJ

Testing Done:
There should be no code changes.  running CI

Bugs closed: 707

Reviewed at https://rbcommons.com/s/twitter/r/1196/
Print a diagnostic if a BUILD file references a source file that does not exist

Testing Done:
Edited a BUILD file, added a reference to a source file that does not exist.

```
diff --git a/src/python/pants/backend/core/BUILD b/src/python/pants/backend/core/BUILD
index 2da9682..8b1e827 100644
--- a/src/python/pants/backend/core/BUILD
+++ b/src/python/pants/backend/core/BUILD
@@ -4,7 +4,7 @@

 python_library(
   name = 'plugin',
-  sources = ['register.py'],
+  sources = ['register.py', 'does-not-exist.py'],
   dependencies = [
     'src/python/pants/backend/core/targets:all',
     'src/python/pants/backend/core/tasks:all',

```

Before this change, when running

```
PANTS_DEV=1 goal test tests/python/pants_test:all
```

the build would end with an error but not give any further information:

```
Exception message: [Errno 2] No such file or directory
```

After this change, you now get a diagnostic message that gives you a hint why the test failed:

```
 ERROR] Failed to copy src/python/pants/backend/core/does-not-exist.py for library PythonLibrary(BuildFileAddress(/Users/zundel/Src/pants/src/python/pants/backend/core/BUILD, plugin))

               FAILURE

Exception message: [Errno 2] No such file or directory
```

Reviewed at https://rbcommons.com/s/twitter/r/1198/