Switch from os.rename to shutil.move to support cross-fs renames when needed.

In each of these cases dst is a file or a new dir.  In these 2 cases the ouput of os.rename and shutil.move are identical (they are only different when dst is an existing dir).

Testing Done:
CI went green here: https://travis-ci.org/pantsbuild/pants/builds/38112860

Bugs closed: 671

Reviewed at https://rbcommons.com/s/twitter/r/1157/
A base class for tests that test a single task type.

Subclasses advertise the task type they care about in setupClass().

Deals with option registration and scoping for the task under test.

Also addresses some small code review comments from a previous commit.

Testing Done:
ci.sh passed.

Reviewed at https://rbcommons.com/s/twitter/r/1168/
Fix a test.

Should have been in the previous commit.
Move workdirs creation from __init__ to pre_execute in jvm_compile & Remove QuietTaskMixin from several tasks.

- jvm_compile creates a bunch of dirs in its __init__ method which gets wiped out by clean-all. Instead, move the dirs creation to pre_execute().
- Another issue is clean-all/clean-all-async/invalidate/ng-killall are with QuietTaskMixin, and once there is one task is QuietTaskMixin, console output will be all suppressed by reporter. This is a fairly common use of combining these goals with others like compile/publish/test/etc.

This is because we only have one reporter for all tasks. A long term solution could be that we have two reporters (or more), one for quiet or one verbose. And tasks can each use its own reporter (not sure if there is any output lineage problem if combined).

Or we can simply make these 3 tasks non quiet. Rest of the ConsoleTask(QuietTaskMixin) derived goals/tasks, their usage is generally standalone. Thus no need for changes.

Testing Done:
PANTS_DEV=1 ./pants goal test tests::

Bugs closed: 325

Reviewed at https://rbcommons.com/s/twitter/r/1173/
Add PantsRunIntegrationTest into pantsbuild.pants.testinfra package.

Testing Done:
ran `build-support/bin/release.sh -n` and verified the inclusion.

Bugs closed: 696

Reviewed at https://rbcommons.com/s/twitter/r/1185/
doc: dust off the Task Developer's Guide :-) rm the "pagerank" example

Our page about how to develop a Task described "science-era" Pants.

Removed a lot of what's there; it talked about APIs that have since become new and improved.

Removed the "pagerank" example. :-( When I started on this change, I thought I was gonna update this example and maybe move it to examples/src/... so it would get exercised and not drift back into b0rken-ness . But it's kind of a weird example--it doesn't really care about products, it doesn't do "the usual" things with deps. So I find myself deleting it... but with some misgivings, because it's cool. If someone wants to do the work to modernize it and move it to examples, I'll happily code review :-)

Replaced copy-pasted code snippets with "includes" from live tasks.
Maybe this won't drift out of date so easily from now on.
(Didn't add includes for the JVM section, just a TODO. But did remove now-obsolete sample code.)

rm'd some redundant-and-obsolete info from the Internals page, replacing w/link to Task Developer's guide. (Thanks to Mateo for spotting this!) The basic concepts were OK, but the details, yikes. Internals page had info about GroupTask, which Task Developers page lacked... so move it over.

Bugs closed: 433

Reviewed at https://rbcommons.com/s/twitter/r/1074/
Use a Session to enable connection pooling

Requests and urllib3 automatically reuse connections if using a Session, which otherwise has all the same methods as the main Requests API, making this a drop-in replacement: http://docs.python-requests.org/en/latest/user/advanced/

In my tests, this change cut 830 HEAD requests to Foursquare's internal artifact cache from 22s to 11s.

Testing Done:
Using 830 cache keys copied from a real build, this change cut the time to do HEAD requests for each from 22s to 11s.

https://travis-ci.org/pantsbuild/pants/builds/38515947

Reviewed at https://rbcommons.com/s/twitter/r/981/
Merge branch 'master' into sitegen_increment
excellent feedback is excellent