sitegen task, create site with better navigation.

Still not everything we want from a basic site generator,
but some steps in the right direction:

A site-wide table of contents, an outline in which the "nodes" are pages.

For each page,
+ page-wide table of contents, a list of headings.
+ "breadcrumbs" up top, that show how/if this page fits into outline

Testing Done:
added some tests. mostly "normal" cases, didn't try to dream up corner cases.

added some non-ASCII text to some test data, inspired by recent Unicode-mangling stories on the mailing list.

started https://travis-ci.org/pantsbuild/pants/builds/36613009

Reviewed at https://rbcommons.com/s/twitter/r/1086/
Ensure that we don't create a forwarding cycle.

Testing Done:
ci.sh passed.

Reviewed at https://rbcommons.com/s/twitter/r/1124/
Updated the src/python/pants/bin/BUILD file to use a minimum of Python 2.7 and include linux deps

(cherry picked from commit 6967c1412f0259d831585d622e91754dd9063cb9)
Filter out None scopes, to avoid trouble later.

I'm not sure why we have tasks with no set scope, but we've seen
this with custom tasks at Foursqare. Will figure that out later.

Testing Done:
Running ci.

Reviewed at https://rbcommons.com/s/twitter/r/1125/
Migrate all jvm compilation options to the new registration style.

Testing Done:
CI passed. Manual verification that flags still work.

Reviewed at https://rbcommons.com/s/twitter/r/1127/
Update help message for ./pants goal goals  (was ./pants goals)

./pants goals fails with a message:

BadSpecError: BUILD file does not exist at: /Users/zundel/Src/pants/goals/BUILD

Testing Done:
manual inspection of the help output.

Bugs closed: 644

Reviewed at https://rbcommons.com/s/twitter/r/1128/
Publishing doc fixes

Further cleanups to the publishing doc to avoid sharp edges and fix a formatting error.

Testing Done:
Eyeballed:
./build-support/bin/publish_docs.sh -o

Reviewed at https://rbcommons.com/s/twitter/r/1136/
Remove hack to add java_sources in context

Testing Done:
Yes:
https://travis-ci.org/pantsbuild/pants/builds/37349364

Maunally generated idea for
./pants goal idea testprojects/src/scala/com/pants/testproject/javasources:javasources and
./pants goal idea testprojects/src/scala/com/pants/testproject/javasources:javasources --idea-intransitive

Both have dependent java_sources targets configured in generated Project Structure.

With Eric's hack to add java_sources in ide_gen for scala_library, we no longer need the java_sources to be added to the self.context targets.
https://github.com/pantsbuild/pants/blob/master/src/python/pants/backend/jvm/tasks/ide_gen.py#L552

Reviewed at https://rbcommons.com/s/twitter/r/1130/
When a jvm doc tool (e.g. scaladoc) fails in combined mode, throw an exception.  Log runs of jvmdoc.  Eliminate unnecessary dict lookup.

Testing Done:
Ran tests.

Reviewed at https://rbcommons.com/s/twitter/r/1116/
Added documentation on the difference between a pants.pex for testing and production

Bugs closed: 648

Reviewed at https://rbcommons.com/s/twitter/r/1134/
BUILD Dictionary vs factory functions: doc the class, not the factory

Wondering why the docs for 'bundle' say 'Return a factory method that can
create bundles rooted at the parse context path.' ? This is why.

This change...
introduces another @manaul.builddict() param, factory. The builddict
goal, if it sees a symbol tagged @manual.builddict(factory=True), hopes
the factory is a class method and doc's the attached class instead of
the factory function itself.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/37459441

Bugs closed: 580

Reviewed at https://rbcommons.com/s/twitter/r/1126/
no longer git-ignore these no-longer-generated files

We stopped generating these back in July. Surely everybody's done a 'git clean' since then. Surely.

Testing Done:
git status

Bugs closed: 371

Reviewed at https://rbcommons.com/s/twitter/r/1144/
Changes dependees task to register options using the new system

Testing Done:
ci.sh and manual testing of options.

Reviewed at https://rbcommons.com/s/twitter/r/1138/
Remove debug_args from pants.ini.  Fixes issue 650 by removing string interpolation of debug_port

Testing Done:
Added a unit test for new jvm_debug_config.py

Ran the following tests with `debug_port` unset and set in pants.ini to make sure debug port was actually overridden

./pants goal test ./examples/tests/java/com/pants/examples/hello/greet --test-junit-debug
./pants goal test ./examples/tests/java/com/pants/examples/hello/greet --test-junit-debug --test-junit-coverage --test-junit-coverage-processor=emma
./pants goal test ./examples/tests/java/com/pants/examples/hello/greet --test-junit-debug --test-junit-coverage --test-junit-coverage-processor=cobertura
./pants goal run testprojects/src/java/com/pants/testproject/unicode/main --run-jvm-run-debug
./pants goal idea ./src:: # checked the project.i* file for the debug port

I could not find examples or testprojects for goals benchmark and spec to test them out.

Bugs closed: 650, 652

Reviewed at https://rbcommons.com/s/twitter/r/1137/
list assertion for python_requirement_library's requirements

Testing Done:
Ran new tests.

Reviewed at https://rbcommons.com/s/twitter/r/1142/
Fix issue with appendable options in new options system.

Reviewed at https://rbcommons.com/s/twitter/r/1131/
Read jvm compilation options the new way.

Also fix a bug where we weren't detecting legacy inverse boolean flags correctly.

Testing Done:
CI running. Manual use of relevant flags.

Reviewed at https://rbcommons.com/s/twitter/r/1129/
A unicode-safe wrapper around shlex.split().
Temporarily move nailgun flag registration to be global.

It was causing problems during option migration. This temporarily
widens an existing abstraction leak, but once we're fully over to
the new options system this naturally goes away.

Also - ensure task options are registered in proper scope order.

Testing Done:
ci is baking.

Reviewed at https://rbcommons.com/s/twitter/r/1140/
Fix a BUILD file bug.