Publishing doc fixes

Further cleanups to the publishing doc to avoid sharp edges and fix a formatting error.

Testing Done:
Eyeballed:
./build-support/bin/publish_docs.sh -o

Reviewed at https://rbcommons.com/s/twitter/r/1136/
Remove hack to add java_sources in context

Testing Done:
Yes:
https://travis-ci.org/pantsbuild/pants/builds/37349364

Maunally generated idea for
./pants goal idea testprojects/src/scala/com/pants/testproject/javasources:javasources and
./pants goal idea testprojects/src/scala/com/pants/testproject/javasources:javasources --idea-intransitive

Both have dependent java_sources targets configured in generated Project Structure.

With Eric's hack to add java_sources in ide_gen for scala_library, we no longer need the java_sources to be added to the self.context targets.
https://github.com/pantsbuild/pants/blob/master/src/python/pants/backend/jvm/tasks/ide_gen.py#L552

Reviewed at https://rbcommons.com/s/twitter/r/1130/
When a jvm doc tool (e.g. scaladoc) fails in combined mode, throw an exception.  Log runs of jvmdoc.  Eliminate unnecessary dict lookup.

Testing Done:
Ran tests.

Reviewed at https://rbcommons.com/s/twitter/r/1116/
Added documentation on the difference between a pants.pex for testing and production

Bugs closed: 648

Reviewed at https://rbcommons.com/s/twitter/r/1134/
BUILD Dictionary vs factory functions: doc the class, not the factory

Wondering why the docs for 'bundle' say 'Return a factory method that can
create bundles rooted at the parse context path.' ? This is why.

This change...
introduces another @manaul.builddict() param, factory. The builddict
goal, if it sees a symbol tagged @manual.builddict(factory=True), hopes
the factory is a class method and doc's the attached class instead of
the factory function itself.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/37459441

Bugs closed: 580

Reviewed at https://rbcommons.com/s/twitter/r/1126/
no longer git-ignore these no-longer-generated files

We stopped generating these back in July. Surely everybody's done a 'git clean' since then. Surely.

Testing Done:
git status

Bugs closed: 371

Reviewed at https://rbcommons.com/s/twitter/r/1144/
Changes dependees task to register options using the new system

Testing Done:
ci.sh and manual testing of options.

Reviewed at https://rbcommons.com/s/twitter/r/1138/
Remove debug_args from pants.ini.  Fixes issue 650 by removing string interpolation of debug_port

Testing Done:
Added a unit test for new jvm_debug_config.py

Ran the following tests with `debug_port` unset and set in pants.ini to make sure debug port was actually overridden

./pants goal test ./examples/tests/java/com/pants/examples/hello/greet --test-junit-debug
./pants goal test ./examples/tests/java/com/pants/examples/hello/greet --test-junit-debug --test-junit-coverage --test-junit-coverage-processor=emma
./pants goal test ./examples/tests/java/com/pants/examples/hello/greet --test-junit-debug --test-junit-coverage --test-junit-coverage-processor=cobertura
./pants goal run testprojects/src/java/com/pants/testproject/unicode/main --run-jvm-run-debug
./pants goal idea ./src:: # checked the project.i* file for the debug port

I could not find examples or testprojects for goals benchmark and spec to test them out.

Bugs closed: 650, 652

Reviewed at https://rbcommons.com/s/twitter/r/1137/
list assertion for python_requirement_library's requirements

Testing Done:
Ran new tests.

Reviewed at https://rbcommons.com/s/twitter/r/1142/
Fix issue with appendable options in new options system.

Reviewed at https://rbcommons.com/s/twitter/r/1131/
Read jvm compilation options the new way.

Also fix a bug where we weren't detecting legacy inverse boolean flags correctly.

Testing Done:
CI running. Manual use of relevant flags.

Reviewed at https://rbcommons.com/s/twitter/r/1129/
A unicode-safe wrapper around shlex.split().
Temporarily move nailgun flag registration to be global.

It was causing problems during option migration. This temporarily
widens an existing abstraction leak, but once we're fully over to
the new options system this naturally goes away.

Also - ensure task options are registered in proper scope order.

Testing Done:
ci is baking.

Reviewed at https://rbcommons.com/s/twitter/r/1140/
Fix a BUILD file bug.
Make old-style flag scoping conform to the new style rules.

Specifically, a singleton task in a goal must be fully qualified
by the goal name, unless the task has the same name as the goal.

E.g., --ivy-report becomes --resolve-ivy-report.

This only affects a small number of flags, and has to happen
sooner or later anyway.

Reviewed at https://rbcommons.com/s/twitter/r/1146/
Simpler implementation of flag prefix gathering.

The previous implementation detected "leaf scopes", to avoid matching
a prefix of a longer prefix. However this was more trouble than it
was worth, and the same effect can be achieved by simply sorting
the scopes.

Reviewed in https://rbcommons.com/s/twitter/r/1147/
register_options for some help-ish and doc-ish tasks.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/37842836

Reviewed at https://rbcommons.com/s/twitter/r/1135/
Add payload.get_field_value() to fix KeyError from pants goal idea testprojects::

There are cases where we access a field on a payload that doesn't exist.

Rather than try to figure out when target.payload.excludes exists,
I added a method Payload.get_field_value() that directly access the
value and has the capability to return 'None' for a missing field.

```
./pants goal idea testprojects:: --print-exception-stacktrace
...

Exception caught:
  ...
  File "/Users/zundel/Src/pants/src/python/pants/backend/codegen/tasks/apache_thrift_gen.py", line 183, in createtarget
    return self._create_java_target(gentarget, dependees)
  File "/Users/zundel/Src/pants/src/python/pants/backend/codegen/tasks/apache_thrift_gen.py", line 201, in _create_java_target
    return self._inject_target(target, dependees, self.gen_java, 'java', create_target)
  File "/Users/zundel/Src/pants/src/python/pants/backend/codegen/tasks/apache_thrift_gen.py", line 223, in _inject_target
    tgt = create_target(files, deps)
  File "/Users/zundel/Src/pants/src/python/pants/backend/codegen/tasks/apache_thrift_gen.py", line 200, in create_target
    excludes=target.payload.excludes)
  File "/Users/zundel/Src/pants/src/python/pants/base/payload.py", line 98, in __getattr__
    field = self._fields[attr]

Exception message: 'excludes'
```

Testing Done:
Added unit test, CI passed.

Bugs closed: 659

Reviewed at https://rbcommons.com/s/twitter/r/1150/
move the publish-extras-task doc to its own page. mention repo in register.py

moved some info around in hopes folks can find it more easily
mention repo-in-register.py

before:
setup_repo had some info about setting up publishing, but the publish page didn't link to it.
setup_repo didn't mention something new and important: setting up a repo in a plugin/register.py
publish page had a section about setting up an extras-with-publish task

after:
link to setup_repo's publish section from the publish page
mention repo-in-register.py in setup_repo
mv 'extras-with-publish task' to its own page, linked from setup_repo

Bugs closed: 602

Reviewed at https://rbcommons.com/s/twitter/r/1090/
get the name right, no stray quote mark, sheesh

Testing Done:
monkey-patched in Twitter's pants. hand-copied over. what coudl go wrong?

Bugs closed: 679

Reviewed at https://rbcommons.com/s/twitter/r/1164/