register_options for some help-ish and doc-ish tasks.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/37842836

Reviewed at https://rbcommons.com/s/twitter/r/1135/
Add payload.get_field_value() to fix KeyError from pants goal idea testprojects::

There are cases where we access a field on a payload that doesn't exist.

Rather than try to figure out when target.payload.excludes exists,
I added a method Payload.get_field_value() that directly access the
value and has the capability to return 'None' for a missing field.

```
./pants goal idea testprojects:: --print-exception-stacktrace
...

Exception caught:
  ...
  File "/Users/zundel/Src/pants/src/python/pants/backend/codegen/tasks/apache_thrift_gen.py", line 183, in createtarget
    return self._create_java_target(gentarget, dependees)
  File "/Users/zundel/Src/pants/src/python/pants/backend/codegen/tasks/apache_thrift_gen.py", line 201, in _create_java_target
    return self._inject_target(target, dependees, self.gen_java, 'java', create_target)
  File "/Users/zundel/Src/pants/src/python/pants/backend/codegen/tasks/apache_thrift_gen.py", line 223, in _inject_target
    tgt = create_target(files, deps)
  File "/Users/zundel/Src/pants/src/python/pants/backend/codegen/tasks/apache_thrift_gen.py", line 200, in create_target
    excludes=target.payload.excludes)
  File "/Users/zundel/Src/pants/src/python/pants/base/payload.py", line 98, in __getattr__
    field = self._fields[attr]

Exception message: 'excludes'
```

Testing Done:
Added unit test, CI passed.

Bugs closed: 659

Reviewed at https://rbcommons.com/s/twitter/r/1150/
move the publish-extras-task doc to its own page. mention repo in register.py

moved some info around in hopes folks can find it more easily
mention repo-in-register.py

before:
setup_repo had some info about setting up publishing, but the publish page didn't link to it.
setup_repo didn't mention something new and important: setting up a repo in a plugin/register.py
publish page had a section about setting up an extras-with-publish task

after:
link to setup_repo's publish section from the publish page
mention repo-in-register.py in setup_repo
mv 'extras-with-publish task' to its own page, linked from setup_repo

Bugs closed: 602

Reviewed at https://rbcommons.com/s/twitter/r/1090/
get the name right, no stray quote mark, sheesh

Testing Done:
monkey-patched in Twitter's pants. hand-copied over. what coudl go wrong?

Bugs closed: 679

Reviewed at https://rbcommons.com/s/twitter/r/1164/
Fix issue 668: make release.sh execute packaged pants without loading internal backends during testing

The introduction of src/python/internal_backend plugins broke build-support/bin/release.sh.
Release.sh, during dry-run and real testing, fails to execute packaged pants, because it
doesn't have internal_backend packaged within but the repo root pants.ini still asks it
to load them.

The fix is to use PANTS_CONFIG_OVERRIDE during such execution with an empty plugin list.

Testing Done:
ran and passed:

- build-support/bin/release.sh -n
- build-support/bin/release.sh -t

CI passed: https://travis-ci.org/jinfeng/jinfeng-pants-fork/builds/37896390

Bugs closed: 668

Reviewed at https://rbcommons.com/s/twitter/r/1158/
A convenient way to set new-style options in tests.

Testing Done:
CI passed.

Merged this into a branch where I switched a bunch of JvmTask-related tests over to new options, and checked that this worked.

Reviewed at https://rbcommons.com/s/twitter/r/1163/
comments attempting to clear up: what goes in BUILD.tools ? what goes in 3rdparty/BUILD ?

Inspired by conversation on https://rbcommons.com/s/twitter/r/1151/ , I bet we want comments on these files helping folks to figure out which deps go where.

Reviewed at https://rbcommons.com/s/twitter/r/1155/
Fix scalastyle task, wire it up, make configs optional + refactoring and unit tests

- fix the task, which has slipped out of maintenance.
- wire it up in OS pants.
- but make it flexible so that pants_internal or other private wrapper of OS pants won't fail if they don't have scalastyle config and excludes present (making config/exclude optional)
- some refactoring so I can write unit tests easily.
- unit tests and end to end tests

Testing Done:
- CI passed: https://travis-ci.org/jinfeng/jinfeng-pants-fork/builds/37824308

- - PANTS_DEV=1 ./pants goal test tests::

- positive manual test:

$ PANTS_DEV=1 ./pants goal compile examples/src/scala/com/pants/example/hello/welcome
...
...
...
16:00:13 00:02   [compile]
16:00:13 00:02     [jvm]
...
...
16:00:22 00:11     [scalastyle]
16:00:22 00:11       [bootstrap-scalastyle]
16:00:23 00:12       [org.scalastyle.Main]
               Waiting for background workers to finish.
               SUCCESS

- negative manual test (with the examples/src/scala/com/pants/example/Welcome.scala to break the ImportGroupingChecker rule:

$ PANTS_DEV=1 ./pants goal compile examples/src/scala/com/pants/example/hello/welcome
...
...
...
16:03:42 00:00   [compile]
16:03:42 00:00     [jvm]
...
...
16:03:44 00:02     [scalastyle]
16:03:44 00:02       [org.scalastyle.Main]
                     ==== stderr ====

                     ==== stdout ====
                     error file=/Users/jinfeng/workspace/github-pants-jinfeng/examples/src/scala/com/pants/example/hello/welcome/Welcome.scala message=Imports should be grouped together  line=13 column=0
                     Processed 1 file(s)
                     Found 1 errors
                     Found 0 warnings
                     Finished in 56 ms

FAILURE: java org.scalastyle.Main ... exited non-zero (1)

               Waiting for background workers to finish.
               FAILURE

Bugs closed: 647

Reviewed at https://rbcommons.com/s/twitter/r/1145/
doc: if you're adopting Pants, and need to integrate that one weird tool

without an org-custom plugin, Pants does many amazing things. but it seems every organization has that handful of tools to integrate

Bugs closed: 105

Reviewed at https://rbcommons.com/s/twitter/r/1166/
sitegen += section links and xrefs

xmarks, xrefs:
.html files can have xmark anchors, like htmls anchor, except can be referenced from other pages.
xref is like href, but references xmarks

"pilcrow" section links: when mouse hovers over a heading, show a clickable link to that section. handy for when you want to answer a question w/a link to place in the docs.

fix to "breadcrumb" navigation: just show "ancestors", not also "older siblings"

added the start of some docs-for-docs

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/37849182

Reviewed at https://rbcommons.com/s/twitter/r/1154/
Adds a junit test integration test, including coverage options.

(Is cobertura working? I don't see any output in the cobertura reports.)

Testing Done:
ci is running.

Bugs closed: 675

Reviewed at https://rbcommons.com/s/twitter/r/1161/
(Adds missing fixes from new junit test integration test,  https://rbcommons.com/s/twitter/r/1161/)
Switch travis email notifications back to the default.

Emails now go to author (+comitter).

 .travis.yml                  | 14 ++++----------
 build-support/bin/ci-sync.sh |  5 +++--
 2 files changed, 7 insertions(+), 12 deletions(-)

Testing Done:
$ travis lint
Hooray, .travis.yml looks valid :)

And CI went green here: https://travis-ci.org/pantsbuild/pants/builds/38073663

I enabled ci for my pantsbuild/pants fork and I added a commit of this:
```console
$ cat << EOF > build-support/bin/ci.sh
#!/usr/bin/env bash
exit 1
EOF
```
That CI went red and emailed just me for both the pantsbuild/pants and jsirois/pants jobs:
  https://travis-ci.org/jsirois/pants/builds/38087136

Email to just me:
 jsirois / pants (jsirois/travis/email_simple)

Build #4 is still failing.	 38 seconds
	John Sirois	f727f99 Changeset →
 	Small workding fixes.

Bugs closed: 682

Reviewed at https://rbcommons.com/s/twitter/r/1169/
When resolving a tool dep, report back the pants.ini section with a reference that is failing.
Added a convenience method to pull a dependency list from pants.ini and   report the section/key info along to register_jvm_tool()

Testing Done:
ci is baking.

Manual Testing:

1) renamed target 3rdparty:protobuf-2.4.1 to 3rdparty:protobuf-2.4.2 and ran example protobuf compile.

``
$ ./pants goal compile examples/src/java/com/pants/examples/protobuf/distance
  ...
Exception message: protobuf-2.4.1 was not found in BUILD file /Users/zundel/Src/Pants/3rdparty/BUILD. Perhaps you meant one of:
  :checkstyle
  ...
  :util-core
  referenced from [protobuf-gen] key: javadeps in pants.ini
```

2) In BUILD.tools, I renamed '//:jar-tool to //:jar-tool2

```
$ ./pants goal bundle examples/src/java/com/pants/examples/hello/main
  ...
                     Failed to resolve target for bootstrap tool: //:jar-tool. You probably need to add this dep to your tools BUILD file(s), usually located in the root of the build.
                     See pants.ini section: [jar-tool] key: bootstrap-tools
                     Error: jar-tool was not found in BUILD file /Users/zundel/Src/Pants/BUILD. Perhaps you meant one of:
                       BUILD:readme
                       ...
                       BUILD.tools:zinc

```

Bugs closed: 676

Reviewed at https://rbcommons.com/s/twitter/r/1162/
Relabel the thrift_linter test to an integration test

test_thrift_linter is actually an integration test.  It is fast for an integration test, but slow as a unit test.

Testing Done:
ci si running

Bugs closed: 684

Reviewed at https://rbcommons.com/s/twitter/r/1171/
Refactor jar_library to be able to unwrap its list of jar_dependency objects

This change is intended to support the external_archive() sharing the same logic as java_protobuf_library() for processing
a list of addresses to JarLibrary targets.

Testing Done:
Added new unit test for new method in test_jvm_target
Also beefed up the test_jar_library unit test

CI is running

Bugs closed: 680

Reviewed at https://rbcommons.com/s/twitter/r/1165/
Move workdirs creation from __init__ to pre_execute in jvm_compile & Remove QuietTaskMixin from several tasks