comments attempting to clear up: what goes in BUILD.tools ? what goes in 3rdparty/BUILD ?

Inspired by conversation on https://rbcommons.com/s/twitter/r/1151/ , I bet we want comments on these files helping folks to figure out which deps go where.

Reviewed at https://rbcommons.com/s/twitter/r/1155/
Fix scalastyle task, wire it up, make configs optional + refactoring and unit tests

- fix the task, which has slipped out of maintenance.
- wire it up in OS pants.
- but make it flexible so that pants_internal or other private wrapper of OS pants won't fail if they don't have scalastyle config and excludes present (making config/exclude optional)
- some refactoring so I can write unit tests easily.
- unit tests and end to end tests

Testing Done:
- CI passed: https://travis-ci.org/jinfeng/jinfeng-pants-fork/builds/37824308

- - PANTS_DEV=1 ./pants goal test tests::

- positive manual test:

$ PANTS_DEV=1 ./pants goal compile examples/src/scala/com/pants/example/hello/welcome
...
...
...
16:00:13 00:02   [compile]
16:00:13 00:02     [jvm]
...
...
16:00:22 00:11     [scalastyle]
16:00:22 00:11       [bootstrap-scalastyle]
16:00:23 00:12       [org.scalastyle.Main]
               Waiting for background workers to finish.
               SUCCESS

- negative manual test (with the examples/src/scala/com/pants/example/Welcome.scala to break the ImportGroupingChecker rule:

$ PANTS_DEV=1 ./pants goal compile examples/src/scala/com/pants/example/hello/welcome
...
...
...
16:03:42 00:00   [compile]
16:03:42 00:00     [jvm]
...
...
16:03:44 00:02     [scalastyle]
16:03:44 00:02       [org.scalastyle.Main]
                     ==== stderr ====

                     ==== stdout ====
                     error file=/Users/jinfeng/workspace/github-pants-jinfeng/examples/src/scala/com/pants/example/hello/welcome/Welcome.scala message=Imports should be grouped together  line=13 column=0
                     Processed 1 file(s)
                     Found 1 errors
                     Found 0 warnings
                     Finished in 56 ms

FAILURE: java org.scalastyle.Main ... exited non-zero (1)

               Waiting for background workers to finish.
               FAILURE

Bugs closed: 647

Reviewed at https://rbcommons.com/s/twitter/r/1145/
doc: if you're adopting Pants, and need to integrate that one weird tool

without an org-custom plugin, Pants does many amazing things. but it seems every organization has that handful of tools to integrate

Bugs closed: 105

Reviewed at https://rbcommons.com/s/twitter/r/1166/
sitegen += section links and xrefs

xmarks, xrefs:
.html files can have xmark anchors, like htmls anchor, except can be referenced from other pages.
xref is like href, but references xmarks

"pilcrow" section links: when mouse hovers over a heading, show a clickable link to that section. handy for when you want to answer a question w/a link to place in the docs.

fix to "breadcrumb" navigation: just show "ancestors", not also "older siblings"

added the start of some docs-for-docs

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/37849182

Reviewed at https://rbcommons.com/s/twitter/r/1154/
Adds a junit test integration test, including coverage options.

(Is cobertura working? I don't see any output in the cobertura reports.)

Testing Done:
ci is running.

Bugs closed: 675

Reviewed at https://rbcommons.com/s/twitter/r/1161/
(Adds missing fixes from new junit test integration test,  https://rbcommons.com/s/twitter/r/1161/)
Switch travis email notifications back to the default.

Emails now go to author (+comitter).

 .travis.yml                  | 14 ++++----------
 build-support/bin/ci-sync.sh |  5 +++--
 2 files changed, 7 insertions(+), 12 deletions(-)

Testing Done:
$ travis lint
Hooray, .travis.yml looks valid :)

And CI went green here: https://travis-ci.org/pantsbuild/pants/builds/38073663

I enabled ci for my pantsbuild/pants fork and I added a commit of this:
```console
$ cat << EOF > build-support/bin/ci.sh
#!/usr/bin/env bash
exit 1
EOF
```
That CI went red and emailed just me for both the pantsbuild/pants and jsirois/pants jobs:
  https://travis-ci.org/jsirois/pants/builds/38087136

Email to just me:
 jsirois / pants (jsirois/travis/email_simple)

Build #4 is still failing.	 38 seconds
	John Sirois	f727f99 Changeset →
 	Small workding fixes.

Bugs closed: 682

Reviewed at https://rbcommons.com/s/twitter/r/1169/
When resolving a tool dep, report back the pants.ini section with a reference that is failing.
Added a convenience method to pull a dependency list from pants.ini and   report the section/key info along to register_jvm_tool()

Testing Done:
ci is baking.

Manual Testing:

1) renamed target 3rdparty:protobuf-2.4.1 to 3rdparty:protobuf-2.4.2 and ran example protobuf compile.

``
$ ./pants goal compile examples/src/java/com/pants/examples/protobuf/distance
  ...
Exception message: protobuf-2.4.1 was not found in BUILD file /Users/zundel/Src/Pants/3rdparty/BUILD. Perhaps you meant one of:
  :checkstyle
  ...
  :util-core
  referenced from [protobuf-gen] key: javadeps in pants.ini
```

2) In BUILD.tools, I renamed '//:jar-tool to //:jar-tool2

```
$ ./pants goal bundle examples/src/java/com/pants/examples/hello/main
  ...
                     Failed to resolve target for bootstrap tool: //:jar-tool. You probably need to add this dep to your tools BUILD file(s), usually located in the root of the build.
                     See pants.ini section: [jar-tool] key: bootstrap-tools
                     Error: jar-tool was not found in BUILD file /Users/zundel/Src/Pants/BUILD. Perhaps you meant one of:
                       BUILD:readme
                       ...
                       BUILD.tools:zinc

```

Bugs closed: 676

Reviewed at https://rbcommons.com/s/twitter/r/1162/
Relabel the thrift_linter test to an integration test

test_thrift_linter is actually an integration test.  It is fast for an integration test, but slow as a unit test.

Testing Done:
ci si running

Bugs closed: 684

Reviewed at https://rbcommons.com/s/twitter/r/1171/
Refactor jar_library to be able to unwrap its list of jar_dependency objects

This change is intended to support the external_archive() sharing the same logic as java_protobuf_library() for processing
a list of addresses to JarLibrary targets.

Testing Done:
Added new unit test for new method in test_jvm_target
Also beefed up the test_jar_library unit test

CI is running

Bugs closed: 680

Reviewed at https://rbcommons.com/s/twitter/r/1165/
Missed some style comments from code reviews earlier today, trying to clean up.

Missed some style comments from code reviews yesterday, trying to clean up.

Testing Done:
CI passed.

Bugs closed: 686

Reviewed at https://rbcommons.com/s/twitter/r/1174/
Move registration of --no-pantsrc to global_options.py to resurrect it

It's a bit of a special case since it is used before the options are parsed. I left that part as is.

Testing Done:
Manual

Reviewed at https://rbcommons.com/s/twitter/r/1170/
Ensure new options values are always properly typed.

Previously values read from pants.ini were always strings.

Amend the relevant unit test to detect a regression of this.

Testing Done:
Fixed the test, then ran it before and after the fix.

ci baking.

Reviewed at https://rbcommons.com/s/twitter/r/1175/
Switch from os.rename to shutil.move to support cross-fs renames when needed.

In each of these cases dst is a file or a new dir.  In these 2 cases the ouput of os.rename and shutil.move are identical (they are only different when dst is an existing dir).

Testing Done:
CI went green here: https://travis-ci.org/pantsbuild/pants/builds/38112860

Bugs closed: 671

Reviewed at https://rbcommons.com/s/twitter/r/1157/
A base class for tests that test a single task type.

Subclasses advertise the task type they care about in setupClass().

Deals with option registration and scoping for the task under test.

Also addresses some small code review comments from a previous commit.

Testing Done:
ci.sh passed.

Reviewed at https://rbcommons.com/s/twitter/r/1168/
Fix a test.

Should have been in the previous commit.
A base class for tests that test a single task type.

Subclasses advertise the task type they care about in setupClass().

Deals with option registration and scoping for the task under test.

Also addresses some small code review comments from a previous commit.
Move prepare_task() into TaskTest.

Make all the tests that used it as a standalone method be
subclass TaskTest.

All tests pass after this change.
WIP
WIP