convert more .rst files to .md

convert more .rst docs to .md. With this, src/python/pants/docs/*.rst have been converted.

In some pages that were already converted, some links were "TODO" because their destinations weren't converted yet. So fix those up.

Mostly formatting changes, no content changes. except...

src/python/pants/docs/docs.rst -> src/python/pants/docs/docs.md : this page said that docs were .rst format with some details. switched to saying .md format, left out new details.

src/python/pants/docs/with_emacs.rst -> src/docs/with_emacs.md : Changed title to mention "Emacs" instead of just "Ensime".

more small tweaks in response to reviewer feedback

Testing Done:
eyeballing. To eyeball this change along with a couple of sitegen-fix reviewboards, see http://pantsbuild.github.io/sitegen-test/

grepping for some common mistakes from previous batch

Reviewed at https://rbcommons.com/s/twitter/r/1219/
git root calculation for idea goal

Testing Done:
tested in dev mode locally. ci passes.

Reviewed at https://rbcommons.com/s/twitter/r/1189/
Revert "convert more .rst files to .md"

This reverts commit b7967f52c1b70aa0c7c297633438fafbfd23164a.

This commit broke CI :-(

Submitting this reversion TBR

Reviewed at https://rbcommons.com/s/twitter/r/1235/
Some small fixes to the new options implementation.

These all relate to options whose action is 'append', and address rare corner cases.

1. Switch some invocations of copy() to deepcopy(), to ensure e.g., that
   default values of list type are not shared between the
   parser we use to generate help and the parser we use to actually
   parse args.

2. Distinguish between an empty list value in config, and no value at all.
   This ensures that "no value" doesn't trump a hard-coded default because
   we mistake it for an explicit config value.

3. Don't special-case appends to RankedValue. Instead make copy()
   return the underlying value.  argparse is the only thing that calls copy()
   on a RankedValue, and this will ensure that argparse always works with
   raw values, both in the list case and (as before) in the single value case.

4. Nits: Remove an unused import, break up an overlong line.

Testing Done:
Various manual tests of corner cases that could trigger these issues. CI running.

Reviewed at https://rbcommons.com/s/twitter/r/1236/
Migrate the filter task to new-style options.

Testing Done:
Ran all of filter's unittests. CI pending.

Reviewed at https://rbcommons.com/s/twitter/r/1232/
Add the environment's PYTHONPATH to sys.path when running dev pants.

This is helpful when running a graphical debugger: you need to ensure that
the debugger's code (pydevd in my case) is on your path so you can import
it at the tracepoint.

Reviewed at https://rbcommons.com/s/twitter/r/1237/
Migrate checkstyle and scalastyle to new-style options.

Testing Done:
CI passed.

Reviewed at https://rbcommons.com/s/twitter/r/1231/
Fix a test snafu.
Migrate python tasks to new-style options.

Testing Done:
CI passed.

Manually verified that flags work as expected, especially the global --interpreter flag, as used by these tasks.

Reviewed at https://rbcommons.com/s/twitter/r/1244/
Temporarily change stderr log level to silence log.init if --quiet

Otherwise, if disk logging is configured, it prints 'logging to /path/to/something' even when in quiet mode (https://github.com/twitter/commons/blob/master/src/python/twitter/common/log/initialize.py#L285)

Testing Done:
https://github.com/pantsbuild/pants/pull/743

https://travis-ci.org/pantsbuild/pants/builds/39412239

Reviewed at https://rbcommons.com/s/twitter/r/1243/
Add notes about JDK versions supported.

Bugs closed: https://github.com/pantsbuild/pants/issues/739

Reviewed at https://rbcommons.com/s/twitter/r/1241/
Migrate ivy_resolve options to new options system

Testing Done:
inspected output of pants goal resolve.ivy examples/src/java:: --help
ran several invocations by hand.

Bugs closed: 738

Reviewed at https://rbcommons.com/s/twitter/r/1234/
Fix the issue of seeing "Error while writing to artifact cache: an integer is required" during [cache check]

Apparently the order of args was swapped by mistake. Fixed the issue by using named param calling style and updated the arg validation which would fail tests if similar problem occurs.

Testing Done:
https://travis-ci.org/jinfeng/jinfeng-pants-fork/builds/39441973 has passed.

local tests:

PANTS_DEV=1 ./pants goal test ./tests/python/pants_test/cache/

Reviewed at https://rbcommons.com/s/twitter/r/1249/
Fix the regression introduced in https://rbcommons.com/s/twitter/r/1186/

The original code in benchmark_run.py was:

  option_group.add_option(mkflag("target"), dest="target_class", action="append",
                            help="Name of the benchmark class.")
  ...
  ...
  self.caliper_args = self.context.options.target_class
  self.caliper_args += [ str1, str2 ]
  self.caliper_args.extend([str3, str4])
  ...
  ...
  # eventually deep in the callstack, a maybe_list(caliper_args) is called.

For whatever reason, we used "append" and the target_class is a list (even though
it should be a single string argument). But anyhow, it works, since the it was an
assignment "self.caliper_args = self.context.options.target_class"

Now the change introduced in the RB, while keeping the target still a list, it
changes the line "self.caliper_args = self.context.options.target_class" to

  self.args.insert(0, self.get_options().target)

Note even though it's called "target", it's actually an list. Thus we're inserting
a list as an element into a list. Later the maybe_list failed.

- also fixed the pants.ini benchmark setting name mismatch.

Testing Done:
local benchmark run worked.

CI (with all 3 commits) is baking: https://travis-ci.org/jinfeng/jinfeng-pants-fork/builds/39519055

Reviewed at https://rbcommons.com/s/twitter/r/1254/
Migrate ensime_gen.py, reporting_server.py to new options format.

Testing Done:
manually invoking tasks w/ command line flags.
ci is running

Bugs closed: 746

Reviewed at https://rbcommons.com/s/twitter/r/1251/
fix signature in 'goal targets' and BUILD dictionary

We construct the arg signature for Targets by walking up the class inheritance chain, accumulating info. But if more than one class in the chain mentions a param, then... we were showing it more than once in the arg signature, whoops. This had other side effects: we doc'd some params we meant to hide.

With this change, the Target-class-walking code got big enough such that I separated it out into its own function.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/39509603

Reviewed at https://rbcommons.com/s/twitter/r/1253/
RESTfulCache delegates tarfile handling to LocalArtifactCache

The motivation for this change is to allow artifacts downloaded by RESTfulCache to be directly added to the LocalArtifactCache without needing to decompress and recompress them as we do now.

At a high level, this is done by refactoring RESTfulCache such that it no longer handles creating or extracting tarfiles directly and instead delegates those operations to a LocalArtifactCache instance passed in its constructor, which may store them in the process.

This is a revision of a previous proposal (https://rbcommons.com/s/twitter/r/985/) along similar lines, though this iteration has RESTful artifact cache always passed a LocalArtifactCache at construction, to which it can always delegate all local file interaction (tarfile creation and extraction)

This simplifies RESTful cache, reducing it to only the HTTP-related functionality. This also eliminates duplication of tarfile creation and extraction logic, which now justs lives in the LocalArtifactCache.

TempLocalArtifactCache provides an alternate implementation of LocalArtifactCache, backed by tmpfiles instead of a cache_root. This can be passes to RESTfulCache to handle artifacts when running in a remote-only cache configuration.

Notes:

This change removes all usage of CombinedCache -- the only case where it was used,
composing an HTTP and local cache, is covered by the above changes.

It also removes the  "backfill" behavior from CombinedCache, which only made sense in the above setup: two remote caches should likely use rsyncd or whatever to share files rather than relying on clients building to shuttle artifacts between them and local caches can use symlinks or mounts.

Caches also now use stdlib logging instead of context.log since the latter
has references to the runtracker making it almost impossible to pickle.

Caches now use string.format instead of % for composing log messages.

Add more Sphinx style docstrings.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/39519073 passed

Bugs closed: 737

Reviewed at https://rbcommons.com/s/twitter/r/1233/
new param for page target, links. List of hyperlinked-to targets

List of hyperlinked-to targets. We use dependencies for this now.
But circular dependencies are an error while hyperlink-cycles should be OK.

So param page(links=[...]) feeds into traversable_specs (versus traversable_dependency_specs).

While I was looking at the tests, un-commented-in some commented-out assertions that had a done TODO.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/39520576

Bugs closed: 253, 742

Reviewed at https://rbcommons.com/s/twitter/r/1242/
Migrate some utility tasks to new-style options.

Reviewed at https://rbcommons.com/s/twitter/r/1248/
fix [run.junit] -> [test.junit]

Testing Done:
https://travis-ci.org/jinfeng/jinfeng-pants-fork/builds/39534255

Reviewed at https://rbcommons.com/s/twitter/r/1256/