Fix the analysis case for when a scala_library has a java_source that depends back on the scala_library

Reviewed at https://rbcommons.com/s/twitter/r/507/
Make the GroupTaskTest more tolerant of the ordering of prepare_execute_action().

RB_ID=https://rbcommons.com/s/twitter/r/504
Fixup the IntelliJ doc, after actually using it to setup pants/IntelliJ on my Mac

RB_ID=https://rbcommons.com/s/twitter/r/379/
Adding a dependency from a java_library to a scala_library, this should not trigger the build dependency missing warning.

Jmake's analysis file only contains the dependency class names, not the full path to the class file. This means it assumes
they are in the compiled java directory, meaning we get missing dependency warnings for scala files.

Changed to pick those up in the correct classes directory.

Reviewed at https://rbcommons.com/s/twitter/r/474/
Fix erroneous binary dependency for the bundle goal.

Testing Done:
Using `PANTS_DEV=1 ./pants goal bundle {src,tests}/{java,scala}::`

Before:
=======
3.862	main:binary
3.540	main:binary:binary
3.517	main:binary:binary:create-monolithic-jar
3.266	main:binary:binary:create-monolithic-jar:jar-tool
1.631	main:jar
1.624	main:jar:jar
1.618	main:jar:jar:jar-create
1.505	main:jar:jar:jar-create:jar-tool
1.368	main:resolve
1.363	main:resolve:ivy
1.280	main:resolve:ivy:map-jars
0.661	main:bundle
0.654	main:bundle:bundle
0.624	main:bundle:bundle:create-monolithic-jar
0.464	main:bundle:bundle:create-monolithic-jar:jar-tool

After:
======
1.419	main:jar
1.413	main:jar:jar
1.402	main:jar:jar:jar-create
1.327	main:resolve
1.322	main:resolve:ivy
1.295	main:jar:jar:jar-create:jar-tool
1.236	main:resolve:ivy:map-jars
0.638	main:bundle
0.632	main:bundle:bundle
0.605	main:bundle:bundle:create-monolithic-jar
0.442	main:bundle:bundle:create-monolithic-jar:jar-tool

Bugs closed: 219

Reviewed at https://rbcommons.com/s/twitter/r/513/
Divorce 'binary' and 'bundle' goals from the 'jar' goal.

Save effort and time by extracting classfile jar creation
logic to JarTask and using this in JarCreate and JvmBinaryTask
to create classfile jars directly.  Previously BinaryCreate and
BundleCreate re-packed jars containing internal classes created
by the JarCreate Task.  Now time is saved by not creating these
middle-man jars at all.

Testing Done:
$ PANTS_DEV=1 ./pants goal test tests/python/::

Then:
Using `PANTS_DEV=1 ./pants goal bundle {src,tests}/{java,scala}::`

Before:
=======
1.419	main:jar
1.413	main:jar:jar
1.402	main:jar:jar:jar-create
1.327	main:resolve
1.322	main:resolve:ivy
1.295	main:jar:jar:jar-create:jar-tool
1.236	main:resolve:ivy:map-jars
0.638	main:bundle
0.632	main:bundle:bundle
0.605	main:bundle:bundle:create-monolithic-jar
0.442	main:bundle:bundle:create-monolithic-jar:jar-tool

After:
======
1.479	main:resolve
1.473	main:resolve:ivy
1.383	main:resolve:ivy:map-jars
0.708	main:bundle
0.701	main:bundle:bundle
0.669	main:bundle:bundle:create-monolithic-jar
0.493	main:bundle:bundle:create-monolithic-jar:jar-tool

IE: 1.419 seconds of JarCreate time cut out, and 0.063 seconds of Bundle time added.

I also verified binaries and bundles were still runnable and had embedded resources where they did before this change.

Bugs closed: 220

Reviewed at https://rbcommons.com/s/twitter/r/514/
Bump default versions of setuptools and wheel

Testing Done:
CI passes

Reviewed at https://rbcommons.com/s/twitter/r/517/
Relativize java classpaths relative to the build root

JIs original relativization code didn't work because java was executed with subprocess.call, and this call was given a different cwd argument (the build root) than the user's actual current working directory. I have made the classpaths relativized relative to the build root as opposed to the current working directory, and this seems to work.

Also added classpath length logging (so we can know when we are getting close to the maximum argument length).

Testing Done:
./build-support/bin/ci.sh all tests passed

Reviewed at https://rbcommons.com/s/twitter/r/516/
doc: Pants Internals += source roots

attempting to capture some generally-useful info from https://groups.google.com/forum/#!topic/pants-devel/FhS8k4QDJN8

Bugs closed: 186

Reviewed at https://rbcommons.com/s/twitter/r/496/
Use setuptools 5.0 (==3.8.1) because of a regression in 4.0.1

See https://rbcommons.com/s/twitter/r/517/ and
https://mail.python.org/pipermail/distutils-sig/2014-June/024478.html

Testing Done:
CI passed

Reviewed at https://rbcommons.com/s/twitter/r/520/
Remove codegen tags

It is in the android_resources target definition.
Added utf-8 headers to all python files

Added # content=utf-8 headers to all python files

If you use any non-ascii unicode in a file, this is required.

See a follow on patch  https://rbcommons.com/s/twitter/r/524/ that adds a test so we will check to see that new files contain the header.

This is a mechanical change, I used this script:

#!/bin/bash

for i in $(cat /tmp/files.out) ; do
  FILE=$i
  HEAD=$(head -1 ${FILE})
  if [ -f "${FILE}" -a "# coding=utf-8" != "${HEAD}" ] ; then
    mv "${FILE}" "${FILE}.tmp"
    if [ $? != 0 ] ; then
      echo "Couldn't move '${FILE}'"
      exit 1
    fi
    echo '# coding=utf-8' > ${FILE}
    cat ${FILE}.tmp >> ${FILE}
    rm ${FILE}.tmp
  else
    echo "********* HEAD was ${HEAD}"
  fi
done

Testing Done:
running ci.sh

See https://rbcommons.com/s/twitter/r/524/

Bugs closed: 227

Reviewed at https://rbcommons.com/s/twitter/r/523/
Kill the now unused Group class.

Testing Done:
$ ./build-support/bin/ci.sh

Bugs closed: 229

Reviewed at https://rbcommons.com/s/twitter/r/527/
Added a test to show that all files have 'coding=utf8' on them

This is a followon to https://rbcommons.com/s/twitter/r/523

which adds a test to show that all python files under the repo have '# coding=utf8' as the first line

Testing Done:
ci.sh passed

Bugs closed: 227

Reviewed at https://rbcommons.com/s/twitter/r/524/
    (Accidentally omitted last change from review)

    Added a test to show that all files have 'coding=utf8' on them

    This is a followon to https://rbcommons.com/s/twitter/r/523

    which adds a test to show that all python files under the repo have '# coding=utf8' as the first line

    Testing Done:
    ci.sh passed

    Bugs closed: 227

    Reviewed at https://rbcommons.com/s/twitter/r/524/
Fixup Utf8HeaderTest - use original python target sources only.

Restrict the Utf8HeaderTest to python files checked in on disk
by converting from an fs walk to a build file walk.

Button up the build file walk machinery in the build file parser
and convert existing usages to this new api.

Fixup BUILD dependencies.

Testing Done:
Currently red on travis-ci: https://travis-ci.org/pantsbuild/pants/jobs/27673541 :
...
E       AssertionError: Expected these files to contain first line "# coding=utf8": ['/home/travis/build/pantsbuild/pants/.pants.d/gen/thrift/combined/gen-py/__init__.py',

Now passes:
$ ./build-support/bin/ci.sh

Also ran refactored task tests individually to help confirm BUILD deps:
$ PANTS_DEV=1 ./pants goal test tests/python/pants_test/tasks/:listtargets
$ PANTS_DEV=1 ./pants goal test tests/python/pants_test/tasks/:filemap
$ PANTS_DEV=1 ./pants goal test tests/python/pants_test/tasks/:check_published_deps
$ PANTS_DEV=1 ./pants goal test tests/python/pants_test:test_utf8_header

Bugs closed: 236

Reviewed at https://rbcommons.com/s/twitter/r/537/
Add missing python backend plugin dep on the old py commands.

Fixup listtargets test - use uniform address.spec.

Also do the same for filemap as a cleanup sweep on
  https://rbcommons.com/s/twitter/r/537/

Testing Done:
$ ./build-support/bin/ci.sh

Travis went green: https://travis-ci.org/pantsbuild/pants/builds/27698663

Bugs closed: 239

Reviewed at https://rbcommons.com/s/twitter/r/540/
Kill unused coverage dep - this is an indirect dep we need not specify.

Also fixup coverage handling in the pytest goal:
+ Fixup cov_setup's use of itertools.chain.
+ Fix config template encoding, coverage was failing under ci with:
$ PANTS_PY_COVERAGE=1 ./build-support/bin/ci.sh
...
Traceback (most recent call last):
  File "/home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/pants.pex/pants/backend/python/test_builder.py", line 186, in _run_python_tests
    coverage_rc, args = self.cov_setup(targets)
  File "/home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/pants.pex/pants/backend/python/test_builder.py", line 140, in cov_setup
    cp = generate_coverage_config(targets)
  File "/home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/pants.pex/pants/backend/python/test_builder.py", line 66, in generate_coverage_config
    cp.readfp(Compatibility.StringIO(DEFAULT_COVERAGE_CONFIG))
  File "/home/jsirois/.pyenv/versions/2.6.8/lib/python2.6/ConfigParser.py", line 305, in readfp
    self._read(fp, filename)
  File "/home/jsirois/.pyenv/versions/2.6.8/lib/python2.6/ConfigParser.py", line 482, in _read
    raise MissingSectionHeaderError(fpname, lineno, line)
  MissingSectionHeaderError: File contains no section headers.
  file: <???>, line: 2
  '\x00[\x00r\x00u\x00n\x00]\x00\n'
  Failed to run test!

  FAILURE

Testing Done:
There are still issues here on the actual coverage - notice its not finding the src/python and thats because we now defeat the parallel src/tests assumptions of test_builder.py and we do not manually override this with coverage kwargs in out PythonTests targets.  I've filed this issue to track fixing that: https://github.com/pantsbuild/pants/issues/231

Also, the integration tests fail with coverage enabled - I've filed this issue to track that: https://github.com/pantsbuild/pants/issues/232

That said - this is forward progress, we have minimal deps and coverage doesn't blow up on unicode/code errors any longer:
$ PANTS_PY_COVERAGE=1 ./build-support/bin/ci.sh
...
23:40:49 00:01   [test]
23:40:49 00:01     [pytest]
23:40:49 00:01       [run]
...
============== test session starts ===============
                     platform linux3 -- Python 2.6.8 -- py-1.4.20 -- pytest-2.5.2
                     plugins: timeout
                     collected 293 items

                     tests/python/pants_test/base/test_double_dag.py ....
...
                     tests/python/pants_test/tasks/test_listtargets.py .......Coverage.py warning: Module pants_test.cache was never imported.
                     Coverage.py warning: Module example_test.hello.greet was never imported.
                     Coverage.py warning: Module pants_test.commands was never imported.
                     Coverage.py warning: Module pants_test.targets was never imported.
                     ..
                      coverage: platform linux3, python 2.6.8-final-0 -
                     Name                                                                    Stmts   Miss Branch BrMiss  Cover
                     ---------------------------------------------------------------------------------------------------------
                     tests/python/pants_test/__init__                                            0      0      0      0   100%
                     tests/python/pants_test/authentication/__init__                             0      0      0      0   100%
                     tests/python/pants_test/authentication/test_netrc_util                     40      2      6      2    91%
...
                     tests/python/pants_test/testutils/mock_logger                              14      0     12      3    88%
                     ---------------------------------------------------------------------------------------------------------
                     TOTAL                                                                    3943    264    323     86    92%
                     Coverage HTML written to dir /home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/dist/coverage/3bb346293fe9bb96c6f9d6f49e2a992d024617a1

                     ===== 292 passed, 1 xfailed in 27.19 seconds =====

23:42:01 01:13     [junit]
23:42:01 01:13     [specs]
               SUCCESS

Bugs closed: 230

Reviewed at https://rbcommons.com/s/twitter/r/532/
Upgrade setuptools in-line with the interpreter cache.

Testing Done:
$ ./build-support/bin/ci.sh

Bugs closed: 233

Reviewed at https://rbcommons.com/s/twitter/r/533/
Fixup ExampleProcessor to output its resource to the current compile class output dir.

This alleviates the untracked examples.txt left loose in the repo root after compiling
the example and its also more in line with how a typical resource generating apt
processor would locate its output.

Testing Done:
Prior to this change, ci.sh would leave examples.txt laying about untracked at the repo root.
After this change:
$ ./build-support/bin/ci.sh
...
02:26:44 00:22         [compile]
02:26:44 00:22           [jmake]
                         warning: bootstrap class path not set in conjunction with -source 1.6
                         Jmake version 1.3.8-3
                         Jun 16, 2014 2:26:45 AM com.twitter.common.args.apt.Configuration load
                         INFO: Loading @CmdLine config from: [jar:file:/home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/.pants.d/ivy/jars/com.twitter.common/application/jars/application-0.0.72.jar!/com/twitter/common/args/apt/cmdline.arg.info.txt.1, jar:file:/home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/.pants.d/ivy/jars/com.twitter.common/args/jars/args-0.2.8.jar!/com/twitter/common/args/apt/cmdline.arg.info.txt.1, jar:file:/home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/.pants.d/ivy/jars/com.twitter.common/application-module-log/jars/application-module-log-0.0.55.jar!/com/twitter/common/args/apt/cmdline.arg.info.txt.1, jar:file:/home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/.pants.d/ivy/jars/com.twitter.common/logging/jars/logging-0.0.59.jar!/com/twitter/common/args/apt/cmdline.arg.info.txt.1, jar:file:/home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/.pants.d/ivy/jars/com.twitter.common/application-module-stats/jars/application-module-stats-0.0.43.jar!/com/twitter/common/args/apt/cmdline.arg.info.txt.1, jar:file:/home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/.pants.d/ivy/jars/com.twitter.common/application-module-http/jars/application-module-http-0.0.61.jar!/com/twitter/common/args/apt/cmdline.arg.info.txt.1]
                         note: Generated resource 'file:/home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/.pants.d/compile/jvm/java/classes/examples.txt'
                         note: Writing file:/home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/.pants.d/compile/jvm/java/classes/com/twitter/common/args/apt/cmdline.arg.info.txt.2
                         Warning: unable to find .class file corresponding to source /home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/src/java/com/pants/testproject/packageinfo/package-info.java
                         Writing project database...  Done.
...

$ cat /home/jsirois/dev/3rdparty/jsirois-pantsbuild-pants/.pants.d/compile/jvm/java/classes/examples.txt
{
  {'type' : 'Main', 'value': 'Annotation Processing Example'},
}

Bugs closed: 234

Reviewed at https://rbcommons.com/s/twitter/r/535/