Fixed a syntax error preventing compiling in test_list_goals.py, and added it to the pants_test/tasks/BUILD testing suite.
Fix the analysis case for when a scala_library has a java_source that depends back on the scala_library

Reviewed at https://rbcommons.com/s/twitter/r/507/
Make the GroupTaskTest more tolerant of the ordering of prepare_execute_action().

RB_ID=https://rbcommons.com/s/twitter/r/504
Fixup the IntelliJ doc, after actually using it to setup pants/IntelliJ on my Mac

RB_ID=https://rbcommons.com/s/twitter/r/379/
Adding a dependency from a java_library to a scala_library, this should not trigger the build dependency missing warning.

Jmake's analysis file only contains the dependency class names, not the full path to the class file. This means it assumes
they are in the compiled java directory, meaning we get missing dependency warnings for scala files.

Changed to pick those up in the correct classes directory.

Reviewed at https://rbcommons.com/s/twitter/r/474/
Fix erroneous binary dependency for the bundle goal.

Testing Done:
Using `PANTS_DEV=1 ./pants goal bundle {src,tests}/{java,scala}::`

Before:
=======
3.862	main:binary
3.540	main:binary:binary
3.517	main:binary:binary:create-monolithic-jar
3.266	main:binary:binary:create-monolithic-jar:jar-tool
1.631	main:jar
1.624	main:jar:jar
1.618	main:jar:jar:jar-create
1.505	main:jar:jar:jar-create:jar-tool
1.368	main:resolve
1.363	main:resolve:ivy
1.280	main:resolve:ivy:map-jars
0.661	main:bundle
0.654	main:bundle:bundle
0.624	main:bundle:bundle:create-monolithic-jar
0.464	main:bundle:bundle:create-monolithic-jar:jar-tool

After:
======
1.419	main:jar
1.413	main:jar:jar
1.402	main:jar:jar:jar-create
1.327	main:resolve
1.322	main:resolve:ivy
1.295	main:jar:jar:jar-create:jar-tool
1.236	main:resolve:ivy:map-jars
0.638	main:bundle
0.632	main:bundle:bundle
0.605	main:bundle:bundle:create-monolithic-jar
0.442	main:bundle:bundle:create-monolithic-jar:jar-tool

Bugs closed: 219

Reviewed at https://rbcommons.com/s/twitter/r/513/
Divorce 'binary' and 'bundle' goals from the 'jar' goal.

Save effort and time by extracting classfile jar creation
logic to JarTask and using this in JarCreate and JvmBinaryTask
to create classfile jars directly.  Previously BinaryCreate and
BundleCreate re-packed jars containing internal classes created
by the JarCreate Task.  Now time is saved by not creating these
middle-man jars at all.

Testing Done:
$ PANTS_DEV=1 ./pants goal test tests/python/::

Then:
Using `PANTS_DEV=1 ./pants goal bundle {src,tests}/{java,scala}::`

Before:
=======
1.419	main:jar
1.413	main:jar:jar
1.402	main:jar:jar:jar-create
1.327	main:resolve
1.322	main:resolve:ivy
1.295	main:jar:jar:jar-create:jar-tool
1.236	main:resolve:ivy:map-jars
0.638	main:bundle
0.632	main:bundle:bundle
0.605	main:bundle:bundle:create-monolithic-jar
0.442	main:bundle:bundle:create-monolithic-jar:jar-tool

After:
======
1.479	main:resolve
1.473	main:resolve:ivy
1.383	main:resolve:ivy:map-jars
0.708	main:bundle
0.701	main:bundle:bundle
0.669	main:bundle:bundle:create-monolithic-jar
0.493	main:bundle:bundle:create-monolithic-jar:jar-tool

IE: 1.419 seconds of JarCreate time cut out, and 0.063 seconds of Bundle time added.

I also verified binaries and bundles were still runnable and had embedded resources where they did before this change.

Bugs closed: 220

Reviewed at https://rbcommons.com/s/twitter/r/514/
Bump default versions of setuptools and wheel

Testing Done:
CI passes

Reviewed at https://rbcommons.com/s/twitter/r/517/
Relativize java classpaths relative to the build root

JIs original relativization code didn't work because java was executed with subprocess.call, and this call was given a different cwd argument (the build root) than the user's actual current working directory. I have made the classpaths relativized relative to the build root as opposed to the current working directory, and this seems to work.

Also added classpath length logging (so we can know when we are getting close to the maximum argument length).

Testing Done:
./build-support/bin/ci.sh all tests passed

Reviewed at https://rbcommons.com/s/twitter/r/516/
doc: Pants Internals += source roots

attempting to capture some generally-useful info from https://groups.google.com/forum/#!topic/pants-devel/FhS8k4QDJN8

Bugs closed: 186

Reviewed at https://rbcommons.com/s/twitter/r/496/
Use setuptools 5.0 (==3.8.1) because of a regression in 4.0.1

See https://rbcommons.com/s/twitter/r/517/ and
https://mail.python.org/pipermail/distutils-sig/2014-June/024478.html

Testing Done:
CI passed

Reviewed at https://rbcommons.com/s/twitter/r/520/
Added utf-8 headers to all python files

Added # content=utf-8 headers to all python files

If you use any non-ascii unicode in a file, this is required.

See a follow on patch  https://rbcommons.com/s/twitter/r/524/ that adds a test so we will check to see that new files contain the header.

This is a mechanical change, I used this script:

#!/bin/bash

for i in $(cat /tmp/files.out) ; do
  FILE=$i
  HEAD=$(head -1 ${FILE})
  if [ -f "${FILE}" -a "# coding=utf-8" != "${HEAD}" ] ; then
    mv "${FILE}" "${FILE}.tmp"
    if [ $? != 0 ] ; then
      echo "Couldn't move '${FILE}'"
      exit 1
    fi
    echo '# coding=utf-8' > ${FILE}
    cat ${FILE}.tmp >> ${FILE}
    rm ${FILE}.tmp
  else
    echo "********* HEAD was ${HEAD}"
  fi
done

Testing Done:
running ci.sh

See https://rbcommons.com/s/twitter/r/524/

Bugs closed: 227

Reviewed at https://rbcommons.com/s/twitter/r/523/
Kill the now unused Group class.

Testing Done:
$ ./build-support/bin/ci.sh

Bugs closed: 229

Reviewed at https://rbcommons.com/s/twitter/r/527/
Added a test to show that all files have 'coding=utf8' on them

This is a followon to https://rbcommons.com/s/twitter/r/523

which adds a test to show that all python files under the repo have '# coding=utf8' as the first line

Testing Done:
ci.sh passed

Bugs closed: 227

Reviewed at https://rbcommons.com/s/twitter/r/524/
    (Accidentally omitted last change from review)

    Added a test to show that all files have 'coding=utf8' on them

    This is a followon to https://rbcommons.com/s/twitter/r/523

    which adds a test to show that all python files under the repo have '# coding=utf8' as the first line

    Testing Done:
    ci.sh passed

    Bugs closed: 227

    Reviewed at https://rbcommons.com/s/twitter/r/524/
Fixup Utf8HeaderTest - use original python target sources only.

Restrict the Utf8HeaderTest to python files checked in on disk
by converting from an fs walk to a build file walk.

Button up the build file walk machinery in the build file parser
and convert existing usages to this new api.

Fixup BUILD dependencies.

Testing Done:
Currently red on travis-ci: https://travis-ci.org/pantsbuild/pants/jobs/27673541 :
...
E       AssertionError: Expected these files to contain first line "# coding=utf8": ['/home/travis/build/pantsbuild/pants/.pants.d/gen/thrift/combined/gen-py/__init__.py',

Now passes:
$ ./build-support/bin/ci.sh

Also ran refactored task tests individually to help confirm BUILD deps:
$ PANTS_DEV=1 ./pants goal test tests/python/pants_test/tasks/:listtargets
$ PANTS_DEV=1 ./pants goal test tests/python/pants_test/tasks/:filemap
$ PANTS_DEV=1 ./pants goal test tests/python/pants_test/tasks/:check_published_deps
$ PANTS_DEV=1 ./pants goal test tests/python/pants_test:test_utf8_header

Bugs closed: 236

Reviewed at https://rbcommons.com/s/twitter/r/537/
Added support for including .proto files in remote .jars on the --proto_path when compiling with protoc.

This is done by way of a new 'imports' parameter added to the java_protobuf_library target, which accepts jar dependencies as inputs. The syntax for this is:

java_protobuf_library(name='foo',
  imports=jar_sources(jar(org='foo.com', name='bar', rev='1.2.3'), jar(org='bar.com', name='foo', rev='3.2.1')),
  ...,
)

An extra ivy step called 'ivy-imports' (rather than ivy-resolve) has been added to accomodate this. It occurs after bootstrap and before codegen. The build was failing for a while on my machine due unrecognized symbols generated by protoc; this was fixed by deleting ~/.pants.d and re-running (for reasons that are still opaque to me).