add python checkstyle tool to science

code lives in {src,tests}/python/twitter/checkstyle
checkstyle plugins live in src/python/twitter/checkstyle/plugins and are dynamically loaded (but can be toggled with the -p option)

there are four binary targets provided:
  src/python/twitter/checkstyle:check => checker with all checks linked in
  src/python/twitter/checkstyle:check_pep8 => checker with only pep8 checks linked in
  src/python/twitter/checkstyle:check_pyflakes => checker with only pyflakes checks linked in
  src/python/twitter/checkstyle:check_twitter => checker with only twitter styleguide-specific checks linked in

running:

dist/check.pex [-p PLUGIN1 [-p PLUGIN2] ...]
               [--diff=branch]
               [-s SEVERITY]
               [--strict]
               glob1 glob2 ...

  -p allows you to toggle individual auto-loaded plugins, e.g. ExceptStatements.  if -p unspecified, all are selected.
  if --diff=branch is supplied, only run checkstyle against a diff (e.g. i've been testing this code by running dist/check.pex --diff=master)  NOTE this disables glob checking.
  severity levels are COMMENT, WARNING, ERROR.  -s allows you to toggle the level at which we log nits.
  if --strict is supplied, it exits non-zero if you get any nit at WARNING or higher.
  if globs specified, zsh glob (you probably want to escape these if you're already in zsh)

RB_ID=151272

(sapling split of 73e80ba512c854135544624f2baa6d6fca4eaddf)
add more checkstyle test coverage, integrate issue taxonomy

RB_ID=152529

(sapling split of d5b887947f86a34d40c05616cb2816db64132b9d)
do not discourage single line exception classes in checkstyle

single line: pass and hierarchy checking coming in next diff

RB_ID=153609
NO-QUEUE=true

(sapling split of 088ee346232a2869e6bb698a2f75732be7f840d2)
Streamlines Python checkstyle output

RB_ID=156640
NO-QUEUE=true

(sapling split of 57a0c45cb78dd99691a903d0c9bcf648e06fa4d6)
Sets checkstyle diff branch to master if no args or branch are set

RB_ID=156749
NO-QUEUE=true

(sapling split of 71613a83c5dfcc7b2f405d69a221f3db655e51f5)
AWESOME-4695 Fix checkstyle when renaming python files

RB_ID=170611

(sapling split of 6c2832f74059eeed0313e9e9c184d4d16dfe8cb1)
rename src/python/twitter/mesos to src/python/twitter/aurora

rename src/python/twitter/mesos to src/python/twitter/aurora
 - change sandbox code
 - rename recipes
 - s/MesosClientAPI/AuroraClientAPI/
 - update thrift namespaces, straggling missed changes and entry_points
 - import reordering
 - search and replace for twitter/mesos to twitter/aurora
 - search and replace for twitter.mesos to twitter.aurora
 - rewrite BUILD files

RB_ID=175784

(sapling split of d882f27457be9f6918c4eda1d66c7e415af28eca)
Fix checkstyle errors for checkstyle.

RB_ID=195707

(sapling split of 19d8d682d5f4b6721aa26cc4fc592cc100d5f990)
Alphabetize target dependencies
This looks like a total yak shave, but it will make automating the moving of dependencies to 3rdparty/... easier.

RB_ID=285621
TBR=true

(sapling split of 3189674498304979ce21dbd2e5165593064f7984)
Fix a few twitter.checkstyle bugs.  Add setup.py project for OSS.

- Add setup.py project.  Add more docstrings + check --list
- Add noqa.  Skip camelCase names if derived from non-object superclass.
- Get CheckstylePlugin out of the business of line filtering

RB_ID=353163

(sapling split of ce4d81d9762b850ae60c82a9d933dbb803275692)
Apply twitter.checkstyle split to commons repository.

Reviewed at https://rbcommons.com/s/twitter/r/341/
Apply checkstyle license headers.

Testing Done:
mba=commons=; ./pants py src/python/twitter/checkstyle:check src/python/twitter/checkstyle
mba=commons=; echo $?
0

Reviewed at https://rbcommons.com/s/twitter/r/343/
migrate-history: clean up text

Rearrange to move usage first; explain missing repeat-import feature.
Adds an optimized copy from one jar file to another.

The SuperShady.java code should be attributed to scottb@squareup.com

Testing Done:
To test the updated jar-tool, I used one of our targets with about 100 jar files. The unpacked size is 303MB. The super-shady version takes under 7 seconds as compared to the default version’s time of a little over 3 minutes:

jsirois/twitter-sync ./super-shady-exemplar
real    3m4.610s
user    14m10.336s

sys    0m12.502s

zundel/super-shady-jar-tool ./super-shady-exemplar
real    0m6.256s
user    0m5.061s

sys    0m4.216s
Also, the super-shady version compression is much better.

-rw-r--r--  1 zundel  zundel  198500393 May 11 11:51 /tmp/exemplar-orig.jar
-rw-r--r--  1 zundel  zundel   84200072 May 12 04:55 /tmp/exemplar-super-shady.jar

I don’t know why that is, maybe a difference in the compression level passed to the zip algorithm. I did have to fix the unit test because it expected all entries in the zip file to be deflated. It might also be that using DEFLATE on directory entries and other zipfile entries that don’t need it is mushrooming the size of the file in the original jar-tool version. I did a check to make sure that all the extracted files were the same between both versions. I unpacked the jars with ‘unzip’ into 2 directories and ran ‘md5’ on each file in each unpacked dir

Bugs closed: 290

Reviewed at https://rbcommons.com/s/twitter/r/345/
Small style fixes and a descriptive re-name for the SuperShady
contribution after internal review.
Print bad keys when erroring out on new reviewboard creation
Remove ShutdownHook from jar-tool to prevent memory leak when running under nailgun.

This fixes a memory leak using nailgun with jar-tool caused by adding a Shutdown hook.

Testing Done:
Inspected with jvisualvm to find memory leak

In pants.ini, crank down the heap size:

 while [ t ] ; do PANTS_DEV=1 ~/Src/pants/pants goal bundle <target> --bundle-deployjar ; done

ran for a while and heap did not grow, whereas before it would die after just a few iterations with a 64 MB heap.

Bugs closed: 293

Reviewed at https://rbcommons.com/s/twitter/r/428/
Add pex_bootstrapper.bootstrap_pex_env() for simplified external pex bootstrapping.
In jar-tool, use apache commons FileUtils.moveFile to use copy when target of jar write is across filesystem boundary

This change updates File.rename() to use the apache commons FileUtils.moveFile() when writing out a jar file in JarBuilder.write().  This change uses a simple file rename when possible, or a copy if the two files can't be renamed (if the destination is on a different filesystem, for example.)

Testing Done:
Ran ./pants goal test tests/java/com/twitter/common/jar/tool

It would be great if Patrick could test out in their environment where this is a problem:

1.) in twitter/commons use `./pants goal binary --binary-deployjar src/java/com/twitter/common/jar/tool/:main` to produce a self-contained tool jar at dist/jar-tool.jar
2.) in pantsbuild/pants edit like so adapted for your local paths:
$ git diff
diff --git a/BUILD.tools b/BUILD.tools
index 0e39737..0562831 100644
--- a/BUILD.tools
+++ b/BUILD.tools
@@ -95,6 +95,7 @@ jar_library(name = 'scrooge-legacy-gen',

 jar_library(name = 'jar-tool',
             jars = [
-              jar(org = 'com.twitter.common', name = 'jar-tool', rev = '0.1.5')
+              jar(org = 'com.twitter.common', name = 'jar-tool', rev = '0.1.6-SNAPSHOT',
+                  mutable=True, url="file:///home/<your path here>/commons/dist/jar-tool.jar")
             ])

Bugs closed: 295

Reviewed at https://rbcommons.com/s/twitter/r/440/
In jar-tool, use apache commons FileUtils.moveFile to use copy when target of jar write is across filesystem boundary

This change updates File.rename() to use the apache commons FileUtils.moveFile() when writing out a jar file in JarBuilder.write().  This change uses a simple file rename when possible, or a copy if the two files can't be renamed (if the destination is on a different filesystem, for example.)

Testing Done:
Ran ./pants goal test tests/java/com/twitter/common/jar/tool

Bugs closed: 295

Reviewed at https://rbcommons.com/s/twitter/r/440/

Signed-off-by: John Sirois <jsirois@twitter.com>

RB_ID=376301