This month brings the release candidate for a new pants build system and many additions and improvements to the python application framework.

pants:
  + Add support for excluding java or scala from ide configurations
  + Fix idea checkstyle integration to include all suppression files
  + Support forward references in BUILD files
  + Fixup eclipse support so that the generated project recognizes any annotation_processors
  + Add a PANTS_DEV option to run pants straight from the repo sources to ease the pants dev cycle
  + Cache git build info to avoid running expensive git commands on every build
  + Add marker files for checkstyle runs to eliminate unnecessary checkstyle invocations
  + Add a doc target that supports conversion of markdown to html and nesting other doc dependencies

python:
  + Added support for subcommands and ini file based flag defaulting for subcommands
  + Added support for installable modules and provide logging and an http server in this form
  + Added jvm Amount analog along with a metrics module and /vars HTTP handler
  + Add glog style log file rolling
  + Added an Atlassian Confluence wiki posting component
  + Upgrade pytest to 2.6
  + Resource pool implementation
  + Added caching support to pants python builds
  + Basic ServerSet client
  + Improvements to java classfile parser
  + Add a file tailer
  + Support PEX_INHERIT_PATH to allow a pex to inherit the PYTHONPATH

jvm:
  Upgrade to guava 11.0.1, zookeeper 3.3.4 and finaglized thrift compiler

  app:
    + Implement a local service registry for startup actions, use this to boot the HTTP server
    + Several fixes and improvements to ShutdownRegistry ands its integration with AppLauncher
    + ExecutorServiceShutdown to encapsulate an orderly shutdown of an ExecutorService

  args:
    + @Positional Arg<List<T>> is now supported for gathering positional args
    + Added a twitter util Duration parser that uses Amount<Long, Time> syntax
    + Added a RangeParser to parse guava Ranges
    + Added support for registering custom Parsers and Verifiers globally via annotations+apt
    + Add CanExecute Verifier for files

  base:
    + Introduce a CachingSupplier

  objectsize:
    + Now has a public artifact: com.twitter.common#objectsize
    + Refactor to be parameterizable to arbitrary JVM memory layouts by the user

  reflect:
    + Added simple TypeToken based on guice's TypeLiteral but without the dependency burden

  stats:
    + Added ReservoirSampler - an implementation of http://en.wikipedia.org/wiki/Reservoir_sampling
    + Added Elapsed counting stat that reports the time since it was reset.
    + Added a /vars handler that exports data in JSON format and sort /vars text handler output
    + Added per pool gc cycle count and gc time, fd stats and more thread stats to JvmStats
    + Percentile improvements including support for multiple sampling windows

  testing:
    + TearDownRegistry to adapt a TearDownTestCase to an ShutdownRegistry used for shutdowns

  text:
    + Add a TokenStream(De)Serializer for serialization of TokenStreams and arbitrary attributes
    + Better definitions of punctuation
    + Rename PossetiveTokenCombiner to PossessiveContractionTokenCombiner and modify it to combine tokens in contraction form.
    + Implement JapaneseGomokuTokenizer, and remove JapaneseMeCabTokenizer

  util:
    + Utilities for working with temporary files and directories

  zookeeper:
    + The com.twitter.common#zookeeper jar has been broken up into one jar per major component

(sapling split of 9fc05c8cc927ade33cb14ee21d433669af218c91)
This push brings several improvements to the release candidate for the new pants build system as well as continued improvements in the python application framework and pex support.

pants:
  + Add a dependency tracking java compiler for faster mapping of classfiles to sources.
  + Improve the junit runner to handle scala tests, abstract classes and classes with no tests.
  + Improve scala_library and scala_tests targets to accept java_sources for circular dependencies via pants() target pointers to java_libraries instead of using globs on a rigid parallel filesystem layout.
  + Fixup pants list to work against jar_library and other aggregator/aliasing targets.
  + Add support for generating an ivy resolve report.
  + Add --pex option to pants py and add PEX_MODULE as an invocation option during bootstrapping.
  + Fix pex generation for python library targets that only have resources.

python:
  + Add CherryPy and tornado eggs and upgrade pystachio egg.
  + Implement a TFinagleProtocol.
  + Fixup touch to use safe_open and factor this up.
  + Add parse_date/parse_time to twitter.common.quantity.
  + Add support for quantity __add__/__mul__.

jvm:
  Upgrade to guava 11.0.1, zookeeper 3.3.4 and finaglized thrift compiler

  app:
    + Fix and simplify LocalServiceRegistry and service launching.

  text:
    + Implement CapitalLetterDotTokenCombiner.
    + Implement Traditional2SimplifiedHanTransformer and LuceneSmartChineseTokenizer.

  zookeeper:
    + Handle interrupted exception on blocking get() in ZooKeeperClient.
    + Expose number of partitions in zookeeper/Partition.

(sapling split of 49d8f99f554457cedb9002ba79c51ea3d11c4850)
This push brings more improvements to the release candidate for the new pants build system as well as a good start towards full py 3.x compatibility.

    pants:
      + Get depmap graphviz support working on dotty - omnigraffle was hiding a few bugs.
      + Add support for passing tests to the junit runner and the specs runner via @argfiles for command lines that are beyond a size threshold.
      + Divorce the ide algorithm from bang so it can operate on unchunked targets and avoid cycles that can result in aggregated ide targets.
      + Add a markdown_to_html task.
      + Make javac and scalac tasks resilient to targets with no files.
      + Fix jar_publish change tracking to include internal and external dependency changes.
      + Fix codegen tasks to always link synthetic targets for upstream even when actually skipping codegen.
      + Move mostly to __future__ print_function make pants bootstrappable in python 3.x add pants.multi to manage pants with multiple python interpreters/
      + Harden BuildFile against valid BUILD file paths that are in fact dirs.
      + Add Phase.uninstall to support trimming away un-needed pants goals.
      + Fixup target ids to handle a top-level BUILD and compiler class mapping to normalize path names for comparison.
      + Change the 1-arg FuncTask wrapper to take a context - this will be a much more common use-case for func actions.
      + Add deploy_excludes to jvm_binary and respect these in the binary and bundle tasks as a short-term hack to be replaced with proper ivy configuration/maven scope support.
      + Add support for globs and rglobs when defining bundles

    python:
      + Make python twitter.commons more py3x friendly.
      + Changes to python common log:
        + change logs from going to stdout to going to stderr
        + change the default log level for stderr to ERROR rather than NONE
        + if we are writing both to disk and to stderr, write a message stating where we write the log files

    jvm:
      Upgrade to guava 11.0.2, twitter-text 1.4.8 and finagle/util 2.0.0.

      app:
        + Log more exception info in applauncher.

      args:
        + Refactoring of command line args parsing.
        + Scala prototype command line parsing implementation.

(sapling split of 522438231c1cb9d72dc3fd1ae23eb79d6cd31298)
This push brings more improvements to the release candidate for the new pants build system as well as a major revamp of python .pex support.

 New python toolchain:
   The major components:
     PythonLauncher => PEX
     Environment => PEXBuilder
     Almost completely rewritten twitter.common.python
     First class PexInfo manifest for doing thin pexes
     PEX files now embed enough portions of pip/distribute/twitter.common.* rather than dumping out eggs
    No longer use EggParser + EggCache; instead use the "proper" pkg_resources Metadata+Distribution components

  pants:
    + Fix resolve cp construction to always occur
    + Add a comment to make it clear pants confluence pages should not be edited directly - give a pointer to the target to use instead
    + Fix markdown open to display isolated pages in addition to page graph roots
    + Remove -types from ivy resolves and instead uniformly use extension matching against jar to cull classpath mappings
    + Fix help - don't reuse the initial option parser - just seed a new one
    + Add support in test runner for failing fast and plumb this option through to pants.new's junit task
    + Fix compiler diagnostic filters to only apply to warnings - was squashing error messages as well
    + Pass return codes back up from the pants py command
    + Support jvm_args for jmake
    + Add orphan pid reaping to python tests
    + Add support regex excludes to old and new pants
    + Add configurable test timeouts to python test targets
    + Add basic command line pre-parsing support to enable free mixing of goals, targets and flags for 99.9% of cases
    + Fix AnsiColorDiagnosticListener to handle warnings and errors at eol - a common case being missing ;   + add a failing test and fix
    + Add support for : and :: syntax to select all sibling and all descendant targets, deprecate the corresponding --all and --all-recursive flags
    + Add a check that jars never get over-written
    + Add rcfile support to pants.new for defaulting flags
    + Fix dependency file flushing for cases when the compiler hits an error before ever flushing a class file
    + Support python_binary targets without sources
    + Fixup JUnitConsoleRunner to count all public methods of a concrete class' hierarchy to model junit4 native runner behavior
    + Protect pants/new java processes from CLASSPATH environment variables - scrub these and warn the user
    + Cut off BUILD file ancestor walks at the build root to prevent a large class of hard to diagnose errors
    + Add an option to force NailgunTasks to globally revert to subprocess java spawns instead of starting up a nailgun server
    + Add support for publishing snapshots to a local maven repo. + Fix latent bug when publishing targets with thrift or protobuf dependencies
    + Map jmake error codes to meaningful (for the most part!) messages
    + Use a single --specs flag to collect both explicit specs file paths and classnames
    + Add basic emma code coverage support for java_tests
    + Add a custom DiagnosticListener to the dependency tracking compiler to enable targeted supression of warnings and colorized build output
    + Add an option to log success/failure after execution
    + Fix missing provides transfer from protobuf target to synthesized java_library target
    + Add idea and eclipse goals to generate IntelliJ IDEA and Eclipse projects respectively
    + Adds filedeps and pathdeps goals to pants
    + Improve bundle mapping flexibility with support for relative_to
    + Remove duplicate pants BUILD definition

  python:
    + Have the log module respond to a NONE disk level by turning of disk logging completely and add a LogOptions method to allow apps to do this programmatically
    + Add an lru cache decorator
    + Fix wall_time handling for procfs, fixes thermos kills on linux
    + Add ignore_errors option to pex dependency resolution
    + Support initializing common log without disk logs
    + Fix error in python macosx platform compatibility logic, add tests
    + Add support for pex pkg_resources entry points
    + Safer package caching for .pex
    + Add daemonization to twitter.common.app

  jvm:
    upgrades:
      mahout-0.6-T1, finagle 2.0.1, twitter-text 1.4.10, gson 2.1

    + Add 99.9 and 99.99 as default percentiles in RequestStats
    + Just log changes in a serverset when they occur
    + Kill twitter.thrift/ThriftService
    + Introduce an interface to abstract TimeSeriesRepositoryImpl from Stats static badness
    + Unsuppress checkstyle for args, common/application and httphandlers
    + Expose Supplier<EndpointStatus> in ServerSetModule
    + Changes to the TokenStreamSerializer
      add a fingerprint of the attributes used in the serializer so that mismatches in attributes are detectable
      add a ByteRef-like interface for deserializing a snippet of a byte array
      ensure that TokenTypes are serialized to a single byte
    + Introduce a glog style formatted in common.logging
    + Narrow finagle-thrift dependencies to just what is required
    + Expose per event latency stat in SlidingStats
    + Scala command line parsing using case classes

(sapling split of ea1b38ae856953d9f11939c5466bb88304e58d8b)
Pythonpath bugfix in pants.

(sapling split of 32cc24b5e202d543f1d0c6ab73d41cac09ae113d)
BUILD lint fixes.

(sapling split of 682d0ca38f8a01401e9b21cc874e1c295e95281c)
remove pants() boilerplate from BUILD dependencies

Summary: saving much RSI from all who would write BUILD files!

Reviewers: benjy

Differential Revision: http://phabricator.prod.foursquare.com/D1879

Auditors: benjy

(sapling split of 334bff471caf0f7c4d39adc202fb6d2ea693d10b)
This release brings further work towards the new pants antless infrastructure
and many python and java library improvements from many new contributors.

Some noteworthy changes are highlighted below.

upgrades:
 + scala 2.9.2
 + guava 14.0.1
 + protobuf-2.4.1
 + lucene 4.1.0
 + gson 2.2.2
 + easymock 3.1
 + commons-codec 1.6
 + checkstyle 5.6
 + nailgun 0.9.1
 + spymemcached-2.8.1
 + joda-time-2.1

pants:
 + kill the pants ant backend
 + new metadata goals:
   depmap, dependencies, dependees, list, changed, filter, minimize
 + a plugin to handle thrift code generation using scrooge (supports 2.x and
   3.x series)
 + support for scaladoc gen via the renamed 'doc' goal (was 'javadoc')
 + antlr 3 & 4 support for java codegen
 + support for caliper benchmark targets
 + various *.md's filled out and corrected
 + improved multiplatform pex support: ./pants.multi and ./pants setup command
 + a concurrent junit runner upgrade
 + support for junit tests written in scala or java and marked via @RunWith
 + resources become first class targets
 + support for IDEA 12
 + upgrade jmake to get java 8 classfile support
 + optional support for pants usage logging to a remote service
 + python code coverage support
 + setup.py artifact exporting for python libraries
 + support zip64 for jars
 + eclipse goal configured java formatter fixes

python:
 + a versatile pex utility
 + a new jira library
 + many small imporvements to common.app and common.log
 + http support for common
 + EventMuxer added to common.concurrent to multiplex threading.Events along
   with several other concurrency utilities
 + RingBuffer added to common.collections
 + scribe support for common.log
 + hardening and improvement of common.zookeeper libs
 + a new common.net ssh tunnel helper
 + basic finagle client support

java:
 + a new concurrent junit runner that respects @TestParallel and @TestSerial
 + a much improved /graphview interface
 + many HTTP stack imporvements including a request logger
 + support for /pprof/heap (heapster) and /pprof/profile gperftools compatible
   endpoints
 + several improvements to common.{inject,quantity,stats,zookeeper}
 + glog support for log4j
 + a simple utility to unpack and link pre-compiled native libraries packaged
   in jars
 + added increment and group attribute support to TokenizedCharSequence
 + refactored common.text for performance optimizations and run-time
   configuration support

(sapling split of 170192210e4588cff99e8b3560cf2461ca55778f)
A RunInfo file.

This is a small textfile-backed data structure containing info
about a pants run, such as the timestamp, commit sha etc.

This supercedes the mostly unused BuildInfo functionality, which was
removed, to avoid confusion.

Auditors: markcc

(sapling split of d74c0960a79c1a60aa9293cf6f627fc5b654f899)
Attempt #2 at proper multiplatform distillation for pants.pex.

(sapling split of 50491bc04eaf3085d9e8b7f018edfbce68d92da1)
Fixup SourceTranslator to refuse translation for anything but the current platform and python.

(sapling split of fcc7aa5de0ac383e2b8a3916e5e3f126279e8cf3)
Be resilent to lack of a configured SCM.

(sapling split of 8ca946c6ceb4c04bc729679d3b8411ceea631a6e)
Do a better job scrubbing PYTHONPATH.

(sapling split of 7c3a12f117c7ac449db739be6f4bbc506f448642)
fix includes

(sapling split of 1499ce31aba4d679e52de67e7ae1a7ea5cc3dc2a)
force pex builder to use a safe pkg_resources

(sapling split of 05ae1fb3d805855d71834a1a16e9050bf819a142)
smdh

(sapling split of 6d262255ab3f78c040abb3bb7765ba8e0ba42952)
Missing imports

(sapling split of 8a3a31613b92e8e69281d4d4d9db85141ca2abcb)
Fixup bootstrap script to point to https://pypi.python.org and fixup
plumbing to allow said-same for distribution resolution.

Also fixup an over-aggressive assert.  Some distributions, like
distribute 0.7.3, just serve as a dependency alias and export no top
level packages.

(sapling split of 45ca963a024f3a648691c7979913f42bdfd14f41)
Remove a superfluous encode() call.

dist.get_metadata() returns a string (NOT a unicode). It is not
logically sound to call encode('utf-8') on a string, only on a
unicode. In fact, string probably shouldn't have an encode() method
at all.

(sapling split of 783fa7b33a988019f79685c10eda6441e0de69c7)
raise URLError when CachedWeb gets non-200 code

This is to stop bad responses from being put into the python cache.

(sapling split of c00ac092f285d5a71fc92c494b1890b6b89b43fc)