Repository: pantsbuild/pants

Issue:
Fix developer thrash on the `native_engine_version` file
Right now, we deploy the native engine code as a seperate binary and version the binary via a hash of its sources, the value being stored in `src/python/pants/engine/native_engine_version`. With multiple developers starting to hack on the native engine more regularly, this files contents conflict and the dev experience is suboptimal.


Discussion:
One way to fix this is eliminate a version and separate distribution of the binary altogether. Today we build the native engine in-line in the `./pants` script so it has a 1-1 mapping with current source code. If we switch to accessing the binary as a resource file instead of using BinaryUtil to fetch it, we could modify the current inline building `./pants` does to place the native engine binary file on the `PYTHONPATH` for access as a resource when developing pants. For releasing pants we'll need to embed the native binary in a platform-specific wheel, whether that be a wheel of `pantsbuild.pants` itself, or an auxillary resource-only wheel. Either way, the wheel will need to be prepared - if not deployed - by CI to ensure we uniformly get the desired OSX and linux variants.
Alright, I have the native_engine.so scheme working via `./pants` and `./build-support/bin/release.sh -n`, which generates:
```
$ find dist/ -name "*.whl"
dist/pantsbuild.pants-1.4.0.dev13/dist/pantsbuild.pants-1.4.0.dev13-py2-none-linux_x86_64.whl
dist/pantsbuild.pants.contrib.jax_ws-1.4.0.dev13/dist/pantsbuild.pants.contrib.jax_ws-1.4.0.dev13-py2-none-any.whl
dist/pantsbuild.pants.contrib.errorprone-1.4.0.dev13/dist/pantsbuild.pants.contrib.errorprone-1.4.0.dev13-py2-none-any.whl
dist/pantsbuild.pants.contrib.scalajs-1.4.0.dev13/dist/pantsbuild.pants.contrib.scalajs-1.4.0.dev13-py2-none-any.whl
dist/pantsbuild.pants.contrib.go-1.4.0.dev13/dist/pantsbuild.pants.contrib.go-1.4.0.dev13-py2-none-any.whl
dist/pantsbuild.pants.contrib.android-1.4.0.dev13/dist/pantsbuild.pants.contrib.android-1.4.0.dev13-py2-none-any.whl
dist/pantsbuild.pants.contrib.scrooge-1.4.0.dev13/dist/pantsbuild.pants.contrib.scrooge-1.4.0.dev13-py2-none-any.whl
dist/pantsbuild.pants.contrib.findbugs-1.4.0.dev13/dist/pantsbuild.pants.contrib.findbugs-1.4.0.dev13-py2-none-any.whl
dist/pantsbuild.pants.contrib.buildgen-1.4.0.dev13/dist/pantsbuild.pants.contrib.buildgen-1.4.0.dev13-py2-none-any.whl
dist/pantsbuild.pants.contrib.node-1.4.0.dev13/dist/pantsbuild.pants.contrib.node-1.4.0.dev13-py2-none-any.whl
dist/pantsbuild.pants.contrib.python.checks-1.4.0.dev13/dist/pantsbuild.pants.contrib.python.checks-1.4.0.dev13-py2-none-any.whl
dist/pantsbuild.pants.contrib.cpp-1.4.0.dev13/dist/pantsbuild.pants.contrib.cpp-1.4.0.dev13-py2-none-any.whl
dist/pantsbuild.pants.testinfra-1.4.0.dev13/dist/pantsbuild.pants.testinfra-1.4.0.dev13-py2-none-any.whl
```

The last change is to have the 2 existing travis binary prep shards generate these wheels and upload them to s3 under `wheels/deploy/[pants_version]` instead of `bin/`. The `./build-support/bin/release.sh` script will be changed to fetch the wheels to deploy (the exact contents of which are now pre-tested on all platforms we support!) and use [`twine`](https://pypi.python.org/pypi/twine) to sign and upload them.
I should change py2 to py27 while I'm at it since we have that constraint to run pants.
Implementation note: thankfully the bucket listing api on s3 accepts prefixes making retrieval of pre-built wheels efficient now and going forward. For example:
```
curl -sSL https://binaries.pantsbuild.org/?prefix=bin/thrift/linux | xmllint --format - | grep "<Key>"
    <Key>bin/thrift/linux/i386/0.10.0/thrift</Key>
    <Key>bin/thrift/linux/i386/0.5.0-finagle/thrift</Key>
    <Key>bin/thrift/linux/i386/0.7.0/thrift</Key>
    <Key>bin/thrift/linux/i386/0.9.1/thrift</Key>
    <Key>bin/thrift/linux/i386/0.9.2/thrift</Key>
    <Key>bin/thrift/linux/i386/0.9.3/thrift</Key>
    <Key>bin/thrift/linux/x86_64/0.10.0/thrift</Key>
    <Key>bin/thrift/linux/x86_64/0.5.0-finagle/thrift</Key>
    <Key>bin/thrift/linux/x86_64/0.7.0/thrift</Key>
    <Key>bin/thrift/linux/x86_64/0.9.1/thrift</Key>
    <Key>bin/thrift/linux/x86_64/0.9.2/thrift</Key>
    <Key>bin/thrift/linux/x86_64/0.9.3/thrift</Key>
```

Resolving patch:
diff --git a/src/python/pants/engine/native.py b/src/python/pants/engine/native.py
--- a/src/python/pants/engine/native.py
+++ b/src/python/pants/engine/native.py
@@ -12,12 +12,13 @@
 import sysconfig
 import threading
 import traceback
+from contextlib import closing
 
 import cffi
+import pkg_resources
 import six
 
-from pants.binaries.binary_util import BinaryUtil
-from pants.util.dirutil import safe_mkdir, touch
+from pants.util.dirutil import safe_mkdir, safe_mkdtemp, touch
 from pants.util.memo import memoized_property
 from pants.util.objects import datatype
 
@@ -562,22 +563,12 @@ class Native(object):
   @staticmethod
   def create(options):
     """:param options: Any object that provides access to bootstrap option values."""
-    binary_util = BinaryUtil.Factory.create()
-    return Native(binary_util,
-                  options.native_engine_version,
-                  options.native_engine_supportdir,
-                  options.native_engine_visualize_to)
+    return Native(options.native_engine_visualize_to)
 
-  def __init__(self, binary_util, version, supportdir, visualize_to_dir):
+  def __init__(self, visualize_to_dir):
     """
-    :param binary_util: The BinaryUtil subsystem instance for binary retrieval.
-    :param version: The binary version of the native engine.
-    :param supportdir: The supportdir for the native engine.
     :param visualize_to_dir: An existing directory (or None) to visualize executions to.
     """
-    self._binary_util = binary_util
-    self._version = version
-    self._supportdir = supportdir
     self._visualize_to_dir = visualize_to_dir
 
   @property
@@ -587,9 +578,12 @@ def visualize_to_dir(self):
   @memoized_property
   def binary(self):
     """Load and return the path to the native engine binary."""
-    return self._binary_util.select_binary(self._supportdir,
-                                           self._version,
-                                           '{}.so'.format(NATIVE_ENGINE_MODULE))
+    lib_name = '{}.so'.format(NATIVE_ENGINE_MODULE)
+    lib_path = os.path.join(safe_mkdtemp(), lib_name)
+    with closing(pkg_resources.resource_stream(__name__, lib_name)) as input_fp:
+      with open(lib_path, 'wb') as output_fp:
+        output_fp.write(input_fp.read())
+    return lib_path
 
   @memoized_property
   def lib(self):
diff --git a/src/python/pants/option/global_options.py b/src/python/pants/option/global_options.py
--- a/src/python/pants/option/global_options.py
+++ b/src/python/pants/option/global_options.py
@@ -9,8 +9,6 @@
 import os
 import sys
 
-import pkg_resources
-
 from pants.base.build_environment import (get_buildroot, get_default_pants_config_file,
                                           get_pants_cachedir, get_pants_configdir, pants_version)
 from pants.option.arg_splitter import GLOBAL_SCOPE
@@ -135,10 +133,13 @@ def register_bootstrap_options(cls, register):
              help='Enables use of the v2 engine.')
 
     # These facilitate configuring the native engine.
-    register('--native-engine-version', advanced=True,
-             default=pkg_resources.resource_string('pants.engine', 'native_engine_version').strip(),
+    register('--native-engine-version', advanced=True, default='DEPRECATED',
+             removal_version='1.6.0.dev0',
+             removal_hint='Unused, the native engine is now embedded in the pantsbuild.pants wheel',
              help='Native engine version.')
-    register('--native-engine-supportdir', advanced=True, default='bin/native-engine',
+    register('--native-engine-supportdir', advanced=True, default='DEPRECATED',
+             removal_version='1.6.0.dev0',
+             removal_hint='Unused, the native engine is now embedded in the pantsbuild.pants wheel',
              help='Find native engine binaries under this dir. Used as part of the path to '
                   'lookup the binary with --binary-util-baseurls and --pants-bootstrapdir.')
     register('--native-engine-visualize-to', advanced=True, default=None, type=dir_option,


Test patch: