Repository: pantsbuild/pants

Issue:
Git hooks fail in non-master branches
The 1.14.x branch currently has a failure like:
```
[=== 00:00 Running pre-commit checks ===]
* Checking packages
* Checking headers
* Checking for banned imports
fatal: ambiguous argument 'master': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
* Checking for bad shell patterns
The command "./build-support/bin/travis-ci.sh -fmrt" exited with 1.
```
... that is potentially related to #7210, or if not that change than some other hook-related change since the previous stable branch.
Replace binary_mode flag on safe_file_dump with direct usage of `mode`
As mentioned on #6539: `safe_file_dump` has a `binary_mode` flag that should likely be replaced with direct/defaulted usage of `mode`.

While this method is not technically `:API public:`, it is widely used. It might be worth a deprecation cycle before actually removing `binary_mode`.
Remove ordering from PathGlobs / Snapshot
`PathGlobs` and `Snapshot` currently preserve ordering of input globs and output `PathStats` because there was one usecase (iirc) that cared about the ordering of the result ([wire](https://github.com/pantsbuild/pants/blob/cace0c4600348a39949766d6afc35ef8866c58e4/examples/src/wire/org/pantsbuild/example/element/BUILD#L6)). @illicitonion pointed out that that one usecase could be special cased to directly consume the filespecs/pathglobs from the target's `sources` field instead.

Removing ordering will involve:

1. switching from `OrderMap` to `HashSet` in `VFS::expand`
2. adjusting the pants `wire` task to inspect the filespecs of the targets it is compiling
3. removing a few tests that pants has that cover ordering ([including](https://github.com/pantsbuild/pants/blob/cace0c4600348a39949766d6afc35ef8866c58e4/tests/python/pants_test/engine/legacy/test_graph.py#L109))


Discussion:

@stuhood what about instead of deprecating `binary_mode` in favor of `mode`, we keep `binary_mode` and add `append: bool = False`?

Some benefits to this approach
- no deprecation necessary, and you can get the behavior we want (appending) in #6539 without additional work
- don't need to know what options mode accepts
- avoid allowing modes we probably don't want to accept like `r`.
> @stuhood what about instead of deprecating binary_mode in favor of mode, we keep binary_mode and add append: bool = False?

Mm. I think there is a definite grokkability benefit to just aligning with `open` args here. Sure, if you `open` a file with `'r'` it will fail, but that's the existing API... nothing new to understand.

But also, there are other modes: https://docs.python.org/2/library/functions.html#open
Also, by aligning with `open` args, we don't even have to interpret them: can just pass them through to `open`.
Okay, I see benefits to both approaches. I'll remove the requested changes.
I'm a little unclear on this. Could you note why removing ordering is valuable? Is it because without ordering, we could fingerprint based on set contents without ordering and avoid invalidating on order changes?
I found the slack conversation about it
Removing ordering allows us to construct any `Snapshot` from a `Digest`/`Directory`, which means we can have uniformity of the API with remote execution and local execution. My hope is that we can pass around `SnapshotDigests` (or some other thin wrapper type) in more cases to reduce memory usage.
Looked at this briefly tonight, and then realized that because `eager_fileset_with_spec` actually needs file listings (and thus cannot use only `DirectoryDigest`), there is relatively little performance benefit to providing intrinsics like `PathGlobs -> DirectoryDigest -> Snapshot`... either way, we need to store the `Snapshot` object.
Thanks for looking!

I can imagine a world where `EagerFilesetWithSpec` can make the more expensive "look up files" call if it needs it, but where most of the time an opaque `DirectoryDigest` is sufficient, and the need to make sure a call is rare. But I don't know how rare that would actually be in practice.

Resolving patch:
diff --git a/src/python/pants/backend/python/pants_requirement.py b/src/python/pants/backend/python/pants_requirement.py
--- a/src/python/pants/backend/python/pants_requirement.py
+++ b/src/python/pants/backend/python/pants_requirement.py
@@ -50,14 +50,7 @@ def __call__(self, name=None, dist=None):
                                       msg='The {} target only works for pantsbuild.pants '
                                           'distributions, given {}'.format(self.alias, dist))
 
-    # Update the environment marker in lockstep with other changes as described in
-    #   https://github.com/pantsbuild/pants/issues/6450
-    env_marker = "python_version>='2.7' and python_version<'3'"
-
-    requirement = PythonRequirement(requirement="{key}=={version} ; {env_marker}"
-                                    .format(key=dist,
-                                            version=pants_version(),
-                                            env_marker=env_marker))
+    requirement = PythonRequirement(requirement="{key}=={version}".format(key=dist, version=pants_version()))
 
     self._parse_context.create_object('python_requirement_library',
                                       name=name,
diff --git a/src/python/pants/releases/packages.py b/src/python/pants/releases/packages.py
--- a/src/python/pants/releases/packages.py
+++ b/src/python/pants/releases/packages.py
@@ -14,11 +14,15 @@
 from distutils.util import get_platform
 from functools import total_ordering
 
-import subprocess32 as subprocess
 from bs4 import BeautifulSoup
 from future.moves.urllib.error import HTTPError
 from future.moves.urllib.request import Request, urlopen
+from future.utils import PY2
 
+if PY2:
+  import subprocess32 as subprocess
+else:
+  import subprocess
 
 COLOR_BLUE = "\x1b[34m"
 COLOR_RESET = "\x1b[0m"
@@ -34,9 +38,7 @@ class Package(object):
   def __init__(self, name, target, bdist_wheel_flags=None):
     self.name = name
     self.target = target
-    # Update the --python-tag default in lockstep with other changes as described in
-    #   https://github.com/pantsbuild/pants/issues/6450
-    self.bdist_wheel_flags = bdist_wheel_flags or ("--python-tag", "py27")
+    self.bdist_wheel_flags = bdist_wheel_flags or ("--python-tag", "py27.py36.py37")
 
   def __lt__(self, other):
     return self.name < other.name
@@ -89,14 +91,19 @@ def find_platform_name():
   return get_platform().replace("-", "_").replace(".", "_")
 
 
-core_packages = {
-  Package(
-    "pantsbuild.pants",
-    "//src/python/pants:pants-packaged",
-    bdist_wheel_flags=("--python-tag", "cp27", "--plat-name", find_platform_name()),
-  ),
-  Package("pantsbuild.pants.testinfra", "//tests/python/pants_test:test_infra"),
-}
+def core_packages(py3):
+  # N.B. When releasing with Python 3, we constrain the ABI (Application Binary Interface) to cp36 to allow
+  # pantsbuild.pants to work with any Python 3 version>= 3.6. We are able to get this future compatibility by
+  # specifing `abi3`, which signifies any version >= 3.6 must work. This is possible to set because in
+  # `src/rust/engine/src/cffi/native_engine.c` we set up `Py_LIMITED_API` and in `src/python/pants/BUILD` we
+  # set ext_modules, which together allows us to mark the abi tag. See https://docs.python.org/3/c-api/stable.html
+  # for documentation and https://bitbucket.org/pypa/wheel/commits/1f63b534d74b00e8c2e8809f07914f6da4502490?at=default#Ldocs/index.rstT121
+  # for how to mark the ABI through bdist_wheel.
+  bdist_wheel_flags = ("--py-limited-api", "cp36") if py3 else ("--python-tag", "cp27", "--plat-name", find_platform_name())
+  return {
+    Package("pantsbuild.pants", "//src/python/pants:pants-packaged", bdist_wheel_flags=bdist_wheel_flags),
+    Package("pantsbuild.pants.testinfra", "//tests/python/pants_test:test_infra"),
+  }
 
 
 def contrib_packages():
@@ -177,13 +184,13 @@ def contrib_packages():
   }
 
 
-def all_packages():
-  return core_packages.union(contrib_packages())
+def all_packages(py3):
+  return core_packages(py3).union(contrib_packages())
 
 
-def build_and_print_packages(version):
+def build_and_print_packages(version, py3=False):
   packages_by_flags = defaultdict(list)
-  for package in sorted(all_packages()):
+  for package in sorted(all_packages(py3)):
     packages_by_flags[package.bdist_wheel_flags].append(package)
 
   for (flags, packages) in packages_by_flags.items():
@@ -210,9 +217,9 @@ def get_pypi_config(section, option):
   return config.get(section, option)
 
 
-def check_ownership(users, minimum_owner_count=3):
+def check_ownership(users, minimum_owner_count=3, py3=False):
   minimum_owner_count = max(len(users), minimum_owner_count)
-  packages = sorted(all_packages())
+  packages = sorted(all_packages(py3))
   banner("Checking package ownership for {} packages".format(len(packages)))
   users = {user.lower() for user in users}
   insufficient = set()
@@ -248,6 +255,9 @@ def check_ownership(i, package):
 
 def _create_parser():
   parser = argparse.ArgumentParser()
+  # Note because of how argparse handles subparsers, the --py3 flag must be passed before any of the subparser
+  # flags to resolve properly.
+  parser.add_argument("-3", "--py3", action="store_true", default=False, help="Release any non-universal packages as Python 3.")
   subparsers = parser.add_subparsers(dest="command")
   # list
   parser_list = subparsers.add_parser('list')
@@ -268,11 +278,11 @@ def _create_parser():
   if args.with_packages:
     print('\n'.join(
       '{} {} {}'.format(package.name, package.target, " ".join(package.bdist_wheel_flags))
-      for package in sorted(all_packages())))
+      for package in sorted(all_packages(args.py3))))
   else:
-    print('\n'.join(package.name for package in sorted(all_packages())))
+    print('\n'.join(package.name for package in sorted(all_packages(args.py3))))
 elif args.command == "list-owners":
-  for package in sorted(all_packages()):
+  for package in sorted(all_packages(args.py3)):
     if not package.exists():
       print("The {} package is new!  There are no owners yet.".format(package.name), file=sys.stderr)
       continue
@@ -281,8 +291,8 @@ def _create_parser():
       print("{}".format(owner))
 elif args.command == "check-my-ownership":
   me = get_pypi_config('server-login', 'username')
-  check_ownership({me})
+  check_ownership({me}, py3=args.py3)
 elif args.command == "build_and_print":
-  build_and_print_packages(args.version)
+  build_and_print_packages(args.version, py3=args.py3)
 else:
   raise argparse.ArgumentError("Didn't recognise arguments {}".format(args))


Test patch:
diff --git a/tests/python/pants_test/backend/python/test_pants_requirement.py b/tests/python/pants_test/backend/python/test_pants_requirement.py
--- a/tests/python/pants_test/backend/python/test_pants_requirement.py
+++ b/tests/python/pants_test/backend/python/test_pants_requirement.py
@@ -31,15 +31,6 @@ def key(python_requirement):
     self.assertEqual([key(expected)],
                      [key(pr) for pr in python_requirement_library.payload.requirements])
 
-    req = list(python_requirement_library.payload.requirements)[0]
-    self.assertIsNotNone(req.requirement.marker)
-
-    def evaluate_version(version):
-      return req.requirement.marker.evaluate({'python_version': version})
-
-    self.assertTrue(evaluate_version('2.7'))
-    self.assertFalse(all(evaluate_version(v) for v in ('2.6', '3.4', '3.5', '3.6', '3.7')))
-
   def test_default_name(self):
     self.add_to_build_file('3rdparty/python/pants', 'pants_requirement()')