Repository: pantsbuild/pants

Issue:
Workdirs path length can violate FS constraints with no obvious workarounds for users. 
In particular, Aurora hit an issue with thrift codegen as described in https://issues.apache.org/jira/browse/AURORA-1980:

```
Executing tasks in goals: gen -> pyprep -> test
17:13:42 00:01   [gen]
17:13:42 00:01     [thrift-py]
17:13:42 00:01       [cache]    
                   No cached artifacts for 4 targets.
                   Invalidated 4 targets.....
17:13:42 00:01   [pyprep]
17:13:42 00:01     [interpreter]
17:13:46 00:05     [requirements]
17:13:46 00:05       [cache]                                     
                   No cached artifacts for 37 targets.
                   Invalidated 37 targets.
17:14:06 00:25     [sources]
               Waiting for background workers to finish.
17:14:06 00:25   [complete]
               FAILURE
Exception caught: (<type 'exceptions.IOError'>)
 
Exception message: [Errno 36] File name too long: u'/home/<username>/aurora/.pants.d/build_invalidator/7/pants_backend_python_tasks2_gather_sources_GatherSources/.pants.d.gen.thrift-py.252d64521cf9.api.src.main.thrift.org.apache.aurora.gen._storage.current.api.src.main.thrift.org.apache.aurora.gen._storage.hash'
```

Here `<username>` was more than 5 characters, causing the filename to exceed 255 characters, violating a pretty standard 255 character max filesystem path-length constraint. We should either trap this sort of error centrally and suggest a workaround, or uniformly check for this ourselves and indirect through a fs layer that hashes paths down to a guaranteed length or else, at the very least, patch up for this particular case.


Discussion:
Oof.

I've seen these length restrictions be even shorter in the case of encrypted filesystems and docker... are we able to nail down exactly what the limit was in this case?
> are we able to nail down exactly what the limit was in this case?

It was linux with the standard 255 character limit in this case.
Any update on a solution for this issue? I just had to resort to building Aurora in the `/tmp` to save enough characters for the pants build to succeed 😅 

Interestingly enough, this isn't an issue on MacOS for some reason.
I'm going to look into adding a setting for this.

The strategy would be add the setting without changing the default and cherry-pick that to a `1.4.x`/`1.5.x` release (preferably the latter). You guys could then decrease the limit in your usage.

We can then land a separate un-cherrypicked change in the `1.6.0dev` series to lower the limit.
...well. Hm. I think it should be possible to change this particular path safely. My concern with the setting was just avoiding invalidating caches. But that's probably a non-issue... we don't make any guarantees about cache invalidation.
Makes sense, appreciate you looking into this @stuhood .  I'd offer to submit a patch myself but since I'm not too familiar with the codebase, it would take way longer than necessary.
Any updates on this? Unfortunately the issue is worse than I originally thought because when we try to run our aurora.pex, it'll hit the filename too long issue as well: 

```
$ aurora
Traceback (most recent call last):
  File "/usr/bin/aurora/.bootstrap/_pex/pex.py", line 320, in execute
  File "/usr/bin/aurora/.bootstrap/_pex/pex.py", line 78, in _activate
  File "/usr/bin/aurora/.bootstrap/_pex/environment.py", line 135, in activate
  File "/usr/bin/aurora/.bootstrap/_pex/environment.py", line 179, in _activate
  File "/usr/bin/aurora/.bootstrap/_pex/environment.py", line 124, in update_candidate_distributions
  File "/usr/bin/aurora/.bootstrap/_pex/environment.py", line 110, in load_internal_cache
  File "/usr/bin/aurora/.bootstrap/_pex/environment.py", line 97, in write_zipped_internal_cache
  File "/usr/bin/aurora/.bootstrap/_pex/util.py", line 176, in cache_distribution
  File "/usr/bin/aurora/.bootstrap/_pex/common.py", line 124, in safe_open
  File "/usr/bin/aurora/.bootstrap/_pex/common.py", line 112, in safe_mkdir
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 36] File name too long: '/home/rdelvalle/.pex/install/api.src.main.thrift.org.apache.thermos.thermos_c56dfbc5-0.0.0-py2-none-any.whl.98ef3c4bbe13057017dead11fa7089729df3d497.08c11c7b33c7485b842b5198dfa86b29'
```

Should I try to take a stab at fixing this issue and submitting a patch @jsirois ?
@rdelval : That second error is a different one... the total path length is 182 characters, and I don't think the issue in the original description is related. I'll have a patch out for the issue in the description in a minute.

Can you open a new issue for that one, and see if you can do some experiments to figure out what the limit is on that machine by creating some nested files/directories? It's also possible that the final path component is tripping some other limit.

Resolving patch:
diff --git a/src/python/pants/build_graph/target.py b/src/python/pants/build_graph/target.py
--- a/src/python/pants/build_graph/target.py
+++ b/src/python/pants/build_graph/target.py
@@ -23,6 +23,7 @@
 from pants.build_graph.address_lookup_error import AddressLookupError
 from pants.build_graph.target_addressable import TargetAddressable
 from pants.build_graph.target_scopes import Scope
+from pants.fs.fs import safe_filename
 from pants.source.payload_fields import SourcesField
 from pants.source.wrapped_globs import Files, FilesetWithSpec, Globs
 from pants.subsystem.subsystem import Subsystem
@@ -173,11 +174,7 @@ def maybe_readable_identify(cls, targets):
   @classmethod
   def compute_target_id(cls, address):
     """Computes a target id from the given address."""
-    id_candidate = address.path_safe_spec
-    if len(id_candidate) >= 200:
-      # two dots + 79 char head + 79 char tail + 40 char sha1
-      return '{}.{}.{}'.format(id_candidate[:79], sha1(id_candidate).hexdigest(), id_candidate[-79:])
-    return id_candidate
+    return safe_filename(address.path_safe_spec)
 
   @staticmethod
   def combine_ids(ids):
diff --git a/src/python/pants/fs/fs.py b/src/python/pants/fs/fs.py
--- a/src/python/pants/fs/fs.py
+++ b/src/python/pants/fs/fs.py
@@ -9,10 +9,9 @@
 import os
 
 
-# This is the max filename length for HFS+, extX and NTFS - the most likely filesystems pants will
-# be run under.
-# TODO(John Sirois): consider a better isolation layer
-_MAX_FILENAME_LENGTH = 255
+# The max filename length for HFS+, extX and NTFS is 255, but many systems also have limits on the
+# total path length (made up of multiple filenames), so we include some additional buffer.
+_MAX_FILENAME_LENGTH = 100
 
 
 def safe_filename(name, extension=None, digest=None, max_length=_MAX_FILENAME_LENGTH):
@@ -43,8 +42,16 @@ def safe_filename(name, extension=None, digest=None, max_length=_MAX_FILENAME_LE
     return filename
   else:
     digest = digest or hashlib.sha1()
-    digest.update(name)
-    safe_name = digest.hexdigest() + ext
+    digest.update(filename)
+    hexdigest = digest.hexdigest()[:16]
+
+    # Prefix and suffix length: max length less 2 periods, the extension length, and the digest length.
+    ps_len = max(0, (max_length - (2 + len(ext) + len(hexdigest))) // 2)
+    sep = '.' if ps_len > 0 else ''
+    prefix = name[:ps_len]
+    suffix = name[-ps_len:] if ps_len > 0 else ''
+
+    safe_name = '{}{}{}{}{}{}'.format(prefix, sep, hexdigest, sep, suffix, ext)
     if len(safe_name) > max_length:
       raise ValueError('Digest {} failed to produce a filename <= {} '
                        'characters for {} - got {}'.format(digest, max_length, filename, safe_name))


Test patch:
diff --git a/tests/python/pants_test/backend/jvm/tasks/test_jvm_bundle_integration.py b/tests/python/pants_test/backend/jvm/tasks/test_jvm_bundle_integration.py
--- a/tests/python/pants_test/backend/jvm/tasks/test_jvm_bundle_integration.py
+++ b/tests/python/pants_test/backend/jvm/tasks/test_jvm_bundle_integration.py
@@ -28,9 +28,8 @@ def test_bundle_colliding_resources(self):
       target = ('testprojects/maven_layout/resource_collision/example_{name}/'
                 'src/main/java/org/pantsbuild/duplicateres/example{name}/'
                 .format(name=name))
-      bundle_name = ('testprojects.maven_layout.resource_collision.example_{name}.'
-                     'src.main.java.org.pantsbuild.duplicateres.example{name}.example{name}'
-                     .format(name=name))
-      bundle_jar_name = 'example{proj}'.format(proj=name)
-      stdout = self.bundle_and_run(target, bundle_name, bundle_jar_name=bundle_jar_name)
+      bundle_name = 'example{name}'.format(name=name)
+      stdout = self.bundle_and_run(target, bundle_name, bundle_jar_name=bundle_name, bundle_options=[
+          '--bundle-jvm-use-basename-prefix',
+        ])
       self.assertEquals(stdout, 'Hello world!: resource from example {name}\n'.format(name=name))
diff --git a/tests/python/pants_test/build_graph/test_target.py b/tests/python/pants_test/build_graph/test_target.py
--- a/tests/python/pants_test/build_graph/test_target.py
+++ b/tests/python/pants_test/build_graph/test_target.py
@@ -115,11 +115,9 @@ def test_target_id_long(self):
       long_path = os.path.join(long_path, 'dummy{}'.format(i))
     long_target = self.make_target('{}:foo'.format(long_path), Target)
     long_id = long_target.id
-    self.assertEqual(len(long_id), 200)
-    self.assertEqual(long_id,
-      'dummy.dummy1.dummy2.dummy3.dummy4.dummy5.dummy6.dummy7.dummy8.dummy9.dummy10.du.'
-      'c582ce0f60008b3dc8196ae9e6ff5e8c40096974.y20.dummy21.dummy22.dummy23.dummy24.dummy25.'
-      'dummy26.dummy27.dummy28.dummy29.foo')
+    self.assertEqual(len(long_id), 100)
+    self.assertTrue(long_id.startswith('dummy.dummy1.'))
+    self.assertTrue(long_id.endswith('.dummy28.dummy29.foo'))
 
   def test_target_id_short(self):
     short_path = 'dummy'
diff --git a/tests/python/pants_test/fs/test_safe_filename.py b/tests/python/pants_test/fs/test_safe_filename.py
--- a/tests/python/pants_test/fs/test_safe_filename.py
+++ b/tests/python/pants_test/fs/test_safe_filename.py
@@ -34,6 +34,10 @@ def test_shorten(self):
     self.assertEqual('**.jill',
                      safe_filename('jack', '.jill', digest=self.FixedDigest(2), max_length=8))
 
+  def test_shorten_readable(self):
+    self.assertEqual('j.**.e.jill',
+                     safe_filename('jackalope', '.jill', digest=self.FixedDigest(2), max_length=11))
+
   def test_shorten_fail(self):
     with self.assertRaises(ValueError):
       safe_filename('jack', '.beanstalk', digest=self.FixedDigest(3), max_length=12)