Repository: DataBiosphere/toil

Issue:
File name too long error 
I'm running toil on a CWL file (this is an combined single cwl), but I would get the following error message about file name being too long

OSError: [Errno 36] File name too long: '/tmp/14815.1.all.q/tmp37tb1nut/jobs/kind-file_data_users_bz_projects_pilots_toiltest_test_overall__exome.cwl_variants_fe53c68a-8e57-4b6c-8182-1ec7149ba27d_varscan_4c31b316-740c-4cb4-a1e1-62c889918c82_caller_271223b8-674e-48bb-96ce-ff65f4a0792e_filter_9896296d-b58a-4ac3-9a57-694d80e59580'

It seems that toil is trying to create a new filename/folder that is more than the allowable filename length on the system, this is maybe because of a rather hierarchical structure of the CWL and the long job hash codes. Is there a way to resolve this? We cannot really modify the OS filename limit. We have other CWLs that many deeper subworkflows so renaming will not really resolve this given these hash values take up most of the filename length.  Or how do we call a single cwl that points to other subworkflow dependencies (other CWL files) (right now it's all exported as a single CWL) - not sure if this will fix the issue.

toil version: 4.2.0
python version: 3.7.9

Thank you!

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-663)
┆Issue Number: TOIL-663



Discussion:
We need to modify `_makeStringFilenameSafe` to make sure that it never produces a too-long path component.

https://github.com/DataBiosphere/toil/blob/cb1f59af0f60dadd8349edc8ee21292349cdd9e9/src/toil/jobStores/fileJobStore.py#L358-L382

We could also try and update the job naming for CWL jobs to include fewer UUIDs, because I'm not sure they really help anyone for debugging.

Resolving patch:
diff --git a/src/toil/jobStores/fileJobStore.py b/src/toil/jobStores/fileJobStore.py
--- a/src/toil/jobStores/fileJobStore.py
+++ b/src/toil/jobStores/fileJobStore.py
@@ -355,7 +355,7 @@ def _extractPathFromUrl(url):
     def _supportsUrl(cls, url, export=False):
         return url.scheme.lower() == 'file'
 
-    def _makeStringFilenameSafe(self, arbitraryString):
+    def _makeStringFilenameSafe(self, arbitraryString, maxLength=240):
         """
         Given an arbitrary string, produce a filename-safe though not
         necessarily unique string based on it.
@@ -364,6 +364,9 @@ def _makeStringFilenameSafe(self, arbitraryString):
         other nonempty filename-safe string.
 
         :param str arbitraryString: An arbitrary string
+        :param int maxLength: Maximum length of the result, to keep it plus
+                              any prefix or suffix under the filesystem's
+                              path component length limit
 
         :return: A filename-safe string
         """
@@ -378,8 +381,8 @@ def _makeStringFilenameSafe(self, arbitraryString):
         if len(parts) == 0:
             parts.append("UNPRINTABLE")
 
-        # Glue it all together
-        return '_'.join(parts)
+        # Glue it all together, and truncate to length
+        return '_'.join(parts)[:maxLength]
 
     def writeFile(self, localFilePath, jobStoreID=None, cleanup=False):
         absPath = self._getUniqueFilePath(localFilePath, jobStoreID, cleanup)


Test patch: