Repository: DataBiosphere/toil

Issue:
fileJobStore needs to handle arbitrary (or permission denied) os.link errors to work on some filesystems
I'm trying to run this simple workflow.

**cwltoil version**: 3.15.0
**cwltool version**: 1.0.20180525185854

cwlfile [touchfile.cwl]

```
cwlVersion: v1.0
class: CommandLineTool

baseCommand: touch

hints:
  DockerRequirement:
      dockerPull: ubuntu:18.04

arguments: 
  - output-$(inputs.num).txt


inputs:
  infile:
    type: File
  num: 
    type: int 

outputs:
  outfile:
    type: File
    outputBinding:
      glob: output-$(inputs.num).txt
```

workflow file [multi.cwl]

```
cwlVersion: v1.0
class: Workflow

requirements:
  - class: SubworkflowFeatureRequirement
  - class: ScatterFeatureRequirement

inputs:
  nums: int[]
  infile: File

outputs:
  output:
    type: File[]
    outputSource: touchfile/outfile

steps:
  touchfile:
    run: touchfile.cwl
    in:
      infile: infile
      num: nums

    scatter: num

    out:
      [outfile]
```

job file [job.yml]

```
nums: [1,2,3,4,5]
infile:
  class: File
  path: message.txt
```

I then run this as

    $ cwltoil --singularity --outdir `pwd`/outdir --jobStore `pwd`/JobStore  --workDir work mutli.cwl job.ymland get the following error:

```
=========> Failed job 'file:///users/sphe/cwl-crash/touchfile.cwl' touch T/9/jobyXgJmQ 
INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
INFO:toil:Running Toil version 3.15.0-0e3a87e738f5e0e7cff64bfdad337d592bd92704.
Got workflow error
Traceback (most recent call last):
  File "/users/sphe/spiel/.virtualenv/local/lib/python2.7/site-packages/cwltool/executors.py", line 100, in run_jobs
    for r in jobiter:
  File "/users/sphe/spiel/.virtualenv/local/lib/python2.7/site-packages/cwltool/command_line_tool.py", line 382, in job
    builder.pathmapper = self.makePathMapper(reffiles, builder.stagedir, **make_path_mapper_kwargs)
  File "/users/sphe/spiel/.virtualenv/local/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 233, in makePathMapper
    get_file=kwargs["toil_get_file"])
  File "/users/sphe/spiel/.virtualenv/local/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 196, in __init__
    stagedir, separateDirs=separateDirs)
  File "/users/sphe/spiel/.virtualenv/local/lib/python2.7/site-packages/cwltool/pathmapper.py", line 218, in __init__
    self.setup(dedup(referenced_files), basedir)
  File "/users/sphe/spiel/.virtualenv/local/lib/python2.7/site-packages/cwltool/pathmapper.py", line 271, in setup
    self.visit(fob, stagedir, basedir, copy=fob.get("writable"), staged=True)
  File "/users/sphe/spiel/.virtualenv/local/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 217, in visit
    resolved = self.get_file(loc) if self.get_file else loc
  File "/users/sphe/spiel/.virtualenv/local/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 268, in toilGetFile
    srcPath = fileStore.readGlobalFile(fileStoreID[7:])
  File "/users/sphe/spiel/.virtualenv/local/lib/python2.7/site-packages/toil/fileStore.py", line 1659, in readGlobalFile
    self.jobStore.readFile(fileStoreID, localFilePath, symlink=symlink)
  File "/users/sphe/spiel/.virtualenv/local/lib/python2.7/site-packages/toil/jobStores/fileJobStore.py", line 309, in readFile
    os.link(jobStoreFilePath, localFilePath)
OSError: [Errno 1] Operation not permitted
```

But not if I don't specify --workDir



┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-262)
┆friendlyId: TOIL-262



Discussion:
I get the same issue. Has this been addressed? It seems I have toil version 3.14.0.
We substantially revised how we attempt to make hard links from the fileJobStore in Toil 3.20.0. It looks like we don't handle `[Errno 1] Operation not permitted` specifically, but we use a more robust approach and we print some important debugging info (i.e. what it is trying to link to where) that could be used to solve the problem.

@Tierhon Could you retry with Toil 3.20.0 or later?

If that doesn't help, we could add this error to the ones that mean a hard link just isn't possible, rather than that something has gone wrong in Toil's internals. It might be that you happen to have a file system where for whatever reason hardlinking can be disallowed by permissions. Do you happen to be working with input files that aren't owned by the user running Toil?
@adamnovak I'm updating to 3.24.0 now, and restarting cactus, so hopefully that will work.

The filesystem is BeeGFS. It seems that versions previous to 7 could only create hardlinks in the same folder, and not across folders. We have 7.1.3 which should not have that limitation. Fingers crossed. 

Ole
@adamnovak At least I do not get this specific error anymore, but another:
"TypeError: _runner() got an unexpected keyword argument 'defer'" This is the same as #2854 I guess. 

Our computing cluster do not have Docker or Singularity set up, but I was glad when I saw cactus in conda. However, that version seems a bit outdated, and I am not really sure I would install cactus all on my own. The cluster should implement Docker/Singularity soon, and I guess I can revisit cactus then.

Ole
OK. Sorry Cactus is giving you trouble now.

I'll change this issue to track that some filesystems exist with this weird hard link limitation; I wasn't aware of that. We probably need to change the code here to account for permission or whatever other weird errors they raise to enforce that and fall back on copying in those cases instead of complaining that something is wrong:

https://github.com/DataBiosphere/toil/blob/993be0c3d95c83ca969d783e708c48d918407b16/src/toil/jobStores/fileJobStore.py#L442-L460

Resolving patch:
diff --git a/src/toil/jobStores/fileJobStore.py b/src/toil/jobStores/fileJobStore.py
--- a/src/toil/jobStores/fileJobStore.py
+++ b/src/toil/jobStores/fileJobStore.py
@@ -492,6 +492,7 @@ def read_file(self, file_id, local_path, symlink=False):
                 # It worked!
                 return
             except OSError as e:
+                # For the list of the possible errno codes, see: https://linux.die.net/man/2/symlink
                 if e.errno == errno.EEXIST:
                     # Overwrite existing file, emulating shutil.copyfile().
                     os.unlink(local_path)
@@ -500,7 +501,12 @@ def read_file(self, file_id, local_path, symlink=False):
                     os.symlink(jobStoreFilePath, local_path)
                     # Now we succeeded and don't need to copy
                     return
+                elif e.errno == errno.EPERM:
+                    # On some filesystems, the creation of symbolic links is not possible.
+                    # In this case, we try to make a hard link.
+                    pass
                 else:
+                    logger.error(f"Unexpected OSError when reading file '{jobStoreFilePath}' from job store")
                     raise
 
         # If we get here, symlinking isn't an option.
@@ -518,6 +524,7 @@ def read_file(self, file_id, local_path, symlink=False):
                 # It worked!
                 return
             except OSError as e:
+                # For the list of the possible errno codes, see: https://linux.die.net/man/2/link
                 if e.errno == errno.EEXIST:
                     # Overwrite existing file, emulating shutil.copyfile().
                     os.unlink(local_path)
@@ -530,10 +537,20 @@ def read_file(self, file_id, local_path, symlink=False):
                     # It's a cross-device link even though it didn't appear to be.
                     # Just keep going and hit the file copy case.
                     pass
+                elif e.errno == errno.EPERM:
+                    # On some filesystems, hardlinking could be disallowed by permissions.
+                    # In this case, we also fall back to making a complete copy.
+                    pass
+                elif e.errno == errno.ELOOP:
+                    # Too many symbolic links were encountered. Just keep going and hit the
+                    # file copy case.
+                    pass
+                elif e.errno == errno.EMLINK:
+                    # The maximum number of links to file is reached. Just keep going and
+                    # hit the file copy case.
+                    pass
                 else:
-                    logger.critical('Unexpected OSError when reading file from job store')
-                    logger.critical('jobStoreFilePath: ' + jobStoreFilePath + ' ' + str(os.path.exists(jobStoreFilePath)))
-                    logger.critical('localFilePath: ' + local_path + ' ' + str(os.path.exists(local_path)))
+                    logger.error(f"Unexpected OSError when reading file '{jobStoreFilePath}' from job store")
                     raise
 
         # If we get here, neither a symlink nor a hardlink will work.


Test patch: