Repository: DataBiosphere/toil

Issue:
Issue with toil-mutex-toil_miniwdl_sif_cache_mutex
When running the DeepPolisher WDL I get an error that does not seem to stem from the workflow itself.

I'm running this with Toil  5.11.0 and miniwdl 1.9.1. 

Here is the command I used:
```
mkdir toil_logs 
mkdir analysis

SINGULARITY_CACHEDIR=`pwd`/outputs/cache/.singularity/cache 
MINIWDL__SINGULARITY__IMAGE_CACHE=`pwd`/outputs/cache/.cache/miniwdl 

toil-wdl-runner \
    --jobStore ./big_store \
    --batchSystem slurm \
    --batchLogsDir ./toil_logs \
    /private/groups/hprc/scaff_test/HG01099/github/hpp_production_workflows/QC/wdl/workflows/hprc_DeepPolisher.wdl \
    HG01099_polishing_inputs.json \
    --outputDirectory analysis/polishing \
    --outputFile HG01099_polishing_outputs.json \
    --runLocalJobsOnWorkers \
    --retryCount 0 \
    --disableProgress \
    2>&1 | tee log.txt
```
And here is the log output...
```
toil.exceptions.FailedJobsException: The job store '/private/groups/hprc/scaff_test/HG01099/toil/big_store' contains 13 failed jobs: 'WDLScatterJob' scatter-L40C5-readFile kind-WDLWorkflowNodeJob/instance-f8xp6wca v4, 'WDLWorkflowNodeJob' call-extractReads kind-WDLWorkflowNodeJob/instance-h1wc8g2q v2, 'WDLWorkflowNodeJob' call-alignONTToMat kind-WDLWorkflowNodeJob/instance-16t1jrg_ v2, 'WDLWorkflowNodeJob' call-alignHifiToDiploid kind-WDLWorkflowNodeJob/instance-pgsrk_co v2, 'WDLWorkflowNodeJob' call-extractReads kind-WDLWorkflowNodeJob/instance-ptr0_hk9 v2, 'WDLCombineBindingsJob' kind-WDLCombineBindingsJob/instance-svuk5r6q v3, 'WDLWorkflowJob' kind-WDLRootJob/instance-xqrk3eai v4, 'WDLTaskJob' hprc_DeepPolisher.alignHifiToDiploid.extractReads kind-WDLTaskJob/instance-cczh3854 v3, 'WDLWorkflowNodeJob' call-extractReads kind-WDLWorkflowNodeJob/instance-fb6geiri v2, 'WDLWorkflowJob' kind-WDLWorkflowJob/instance-quq08neh v2, 'WDLTaskJob' hprc_DeepPolisher.alignHifiToDiploid.extractReads kind-WDLTaskJob/instance-gwsl6nwk v2, 'WDLCombineBindingsJob' kind-WDLCombineBindingsJob/instance-0tseqm3x v3, 'WDLWorkflowNodeJob' call-alignONTToPat kind-WDLWorkflowNodeJob/instance-7gvvjkof v2
Log from job "'WDLTaskJob' hprc_DeepPolisher.alignHifiToDiploid.extractReads kind-WDLTaskJob/instance-cczh3854 v3" follows:
=========>
        [2023-10-24T21:09:10-0700] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
        [2023-10-24T21:09:10-0700] [MainThread] [I] [toil] Running Toil version 5.11.0-9a04dabb36d6ab13ed1ac7c711dbdc8c71724dc9 on host phoenix-08.prism.
        [2023-10-24T21:09:10-0700] [MainThread] [I] [toil.worker] Working on job 'WDLTaskJob' hprc_DeepPolisher.alignHifiToDiploid.extractReads kind-WDLTaskJob/instance-cczh3854 v1
        [2023-10-24T21:09:10-0700] [MainThread] [I] [toil.worker] Loaded body Job('WDLTaskJob' hprc_DeepPolisher.alignHifiToDiploid.extractReads kind-WDLTaskJob/instance-cczh3854 v1) from description 'WDLTaskJob' hprc_DeepPolisher.alignHifiToDiploid.extractReads kind-WDLTaskJob/instance-cczh3854 v1
        [2023-10-24T21:09:10-0700] [MainThread] [I] [toil.wdl.wdltoil] Running task extractReads (['extractReads_t', 'extractReads']) called as hprc_DeepPolisher.alignHifiToDiploid.extractReads
        [2023-10-24T21:09:10-0700] [MainThread] [I] [toil.wdl.wdltoil] Defaulting referenceFasta to None
        [2023-10-24T21:09:10-0700] [MainThread] [I] [MiniWDLContainers] no configuration file found
        [2023-10-24T21:09:10-0700] [MainThread] [N] [MiniWDLContainers] Singularity runtime initialized (BETA) :: singularity_version: "singularity-ce version 3.10.3"
        [2023-10-24T21:09:10-0700] [MainThread] [I] [MiniWDLContainers] detected host resources :: cpu: 128, mem_bytes: 2151678160896
        [2023-10-24T21:17:53-0700] [MainThread] [I] [MiniWDLContainers] Singularity SIF found in image cache directory :: sif: "/private/groups/hprc/scaff_test/HG01099/toil/outputs/cache/.cache/miniwdl/docker___mobinasri_bio_base_v0.2.sif"
        [2023-10-24T21:17:54-0700] [MainThread] [W] [toil.fileStores.abstractFileStore] Failed job accessed files:
        [2023-10-24T21:17:54-0700] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/no-job/file-b0b082d222be423399129ff7dc462883/HG01099.m64043_210707_032602.dc.q20.fastq.gz' to path '/data/tmp/0af5f7544b42535fb8aa818e3571d5a0/35fc/b5ff/tmplqsr4ujg/HG01099.m64043_210707_032602.dc.q20.fastq.gz'
        Traceback (most recent call last):
          File "/private/home/juklucas/.local/lib/python3.8/site-packages/toil/worker.py", line 403, in workerScript
            job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer)
          File "/private/home/juklucas/.local/lib/python3.8/site-packages/toil/job.py", line 2774, in _runner
            returnValues = self._run(jobGraph=None, fileStore=fileStore)
          File "/private/home/juklucas/.local/lib/python3.8/site-packages/toil/job.py", line 2691, in _run
            return self.run(fileStore)
          File "/private/home/juklucas/.local/lib/python3.8/site-packages/toil/wdl/wdltoil.py", line 1239, in run
            task_container._pull(miniwdl_logger, cleanup)
          File "/private/home/juklucas/miniconda3/lib/python3.8/contextlib.py", line 120, in __exit__
            next(self.gen)
          File "/private/home/juklucas/.local/lib/python3.8/site-packages/toil/lib/threading.py", line 410, in global_mutex
            os.unlink(lock_filename)
        FileNotFoundError: [Errno 2] No such file or directory: '/private/groups/hprc/scaff_test/HG01099/toil/outputs/cache/.cache/miniwdl/toil-mutex-toil_miniwdl_sif_cache_mutex'
        [2023-10-24T21:17:54-0700] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host phoenix-08.prism
<=========
```

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1435)
┆Issue Number: TOIL-1435

Toil's `global_mutex` needs to tolerate `OSError: [Errno 116] Stale file handle` in its locking code 
The Toil `global_mutex` is designed to work on a lock file that is deleted when the lock is released, so as not to require a separate cleanup step.

To tolerate this, when we open and lock the file, we check to make sure that the file is still in the directory at the name we opened it with. If not, we managed to catch a pre-existing lock file exactly as it was being deleted, and we didn't try to lock it until the previous holder had unlocked it, and we're holding a worthless lock on an invisible file.

https://github.com/DataBiosphere/toil/blob/9126a4e953b5e6225cae3d524f057bad971db554/src/toil/lib/threading.py#L339-L344

But that's only how it works on normal filesystems. On NFS, instead of letting us hold a deleted file open, the file handle goes stale, and when we try to check if the file is in place still we get an error.

We need to catch that error and treat it the same as the file having been removed (since that's what it indicates), and go back to the beginning of the locking loop.



┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1061)
┆Issue Number: TOIL-1061

Toil's `global_mutex` needs to tolerate `OSError: [Errno 116] Stale file handle` in its locking code 
The Toil `global_mutex` is designed to work on a lock file that is deleted when the lock is released, so as not to require a separate cleanup step.

To tolerate this, when we open and lock the file, we check to make sure that the file is still in the directory at the name we opened it with. If not, we managed to catch a pre-existing lock file exactly as it was being deleted, and we didn't try to lock it until the previous holder had unlocked it, and we're holding a worthless lock on an invisible file.

https://github.com/DataBiosphere/toil/blob/9126a4e953b5e6225cae3d524f057bad971db554/src/toil/lib/threading.py#L339-L344

But that's only how it works on normal filesystems. On NFS, instead of letting us hold a deleted file open, the file handle goes stale, and when we try to check if the file is in place still we get an error.

We need to catch that error and treat it the same as the file having been removed (since that's what it indicates), and go back to the beginning of the locking loop.



┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1061)
┆Issue Number: TOIL-1061



Discussion:

Here's @gmloose's example error from Gitter:

```
Traceback (most recent call last):
  File "/project/rapthor/Software/rapthor/bin/_toil_worker", line 8, in <module>
    sys.exit(main())
  File "/project/rapthor/Software/rapthor/lib/python3.6/site-packages/toil/worker.py", line 710, in main
    with in_contexts(options.context):
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/project/rapthor/Software/rapthor/lib/python3.6/site-packages/toil/worker.py", line 684, in in_contexts
    with manager:
  File "/project/rapthor/Software/rapthor/lib/python3.6/site-packages/toil/batchSystems/abstractBatchSystem.py", line 505, in __enter__
    self.arena.enter()
  File "/project/rapthor/Software/rapthor/lib/python3.6/site-packages/toil/lib/threading.py", line 438, in enter
    with global_mutex(self.workDir, self.mutex):
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/project/rapthor/Software/rapthor/lib/python3.6/site-packages/toil/lib/threading.py", line 340, in global_mutex
    fd_stats = os.fstat(fd)
OSError: [Errno 116] Stale file handle
```
There's a related NFS issue in `io.py`, which triggers a "Device busy" error in `robust_rmtree()`, because it tries to clean-up NFS lock files (these are files whose name start with `.nfs` followed by a long hexadecimal string). I reported that error in Gitter as well. 
I would like to create a PR that addresses both the "Stale file handle" error in `threading.py` and the "Device busy error" in `io.py`, If that's OK with you.
@gmloose Please do so, yes. Thanks!
I'm wondering, though, if there's not a more fundamental problem at hand here. These errors seems to be the result of a classic race condition. It seems as if there are multiple processes (maybe even on different hosts) that try to clean up. Is there not a single owner of those files? I'm probably missing some insight knowledge of what and how here.

And I have a question. A stale file handle error means the file is already gone, which is good: no need for further action. A device busy error, on the other hand, indicates that some other process is still having a file handle to the file being deleted. If I ignore that error, the file (or directory) may not be cleaned up at all. Or will the other process also try to clean up?


I think I can safely claim that `global_mutex` is broken when used on an NFS file system. Besides regular `OSError: [Errno 16] Device or resource busy` and `OSError: [Errno 116] Stale file handle` errors, I caught the following trace-back yesterday:
```
        Traceback (most recent call last):
          File "/project/rapthor/Software/rapthor/bin/_toil_worker", line 8, in <module>
            sys.exit(main())
          File "/project/rapthor/Software/rapthor/lib/python3.6/site-packages/toil/worker.py", line 710, in main
            with in_contexts(options.context):
          File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
            return next(self.gen)
          File "/project/rapthor/Software/rapthor/lib/python3.6/site-packages/toil/worker.py", line 684, in in_contexts
            with manager:
          File "/project/rapthor/Software/rapthor/lib/python3.6/site-packages/toil/batchSystems/cleanup_support.py", line 87, in __enter__
            self.arena.enter()
          File "/project/rapthor/Software/rapthor/lib/python3.6/site-packages/toil/lib/threading.py", line 455, in enter
            fcntl.lockf(self.lockfileFD, fcntl.LOCK_EX) # type: ignore
          File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
            next(self.gen)
          File "/project/rapthor/Software/rapthor/lib/python3.6/site-packages/toil/lib/threading.py", line 373, in global_mutex
            os.unlink(lock_filename)
        FileNotFoundError: [Errno 2] No such file or directory: '/project/rapthor/Share/prefactor/L667520/working/toil-mutex-820dc778-1c66-4891-9f18-219ee085fdf5-cleanup-arena-lock'
 ```
This traceback is very worrysome. The code assumes that it has sole ownership over the file, because it's supposed to own an exclusive lock on it, but the file is gone when the call to `os.unlink()` is done.

Furthermore, when googling for `lockf nfs` you can find many links that tell you that `lockf` and `flock` don't work reliably on NFS. See for example https://stackoverflow.com/a/38239456/16350552. The package [flufl.lock](https://pypi.org/project/flufl.lock/) may provide a possible solution to reimplement `global_mutex`. 
Here's @gmloose's example error from Gitter:

```
Traceback (most recent call last):
  File "/project/rapthor/Software/rapthor/bin/_toil_worker", line 8, in <module>
    sys.exit(main())
  File "/project/rapthor/Software/rapthor/lib/python3.6/site-packages/toil/worker.py", line 710, in main
    with in_contexts(options.context):
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/project/rapthor/Software/rapthor/lib/python3.6/site-packages/toil/worker.py", line 684, in in_contexts
    with manager:
  File "/project/rapthor/Software/rapthor/lib/python3.6/site-packages/toil/batchSystems/abstractBatchSystem.py", line 505, in __enter__
    self.arena.enter()
  File "/project/rapthor/Software/rapthor/lib/python3.6/site-packages/toil/lib/threading.py", line 438, in enter
    with global_mutex(self.workDir, self.mutex):
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/project/rapthor/Software/rapthor/lib/python3.6/site-packages/toil/lib/threading.py", line 340, in global_mutex
    fd_stats = os.fstat(fd)
OSError: [Errno 116] Stale file handle
```
There's a related NFS issue in `io.py`, which triggers a "Device busy" error in `robust_rmtree()`, because it tries to clean-up NFS lock files (these are files whose name start with `.nfs` followed by a long hexadecimal string). I reported that error in Gitter as well. 
I would like to create a PR that addresses both the "Stale file handle" error in `threading.py` and the "Device busy error" in `io.py`, If that's OK with you.
@gmloose Please do so, yes. Thanks!
I'm wondering, though, if there's not a more fundamental problem at hand here. These errors seems to be the result of a classic race condition. It seems as if there are multiple processes (maybe even on different hosts) that try to clean up. Is there not a single owner of those files? I'm probably missing some insight knowledge of what and how here.

And I have a question. A stale file handle error means the file is already gone, which is good: no need for further action. A device busy error, on the other hand, indicates that some other process is still having a file handle to the file being deleted. If I ignore that error, the file (or directory) may not be cleaned up at all. Or will the other process also try to clean up?


I think I can safely claim that `global_mutex` is broken when used on an NFS file system. Besides regular `OSError: [Errno 16] Device or resource busy` and `OSError: [Errno 116] Stale file handle` errors, I caught the following trace-back yesterday:
```
        Traceback (most recent call last):
          File "/project/rapthor/Software/rapthor/bin/_toil_worker", line 8, in <module>
            sys.exit(main())
          File "/project/rapthor/Software/rapthor/lib/python3.6/site-packages/toil/worker.py", line 710, in main
            with in_contexts(options.context):
          File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
            return next(self.gen)
          File "/project/rapthor/Software/rapthor/lib/python3.6/site-packages/toil/worker.py", line 684, in in_contexts
            with manager:
          File "/project/rapthor/Software/rapthor/lib/python3.6/site-packages/toil/batchSystems/cleanup_support.py", line 87, in __enter__
            self.arena.enter()
          File "/project/rapthor/Software/rapthor/lib/python3.6/site-packages/toil/lib/threading.py", line 455, in enter
            fcntl.lockf(self.lockfileFD, fcntl.LOCK_EX) # type: ignore
          File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
            next(self.gen)
          File "/project/rapthor/Software/rapthor/lib/python3.6/site-packages/toil/lib/threading.py", line 373, in global_mutex
            os.unlink(lock_filename)
        FileNotFoundError: [Errno 2] No such file or directory: '/project/rapthor/Share/prefactor/L667520/working/toil-mutex-820dc778-1c66-4891-9f18-219ee085fdf5-cleanup-arena-lock'
 ```
This traceback is very worrysome. The code assumes that it has sole ownership over the file, because it's supposed to own an exclusive lock on it, but the file is gone when the call to `os.unlink()` is done.

Furthermore, when googling for `lockf nfs` you can find many links that tell you that `lockf` and `flock` don't work reliably on NFS. See for example https://stackoverflow.com/a/38239456/16350552. The package [flufl.lock](https://pypi.org/project/flufl.lock/) may provide a possible solution to reimplement `global_mutex`. 

Resolving patch:
diff --git a/src/toil/lib/threading.py b/src/toil/lib/threading.py
--- a/src/toil/lib/threading.py
+++ b/src/toil/lib/threading.py
@@ -16,6 +16,7 @@
 # Note: renamed from "threading.py" to "threading.py" to avoid conflicting imports
 # from the built-in "threading" from psutil in python3.9
 import atexit
+import errno
 import fcntl
 import logging
 import math
@@ -358,6 +359,9 @@ def global_mutex(base_dir: str, mutex: str) -> Iterator[None]:
     :param str mutex: Mutex to lock. Must be a permissible path component.
     """
 
+    if not os.path.isdir(base_dir):
+        raise RuntimeError(f"Directory {base_dir} for mutex does not exist")
+
     # Define a filename
     lock_filename = os.path.join(base_dir, 'toil-mutex-' + mutex)
 
@@ -368,18 +372,32 @@ def global_mutex(base_dir: str, mutex: str) -> Iterator[None]:
     # get a lock on the deleted file.
 
     while True:
-        fd = -1
-
-        try:
-            # Try to create the file, ignoring if it exists or not.
-            fd = os.open(lock_filename, os.O_CREAT | os.O_WRONLY)
+        # Try to create the file, ignoring if it exists or not.
+        fd = os.open(lock_filename, os.O_CREAT | os.O_WRONLY)
 
-            # Wait until we can exclusively lock it.
-            fcntl.lockf(fd, fcntl.LOCK_EX)
+        # Wait until we can exclusively lock it.
+        fcntl.lockf(fd, fcntl.LOCK_EX)
 
-            # Holding the lock, make sure we are looking at the same file on disk still.
+        # Holding the lock, make sure we are looking at the same file on disk still.
+        try:
+            # So get the stats from the open file
             fd_stats = os.fstat(fd)
+        except OSError as e:
+            if e.errno == errno.ESTALE:
+                # The file handle has gone stale, because somebody removed the file.
+                # Try again.
+                try:
+                    fcntl.lockf(fd, fcntl.LOCK_UN)
+                except OSError:
+                    pass
+                os.close(fd)
+                continue
+            else:
+                # Something else broke
+                raise
 
+        try:
+            # And get the stats for the name in the directory
             path_stats: Optional[os.stat_result] = os.stat(lock_filename)
         except FileNotFoundError:
             path_stats = None
@@ -389,10 +407,9 @@ def global_mutex(base_dir: str, mutex: str) -> Iterator[None]:
             # any). This usually happens, because before someone releases a
             # lock, they delete the file. Go back and contend again. TODO: This
             # allows a lot of queue jumping on our mutex.
-            if fd != -1:
-                fcntl.lockf(fd, fcntl.LOCK_UN)
-                os.close(fd)
-                continue
+            fcntl.lockf(fd, fcntl.LOCK_UN)
+            os.close(fd)
+            continue
         else:
             # We have a lock on the file that the name points to. Since we
             # hold the lock, nobody will be deleting it or can be in the
@@ -407,14 +424,40 @@ def global_mutex(base_dir: str, mutex: str) -> Iterator[None]:
         # Delete it while we still own it, so we can't delete it from out from
         # under someone else who thinks they are holding it.
         logger.debug('PID %d releasing mutex %s', os.getpid(), lock_filename)
-        os.unlink(lock_filename)
-        if fd != -1:
-            fcntl.lockf(fd, fcntl.LOCK_UN)
-            # Note that we are unlinking it and then unlocking it; a lot of people
-            # might have opened it before we unlinked it and will wake up when they
-            # get the worthless lock on the now-unlinked file. We have to do some
-            # stat gymnastics above to work around this.
-            os.close(fd)
+
+        # We have had observations in the wild of the lock file not exisiting
+        # when we go to unlink it, causing a crash on mutex release. See
+        # <https://github.com/DataBiosphere/toil/issues/4654>.
+        #
+        # We want to tolerate this; maybe unlink() interacts with fcntl() locks
+        # on NFS in a way that is actually fine, somehow? But we also want to
+        # complain loudly if something is tampering with our locks or not
+        # really enforcing locks on the filesystem, so we will notice if it is
+        # the cause of further problems.
+        try:
+            path_stats = os.stat(lock_filename)
+        except FileNotFoundError:
+            path_stats = None
+
+        # Check to make sure it still looks locked before we unlink.
+        if path_stats is None:
+            logger.error('PID %d had mutex %s disappear while locked! Mutex system is not working!', os.getpid(), lock_filename)
+        elif fd_stats.st_dev != path_stats.st_dev or fd_stats.st_ino != path_stats.st_ino:
+            logger.error('PID %d had mutex %s get replaced while locked! Mutex system is not working!', os.getpid(), lock_filename)
+
+        if path_stats is not None:
+            try:
+                # Unlink the file
+                os.unlink(lock_filename)
+            except FileNotFoundError:
+                logger.error('PID %d had mutex %s disappear between stat and unlink while unlocking! Mutex system is not working!', os.getpid(), lock_filename)
+
+        # Note that we are unlinking it and then unlocking it; a lot of people
+        # might have opened it before we unlinked it and will wake up when they
+        # get the worthless lock on the now-unlinked file. We have to do some
+        # stat gymnastics above to work around this.
+        fcntl.lockf(fd, fcntl.LOCK_UN)
+        os.close(fd)
 
 
 class LastProcessStandingArena:


Test patch: