Repository: DataBiosphere/toil

Issue:
NoSuchFileException
I ran into a weird issue with this exception. JobServiceTest passed, I rebased on master, and the test then failed with this exception. I haven't been able to replicate this locally and I seem to remember seeing another unexpected case with this exception - I will look through old issues and link it if I find it. Is it possible that file creation on the file based jobStore is not atomic?

https://github.com/BD2KGenomics/toil/issues/320
WARNING:toil.leader:The job seems to have left a log file, indicating failure: K/S/jobf8UECq
WARNING:toil.leader:Reporting file: K/S/jobf8UECq
WARNING:toil.leader:K/S/jobf8UECq:  ---TOIL WORKER OUTPUT LOG---
WARNING:toil.leader:K/S/jobf8UECq:  Traceback (most recent call last):
WARNING:toil.leader:K/S/jobf8UECq:    File "/mnt/ephemeral/workspace/toil/src/toil/worker.py", line 237, in main
WARNING:toil.leader:K/S/jobf8UECq:      jobStore=jobStore)
WARNING:toil.leader:K/S/jobf8UECq:    File "/mnt/ephemeral/workspace/toil/src/toil/job.py", line 848, in _execute
WARNING:toil.leader:K/S/jobf8UECq:      returnValues = self.run(fileStore)
WARNING:toil.leader:K/S/jobf8UECq:    File "/mnt/ephemeral/workspace/toil/src/toil/job.py", line 1020, in run
WARNING:toil.leader:K/S/jobf8UECq:      while fileStore.globalFileExists(self.stopFileStoreID):
WARNING:toil.leader:K/S/jobf8UECq:    File "/mnt/ephemeral/workspace/toil/src/toil/job.py", line 440, in globalFileExists
WARNING:toil.leader:K/S/jobf8UECq:      return self.jobStore.fileExists(fileStoreID)
WARNING:toil.leader:K/S/jobf8UECq:    File "/mnt/ephemeral/workspace/toil/src/toil/jobStores/fileJobStore.py", line 169, in fileExists
WARNING:toil.leader:K/S/jobf8UECq:      raise NoSuchFileException("Path %s is not a file in the jobStore" % jobStoreFileID)
WARNING:toil.leader:K/S/jobf8UECq:  NoSuchFileException: The file 'Path F/q/jobuD5wtI/g/tmpb2miC_.tmp is not a file in the jobStore' does not exist
WARNING:toil.leader:K/S/jobf8UECq:  Exiting the worker because of a failed job on host ip-172-31-23-35
WARNING:toil.leader:K/S/jobf8UECq:  ERROR:**main**:Exiting the worker because of a failed job on host ip-172-31-23-35
WARNING:toil.leader:K/S/jobf8UECq:  WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job K/S/jobf8UECq to 0
WARNING:toil.leader:Job: K/S/jobf8UECq is completely failed



Discussion:
#146 was about this same exception happening intermittently. I think there was another instance shortly after the DAG-rebased branch was created, but I  couldn't find an issue about it.

Rerunning the same build on jenkins passed this time around.

Just perused the source a bit--I think this is a rare data race in the fileExists method for the file store. First it checks that the file exists, then it checks whether it is actually a file and not a directory. This error can happen if the file is deleted by some other process between the first and second check.

Not sure of a good way of making that series of checks atomic (flock?). But it is probably simpler to not make the second check at all, or possibly just make it return False. I don't see the logic behind raising an exception there.

Good find, Joel.

CJ, just use `os.stat()` as that's what `os.path.exists()` and `.isfile()` use under the hood. Study their source to see how `os.stat()` is used in conjunction with the helpers in the `stat` module. 

Would be more efficient to.

Oh, and search for other places in the file job store code where this might happen.


Resolving patch:
diff --git a/src/toil/jobStores/fileJobStore.py b/src/toil/jobStores/fileJobStore.py
--- a/src/toil/jobStores/fileJobStore.py
+++ b/src/toil/jobStores/fileJobStore.py
@@ -22,6 +22,7 @@
 import shutil
 import os
 import tempfile
+import stat
 from toil.lib.bioio import absSymPath
 from toil.jobStores.abstractJobStore import AbstractJobStore, NoSuchJobException, \
     NoSuchFileException
@@ -163,10 +164,12 @@ def deleteFile(self, jobStoreFileID):
         
     def fileExists(self, jobStoreFileID):
         absPath = self._getAbsPath(jobStoreFileID)
-        if not os.path.exists(absPath):
+        try:
+            st = os.stat(absPath)
+        except os.error:
             return False
-        if not os.path.isfile(absPath):
-            raise NoSuchFileException("Path %s is not a file in the jobStore" % jobStoreFileID) 
+        if not stat.S_ISREG(st.st_mode):
+            raise NoSuchFileException("Path %s is not a file in the jobStore" % jobStoreFileID)
         return True
     
     @contextmanager
@@ -286,11 +289,8 @@ def _checkJobStoreFileID(self, jobStoreFileID):
         """
         Raises NoSuchFileException if the jobStoreFileID does not exist or is not a file.
         """
-        absPath = os.path.join(self.tempFilesDir, jobStoreFileID)
-        if not os.path.exists(absPath):
+        if not self.fileExists(jobStoreFileID):
             raise NoSuchFileException("File %s does not exist in jobStore" % jobStoreFileID)
-        if not os.path.isfile(absPath):
-            raise NoSuchFileException("Path %s is not a file in the jobStore" % jobStoreFileID) 
     
     def _getTempSharedDir(self):
         """


Test patch: