Repository: databiosphere/toil
Issue: NoSuchFileException

I ran into a weird issue with this exception. JobServiceTest passed, I rebased on master, and the test then failed with this exception. I haven't been able to replicate this locally and I seem to remember seeing another unexpected case with this exception - I will look through old issues and link it if I find it. Is it possible that file creation on the file based jobStore is not atomic?

https://github.com/BD2KGenomics/toil/issues/320
WARNING:toil.leader:The job seems to have left a log file, indicating failure: K/S/jobf8UECq
WARNING:toil.leader:Reporting file: K/S/jobf8UECq
WARNING:toil.leader:K/S/jobf8UECq:  ---TOIL WORKER OUTPUT LOG---
WARNING:toil.leader:K/S/jobf8UECq:  Traceback (most recent call last):
WARNING:toil.leader:K/S/jobf8UECq:    File "/mnt/ephemeral/workspace/toil/src/toil/worker.py", line 237, in main
WARNING:toil.leader:K/S/jobf8UECq:      jobStore=jobStore)
WARNING:toil.leader:K/S/jobf8UECq:    File "/mnt/ephemeral/workspace/toil/src/toil/job.py", line 848, in _execute
WARNING:toil.leader:K/S/jobf8UECq:      returnValues = self.run(fileStore)
WARNING:toil.leader:K/S/jobf8UECq:    File "/mnt/ephemeral/workspace/toil/src/toil/job.py", line 1020, in run
WARNING:toil.leader:K/S/jobf8UECq:      while fileStore.globalFileExists(self.stopFileStoreID):
WARNING:toil.leader:K/S/jobf8UECq:    File "/mnt/ephemeral/workspace/toil/src/toil/job.py", line 440, in globalFileExists
WARNING:toil.leader:K/S/jobf8UECq:      return self.jobStore.fileExists(fileStoreID)
WARNING:toil.leader:K/S/jobf8UECq:    File "/mnt/ephemeral/workspace/toil/src/toil/jobStores/fileJobStore.py", line 169, in fileExists
WARNING:toil.leader:K/S/jobf8UECq:      raise NoSuchFileException("Path %s is not a file in the jobStore" % jobStoreFileID)
WARNING:toil.leader:K/S/jobf8UECq:  NoSuchFileException: The file 'Path F/q/jobuD5wtI/g/tmpb2miC_.tmp is not a file in the jobStore' does not exist
WARNING:toil.leader:K/S/jobf8UECq:  Exiting the worker because of a failed job on host ip-172-31-23-35
WARNING:toil.leader:K/S/jobf8UECq:  ERROR:**main**:Exiting the worker because of a failed job on host ip-172-31-23-35
WARNING:toil.leader:K/S/jobf8UECq:  WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job K/S/jobf8UECq to 0
WARNING:toil.leader:Job: K/S/jobf8UECq is completely failed


Changed files:
src/toil/jobStores/fileJobStore.py

diff --git a/src/toil/jobStores/fileJobStore.py b/src/toil/jobStores/fileJobStore.py
index ae13d623..cfdc2b46 100644
--- a/src/toil/jobStores/fileJobStore.py
+++ b/src/toil/jobStores/fileJobStore.py
@@ -22,6 +22,7 @@ import random
 import shutil
 import os
 import tempfile
+import stat
 from toil.lib.bioio import absSymPath
 from toil.jobStores.abstractJobStore import AbstractJobStore, NoSuchJobException, \\
     NoSuchFileException
@@ -163,10 +164,12 @@ class FileJobStore(AbstractJobStore):
         
     def fileExists(self, jobStoreFileID):
         absPath = self._getAbsPath(jobStoreFileID)
-        if not os.path.exists(absPath):
+        try:
+            st = os.stat(absPath)
+        except os.error:
             return False
-        if not os.path.isfile(absPath):
-            raise NoSuchFileException("Path %s is not a file in the jobStore" % jobStoreFileID) 
+        if not stat.S_ISREG(st.st_mode):
+            raise NoSuchFileException("Path %s is not a file in the jobStore" % jobStoreFileID)
         return True
     
     @contextmanager
@@ -286,11 +289,8 @@ class FileJobStore(AbstractJobStore):
         """
         Raises NoSuchFileException if the jobStoreFileID does not exist or is not a file.
         """
-        absPath = os.path.join(self.tempFilesDir, jobStoreFileID)
-        if not os.path.exists(absPath):
+        if not self.fileExists(jobStoreFileID):
             raise NoSuchFileException("File %s does not exist in jobStore" % jobStoreFileID)
-        if not os.path.isfile(absPath):
-            raise NoSuchFileException("Path %s is not a file in the jobStore" % jobStoreFileID) 
     
     def _getTempSharedDir(self):
         """