Repository: DataBiosphere/toil

Issue:
No such file error
WARNING:jobTree.leader:Reporting file: B/Z/jobfW0NrB
WARNING:jobTree.leader:B/Z/jobfW0NrB:   ---JOBTREE WORKER OUTPUT LOG---
WARNING:jobTree.leader:B/Z/jobfW0NrB:   Next available file descriptor: 15
WARNING:jobTree.leader:B/Z/jobfW0NrB:   DEBUG:**main**:Next available file descriptor: 15
WARNING:jobTree.leader:B/Z/jobfW0NrB:   Parsed job
WARNING:jobTree.leader:B/Z/jobfW0NrB:   DEBUG:**main**:Parsed job
WARNING:jobTree.leader:B/Z/jobfW0NrB:   Traceback (most recent call last):
WARNING:jobTree.leader:B/Z/jobfW0NrB:    File "/Users/cj/jobTree/src/jobTree/worker.py", line 365, in main
WARNING:jobTree.leader:B/Z/jobfW0NrB:      jobStore.writeStatsAndLogging(ET.tostring(stats))
WARNING:jobTree.leader:B/Z/jobfW0NrB:    File "/Users/cj/jobTree/src/jobTree/jobStores/fileJobStore.py", line 193, in writeStatsAndLogging
WARNING:jobTree.leader:B/Z/jobfW0NrB:      os.rename(tempStatsFile, tempStatsFile[:-4]) #This operation is atomic
WARNING:jobTree.leader:B/Z/jobfW0NrB:   OSError: [Errno 2] No such file or directory
WARNING:jobTree.leader:B/Z/jobfW0NrB:   Exiting the worker because of a failed job on host dhcp-169-233-153-7.cruznet.ucsc.edu
WARNING:jobTree.leader:B/Z/jobfW0NrB:   ERROR:**main**:Exiting the worker because of a failed job on host dhcp-169-233-153-7.cruznet.ucsc.edu
WARNING:jobTree.leader:B/Z/jobfW0NrB:   WARNING:jobTree.job:Due to failure we are reducing the remaining retry count of job B/Z/jobfW0NrB to 99
DEBUG:jobTree.leader:Added job: B/Z/jobfW0NrB to active jobs



Discussion:
To replicate: use DAG-rebased branch, run StatsTest with success_ratio set to 1 in sort.py. I've run this several times, and seen the failure 1-3 times per test. The test ultimately succeeds. 

This is also similar to the errors that I am seeing when trying to run awsSortTest, although I have not yet pushed the syntax fixes necessary to get awsSortTest to start.

Is this still a bug?

@cket, can you please try to reproduce again?

I just reproduced this following the steps above, so it is still a a bug. Seems to be stats related, although they are being displayed successfully at the end.

I can't reproduce this on the master branch. I use sortTest.py, set retryCount=0 set successRatio=1 and enable stats as an option.

I am assigning to CJ, as I can't reproduce it and I've been through the logic and don't see the bug. As this code hasn't changed I suspect there is still a bug, I just can't trap it.

I tried reproducing this again and I can't. I am seeing new tracebacks complaining about restarting, but it is still completing so these seem to be expected:

Traceback (most recent call last):
  File "/Users/cj/toil/src/toil/test/sort/sort.py", line 113, in <module>
    main()
  File "/Users/cj/toil/src/toil/test/sort/sort.py", line 110, in main
    Job.Runner.startToil(Job.wrapJobFn(setup, options.fileToSort, int(options.N)), options)
  File "/Users/cj/toil/src/toil/job.py", line 354, in startToil
    with setupToil(options, userScript=job.getUserScript()) as (config, batchSystem, jobStore):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17, in **enter**
    return self.gen.next()
  File "/Users/cj/toil/src/toil/common.py", line 407, in setupToil
    jobStore = loadJobStore(config.jobStore, config=config)
  File "/Users/cj/toil/src/toil/common.py", line 376, in loadJobStore
    return FileJobStore( jobStoreArgs, config=config )
  File "/Users/cj/toil/src/toil/jobStores/fileJobStore.py", line 35, in **init**
    self._checkJobStoreCreation(config != None, os.path.exists(self.jobStoreDir), self.jobStoreDir)
  File "/Users/cj/toil/src/toil/jobStores/abstractJobStore.py", line 77, in _checkJobStoreCreation
    "else remove it to start from scratch" % jobStoreString)

Those are expected errors, which are being tested for. I think I worked out the source of the original error. It was stupid. When my PR for #256 is in this will be fixed.


Resolving patch:
diff --git a/src/toil/job.py b/src/toil/job.py
--- a/src/toil/job.py
+++ b/src/toil/job.py
@@ -837,7 +837,7 @@ def _execute(self, jobWrapper, stats, localTempDir, jobStore):
             stats.attrib["time"] = str(time.time() - startTime)
             totalCpuTime, totalMemoryUsage = getTotalCpuTimeAndMemoryUsage()
             stats.attrib["clock"] = str(totalCpuTime - startClock)
-            stats.attrib["class"] = ".".join((self.__class__.__name__,))
+            stats.attrib["class"] = self._jobName()
             stats.attrib["memory"] = str(totalMemoryUsage)
         #Return any logToMaster logging messages
         return fileStore.loggingMessages
@@ -870,6 +870,12 @@ def _resolveMainModule( moduleName ):
                 "Can only handle main modules loaded from .py or .pyc files, but not '%s'" %
                 moduleName)
         return moduleDirPath, moduleName
+    
+    def _jobName(self):
+        """
+        :rtype : string, used as identifier of the job class in the stats report. 
+        """
+        return self.__class__.__name__
 
 class JobGraphDeadlockException( Exception ):
     def __init__( self, string ):
@@ -912,6 +918,9 @@ def run(self,fileStore):
 
     def getUserScript(self):
         return self.userFunctionModule
+    
+    def _jobName(self):
+        return ".".join((self.__class__.__name__,self.userFunctionModule.name,self.userFunctionName))
 
 class JobFunctionWrappingJob(FunctionWrappingJob):
     """
diff --git a/src/toil/jobStores/fileJobStore.py b/src/toil/jobStores/fileJobStore.py
--- a/src/toil/jobStores/fileJobStore.py
+++ b/src/toil/jobStores/fileJobStore.py
@@ -220,7 +220,7 @@ def readStatsAndLogging( self, statsAndLoggingCallBackFn):
                         with open(absTempFile, 'r') as fH:
                             statsAndLoggingCallBackFn(fH)
                         numberOfFilesProcessed += 1
-                    os.remove(absTempFile)
+                        os.remove(absTempFile)
         return numberOfFilesProcessed
     
     ##########################################


Test patch:
diff --git a/src/toil/test/sort/sort.py b/src/toil/test/sort/sort.py
--- a/src/toil/test/sort/sort.py
+++ b/src/toil/test/sort/sort.py
@@ -10,7 +10,7 @@
 from toil.job import Job
 from toil.test.sort.lib import merge, sort, copySubRangeOfFile, getMidPoint
 
-success_ratio = 0.5
+success_ratio = 1.0 #0.5
 
 def setup(job, inputFile, N):
     """Sets up the sort.
diff --git a/src/toil/test/sort/sortTest.py b/src/toil/test/sort/sortTest.py
--- a/src/toil/test/sort/sortTest.py
+++ b/src/toil/test/sort/sortTest.py
@@ -206,18 +206,15 @@ def checkEqual(i, j):
         print "false", j
     assert i == j
 
-
 def loadFile(file):
     with open(file, 'r') as fileHandle:
         return fileHandle.readlines()
 
-
 def getRandomLine(maxLineLength):
     return "".join(
         [random.choice(['a', 'c', 't', 'g', "A", "C", "G", "T", "N", "X", "Y", "Z"]) for i in
          xrange(maxLineLength)]) + "\n"
 
-
 def makeFileToSort(fileName, lines=10, maxLineLength=10):
     with open(fileName, 'w') as fileHandle:
         for line in xrange(lines):
diff --git a/src/toil/test/utils/utilsTest.py b/src/toil/test/utils/utilsTest.py
--- a/src/toil/test/utils/utilsTest.py
+++ b/src/toil/test/utils/utilsTest.py
@@ -10,7 +10,6 @@
 from toil.common import toilPackageDirPath
 from toil.test import ToilTest
 
-
 class UtilsTest(ToilTest):
     """
     Tests the scriptTree toil-script compiler.
@@ -18,82 +17,106 @@ class UtilsTest(ToilTest):
 
     def setUp(self):
         super(UtilsTest, self).setUp()
-        self.testNo = 1
+        self.tempDir = getTempDirectory(os.getcwd())
+        self.tempFile = getTempFile(rootDir=self.tempDir)
+        self.outputFile = getTempFile(rootDir=self.tempDir)
+        self.toilDir = os.path.join(self.tempDir, "testToil")
+        self.assertFalse(os.path.exists(self.toilDir))
+        self.lines = 10000
+        self.maxLineLength = 10
+        self.N = 1000
+        makeFileToSort(self.tempFile, self.lines, self.maxLineLength)
+        # First make our own sorted version
+        with open(self.tempFile, 'r') as fileHandle:
+            self.correctSort = fileHandle.readlines()
+            self.correctSort.sort()
+            
+    def tearDown(self):
+        ToilTest.tearDown(self)
+        system("rm -rf %s" % self.tempDir)
 
     def testUtilsSort(self):
         """
         Tests the restart, status and stats commands of the toil command line utility using the
         sort example.
         """
-        for test in xrange(self.testNo):
-            tempDir = getTempDirectory(os.getcwd())
-            tempFile = getTempFile(rootDir=tempDir)
-            outputFile = getTempFile(rootDir=tempDir)
-            toilDir = os.path.join(tempDir, "testToil")
-            self.assertFalse(os.path.exists(toilDir))
-            lines = 10000
-            maxLineLength = 10
-            N = 1000
-            makeFileToSort(tempFile, lines, maxLineLength)
-
-            # First make our own sorted version
-            with open(tempFile, 'r') as fileHandle:
-                l = fileHandle.readlines()
-                l.sort()
-
-            # Get the sort command to run
-            rootPath = os.path.join(toilPackageDirPath(), "test", "sort")
-            toilCommandString = ("{rootPath}/sort.py "
-                                 "--jobStore {toilDir} "
-                                 "--logLevel=DEBUG "
-                                 "--fileToSort={tempFile} "
-                                 "--N {N} --stats "
-                                 "--retryCount 2".format(**locals()))
-
-            # Try restarting it to check that a JobStoreException is thrown
-            self.assertRaises(CalledProcessError, system, toilCommandString + " --restart")
-            # Check that trying to run it in restart mode does not create the jobStore
-            self.assertFalse(os.path.exists(toilDir))
-
-            # Run the script for the first time
-            system(toilCommandString)
-            self.assertTrue(os.path.exists(toilDir))
-
-            # Try running it without restart and check an exception is thrown
-            self.assertRaises(CalledProcessError, system, toilCommandString)
-
-            # Now restart it until done
-            while True:
-                # Run the status command
-                rootPath = os.path.join(toilPackageDirPath(), "utils")
-                toilStatusString = ("{rootPath}/toilMain.py status "
-                                    "--jobStore {toilDir} --failIfNotComplete".format(**locals()))
-                try:
-                    system(toilStatusString)
-                except CalledProcessError:
-                    # If there is an exception there are still failed jobs then restart
-                    system(toilCommandString + " --restart")
-                else:
-                    break
-
-
-            # Check if we try to launch after its finished that we get a JobException
-            self.assertRaises(CalledProcessError, system, toilCommandString + " --restart")
-
-            # Check we can run 'toil stats'
+        # Get the sort command to run
+        rootPath = os.path.join(toilPackageDirPath(), "test", "sort")
+        toilCommandString = ("{rootPath}/sort.py "
+                             "--jobStore {self.toilDir} "
+                             "--logLevel=DEBUG "
+                             "--fileToSort={self.tempFile} "
+                             "--N {self.N} --stats "
+                             "--retryCount 2".format(**locals()))
+
+        # Try restarting it to check that a JobStoreException is thrown
+        self.assertRaises(CalledProcessError, system, toilCommandString + " --restart")
+        # Check that trying to run it in restart mode does not create the jobStore
+        self.assertFalse(os.path.exists(self.toilDir))
+
+        # Run the script for the first time
+        system(toilCommandString)
+        self.assertTrue(os.path.exists(self.toilDir))
+
+        # Try running it without restart and check an exception is thrown
+        self.assertRaises(CalledProcessError, system, toilCommandString)
+
+        # Now restart it until done
+        while True:
+            # Run the status command
             rootPath = os.path.join(toilPackageDirPath(), "utils")
-            toilStatsString = ("{rootPath}/toilMain.py stats "
-                               "--jobStore {toilDir} --pretty".format(**locals()))
-            system(toilStatsString)
-
-            # Check the file is properly sorted
-            with open(tempFile, 'r') as fileHandle:
-                l2 = fileHandle.readlines()
-                checkEqual(l, l2)
-
-            # Cleanup
-            system("rm -rf %s" % tempDir)
-
+            toilStatusString = ("{rootPath}/toilMain.py status "
+                                "--jobStore {self.toilDir} --failIfNotComplete".format(**locals()))
+            try:
+                system(toilStatusString)
+            except CalledProcessError:
+                # If there is an exception there are still failed jobs then restart
+                system(toilCommandString + " --restart")
+            else:
+                break
+
+
+        # Check if we try to launch after its finished that we get a JobException
+        self.assertRaises(CalledProcessError, system, toilCommandString + " --restart")
+
+        # Check we can run 'toil stats'
+        rootPath = os.path.join(toilPackageDirPath(), "utils")
+        toilStatsString = ("{rootPath}/toilMain.py stats "
+                           "--jobStore {self.toilDir} --pretty".format(**locals()))
+        system(toilStatsString)
+
+        # Check the file is properly sorted
+        with open(self.tempFile, 'r') as fileHandle:
+            l2 = fileHandle.readlines()
+            checkEqual(self.correctSort, l2)
+            
+    def testUtilsStatsSort(self):
+        """
+        Tests the stats commands on a complete run of the stats test.
+        """
+        # Get the sort command to run
+        rootPath = os.path.join(toilPackageDirPath(), "test", "sort")
+        toilCommandString = ("{rootPath}/sort.py "
+                             "--jobStore {self.toilDir} "
+                             "--logLevel=DEBUG "
+                             "--fileToSort={self.tempFile} "
+                             "--N {self.N} --stats "
+                             "--retryCount 99".format(**locals()))
+    
+        # Run the script for the first time
+        system(toilCommandString)
+        self.assertTrue(os.path.exists(self.toilDir))
+
+        # Check we can run 'toil stats'
+        rootPath = os.path.join(toilPackageDirPath(), "utils")
+        toilStatsString = ("{rootPath}/toilMain.py stats "
+                           "--jobStore {self.toilDir} --pretty".format(**locals()))
+        system(toilStatsString)
+
+        # Check the file is properly sorted
+        with open(self.tempFile, 'r') as fileHandle:
+            l2 = fileHandle.readlines()
+            checkEqual(self.correctSort, l2)
 
 if __name__ == '__main__':
     unittest.main()