Repository: DataBiosphere/toil

Issue:
Sporadic NoSuchJobException in sort tests
http://jenkins.cgcloud.info/job/toil-pull-requests/1017/testReport/junit/src.toil.test.sort.sortTest/SortTest/testAwsSingle/

<!---
@huboard:{"milestone_order":253.97460253974603}
-->



Discussion:
Another one: http://jenkins.cgcloud.info/job/toil-pull-requests/1164/testReport/junit/src.toil.test.sort.sortTest/SortTest/testAwsSingle/

AWS with mesos: http://jenkins.cgcloud.info/job/toil-pull-requests/1205/testReport/junit/src.toil.test.sort.sortTest/SortTest/testAwsMesos/

I've added a Lucene indexer to Jenkins so we can do full text search on build results:

http://jenkins.cgcloud.info/search/?q=NoSuchJobException

http://jenkins.cgcloud.info/job/toil-branches/branch/master/167/testReport/junit/src.toil.test.sort.sortTest/SortTest/testAwsMesos/

http://jenkins.cgcloud.info/job/toil-pull-requests/1445/testReport/junit/src.toil.test.sort.sortTest/SortTest/testAwsMesos/

Hmm, I was sort of hoping the fix to #808 would fix this too, but after looking at it, it looks like it's either a race condition (exists check followed right after by a load) or stale reads from SDB. I tried to aggravate the race locally by sleeping in between but no dice.

edit: Now that I think about it, you could just do a single load and catch the NoSuchJobException rather than checking existence separately, but that seems like sweeping the problem under the rug.

I think this is a stale read after delete. We've been in touch with AWS support about this. They didn't outright reject the idea of SDB not being fully consistent and recommended that we upgrade to DynamoDB. 

I wonder is why this happens in the same code location every time.

… meaning that if it is a stale read, the time window should be somewhat flexible and therefore cause different problems in different places.

http://jenkins.cgcloud.info/job/toil-pull-requests/1478/testReport/junit/src.toil.test.sort.sortTest/SortTest/testAwsSingle/

I've been trying to brute-force reproduce this by creating and deleting jobs in quick succession. I tried 16k jobs. No dice. We should entertain the possibility of there being a the possibility of the job being deleted concurrently, maybe by the worker. Then again, why does this only happen with AWS? If there is a race between leader and worker, this should manifest itself in other stores as well/

Punting for next sprint.

Given that this is most likely  a SDB issue, I like @joelarmstrong's solution. We can fix this particular symptom to curb the problem until we switch to DynamoDB, if/when that happens.


Resolving patch:
diff --git a/src/toil/leader.py b/src/toil/leader.py
--- a/src/toil/leader.py
+++ b/src/toil/leader.py
@@ -30,6 +30,8 @@
 from bd2k.util.expando import Expando
 
 from toil import resolveEntryPoint
+from toil.jobStores.abstractJobStore import NoSuchJobException
+from toil.jobStores.aws.jobStore import AWSJobStore
 from toil.lib.bioio import getTotalCpuTime, logStream
 from toil.provisioners.clusterScaler import ClusterScaler
 
@@ -265,13 +267,32 @@ def processFinishedJob(self, jobBatchSystemID, resultStatus, wallTime=None):
         """
         Function reads a processed jobWrapper file and updates it state.
         """
+        def processRemovedJob(jobStoreID):
+            if resultStatus != 0:
+                logger.warn("Despite the batch system claiming failure the "
+                            "jobWrapper %s seems to have finished and been removed", jobStoreID)
+            self._updatePredecessorStatus(jobStoreID)
+
         if wallTime is not None and self.clusterScaler is not None:
             issuedJob = self.jobBatchSystemIDToIssuedJob[jobBatchSystemID]
             self.clusterScaler.addCompletedJob(issuedJob, wallTime)
         jobStoreID = self.removeJobID(jobBatchSystemID)
         if self.jobStore.exists(jobStoreID):
             logger.debug("Job %s continues to exist (i.e. has more to do)" % jobStoreID)
-            jobWrapper = self.jobStore.load(jobStoreID)
+            try:
+                jobWrapper = self.jobStore.load(jobStoreID)
+            except NoSuchJobException:
+                if isinstance(self.jobStore, AWSJobStore):
+                    # We have a ghost job - the job has been deleted but a stale read from
+                    # SDB gave us a false positive when we checked for its existence.
+                    # Process the job from here as any other job removed from the job store.
+                    # This is a temporary work around until https://github.com/BD2KGenomics/toil/issues/1091
+                    # is completed
+                    logger.warn('Got a stale read from SDB for job %s', jobStoreID)
+                    processRemovedJob(jobStoreID)
+                    return
+                else:
+                    raise
             if jobWrapper.logJobStoreFileID is not None:
                 logger.warn("The jobWrapper seems to have left a log file, indicating failure: %s", jobStoreID)
                 with jobWrapper.getLogFileHandle( self.jobStore ) as logFileStream:
@@ -292,10 +313,7 @@ def processFinishedJob(self, jobBatchSystemID, resultStatus, wallTime=None):
             #jobWrapper is done we can add it to the list of updated jobWrapper files
             logger.debug("Added jobWrapper: %s to active jobs", jobStoreID)
         else:  #The jobWrapper is done
-            if resultStatus != 0:
-                logger.warn("Despite the batch system claiming failure the "
-                            "jobWrapper %s seems to have finished and been removed", jobStoreID)
-            self._updatePredecessorStatus(jobStoreID)
+            processRemovedJob(jobStoreID)
 
     def processTotallyFailedJob(self, jobWrapper):
         """


Test patch:
diff --git a/src/toil/test/jobStores/jobStoreTest.py b/src/toil/test/jobStores/jobStoreTest.py
--- a/src/toil/test/jobStores/jobStoreTest.py
+++ b/src/toil/test/jobStores/jobStoreTest.py
@@ -854,10 +854,7 @@ def testInlinedFiles(self):
                     self.assertEqual(s, f.read())
 
     def testInaccessableLocation(self):
-        # This could break if 1000genomes enables GetBucketLocation in their S3 bucket policy,
-        # see https://rt.sanger.ac.uk/Ticket/Display.html?id=534925. If that happens, we'll have
-        # to set up or mock a bucket that disallows it.
-        url = 's3://1000genomes/20131219.populations.tsv'
+        url = 's3://cgl-toil-test-disallow-getbucketlocation/README'
         with patch('toil.jobStores.aws.jobStore.log') as mock_log:
             jobStoreID = self.master.importFile(url)
             self.assertTrue(self.master.fileExists(jobStoreID))