Repository: DataBiosphere/toil

Issue:
Mesos BatchSystem dict changing size during iteration
I haven't had time to update Toil yet; I'm on my fork of 3a81f1114ec7f3 which includes environment serialization, and doesn't yet include the idempotent job deletion.

However, I just went to run a job on Mesos on Azure, and got this on my master:

```
Failed to call scheduler's resourceOffer
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/toil/batchSystems/mesos/batchSystem.py", line 359, in resourceOffers
    self._updateStateToRunning(offer, task)
  File "/usr/local/lib/python2.7/dist-packages/toil/batchSystems/mesos/batchSystem.py", line 310, in _updateStateToRunning
    self._deleteByJobID(int(task.task_id.value))
  File "/usr/local/lib/python2.7/dist-packages/toil/batchSystems/mesos/batchSystem.py", line 301, in _deleteByJobID
    for key, jobType in self.jobQueueList.iteritems():
RuntimeError: dictionary changed size during iteration
I0929 00:22:16.111704 31507 sched.cpp:1623] Asked to abort the driver
INFO:realtime:Aligned /tmp/tmpvAfao6/localTempDir/output.gam
I0929 00:22:16.117781 31507 sched.cpp:856] Aborting framework '20150928-184937-83886090-5050-35771-0006'
```

Is it still possible for that dictionary to change size during iteration in the current version?



Discussion:
Did you modify anything under `src/toil/batchSystems/mesos` ?

I did indeed. I added environment serialization and deserialization, so the worker can have environment variables enabling it to connect to the job store (like the AZURE_ACCOUNT_KEY).

See here: https://github.com/adamnovak/toil/commit/bc0e05d3d1e55c19aeec13e576c349e37c409bfd


Resolving patch:
diff --git a/src/toil/batchSystems/mesos/batchSystem.py b/src/toil/batchSystems/mesos/batchSystem.py
--- a/src/toil/batchSystems/mesos/batchSystem.py
+++ b/src/toil/batchSystems/mesos/batchSystem.py
@@ -150,15 +150,15 @@ def getIssuedBatchJobIDs(self):
         """
         A list of jobs (as jobIDs) currently issued (may be running, or maybe just waiting).
         """
-        # TODO: Ensure jobList holds jobs that have been "launched" from Mesos
-        jobList = []
-        for k, queue in self.jobQueueList.iteritems():
+        # TODO: Ensure jobSet holds jobs that have been "launched" from Mesos
+        jobSet= set()
+        for queue in self.jobQueueList.values():
             for item in queue:
-                jobList.append(item.jobID)
-        for k, v in self.runningJobMap.iteritems():
-            jobList.append(k)
+                jobSet.add(item.jobID)
+        for key in self.runningJobMap.keys():
+            jobSet.add(key)
 
-        return jobList
+        return list(jobSet)
 
     def getRunningBatchJobIDs(self):
         """
@@ -166,7 +166,7 @@ def getRunningBatchJobIDs(self):
         (in seconds).
         """
         currentTime = dict()
-        for jobID, data in self.runningJobMap.iteritems():
+        for jobID, data in self.runningJobMap.items():
             currentTime[jobID] = time.time() - data.startTime
         return currentTime
 
@@ -265,7 +265,7 @@ def registered(self, driver, frameworkId, masterInfo):
         log.debug("Registered with framework ID %s" % frameworkId.value)
 
     def _sortJobsByResourceReq(self):
-        job_types = list(self.jobQueueList.keys())
+        job_types = self.jobQueueList.keys()
         # sorts from largest to smallest core usage
         # TODO: add a size() method to ResourceSummary and use it as the key. Ask me why.
         job_types.sort(key=lambda resourceRequirement: ResourceRequirement.cores)
@@ -297,7 +297,7 @@ def _prepareToRun(self, job_type, offer, index):
 
     def _deleteByJobID(self, jobID, ):
         # FIXME: not efficient, I'm sure.
-        for key, jobType in self.jobQueueList.iteritems():
+        for jobType in self.jobQueueList.values():
             for job in jobType:
                 if jobID == job.jobID:
                     jobType.remove(job)
@@ -456,7 +456,7 @@ def __reconcile(self, driver):
         now = time.time()
         if now > self.lastReconciliation + self.reconciliationPeriod:
             self.lastReconciliation = now
-            driver.reconcileTasks(list(self.runningJobMap.keys()))
+            driver.reconcileTasks(self.runningJobMap.keys())
 
     def reregistered(self, driver, masterInfo):
         """


Test patch: