Repository: DataBiosphere/toil

Issue:
Race condition in Mesos batch system
In (very!) rare cases, the Mesos driver thread can crash with a KeyError. Fortunately, the log file demonstrates exactly the interleaving that needs to happen to cause this error:

```
Launched Mesos task 1667171.
Queueing the job command: _toil_worker CactusHalGeneratorUpWrapper aws:us-west-2:birds-first-jobstore e257211f-7dde-4f82-b2d4-1162f85589fe with
 job id: 1667173 ...
Launched Mesos task 1667172.
Got offer 99e660ab-d9e0-4a70-9eb0-588da54bd4b0-O5620082 for a non-preemptable slave with 12988.00 MiB memory, 31.00 core(s) and 368882.00 MiB o
f disk.
Preparing to launch Mesos task 1667173 using offer 99e660ab-d9e0-4a70-9eb0-588da54bd4b0-O5620082 ...
Offer 99e660ab-d9e0-4a70-9eb0-588da54bd4b0-O5620082 not suitable to run the tasks with requirements {'cores': 1, 'preemptable': True, 'disk': 2
147483648, 'memory': 4089446400}. Mesos offered 13618905088.0 memory, 31.0 cores and 3.86800812032e+11 of disk on a non-preemptable slave.
Offer 99e660ab-d9e0-4a70-9eb0-588da54bd4b0-O5620082 not suitable to run the tasks with requirements {'cores': 1, 'preemptable': True, 'disk': 2
147483648, 'memory': 100000000}. Mesos offered 13618905088.0 memory, 31.0 cores and 3.86800812032e+11 of disk on a non-preemptable slave.
Failed to call scheduler's resourceOffer
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/toil/batchSystems/mesos/batchSystem.py", line 468, in resourceOffers
    self._updateStateToRunning(offer, runnableTasks)
  File "/usr/local/lib/python2.7/dist-packages/toil/batchSystems/mesos/batchSystem.py", line 379, in _updateStateToRunning
    resources = self.taskResources[resourceKey]
KeyError: 1667173
I0120 07:18:14.162212 21183 sched.cpp:2055] Asked to abort the driver
... queued
Issued job 'CactusHalGeneratorUpWrapper' e257211f-7dde-4f82-b2d4-1162f85589fe with job batch system ID: 1667173 and preemptability: True, cores: 1, disk: 2.0 G, and memory: 3.8 G
```

The `Queueing the job command... queued` messages come from `issueBatchJob` on the leader thread. The `Preparing Mesos task` and similar messages come from `resourceOffers` on the driver thread. And if we look at `issueBatchJob`, there's a subtle race:

```python
self.jobQueues.insertJob(job, jobType)
<---------- leader thread was paused here
self.taskResources[jobID] = job.resources
log.debug("... queued")
```

The job was made available for processing before it was entirely ready. If the leader thread gets interrupted *immediately* after putting the job in the queue, and `resourceOffers` gets a chance to run without interruption for a while,`taskResources` won't be filled in properly and the Mesos driver will crash.
Race condition in Mesos batch system
In (very!) rare cases, the Mesos driver thread can crash with a KeyError. Fortunately, the log file demonstrates exactly the interleaving that needs to happen to cause this error:

```
Launched Mesos task 1667171.
Queueing the job command: _toil_worker CactusHalGeneratorUpWrapper aws:us-west-2:birds-first-jobstore e257211f-7dde-4f82-b2d4-1162f85589fe with
 job id: 1667173 ...
Launched Mesos task 1667172.
Got offer 99e660ab-d9e0-4a70-9eb0-588da54bd4b0-O5620082 for a non-preemptable slave with 12988.00 MiB memory, 31.00 core(s) and 368882.00 MiB o
f disk.
Preparing to launch Mesos task 1667173 using offer 99e660ab-d9e0-4a70-9eb0-588da54bd4b0-O5620082 ...
Offer 99e660ab-d9e0-4a70-9eb0-588da54bd4b0-O5620082 not suitable to run the tasks with requirements {'cores': 1, 'preemptable': True, 'disk': 2
147483648, 'memory': 4089446400}. Mesos offered 13618905088.0 memory, 31.0 cores and 3.86800812032e+11 of disk on a non-preemptable slave.
Offer 99e660ab-d9e0-4a70-9eb0-588da54bd4b0-O5620082 not suitable to run the tasks with requirements {'cores': 1, 'preemptable': True, 'disk': 2
147483648, 'memory': 100000000}. Mesos offered 13618905088.0 memory, 31.0 cores and 3.86800812032e+11 of disk on a non-preemptable slave.
Failed to call scheduler's resourceOffer
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/toil/batchSystems/mesos/batchSystem.py", line 468, in resourceOffers
    self._updateStateToRunning(offer, runnableTasks)
  File "/usr/local/lib/python2.7/dist-packages/toil/batchSystems/mesos/batchSystem.py", line 379, in _updateStateToRunning
    resources = self.taskResources[resourceKey]
KeyError: 1667173
I0120 07:18:14.162212 21183 sched.cpp:2055] Asked to abort the driver
... queued
Issued job 'CactusHalGeneratorUpWrapper' e257211f-7dde-4f82-b2d4-1162f85589fe with job batch system ID: 1667173 and preemptability: True, cores: 1, disk: 2.0 G, and memory: 3.8 G
```

The `Queueing the job command... queued` messages come from `issueBatchJob` on the leader thread. The `Preparing Mesos task` and similar messages come from `resourceOffers` on the driver thread. And if we look at `issueBatchJob`, there's a subtle race:

```python
self.jobQueues.insertJob(job, jobType)
<---------- leader thread was paused here
self.taskResources[jobID] = job.resources
log.debug("... queued")
```

The job was made available for processing before it was entirely ready. If the leader thread gets interrupted *immediately* after putting the job in the queue, and `resourceOffers` gets a chance to run without interruption for a while,`taskResources` won't be filled in properly and the Mesos driver will crash.


Discussion:

Resolving patch:
diff --git a/src/toil/batchSystems/mesos/batchSystem.py b/src/toil/batchSystems/mesos/batchSystem.py
--- a/src/toil/batchSystems/mesos/batchSystem.py
+++ b/src/toil/batchSystems/mesos/batchSystem.py
@@ -190,8 +190,8 @@ def issueBatchJob(self, jobNode):
 
         # TODO: round all elements of resources
 
-        self.jobQueues.insertJob(job, jobType)
         self.taskResources[jobID] = job.resources
+        self.jobQueues.insertJob(job, jobType)
         log.debug("... queued")
         return jobID
 


Test patch: