Repository: DataBiosphere/toil

Issue:
Service deadlock detection triggers when there is no service deadlock
The only state the service deadlock detector has is a) the service jobs that are issued during the last "potential deadlock" and b) the time that the "potential deadlock" occurred.

But the deadlock detector doesn't reset the "potential deadlock" information after it knows a deadlock didn't occur. Here's the situation I run into:

- Service job A gets issued. While it's starting, (for e.g. 5 seconds) service job A is the only job issued, so the "potential deadlock" code gets triggered.
- A bunch of jobs B get issued that depend on service A. This takes, say, an hour.
- The last job of batch B gets completed.
- A bunch of jobs C get issued.

The issue is that, between batch B completing and batch C getting issued, even though this may take only a fraction of a second, the service deadlock code can run. If it runs at just the right time, it detects that service job A is the only job issued, just like the last "potential deadlock", and terminates the workflow because it's been an hour since the last "potential deadlock".

I have a fix for this, but I want to double-check that I actually understand the purpose of the service deadlock detection. As of right now I don't think it's too useful, because it works solely on the basis of *issued* jobs. Consider the situation where two cores are available, and 2 service jobs and 1 regular job are issued, but only the 2 service jobs are running. No forward progress can be made, but currently the code wouldn't detect this as a "service deadlock". Is that intentional?
Service deadlock detection triggers when there is no service deadlock
The only state the service deadlock detector has is a) the service jobs that are issued during the last "potential deadlock" and b) the time that the "potential deadlock" occurred.

But the deadlock detector doesn't reset the "potential deadlock" information after it knows a deadlock didn't occur. Here's the situation I run into:

- Service job A gets issued. While it's starting, (for e.g. 5 seconds) service job A is the only job issued, so the "potential deadlock" code gets triggered.
- A bunch of jobs B get issued that depend on service A. This takes, say, an hour.
- The last job of batch B gets completed.
- A bunch of jobs C get issued.

The issue is that, between batch B completing and batch C getting issued, even though this may take only a fraction of a second, the service deadlock code can run. If it runs at just the right time, it detects that service job A is the only job issued, just like the last "potential deadlock", and terminates the workflow because it's been an hour since the last "potential deadlock".

I have a fix for this, but I want to double-check that I actually understand the purpose of the service deadlock detection. As of right now I don't think it's too useful, because it works solely on the basis of *issued* jobs. Consider the situation where two cores are available, and 2 service jobs and 1 regular job are issued, but only the 2 service jobs are running. No forward progress can be made, but currently the code wouldn't detect this as a "service deadlock". Is that intentional?


Discussion:
Thanks for reporting this - @benedictpaten might have better insight but I don't think this is intentional
I wonder if this is related to the test failing from service deadlock in https://github.com/BD2KGenomics/toil-lib/pull/78  
@jvivian I am not a betting man, but I would take that wager, yes.
Given the timing for that test failure (failure almost exactly 1 minute after service job issued), I think it's probably slightly different:

```
ip-172-31-23-244 2017-01-20 09:21:00,049 MainThread INFO toil.leader: Issued job 'SparkService' e/D/jobQeeZfK with job batch system ID: 1 and cores: 1, disk: 2.0 G, and memory: 2.0 G
[ 1 minute later]
ip-172-31-23-244 2017-01-20 09:22:02,057 MainThread INFO toil.serviceManager: Waiting for service manager thread to finish ...
ip-172-31-23-244 2017-01-20 09:22:04,166 MainThread INFO toil.serviceManager: ... finished shutting down the service manager. Took 2.10873413086 seconds
```

What I notice with my issue is a deadlock exception after *less than a second* of a situation where a service job is the only job issued (because of a bug in the deadlock detector).

From reading that test log it seems to me like the Spark service may have taken more than 60 seconds (the default deadlock time) to start up, for any number of reasons (#1503?). Or maybe it's caused by a different bug in the service manager/deadlock detection?
@joelarmstrong good call. The time-based deadlock detection is inherently racey so I'll look into better ways to do it but for now it might make sense to just to bump the default deadlock time
@joelarmstrong just checking in on this - are you working on a PR for this issue?
Thanks for reporting this - @benedictpaten might have better insight but I don't think this is intentional
I wonder if this is related to the test failing from service deadlock in https://github.com/BD2KGenomics/toil-lib/pull/78  
@jvivian I am not a betting man, but I would take that wager, yes.
Given the timing for that test failure (failure almost exactly 1 minute after service job issued), I think it's probably slightly different:

```
ip-172-31-23-244 2017-01-20 09:21:00,049 MainThread INFO toil.leader: Issued job 'SparkService' e/D/jobQeeZfK with job batch system ID: 1 and cores: 1, disk: 2.0 G, and memory: 2.0 G
[ 1 minute later]
ip-172-31-23-244 2017-01-20 09:22:02,057 MainThread INFO toil.serviceManager: Waiting for service manager thread to finish ...
ip-172-31-23-244 2017-01-20 09:22:04,166 MainThread INFO toil.serviceManager: ... finished shutting down the service manager. Took 2.10873413086 seconds
```

What I notice with my issue is a deadlock exception after *less than a second* of a situation where a service job is the only job issued (because of a bug in the deadlock detector).

From reading that test log it seems to me like the Spark service may have taken more than 60 seconds (the default deadlock time) to start up, for any number of reasons (#1503?). Or maybe it's caused by a different bug in the service manager/deadlock detection?
@joelarmstrong good call. The time-based deadlock detection is inherently racey so I'll look into better ways to do it but for now it might make sense to just to bump the default deadlock time
@joelarmstrong just checking in on this - are you working on a PR for this issue?

Resolving patch:
diff --git a/src/toil/leader.py b/src/toil/leader.py
--- a/src/toil/leader.py
+++ b/src/toil/leader.py
@@ -35,6 +35,7 @@
 from toil.serviceManager import ServiceManager
 from toil.statsAndLogging import StatsAndLogging
 from toil.jobGraph import JobNode
+from toil.job import ServiceJobNode
 from toil.toilState import ToilState
 
 logger = logging.getLogger( __name__ )
@@ -140,7 +141,7 @@ def __init__(self, config, batchSystem, provisioner, jobStore, rootJob, jobCache
         self.statsAndLogging = StatsAndLogging(self.jobStore, self.config)
 
         # Set used to monitor deadlocked jobs
-        self.potentialDeadlockedJobs = None
+        self.potentialDeadlockedJobs = set()
         self.potentialDeadlockTime = 0
 
         # internal jobs we should not expose at top level debugging
@@ -488,28 +489,30 @@ def checkForDeadlocks(self):
         """
         Checks if the system is deadlocked running service jobs.
         """
-        # If there are no updated jobs and at least some jobs issued
-        if len(self.toilState.updatedJobs) == 0 and self.getNumberOfJobsIssued() > 0:
-
-            # If all scheduled jobs are services
-            assert self.serviceJobsIssued + self.preemptableServiceJobsIssued <= self.getNumberOfJobsIssued()
-            if self.serviceJobsIssued + self.preemptableServiceJobsIssued == self.getNumberOfJobsIssued():
-
-                # Sanity check that all issued jobs are actually services
-                for jobNode in self.jobBatchSystemIDToIssuedJob.values():
-                    assert jobNode.jobStoreID in self.toilState.serviceJobStoreIDToPredecessorJob
-
-                # An active service job is one that is not in the process of terminating
-                activeServiceJobs = filter(lambda x : self.serviceManager.isActive(x), self.jobBatchSystemIDToIssuedJob.values())
-
-                # If all the service jobs are active then we have a potential deadlock
-                if len(activeServiceJobs) == len(self.jobBatchSystemIDToIssuedJob):
-                    # We wait self.config.deadlockWait seconds before declaring the system deadlocked
-                    if self.potentialDeadlockedJobs != activeServiceJobs:
-                        self.potentialDeadlockedJobs = activeServiceJobs
-                        self.potentialDeadlockTime = time.time()
-                    elif time.time() - self.potentialDeadlockTime >= self.config.deadlockWait:
-                        raise DeadlockException("The system is service deadlocked - all issued jobs %s are active services" % self.getNumberOfJobsIssued())
+        totalRunningJobs = len(self.batchSystem.getRunningBatchJobIDs())
+        totalServicesIssued = self.serviceJobsIssued + self.preemptableServiceJobsIssued
+        # If there are no updated jobs and at least some jobs running
+        if totalServicesIssued >= totalRunningJobs and len(self.toilState.updatedJobs) == 0 and totalRunningJobs > 0:
+            serviceJobs = filter(lambda x : isinstance(x, ServiceJobNode), self.jobBatchSystemIDToIssuedJob.values())
+            runningServiceJobs = set(filter(lambda x : self.serviceManager.isRunning(x), serviceJobs))
+            assert len(runningServiceJobs) <= totalRunningJobs
+
+            # If all the running jobs are active services then we have a potential deadlock
+            if len(runningServiceJobs) == totalRunningJobs:
+                # We wait self.config.deadlockWait seconds before declaring the system deadlocked
+                if self.potentialDeadlockedJobs != runningServiceJobs:
+                    self.potentialDeadlockedJobs = runningServiceJobs
+                    self.potentialDeadlockTime = time.time()
+                elif time.time() - self.potentialDeadlockTime >= self.config.deadlockWait:
+                    raise DeadlockException("The system is service deadlocked - all %d running jobs are active services" % totalRunningJobs)
+            else:
+                # We have observed non-service jobs running, so reset the potential deadlock
+                self.potentialDeadlockedJobs = set()
+                self.potentialDeadlockTime = 0
+        else:
+            # We have observed non-service jobs running, so reset the potential deadlock
+            self.potentialDeadlockedJobs = set()
+            self.potentialDeadlockTime = 0
 
 
     def issueJob(self, jobNode):
diff --git a/src/toil/serviceManager.py b/src/toil/serviceManager.py
--- a/src/toil/serviceManager.py
+++ b/src/toil/serviceManager.py
@@ -125,11 +125,18 @@ def killServices(self, services, error=False):
             
     def isActive(self, serviceJobNode):
         """
-        Returns true is the service job has not been told to terminate.
+        Returns true if the service job has not been told to terminate.
         :rtype: boolean
         """
         return self.jobStore.fileExists(serviceJobNode.terminateJobStoreID)
 
+    def isRunning(self, serviceJobNode):
+        """
+        Returns true if the service job has started and is active
+        :rtype: boolean
+        """
+        return (not self.jobStore.fileExists(serviceJobNode.startJobStoreID)) and self.isActive(serviceJobNode)
+
     def check(self):
         """
         Check on the service manager thread.
@@ -195,4 +202,4 @@ def _startServices(jobGraphsWithServicesToStart,
                             break
 
             # Add the jobGraph to the output queue of jobs whose services have been started
-            jobGraphsWithServicesThatHaveStarted.put(jobGraph)
\ No newline at end of file
+            jobGraphsWithServicesThatHaveStarted.put(jobGraph)


Test patch: