Repository: DataBiosphere/toil

Issue:
Failures in the cluster scaler thread aren't propagated
If an exception is raised that kills the scaler thread Toil will try to continue running, potentially hanging until it is killed.

Failures in the cluster scaler thread aren't propagated
If an exception is raised that kills the scaler thread Toil will try to continue running, potentially hanging until it is killed.



Discussion:
We should probably add a hook into the scaler that attempts to join the scaler threads for a brief amount of time (join() takes a timeout). The threads are already ExceptionalThreads so any exception would be propagated through join.

The hook would have to be invoked by the main loop.

We should probably add a hook into the scaler that attempts to join the scaler threads for a brief amount of time (join() takes a timeout). The threads are already ExceptionalThreads so any exception would be propagated through join.

The hook would have to be invoked by the main loop.


Resolving patch:
diff --git a/src/toil/leader.py b/src/toil/leader.py
--- a/src/toil/leader.py
+++ b/src/toil/leader.py
@@ -905,9 +905,9 @@ def innerLoop(jobStore, config, batchSystem, toilState, jobBatcher, serviceManag
 
                 elif jobWrapper.jobStoreID in toilState.servicesIssued:
                     logger.debug("Telling job: %s to terminate its due to the successful completion of its successor jobs",
-                                jobWrapper.jobStoreID)
+                                 jobWrapper.jobStoreID)
                     serviceManager.killServices(toilState.servicesIssued[jobWrapper.jobStoreID],
-                                                    error=False)
+                                                error=False)
 
                 #There are no remaining tasks to schedule within the jobWrapper, but
                 #we schedule it anyway to allow it to be deleted.
@@ -997,9 +997,12 @@ def innerLoop(jobStore, config, batchSystem, toilState, jobBatcher, serviceManag
                     #in a minute, providing things are quiet
                 logger.info("Rescued any (long) missing jobs")
 
-        # Check on the associated processes and exit if a failure is detected
+        # Check on the associated threads and exit if a failure is detected
         statsAndLogging.check()
         serviceManager.check()
+        # the cluster scaler object will only be instantiated if autoscaling is enabled
+        if jobBatcher.clusterScaler is not None:
+            jobBatcher.clusterScaler.check()
 
     logger.info("Finished the main loop")
 
diff --git a/src/toil/provisioners/clusterScaler.py b/src/toil/provisioners/clusterScaler.py
--- a/src/toil/provisioners/clusterScaler.py
+++ b/src/toil/provisioners/clusterScaler.py
@@ -235,6 +235,23 @@ def __init__(self, provisioner, jobBatcher, config):
         else:
             self.scaler = None
 
+    def check(self):
+        """
+        Attempt to join any existing scaler threads that may have died or finished. This insures
+        any exceptions raised in the threads are propagated in a timely fashion.
+        """
+        exception = False
+        for scalerThread in [self.preemptableScaler, self.scaler]:
+            if scalerThread is not None:
+                try:
+                    scalerThread.join(timeout=0)
+                except Exception as e:
+                    logger.exception(e)
+                    exception = True
+        if exception:
+            raise RuntimeError('The cluster scaler has exited due to an exception')
+
+
     def shutdown(self):
         """
         Shutdown the cluster.


Test patch: