Repository: DataBiosphere/toil

Issue:
Race condition when destroying a running cluster
When destroying a running cluster with worker nodes, all instances (including the leader) are terminated at the same time. However, the processes in the leader may still be running while the workers are being terminated, so the leader (itself in the process of being terminated) may spin up more worker instances to reach the desired cluster size again.

This results in orphan workers being left after the cluster is destroyed[*].

I think the simplest solution is to terminate the leader first and then any workers that may be leftover.

[*] Actually, the destroy cluster fails as the security group is still associated with orphan worker nodes.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-358)
┆Issue Number: TOIL-358



Discussion:

Resolving patch:
diff --git a/src/toil/provisioners/aws/awsProvisioner.py b/src/toil/provisioners/aws/awsProvisioner.py
--- a/src/toil/provisioners/aws/awsProvisioner.py
+++ b/src/toil/provisioners/aws/awsProvisioner.py
@@ -94,6 +94,10 @@ def wrapper(*args, **kwargs):
     return wrapper
 
 
+class InvalidClusterStateException(Exception):
+    pass
+
+
 class AWSProvisioner(AbstractProvisioner):
     """
     Implements an AWS provisioner using the boto libraries.
@@ -208,6 +212,18 @@ def destroyCluster(self):
         def expectedShutdownErrors(e):
             return e.status == 400 and 'dependent object' in e.body
 
+        # We should terminate the leader first in case a workflow is still running in the cluster.
+        # The leader may create more instances while we're terminating the workers.
+        try:
+            leader = self.getLeader(returnRawInstance=True)
+            logger.info('Terminating the leader first ...')
+            self._terminateInstances(instances=[leader])
+            logger.info('Now terminating any remaining workers ...')
+        except (NoSuchClusterException, InvalidClusterStateException):
+            # It's ok if the leader is not found. We'll terminate any remaining
+            # instances below anyway.
+            pass
+
         instances = self._getNodesInCluster(nodeType=None, both=True)
         spotIDs = self._getSpotRequestIDs()
         if spotIDs:
@@ -366,7 +382,7 @@ def _toNameSpace(self):
             namespace = '/' + namespace + '/'
         return namespace.replace('-', '/')
 
-    def getLeader(self, wait=False):
+    def getLeader(self, wait=False, returnRawInstance=False):
         assert self._ctx
         instances = self._getNodesInCluster(nodeType=None, both=True)
         instances.sort(key=lambda x: x.launch_time)
@@ -375,7 +391,7 @@ def getLeader(self, wait=False):
         except IndexError:
             raise NoSuchClusterException(self.clusterName)
         if (leader.tags.get(_TOIL_NODE_TYPE_TAG_KEY) or 'leader') != 'leader':
-            raise RuntimeError(
+            raise InvalidClusterStateException(
                 'Invalid cluster state! The first launched instance appears not to be the leader '
                 'as it is missing the "leader" tag. The safest recovery is to destroy the cluster '
                 'and restart the job. Incorrect Leader ID: %s' % leader.id
@@ -390,7 +406,7 @@ def getLeader(self, wait=False):
             self._waitForIP(leader)
             leaderNode.waitForNode('toil_leader')
 
-        return leaderNode
+        return leader if returnRawInstance else leaderNode
 
     @classmethod
     @awsRetry


Test patch: