Repository: DataBiosphere/toil

Issue:
Race condition when destroying a running cluster
When destroying a running cluster with worker nodes, all instances (including the leader) are terminated at the same time. However, the processes in the leader may still be running while the workers are being terminated, so the leader (itself in the process of being terminated) may spin up more worker instances to reach the desired cluster size again.

This results in orphan workers being left after the cluster is destroyed[*].

I think the simplest solution is to terminate the leader first and then any workers that may be leftover.

[*] Actually, the destroy cluster fails as the security group is still associated with orphan worker nodes.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-358)
┆Issue Number: TOIL-358



Discussion:
I just noticed that the security group is not deleted if the leader is the only instance left in the cluster. The issue is that `vpcId` is set [here](https://github.com/DataBiosphere/toil/blob/b3d132ba9f6475751cc2fa4e634a0c1ccd2cbb07/src/toil/provisioners/aws/awsProvisioner.py#L234) and we ensure that it matches the one in the security group.

Sent #2685 to fix it. Sorry for not catching this earlier!

Resolving patch:
diff --git a/src/toil/provisioners/aws/awsProvisioner.py b/src/toil/provisioners/aws/awsProvisioner.py
--- a/src/toil/provisioners/aws/awsProvisioner.py
+++ b/src/toil/provisioners/aws/awsProvisioner.py
@@ -94,6 +94,10 @@ def wrapper(*args, **kwargs):
     return wrapper
 
 
+class InvalidClusterStateException(Exception):
+    pass
+
+
 class AWSProvisioner(AbstractProvisioner):
     """
     Implements an AWS provisioner using the boto libraries.
@@ -208,16 +212,36 @@ def destroyCluster(self):
         def expectedShutdownErrors(e):
             return e.status == 400 and 'dependent object' in e.body
 
+        def destroyInstances(instances):
+            """
+            Similar to _terminateInstances, except that it also cleans up any
+            resources associated with the instances (e.g. IAM profiles).
+            """
+            self._deleteIAMProfiles(instances)
+            self._terminateInstances(instances)
+
+        # We should terminate the leader first in case a workflow is still running in the cluster.
+        # The leader may create more instances while we're terminating the workers.
+        vpcId = None
+        try:
+            leader = self.getLeader(returnRawInstance=True)
+            vpcId = leader.vpc_id
+            logger.info('Terminating the leader first ...')
+            destroyInstances([leader])
+            logger.info('Now terminating any remaining workers ...')
+        except (NoSuchClusterException, InvalidClusterStateException):
+            # It's ok if the leader is not found. We'll terminate any remaining
+            # instances below anyway.
+            pass
+
         instances = self._getNodesInCluster(nodeType=None, both=True)
         spotIDs = self._getSpotRequestIDs()
         if spotIDs:
             self._ctx.ec2.cancel_spot_instance_requests(request_ids=spotIDs)
         instancesToTerminate = awsFilterImpairedNodes(instances, self._ctx.ec2)
-        vpcId = None
         if instancesToTerminate:
-            vpcId = instancesToTerminate[0].vpc_id
-            self._deleteIAMProfiles(instances=instancesToTerminate)
-            self._terminateInstances(instances=instancesToTerminate)
+            vpcId = vpcId or instancesToTerminate[0].vpc_id
+            destroyInstances(instancesToTerminate)
         if len(instances) == len(instancesToTerminate):
             logger.debug('Deleting security group...')
             removed = False
@@ -366,7 +390,7 @@ def _toNameSpace(self):
             namespace = '/' + namespace + '/'
         return namespace.replace('-', '/')
 
-    def getLeader(self, wait=False):
+    def getLeader(self, wait=False, returnRawInstance=False):
         assert self._ctx
         instances = self._getNodesInCluster(nodeType=None, both=True)
         instances.sort(key=lambda x: x.launch_time)
@@ -375,7 +399,7 @@ def getLeader(self, wait=False):
         except IndexError:
             raise NoSuchClusterException(self.clusterName)
         if (leader.tags.get(_TOIL_NODE_TYPE_TAG_KEY) or 'leader') != 'leader':
-            raise RuntimeError(
+            raise InvalidClusterStateException(
                 'Invalid cluster state! The first launched instance appears not to be the leader '
                 'as it is missing the "leader" tag. The safest recovery is to destroy the cluster '
                 'and restart the job. Incorrect Leader ID: %s' % leader.id
@@ -390,7 +414,7 @@ def getLeader(self, wait=False):
             self._waitForIP(leader)
             leaderNode.waitForNode('toil_leader')
 
-        return leaderNode
+        return leader if returnRawInstance else leaderNode
 
     @classmethod
     @awsRetry


Test patch: