Repository: DataBiosphere/toil

Issue:
Add a tag to distinguish leader vs. worker nodes in AWS
Currently, there is no easy way to distinguish Toil leader(s) from workers in AWS as they all have the same tags (the leader is assumed to be the first instance that was launched).
It would be easier to locate and filter them in the AWS console if we add a special tag (e.g `ToilNodeType`) that equals `leader` or `worker`.

We can optionally make the [`getLeader`](https://github.com/DataBiosphere/toil/blob/a8252277ff814e7bee0971139c2344f88e44b644/src/toil/provisioners/aws/awsProvisioner.py#L367) function more robust by verifying that the leader does indeed have the `leader` tag (e.g. in case the leader has terminated, so we don't accidentally think that a worker is a leader).

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-354)
┆Issue Number: TOIL-354



Discussion:

Resolving patch:
diff --git a/src/toil/provisioners/aws/awsProvisioner.py b/src/toil/provisioners/aws/awsProvisioner.py
--- a/src/toil/provisioners/aws/awsProvisioner.py
+++ b/src/toil/provisioners/aws/awsProvisioner.py
@@ -40,6 +40,10 @@
 
 logger = logging.getLogger(__name__)
 logging.getLogger("boto").setLevel(logging.CRITICAL)
+# The tag key that specifies the Toil node type ("leader" or "worker") so that
+# leader vs. worker nodes can be robustly identified.
+_TOIL_NODE_TYPE_TAG_KEY = 'ToilNodeType'
+
 
 def awsRetryPredicate(e):
     if not isinstance(e, BotoServerError):
@@ -159,7 +163,7 @@ def launchCluster(self, leaderNodeType, leaderStorage, owner, **kwargs):
                           preemptable=False, tags=leader.tags)
         leaderNode.waitForNode('toil_leader')
 
-        defaultTags = {'Name': self.clusterName, 'Owner': owner}
+        defaultTags = {'Name': self.clusterName, 'Owner': owner, _TOIL_NODE_TYPE_TAG_KEY: 'leader'}
         if kwargs['userTags']:
             defaultTags.update(kwargs['userTags'])
 
@@ -292,6 +296,7 @@ def addNodes(self, nodeType, numNodes, preemptable, spotBid=None):
             with attempt:
                 wait_instances_running(self._ctx.ec2, instancesLaunched)
 
+        self._tags[_TOIL_NODE_TYPE_TAG_KEY] = 'worker'
         AWSProvisioner._addTags(instancesLaunched, self._tags)
         if self._sseKey:
             for i in instancesLaunched:
@@ -367,6 +372,12 @@ def getLeader(self, wait=False):
             leader = instances[0]  # assume leader was launched first
         except IndexError:
             raise NoSuchClusterException(self.clusterName)
+        if (leader.tags.get(_TOIL_NODE_TYPE_TAG_KEY) or 'leader') != 'leader':
+            raise RuntimeError(
+                'Invalid cluster state! The first launched instance appears not to be the leader '
+                'as it is missing the "leader" tag. The safest recovery is to destroy the cluster '
+                'and restart the job. Incorrect Leader ID: %s' % leader.id
+            )
         leaderNode = Node(publicIP=leader.ip_address, privateIP=leader.private_ip_address,
                           name=leader.id, launchTime=leader.launch_time, nodeType=None,
                           preemptable=False, tags=leader.tags)


Test patch: