Repository: DataBiosphere/toil

Issue:
Add an option to set custom AWS IAM policies
The AWS IAM policies are currently [hardcoded](https://github.com/DataBiosphere/toil/blob/c483abf5c91894de876f43ab14761238f1021963/src/toil/provisioners/aws/awsProvisioner.py#L563), so it's not possible to use additional services (e.g. KMS, ECR, etc) with default settings

I can think of two options:
* **(preferred)** Allow setting custom IAM role via a flag. This would enable arbitrary role definitions and customizations. We'd need to ensure to not delete the role when `destroy-cluster` is called though. Also, workers would need to get the role via the master rather than assuming it's based on the cluster name.
* Add an option for setting additional policies when creating the custom role. This would work, but has limits (e.g. it can't be used in other places prior to launching the cluster; complex policies would be difficult to specify).

@DailyDreaming @adamnovak : do you have other options/thoughts in mind?

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-41)
┆Issue Number: TOIL-41

Add an option to set custom AWS IAM policies
The AWS IAM policies are currently [hardcoded](https://github.com/DataBiosphere/toil/blob/c483abf5c91894de876f43ab14761238f1021963/src/toil/provisioners/aws/awsProvisioner.py#L563), so it's not possible to use additional services (e.g. KMS, ECR, etc) with default settings

I can think of two options:
* **(preferred)** Allow setting custom IAM role via a flag. This would enable arbitrary role definitions and customizations. We'd need to ensure to not delete the role when `destroy-cluster` is called though. Also, workers would need to get the role via the master rather than assuming it's based on the cluster name.
* Add an option for setting additional policies when creating the custom role. This would work, but has limits (e.g. it can't be used in other places prior to launching the cluster; complex policies would be difficult to specify).

@DailyDreaming @adamnovak : do you have other options/thoughts in mind?

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-41)
┆Issue Number: TOIL-41



Discussion:
@arostamianfar I'm definitely a fan of making this configurable.  I agree with your preferred method though the role should still be deleted from Amazon provided it is set in a config somewhere.  Making a new deployment should make new roles each time.  There's no reason to keep the old ones around if the cluster was deleted.
@arostamianfar I'm definitely a fan of making this configurable.  I agree with your preferred method though the role should still be deleted from Amazon provided it is set in a config somewhere.  Making a new deployment should make new roles each time.  There's no reason to keep the old ones around if the cluster was deleted.

Resolving patch:
diff --git a/src/toil/provisioners/aws/awsProvisioner.py b/src/toil/provisioners/aws/awsProvisioner.py
--- a/src/toil/provisioners/aws/awsProvisioner.py
+++ b/src/toil/provisioners/aws/awsProvisioner.py
@@ -40,6 +40,8 @@
 
 logger = logging.getLogger(__name__)
 logging.getLogger("boto").setLevel(logging.CRITICAL)
+# Role name (used as the suffix) for EC2 instance profiles that are automatically created by Toil.
+_INSTANCE_PROFILE_ROLE_NAME = 'toil'
 # The tag key that specifies the Toil node type ("leader" or "worker") so that
 # leader vs. worker nodes can be robustly identified.
 _TOIL_NODE_TYPE_TAG_KEY = 'ToilNodeType'
@@ -124,6 +126,7 @@ def _readClusterSettings(self):
         self._keyName = list(instanceMetaData['public-keys'].keys())[0]
         self._tags = self.getLeader().tags
         self._masterPublicKey = self._setSSH()
+        self._leaderProfileArn = instanceMetaData['iam']['info']['InstanceProfileArn']
 
     def launchCluster(self, leaderNodeType, leaderStorage, owner, **kwargs):
         """
@@ -135,9 +138,9 @@ def launchCluster(self, leaderNodeType, leaderStorage, owner, **kwargs):
         if 'keyName' not in kwargs:
             raise RuntimeError("A keyPairName is required for the AWS provisioner.")
         self._keyName = kwargs['keyName']
-        self._vpcSubnet = kwargs['vpcSubnet'] if 'vpcSubnet' in kwargs else None
+        self._vpcSubnet = kwargs.get('vpcSubnet')
 
-        profileARN = self._getProfileARN()
+        profileArn = kwargs.get('awsEc2ProfileArn') or self._getProfileArn()
         # the security group name is used as the cluster identifier
         sgs = self._createSecurityGroup()
         bdm = self._getBlockDeviceMapping(E2Instances[leaderNodeType], rootVolSize=leaderStorage)
@@ -147,7 +150,7 @@ def launchCluster(self, leaderNodeType, leaderStorage, owner, **kwargs):
         specKwargs = {'key_name': self._keyName, 'security_group_ids': [sg.id for sg in sgs],
                   'instance_type': leaderNodeType,
                   'user_data': userData, 'block_device_map': bdm,
-                  'instance_profile_arn': profileARN,
+                  'instance_profile_arn': profileArn,
                   'placement': self._zone}
         if self._vpcSubnet:
             specKwargs["subnet_id"] = self._vpcSubnet
@@ -252,7 +255,6 @@ def addNodes(self, nodeType, numNodes, preemptable, spotBid=None):
                 raise RuntimeError("No spot bid given for a preemptable node request.")
         instanceType = E2Instances[nodeType]
         bdm = self._getBlockDeviceMapping(instanceType, rootVolSize=self._nodeStorage)
-        arn = self._getProfileARN()
 
         keyPath = self._sseKey if self._sseKey else None
         userData =  self._getCloudConfigUserData('worker', self._masterPublicKey, keyPath, preemptable)
@@ -262,7 +264,7 @@ def addNodes(self, nodeType, numNodes, preemptable, spotBid=None):
                   'instance_type': instanceType.name,
                   'user_data': userData,
                   'block_device_map': bdm,
-                  'instance_profile_arn': arn,
+                  'instance_profile_arn': self._leaderProfileArn,
                   'placement': self._zone,
                   'subnet_id': self._subnetID}
 
@@ -438,6 +440,11 @@ def _deleteIAMProfiles(self, instances):
             # boto won't look things up by the ARN so we have to parse it to get
             # the profile name
             profileName = profile.rsplit('/')[-1]
+
+            # Only delete profiles that were automatically created by Toil.
+            if profileName != self._ctx.to_aws_name(_INSTANCE_PROFILE_ROLE_NAME):
+                continue
+
             try:
                 profileResult = self._ctx.iam.get_instance_profile(profileName)
             except BotoServerError as e:
@@ -566,14 +573,13 @@ def groupNotFound(e):
         return out
 
     @awsRetry
-    def _getProfileARN(self):
+    def _getProfileArn(self):
         assert self._ctx
         def addRoleErrors(e):
             return e.status == 404
-        roleName = 'toil'
         policy = dict(iam_full=iamFullPolicy, ec2_full=ec2FullPolicy,
                       s3_full=s3FullPolicy, sbd_full=sdbFullPolicy)
-        iamRoleName = self._ctx.setup_iam_ec2_role(role_name=roleName, policies=policy)
+        iamRoleName = self._ctx.setup_iam_ec2_role(role_name=_INSTANCE_PROFILE_ROLE_NAME, policies=policy)
 
         try:
             profile = self._ctx.iam.get_instance_profile(iamRoleName)
diff --git a/src/toil/utils/toilLaunchCluster.py b/src/toil/utils/toilLaunchCluster.py
--- a/src/toil/utils/toilLaunchCluster.py
+++ b/src/toil/utils/toilLaunchCluster.py
@@ -95,6 +95,10 @@ def main():
                         help="The location of the file containing the Azure storage credentials. If not specified,"
                              " the default file is used with Azure provisioning. Use 'None' to disable"
                              " the transfer of credentials.")
+    parser.add_argument('--awsEc2ProfileArn', dest='awsEc2ProfileArn', default=None, type=str,
+                        help="If provided, the specified ARN is used as the instance profile for EC2 instances."
+                             "Useful for setting custom IAM profiles. If not specified, a new IAM role is created "
+                             "by default with sufficient access to perform basic cluster operations.")
     config = parseBasicOptions(parser)
     tagsDict = None if config.tags is None else createTagsDict(config.tags)
     checkValidNodeTypes(config.provisioner, config.nodeTypes)
@@ -154,7 +158,8 @@ def main():
                           userTags=tagsDict,
                           vpcSubnet=config.vpcSubnet,
                           publicKeyFile=config.publicKeyFile,
-                          azureStorageCredentials=config.azureStorageCredentials)
+                          azureStorageCredentials=config.azureStorageCredentials,
+                          awsEc2ProfileArn=config.awsEc2ProfileArn)
 
     for nodeType, workers in zip(nodeTypes, numNodes):
         cluster.addNodes(nodeType=nodeType, numNodes=workers, preemptable=False)


Test patch: