Repository: getmoto/moto

Issue:
Request AWS::EMR::Cluster resource to be added to cloudformation
Request for AWS::EMR::Cluster to be added to cloudformation. This is used for our testing and we would like to request this resource to be added. 



Discussion:
Hi @sli10, welcome to Moto! I'll mark it as an enhancement to add this feature.
Hi @sli10, an initial implementation is now part of `moto >= 5.0.3.dev14` and the latest Docker image.

Are you able to try it out, and see whether this solves your issue?

Not all properties for the CloudFormation-template are processed at the moment, so please let us know if there are any specific properties that you use that are missing.
Hi @bblommers, we were able to try this out and it resolved our issue. On top of that we were encountering an issue with the stack deletion. We were wondering if Cloudformation implementation supports in order deletion of resources?
Happy to hear that @sli10!

I don't think we do anything special with stack deletion - we just delete all all resources in a random order, and re-try to delete them in case of failure, up to 5 times, to try and resolve any dependency problems.

Can you share a bit more about the issues you're running into, ideally with an example template?
Hi @bblommers , we are running into an issue with delete stack where the ordering of how the resources are deleted is causing an issue. When the cloudformation delete it's resources it gets deleted in reverse order it is created, so in our case the random order that moto deletes in gives us an error of `An error occurred (DeleteConflict) when calling the DeleteStack operation: Cannot delete entity, must remove roles from instance profile first.`

Example of the resources we are using:
`Resources:
  Cluster:
    Type: 'AWS::EMR::Cluster'
    Properties:
      :
      :
  emrRole:
    Type: 'AWS::IAM::Role'
      :
      :
  emrEc2Role:
    Condition: CreateEMREc2Role
    Type: 'AWS::IAM::Role'
      :
      :
  emrEc2InstanceProfile:
    Type: 'AWS::IAM::InstanceProfile'
    Properties:
      :
      :`
With that particular example I can't quite reproduce it @sli10 - but I imagine that the actual setup is quite more complex.

Having said that, I was able to improve the logic a bit around stack deletions in #7413. Are you able to test whether `moto >= 5.0.3.dev34` fixes your problem?

If that doesn't work, I would probably need a reproducible example to really understand what's going wrong.
Hi @bblommers, the issue with delete stack is not producible with latest version any more, thank you for your timely response. We have a few more problems, one relating to `AWS::EFS::FileSystem` which complains about the following missing parameters `EFSBackend.create_file_system() missing 7 required positional arguments: 'performance_mode', 'kms_key_id', 'throughput_mode', 'provisioned_throughput_in_mibps', 'availability_zone_name', 'backup', and 'tags'`. According to the aws documentation, these aren't required fields. Are these required for moto?

Example of cf template:
`Resources:
  FileSystem:
    Type: 'AWS::EFS::FileSystem'
    Properties:
      Encrypted: true`

Another issue is related to the emr cluster resource mentioned above, when we create a resource using the latest docker image we get `No Moto CloudFormation support for AWS::EMR::Cluster`.  Create stack request outputs stack id, but doesn't end up creating cluster. 

One more request we have regarding the `AWS::IAM::Role` resource, we were wondering if we can add support for RoleId?



> when we create a resource using the latest docker image

Just to double check - do you mean the latest version (5.0.2), or `motoserver/moto:latest`? We added [support for EMR::Cluster](https://github.com/getmoto/moto/pull/7384) a few weeks ago, and while there hasn't been a proper release yet, it should be part of the `:latest` Docker image.

I'll have a look at the other issues later this week.
We are using `motoserver/moto:latest` as the Docker image.
Update on delete stack issue, sorry for going back and forth. Here is a reproducible example @bblommers , that's causing an issue when trying to delete stack. We were able to reproduce the `An error occurred (DeleteConflict) when calling the DeleteStack operation: Cannot delete entity, must remove roles from instance profile first.` with this cf template.

`
Parameters:
  InputRole:
    Type: String
    Default: "test-emr-role"

Resources:
  CertificateStoreSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupDescription: Allow NFS traffic
      VpcId: "subnet-f90771ae"
  CertificateStoreSecurityGroupIngress:
    Type: 'AWS::EC2::SecurityGroupIngress'
    Properties:
      FromPort: 2049
      ToPort: 2049
      IpProtocol: tcp
      GroupId: !Ref CertificateStoreSecurityGroup
      SourceSecurityGroupId: !Ref CertificateStoreSecurityGroup
  Cluster:
    Type: 'AWS::EMR::Cluster'
    Properties:
      LogUri: "s3://test-bucket/emr-logs"
      EbsRootVolumeSize: 64
      Instances:
        MasterInstanceGroup:
          InstanceCount: 1
          InstanceType: m6g.xlarge
          Market: ON_DEMAND
          Name: cfnMaster
          EbsConfiguration:
            EbsOptimized: true
            EbsBlockDeviceConfigs:
              - VolumeSpecification:
                  SizeInGB: 128
                  VolumeType: gp3
                VolumesPerInstance: 1
        CoreInstanceGroup:
          InstanceCount: 1
          InstanceType: m6g.xlarge
          Market: ON_DEMAND
          Name: cfnCore
          EbsConfiguration:
            EbsOptimized: true
            EbsBlockDeviceConfigs:
              - VolumeSpecification:
                  SizeInGB: 128
                  VolumeType: gp3
                VolumesPerInstance: 1
        TerminationProtected: 'false'
        AdditionalMasterSecurityGroups:
          - !Ref CertificateStoreSecurityGroup
        AdditionalSlaveSecurityGroups:
          - !Ref CertificateStoreSecurityGroup
        HadoopVersion: 3.3.3
      Name: !Ref 'AWS::StackName'
      JobFlowRole: !Ref emrEc2InstanceProfile
      ServiceRole: !Ref emrRole
      ReleaseLabel: emr-6.10.0
      AutoScalingRole: EMR_AutoScaling_DefaultRole
      ScaleDownBehavior: TERMINATE_AT_TASK_COMPLETION
      VisibleToAllUsers: true
      Applications:
        - Name: Hive
        - Name: Spark
  emrRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2008-10-17
        Statement:
          - Sid: ''
            Effect: Allow
            Principal:
              Service: elasticmapreduce.amazonaws.com
            Action: 'sts:AssumeRole'
      Path: /
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole'
  emrEc2Role:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2008-10-17
        Statement:
          - Sid: ''
            Effect: Allow
            Principal:
              Service: ec2.amazonaws.com
            Action: 'sts:AssumeRole'
      Path: /
      Policies:
        - PolicyName: EMRNodeResourcesPolicy
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - 'kms:Decrypt'
                Resource: '*'
              - Effect: Allow
                Action:
                  - 'logs:DescribeLogGroups'
                  - 'logs:DescribeLogStreams'
                  - 'logs:CreateLogGroup'
                  - 'logs:CreateLogStream'
                Resource: '*'
              - Effect: Allow
                Action: 'ssm:StartSession'
                Resource:
                  - 'arn:aws:ec2:*:*:instance/*'
                  - 'arn:aws:ssm:*:*:document/AWS-StartSSHSession'
                  - 'arn:aws:ssm:*:*:document/AWS-StartPortForwardingSession'
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role'
        - 'arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole'
        - 'arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore'
        - 'arn:aws:iam::aws:policy/CloudWatchAgentAdminPolicy'
        - 'arn:aws:iam::aws:policy/CloudWatchLogsFullAccess'
        - 'arn:aws:iam::aws:policy/AmazonSSMReadOnlyAccess'
  emrEc2InstanceProfile:
    Type: 'AWS::IAM::InstanceProfile'
    Properties:
      Path: /
      Roles:
        - !Ref InputRole
  EventBridgeEventBusPolicy:
    Type: AWS::IAM::Policy
    Properties:
      Roles:
        - !Ref InputRole
      PolicyName: EventBridgeEventBusPolicy
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Action:
              - "events:DescribeEventBus"
              - "events:PutEvents"
            Resource: "arn:aws:events:us-west-2:12345612345:event-bus/test-event-bus"
  LambdaPolicy:
    Type: AWS::IAM::Policy
    Properties:
      Roles:
        - !Ref InputRole
      PolicyName: LambdaPolicy
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Action:
              - "lambda:InvokeFunction"
            Resource: "arn:aws:lambda:us-west-2:12345612345:function:test-function"`

Thank you for the example @sli10, that was very useful. I hadn't considered the scenario where a IAM role could be created/controlled outside of CloudFormation.

Creating an `AWS::EMR::Cluster` does work for me - as long as I explicitly do a `docker pull motoserver/moto:latest`. Could there be a caching issue at play somehow?

The latest release (as of this morning) contains:
 - Delete support for `AWS::EMR::Cluster`
 - Create and Delete support for `AWS::EFS::FileSystem` (it was implemented, just incredibly broken - should be fixed now)
 - A fix for the IAM `DeleteConflict`

> One more request we have regarding the AWS::IAM::Role resource, we were wondering if we can add support for RoleId?

Yes - I'll try to get to that later this week.
@bblommers Thank you for fixing the above. The latest release has resolved the issues you've listed above and I've also tried it out and it was working on our end. 

We realized a few more things that we want to request support for. `AWS::Logs::LogGroup` doesn't seem to have delete implemented which I think is cause this issue when we delete the stack and recreate it with the same name. `b'{"__type": "ResourceAlreadyExistsException", "message": "The specified log group already exists"}'` Another resource that isn't supported is the `AWS::EMR::InstanceGroupConfig`, we were wondering if these resources can also be added to moto? 

Thank you so much for your help, we really appreciate it! 

Resolving patch:
diff --git a/moto/emr/models.py b/moto/emr/models.py
--- a/moto/emr/models.py
+++ b/moto/emr/models.py
@@ -53,7 +53,7 @@ def __init__(
         self.instance_fleet_id = instance_fleet_id
 
 
-class FakeInstanceGroup(BaseModel):
+class FakeInstanceGroup(CloudFormationModel):
     def __init__(
         self,
         cluster_id: str,
@@ -82,7 +82,7 @@ def __init__(
         self.name = name
         self.num_instances = instance_count
         self.role = instance_role
-        self.type = instance_type
+        self.instance_type = instance_type
         self.ebs_configuration = ebs_configuration
         self.auto_scaling_policy = auto_scaling_policy
         self.creation_datetime = datetime.now(timezone.utc)
@@ -122,6 +122,45 @@ def auto_scaling_policy(self, value: Any) -> None:
                         ):
                             dimension["value"] = self.cluster_id
 
+    @property
+    def physical_resource_id(self) -> str:
+        return self.id
+
+    @staticmethod
+    def cloudformation_type() -> str:
+        return "AWS::EMR::InstanceGroupConfig"
+
+    @classmethod
+    def create_from_cloudformation_json(  # type: ignore[misc]
+        cls,
+        resource_name: str,
+        cloudformation_json: Any,
+        account_id: str,
+        region_name: str,
+        **kwargs: Any,
+    ) -> "FakeInstanceGroup":
+
+        properties = cloudformation_json["Properties"]
+        job_flow_id = properties["JobFlowId"]
+        ebs_config = properties.get("EbsConfiguration")
+        if ebs_config:
+            ebs_config = CamelToUnderscoresWalker.parse_dict(ebs_config)
+        props = {
+            "instance_count": properties.get("InstanceCount"),
+            "instance_role": properties.get("InstanceRole"),
+            "instance_type": properties.get("InstanceType"),
+            "market": properties.get("Market"),
+            "bid_price": properties.get("BidPrice"),
+            "name": properties.get("Name"),
+            "auto_scaling_policy": properties.get("AutoScalingPolicy"),
+            "ebs_configuration": ebs_config,
+        }
+
+        emr_backend: ElasticMapReduceBackend = emr_backends[account_id][region_name]
+        return emr_backend.add_instance_groups(
+            cluster_id=job_flow_id, instance_groups=[props]
+        )[0]
+
 
 class FakeStep(BaseModel):
     def __init__(
@@ -292,11 +331,15 @@ def instance_groups(self) -> List[FakeInstanceGroup]:
 
     @property
     def master_instance_type(self) -> str:
-        return self.emr_backend.instance_groups[self.master_instance_group_id].type  # type: ignore
+        return self.emr_backend.instance_groups[
+            self.master_instance_group_id  # type: ignore
+        ].instance_type
 
     @property
     def slave_instance_type(self) -> str:
-        return self.emr_backend.instance_groups[self.core_instance_group_id].type  # type: ignore
+        return self.emr_backend.instance_groups[
+            self.core_instance_group_id  # type: ignore
+        ].instance_type
 
     @property
     def instance_count(self) -> int:
diff --git a/moto/emr/responses.py b/moto/emr/responses.py
--- a/moto/emr/responses.py
+++ b/moto/emr/responses.py
@@ -758,7 +758,7 @@ def remove_auto_scaling_policy(self) -> str:
               <InstanceRequestCount>{{ instance_group.num_instances }}</InstanceRequestCount>
               <InstanceRole>{{ instance_group.role }}</InstanceRole>
               <InstanceRunningCount>{{ instance_group.num_instances }}</InstanceRunningCount>
-              <InstanceType>{{ instance_group.type }}</InstanceType>
+              <InstanceType>{{ instance_group.instance_type }}</InstanceType>
               <LastStateChangeReason/>
               <Market>{{ instance_group.market }}</Market>
               <Name>{{ instance_group.name }}</Name>
@@ -1084,7 +1084,7 @@ def remove_auto_scaling_policy(self) -> str:
         {% endif %}
         <Id>{{ instance_group.id }}</Id>
         <InstanceGroupType>{{ instance_group.role }}</InstanceGroupType>
-        <InstanceType>{{ instance_group.type }}</InstanceType>
+        <InstanceType>{{ instance_group.instance_type }}</InstanceType>
         <Market>{{ instance_group.market }}</Market>
         <Name>{{ instance_group.name }}</Name>
         <RequestedInstanceCount>{{ instance_group.num_instances }}</RequestedInstanceCount>


Test patch:
diff --git a/tests/test_emr/test_emr_cloudformation.py b/tests/test_emr/test_emr_cloudformation.py
--- a/tests/test_emr/test_emr_cloudformation.py
+++ b/tests/test_emr/test_emr_cloudformation.py
@@ -632,3 +632,195 @@ def test_create_cluster_with_kerberos_attrs():
         emr.describe_security_configuration(Name="mysecconfig")
     err = exc.value.response["Error"]
     assert err["Code"] == "InvalidRequestException"
+
+
+template_with_simple_instance_group_config = {
+    "Resources": {
+        "Cluster1": {
+            "Type": "AWS::EMR::Cluster",
+            "Properties": {
+                "Instances": {
+                    "CoreInstanceGroup": {
+                        "InstanceCount": 3,
+                        "InstanceType": "m3g",
+                    }
+                },
+                "JobFlowRole": "EMR_EC2_DefaultRole",
+                "Name": "my cluster",
+                "ServiceRole": "EMR_DefaultRole",
+            },
+        },
+        "TestInstanceGroupConfig": {
+            "Type": "AWS::EMR::InstanceGroupConfig",
+            "Properties": {
+                "InstanceCount": 2,
+                "InstanceType": "m3.xlarge",
+                "InstanceRole": "TASK",
+                "Market": "ON_DEMAND",
+                "Name": "cfnTask2",
+                "JobFlowId": {"Ref": "Cluster1"},
+            },
+        },
+    },
+}
+
+
+@mock_aws
+def test_create_simple_instance_group():
+    region = "us-east-1"
+    cf = boto3.client("cloudformation", region_name=region)
+    emr = boto3.client("emr", region_name=region)
+    cf.create_stack(
+        StackName="teststack",
+        TemplateBody=json.dumps(template_with_simple_instance_group_config),
+    )
+
+    # Verify resources
+    res = cf.describe_stack_resources(StackName="teststack")["StackResources"][0]
+    cluster_id = res["PhysicalResourceId"]
+
+    ig = emr.list_instance_groups(ClusterId=cluster_id)["InstanceGroups"][0]
+    assert ig["Name"] == "cfnTask2"
+    assert ig["Market"] == "ON_DEMAND"
+    assert ig["InstanceGroupType"] == "TASK"
+    assert ig["InstanceType"] == "m3.xlarge"
+
+
+template_with_advanced_instance_group_config = {
+    "Resources": {
+        "Cluster1": {
+            "Type": "AWS::EMR::Cluster",
+            "Properties": {
+                "Instances": {
+                    "CoreInstanceGroup": {
+                        "InstanceCount": 3,
+                        "InstanceType": "m3g",
+                    }
+                },
+                "JobFlowRole": "EMR_EC2_DefaultRole",
+                "Name": "my cluster",
+                "ServiceRole": "EMR_DefaultRole",
+            },
+        },
+        "TestInstanceGroupConfig": {
+            "Type": "AWS::EMR::InstanceGroupConfig",
+            "Properties": {
+                "InstanceCount": 1,
+                "InstanceType": "m4.large",
+                "InstanceRole": "TASK",
+                "Market": "ON_DEMAND",
+                "Name": "cfnTask3",
+                "JobFlowId": {"Ref": "Cluster1"},
+                "EbsConfiguration": {
+                    "EbsOptimized": True,
+                    "EbsBlockDeviceConfigs": [
+                        {
+                            "VolumesPerInstance": 2,
+                            "VolumeSpecification": {
+                                "Iops": 10,
+                                "SizeInGB": 50,
+                                "Throughput": 100,
+                                "VolumeType": "gp3",
+                            },
+                        }
+                    ],
+                },
+                "AutoScalingPolicy": {
+                    "Constraints": {"MinCapacity": 1, "MaxCapacity": 4},
+                    "Rules": [
+                        {
+                            "Name": "Scale-out",
+                            "Description": "Scale-out policy",
+                            "Action": {
+                                "SimpleScalingPolicyConfiguration": {
+                                    "AdjustmentType": "CHANGE_IN_CAPACITY",
+                                    "ScalingAdjustment": 1,
+                                    "CoolDown": 300,
+                                }
+                            },
+                            "Trigger": {
+                                "CloudWatchAlarmDefinition": {
+                                    "Dimensions": [
+                                        {
+                                            "Key": "JobFlowId",
+                                            "Value": "${emr.clusterId}",
+                                        }
+                                    ],
+                                    "EvaluationPeriods": 1,
+                                    "Namespace": "AWS/ElasticMapReduce",
+                                    "Period": 300,
+                                    "ComparisonOperator": "LESS_THAN",
+                                    "Statistic": "AVERAGE",
+                                    "Threshold": 15,
+                                    "Unit": "PERCENT",
+                                    "MetricName": "YARNMemoryAvailablePercentage",
+                                }
+                            },
+                        },
+                        {
+                            "Name": "Scale-in",
+                            "Description": "Scale-in policy",
+                            "Action": {
+                                "SimpleScalingPolicyConfiguration": {
+                                    "AdjustmentType": "CHANGE_IN_CAPACITY",
+                                    "ScalingAdjustment": -1,
+                                    "CoolDown": 300,
+                                }
+                            },
+                            "Trigger": {
+                                "CloudWatchAlarmDefinition": {
+                                    "Dimensions": [
+                                        {
+                                            "Key": "JobFlowId",
+                                            "Value": "${emr.clusterId}",
+                                        }
+                                    ],
+                                    "EvaluationPeriods": 1,
+                                    "Namespace": "AWS/ElasticMapReduce",
+                                    "Period": 300,
+                                    "ComparisonOperator": "GREATER_THAN",
+                                    "Statistic": "AVERAGE",
+                                    "Threshold": 75,
+                                    "Unit": "PERCENT",
+                                    "MetricName": "YARNMemoryAvailablePercentage",
+                                }
+                            },
+                        },
+                    ],
+                },
+            },
+        },
+    },
+}
+
+
+@mock_aws
+def test_create_advanced_instance_group():
+    region = "us-east-1"
+    cf = boto3.client("cloudformation", region_name=region)
+    emr = boto3.client("emr", region_name=region)
+    cf.create_stack(
+        StackName="teststack",
+        TemplateBody=json.dumps(template_with_advanced_instance_group_config),
+    )
+
+    # Verify resources
+    res = cf.describe_stack_resources(StackName="teststack")["StackResources"][0]
+    cluster_id = res["PhysicalResourceId"]
+
+    ig = emr.list_instance_groups(ClusterId=cluster_id)["InstanceGroups"][0]
+    assert ig["Name"] == "cfnTask3"
+    assert ig["Market"] == "ON_DEMAND"
+    assert ig["InstanceGroupType"] == "TASK"
+    assert ig["InstanceType"] == "m4.large"
+
+    as_policy = ig["AutoScalingPolicy"]
+    assert as_policy["Status"] == {"State": "ATTACHED"}
+    assert as_policy["Constraints"] == {"MinCapacity": 1, "MaxCapacity": 4}
+
+    ebs = ig["EbsBlockDevices"]
+    assert ebs[0]["VolumeSpecification"] == {
+        "VolumeType": "gp3",
+        "Iops": 10,
+        "SizeInGB": 50,
+    }