Repository: aws-ia/terraform-aws-eks-blueprints

Issue #559: [Bug]: Managed IAM polices are getting detached during Node groups create/update/delete
Created: 2022-05-17T17:23:32Z
Welcome to Amazon EKS Blueprints!

 Yes, I've searched similar issues on GitHub and didn't find any.

Amazon EKS Blueprints Release version
v4.0.4
What is your environment, configuration and the example used?
Managed IAM polices attached to IAM roles(IRSA) are getting detached while adding new Managed node groups, update node groups or delete node groups.  Detached Managed policies are not getting attached only when the TF apply fails during Nodegroup create, update or delete.
e.g., Managed IAM polices -> arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy , arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
TF Plan that shows the force replacement for IRSA Add-ons with managed policies
  # module.eks_blueprints_kubernetes_addons.module.aws_cloudwatch_metrics[0].module.helm_addon.module.irsa[0].aws_iam_role_policy_attachment.irsa[0] must be replaced
+/- resource "aws_iam_role_policy_attachment" "irsa" {
      ~ id         = "aws001-preprod-dev-eks-cloudwatch-agent-irsa-20220517110206724700000008" -> (known after apply)
      ~ policy_arn = "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy" -> (known after apply) # forces replacement
        # (1 unchanged attribute hidden)
    }

 # module.eks_blueprints_kubernetes_addons.module.aws_vpc_cni[0].module.irsa_addon[0].aws_iam_role_policy_attachment.irsa[0] must be replaced
+/- resource "aws_iam_role_policy_attachment" "irsa" {
      ~ id         = "aws001-preprod-dev-eks-aws-node-irsa-20220517110206713100000007" -> (known after apply)
      ~ policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy" -> (known after apply) # forces replacement
        # (1 unchanged attribute hidden)
    }


Steps to reproduce
Step1: Deploy the new VPC example
Step2: Add additional Managed node group to the example and run terraform apply again
    mg5 = {
      node_group_name = "mg5"
      instance_types  = ["m5.large"]
      min_size        = "2"
    }
Check out the Terraform Apply log and you should observe the above TF Plan log that shows force replacement for two managed policies
_Step2 log that shows the Managed policy(aws_vpc_cni[0].module.irsa_addon[0].aws_iam_role_policy_attachment) deletion, node group create and Managed policy creation . This will not be an issue if deletion and creation are being done by one action.
module.eks_blueprints_kubernetes_addons.module.aws_cloudwatch_metrics[0].module.helm_addon.module.irsa[0].aws_iam_role_policy_attachment.irsa[0]: Destroying... [id=aws001-preprod-dev-eks-cloudwatch-agent-irsa-20220517100651834500000009]
module.eks_blueprints_kubernetes_addons.module.aws_vpc_cni[0].module.irsa_addon[0].aws_iam_role_policy_attachment.irsa[0]: Destroying... [id=aws001-preprod-dev-eks-aws-node-irsa-2022051710065183460000000a]
module.eks_blueprints_kubernetes_addons.module.aws_cloudwatch_metrics[0].module.helm_addon.module.irsa[0].aws_iam_role_policy_attachment.irsa[0]: Destruction complete after 2s
module.eks_blueprints_kubernetes_addons.module.aws_vpc_cni[0].module.irsa_addon[0].aws_iam_role_policy_attachment.irsa[0]: Destruction complete after 2s
module.eks_blueprints.kubernetes_config_map.aws_auth[0]: Modifying... [id=kube-system/aws-auth]
module.eks_blueprints.kubernetes_config_map.aws_auth[0]: Modifications complete after 4s [id=kube-system/aws-auth]
module.eks_blueprints.module.aws_eks_managed_node_groups["mg5"].data.aws_iam_policy_document.managed_ng_assume_role_policy: Reading... [id=3778018924]
module.eks_blueprints.module.aws_eks_managed_node_groups["mg4"].data.aws_iam_policy_document.managed_ng_assume_role_policy: Reading...
module.eks_blueprints.module.aws_eks_managed_node_groups["mng_custom_ami"].data.aws_iam_policy_document.managed_ng_assume_role_policy: Reading... [id=3778018924]
module.eks_blueprints.module.aws_eks_managed_node_groups["mng_lt"].data.aws_iam_policy_document.managed_ng_assume_role_policy: Reading... [id=3778018924]
module.eks_blueprints.module.aws_eks_managed_node_groups["mng_lt"].data.aws_iam_policy_document.managed_ng_assume_role_policy: Read complete after 0s [id=3778018924]
module.eks_blueprints.module.aws_eks_managed_node_groups["mng_custom_ami"].data.aws_iam_policy_document.managed_ng_assume_role_policy: Read complete after 0s [id=3778018924]
module.eks_blueprints.module.aws_eks_managed_node_groups["mg4"].data.aws_iam_policy_document.managed_ng_assume_role_policy: Read complete after 0s [id=3778018924]
module.eks_blueprints.module.aws_eks_managed_node_groups["mg5"].data.aws_iam_policy_document.managed_ng_assume_role_policy: Read complete after 0s [id=3778018924]
module.eks_blueprints.module.aws_eks_managed_node_groups["mg4"].aws_eks_node_group.managed_ng: Creating...
module.eks_blueprints.module.aws_eks_managed_node_groups["mg4"].aws_eks_node_group.managed_ng: Still creating... [10s elapsed]
module.eks_blueprints.module.aws_eks_managed_node_groups["mg4"].aws_eks_node_group.managed_ng: Still creating... [20s elapsed]
module.eks_blueprints.module.aws_eks_managed_node_groups["mg4"].aws_eks_node_group.managed_ng: Still creating... [2m30s elapsed]
module.eks_blueprints.module.aws_eks_managed_node_groups["mg4"].aws_eks_node_group.managed_ng: Still creating... [2m40s elapsed]
module.eks_blueprints.module.aws_eks_managed_node_groups["mg4"].aws_eks_node_group.managed_ng: Still creating... [2m50s elapsed]
module.eks_blueprints.module.aws_eks_managed_node_groups["mg4"].aws_eks_node_group.managed_ng: Creation complete after 2m50s [id=aws001-preprod-dev-eks:mg4-20220517102100523200000005]
module.eks_blueprints_kubernetes_addons.data.aws_region.current: Reading... [id=eu-west-1]
module.eks_blueprints_kubernetes_addons.data.aws_partition.current: Reading... [id=aws]
module.eks_blueprints_kubernetes_addons.data.aws_caller_identity.current: Reading... [id=327949925549]
module.eks_blueprints_kubernetes_addons.module.cluster_autoscaler[0].data.aws_iam_policy_document.cluster_autoscaler: Reading... [id=1560876111]
module.eks_blueprints_kubernetes_addons.data.aws_eks_cluster.eks_cluster: Reading... [id=aws001-preprod-dev-eks]
module.eks_blueprints_kubernetes_addons.data.aws_partition.current: Read complete after 0s [id=aws]
module.eks_blueprints_kubernetes_addons.data.aws_region.current: Read complete after 0s [id=eu-west-1]
module.eks_blueprints_kubernetes_addons.module.cluster_autoscaler[0].data.aws_iam_policy_document.cluster_autoscaler: Read complete after 0s [id=1560876111]
module.eks_blueprints_kubernetes_addons.data.aws_eks_cluster.eks_cluster: Read complete after 0s [id=aws001-preprod-dev-eks]
module.eks_blueprints_kubernetes_addons.data.aws_caller_identity.current: Read complete after 2s [id=327949925549]
module.eks_blueprints_kubernetes_addons.module.aws_load_balancer_controller[0].data.aws_iam_policy_document.aws_lb: Reading... [id=2593514191]
module.eks_blueprints_kubernetes_addons.module.aws_cloudwatch_metrics[0].module.helm_addon.module.irsa[0].aws_iam_role_policy_attachment.irsa[0]: Creating...
module.eks_blueprints_kubernetes_addons.module.aws_vpc_cni[0].module.irsa_addon[0].aws_iam_role_policy_attachment.irsa[0]: Creating...
module.eks_blueprints_kubernetes_addons.module.aws_load_balancer_controller[0].data.aws_iam_policy_document.aws_lb: Read complete after 0s [id=2593514191]
module.eks_blueprints_kubernetes_addons.module.aws_vpc_cni[0].module.irsa_addon[0].aws_iam_role_policy_attachment.irsa[0]: Creation complete after 1s [id=aws001-preprod-dev-eks-aws-node-irsa-20220517102354178200000007]
module.eks_blueprints_kubernetes_addons.module.aws_cloudwatch_metrics[0].module.helm_addon.module.irsa[0].aws_iam_role_policy_attachment.irsa[0]: Creation complete after 1s [id=aws001-preprod-dev-eks-cloudwatch-agent-irsa-20220517102354210800000008]
aws_iam_role_policy_attachment force replacement is being executed as two steps (delete and create) but they are not executed sequentially by the terraform aws provider. First it executes delete operation which detaches the policy (e.g., CNI) and the next step is to create a new node group but this fails due to missing CNI policy. Create CNI policy supposed to be executed right after the node group creation but this will never happen due to the above failure.
What did you do and What did you see instead?
Workaround:

Execute TF Apply again which identifies the detached Managed policies and try to apply on the second run.
or
Use custom policies with Add-ons with IRSA instead of Managed policies.

Additional Information
Note: I created this issue to make sure you all aware of the known issue with Managed Policies. We are trying to workout the better way to handle this issue. Ideas are welcome
Related issues are ...
terraform-aws-modules/terraform-aws-eks#1248

Duplicate issue #579: [Bug]: node group update fails when VPC CNI addon is enabled
Created: 2022-05-26T07:02:02Z
Welcome to Amazon EKS Blueprints!

 Yes, I've searched similar issues on GitHub and didn't find any.

Amazon EKS Blueprints Release version
4.0.4
What is your environment, configuration and the example used?
Terraform version:
Terraform v1.1.8
Provider version(s):
Terraform v1.1.8
on linux_amd64
provider registry.terraform.io/gavinbunney/kubectl v1.14.0
provider registry.terraform.io/hashicorp/aws v4.14.0
provider registry.terraform.io/hashicorp/cloudinit v2.2.0
provider registry.terraform.io/hashicorp/helm v2.5.1
provider registry.terraform.io/hashicorp/kubernetes v2.11.0
provider registry.terraform.io/hashicorp/local v2.2.3
provider registry.terraform.io/hashicorp/null v3.1.1
provider registry.terraform.io/hashicorp/time v0.7.2
provider registry.terraform.io/hashicorp/tls v3.4.0
provider registry.terraform.io/terraform-aws-modules/http v2.4.1
What did you do and What did you see instead?
I've found some bug in EKS blueprint when VPC_CNI addon is turned on. Editing a node group (for example adding a new one) fails with the timeout because the role assigned to the nodes has policy attachment removed.
How to reproduce
The code is taken from
https://aws-ia.github.io/terraform-aws-eks-blueprints/getting-started/
and as this tutorial says
terraform-aws-eks-blueprints/examples/eks-cluster-with-new-vpc example is taken as a start
then getting started tutorial is executed but skipping cleanup section - the EKS is working
Steps to reproduce:
execute steps from https://aws-ia.github.io/terraform-aws-eks-blueprints/getting-started/ but skip cleanup section
so now there is a cluster up and running
then:
add the following
mg_test = {
          node_group_name = "managed-test"
          instance_types  = ["m5.large"]
          min_size        = "1"
          subnet_ids      = module.aws_vpc.private_subnets
        }

in that part:
  # EKS MANAGED NODE GROUPS
  managed_node_groups = {

then execute terraform apply
the very first step terraform does is to remove policy attachment from IRSA role:
Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.eks_blueprints_kubernetes_addons.module.aws_cloudwatch_metrics[0].module.helm_addon.module.irsa[0].aws_iam_role_policy_attachment.irsa[0]: Destroying... [id=aws001-preprod-dev-eks-cloudwatch-agent-irsa-2022051909043063440000000e]
module.eks_blueprints_kubernetes_addons.module.aws_vpc_cni[0].module.irsa_addon[0].aws_iam_role_policy_attachment.irsa[0]: Destroying... [id=aws001-preprod-dev-eks-aws-node-irsa-2022051909043068410000000f]
module.eks_blueprints_kubernetes_addons.module.aws_vpc_cni[0].module.irsa_addon[0].aws_iam_role_policy_attachment.irsa[0]: Destruction complete after 1s

( Note that the removed policy is arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy )
then terraform tries to create a new node group
 module.eks_blueprints.module.aws_eks_managed_node_groups["mg_ski"].aws_iam_role.managed_ng: Creating...
but it times out after 20 minutes
│ Error: error waiting for EKS Node Group (aws001-preprod-dev-eks:managed-ski-20220519091022118900000005) to create: unexpected state 'CREATE_FAILED', wanted target 'ACTIVE'. last error: 1 error occurred:
│ 	* i-06576ff0ac5971553, i-0849e995e20cbf19b: NodeCreationFailure: Instances failed to join the kubernetes cluster

Node group gets "Create failed" status and nodes have "Not ready" status
Instances are up and running but:
aws-node pod is restarting with
{"level":"info","ts":"2022-05-19T09:25:57.154Z","caller":"entrypoint.sh","msg":"Retrying waiting for IPAM-D"}
and ipamd.log shows
tail -n 3 /var/log/aws-routed-eni/ipamd.log
{"level":"error","ts":"2022-05-19T09:26:17.788Z","caller":"ipamd/ipamd.go:456","msg":"Failed to call ec2:DescribeNetworkInterfaces for [eni-0721b5e0b27ea7275]: UnauthorizedOperation: You are not authorized to perform this operation.\n\tstatus code: 403, request id: 0e88218f-037c-4838-a347-ee4fa254f495"}
{"level":"error","ts":"2022-05-19T09:26:17.903Z","caller":"ipamd/ipamd.go:456","msg":"Failed to call ec2:DescribeNetworkInterfaces for [eni-0721b5e0b27ea7275]: UnauthorizedOperation: You are not authorized to perform this operation.\n\tstatus code: 403, request id: b8133cd9-a889-4e5c-b191-5d970f0dc7cf"}
{"level":"error","ts":"2022-05-19T09:26:18.013Z","caller":"aws-k8s-agent/main.go:27","msg":"Initialization failure: ipamd init: failed to retrieve attached ENIs info"}

I confirmed that all above happens due to arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy removal from IRSA role used by aws-node pods
Additional Information
This issue doesn't exist when VPC_CNI addon is turned off (aws-node pods don't use IRSA role then)
enable_amazon_eks_vpc_cni = false

Closing comment (2022-05-26T09:42:49Z): Related to #559