Terraformのeks moduleを利用してEKSクラスターのデプロイと踏み台サーバのデプロイ

以下のTerraformのeks moduleを利用してEKSのデプロイを行ってみる
- [Terraform module to create AWS Elastic Kubernetes (EKS) resources](https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest)

合わせてEKSのcontrole_planeに接続する踏み台サーバもデプロイする
今回構築する構成は以下のイメージ
![20240511_AWS-EKS構成図.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3472468/9ad62fc6-6e76-edc6-fecb-0c3c87f9595f.png)


# ローカルPCへAWS CLIインストール
ローカルのWindows PCに以下コマンドでAWS CLIをインストールする
```
$ winget install -e --id Amazon.AWSCLI
```
以下コマンドでAWS認証情報を設定
AWS上にリソースをデプロイ可能なIAMユーザの情報を設定する
```
$ aws configure
AWS Access Key ID [None]: <AWS IAMユーザのAccess Key ID>
AWS Secret Access Key [None]: <AWS IAMユーザのSecret Access Key>
Default region name [None]: ap-northeast-1
Default output format [None]: json
```

# IAM Role/Policyの作成
EKS用のIAM RoleとPolicyを作成して、ユーザに紐づけておく
ここで紐づけたユーザから`kubectl`でK8sクラスタのコントロールが行える

## IAM Roleの作成
- カスタム信頼ポリシー
```
{
"Version": "2012-10-17",
"Statement": [
	{
		"Sid": "Statement1",
		"Effect": "Allow",
		"Principal": {
		    "Service": "ec2.amazonaws.com",
			"AWS": "arn:aws:iam::<AWSアカウントID>:root"
		},
		"Action": "sts:AssumeRole"
	}
]
}
```
- 許可ポリシー
```
AdministratorAccess
```
- ロール名
```
ex-man-nodeG-profile-iam-assume-role
```

## AWSコンソールからIAMポリシーを作成
- アクセス許可
```
{
"Version": "2012-10-17",
"Statement": [
	{
		"Sid": "Statement1",
		"Effect": "Allow",
		"Action": "sts:AssumeRole",
		"Resource": "arn:aws:iam::<AWSアカウントID>:role/ex-man-nodeG-profile-iam-assume-role"
	}
]
}
```
- ポリシー名
```
ex-man-nodeG-profile-iam-assume-policy
```

## AWSコンソールからIAMユーザにAssumeポリシーの許可を追加
- 今回はAWS用にリソースをデプロイするIAMユーザに許可を追加
- ポリシーを直接アタッチする -> `ex-man-nodeG-profile-iam-assume-policy`
- 別途EKSクラスタを管理する専用ユーザを作成してアサインしても大丈夫なはず

# Terraform マニフェストファイル(tfファイル)の作成

## ベースファイルのコピー(fargate_profileフォルダ)
特に決まりはないが、今回は`examples/fargate_profile`フォルダ配下のファイルを作業フォルダにコピー

## EKSクラスターのデプロイ
`main.tf`は[Terraform module to create AWS Elastic Kubernetes (EKS) resources Repository](https://github.com/terraform-aws-modules/terraform-aws-eks)の`usage`の記載をベースとする
以下ファイルを編集
- main.tf
- variables.tf
- terraform.tfvars
- outputs.tf

### main.tfの編集
修正点は以下の通り
- リージョン設定とtfstateをS3上で管理するためのbackend設定の追加
```
# Setting of AWS Provider
provider "aws" {
  region     = var.region
}

terraform {
  backend "s3" {
    bucket = "tfstate-bucket-2024"
    key = "backend/terraform.tfstate"
    region = "ap-northeast-1"
    dynamodb_table = "tfstate-lockdb-2024"
    encrypt = true
  }
}
```
- eks関連のmoduleを外部呼出し可能とするために`module`エントリの`source`フィールドの`../..` -> `terraform-aws-modules/eks/aws`に書き換える
```
module "eks" {
- source = "../.."
+  source = "terraform-aws-modules/eks/aws"
...
```
-  いくつかのパラメータを変数で指定できるように修正
```
locals {
-   name            = "ex-${replace(basename(path.cwd), "_", "-")}"
+  name            = var.eks_cluster_name
 
- cluster_version = "1.29"
+ cluster_version = var.cluster_version

- region          = "eu-west-1"
+ region          = var.region

- vpc_cidr = "10.0.0.0/16"
+  vpc_cidr = var.vpc_cidr
  azs      = slice(data.aws_availability_zones.available.names, 0, 3)

  tags = {
    Example    = local.name
    GithubRepo = "terraform-aws-eks"
    GithubOrg  = "terraform-aws-modules"
  }
}
```

k8sの認証/認可連携はここの部分がポイントと考えられる
- 変更するのはIAM Roleのarn部分を最初に作成したEKS用のRoleのarnに修正する
- この記述がないとeks clusterが作成されても、`kubectl`ができない
```
  # Cluster access entry
  # To add the current caller identity as an administrator
  enable_cluster_creator_admin_permissions = true

  access_entries = {
    # One access entry with a policy associated
    example = {
      kubernetes_groups = []
-  principal_arn     = "arn:aws:iam::123456789012:role/something"
+       principal_arn     = "arn:aws:iam::${var.aws_account_id}:role/${var.aws_iam_role}"

      policy_associations = {
        example = {
          policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy"
          access_scope = {
            namespaces = ["default"]
            type       = "namespace"
          }
        }
      }
    }
  }
```

VPC作成部分は`examples/fargate_profile`の`main.tf`を参考に以下の部分をコピー
動的にアドレスアサインが行えるようにした
```
################################################################################
# Supporting Resources
################################################################################

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = local.name
  cidr = local.vpc_cidr

  azs             = local.azs
  private_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 4, k)]
  public_subnets  = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k + 48)]
  intra_subnets   = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k + 52)]

  enable_nat_gateway = true
  single_nat_gateway = true

  public_subnet_tags = {
    "kubernetes.io/role/elb" = 1
  }

  private_subnet_tags = {
    "kubernetes.io/role/internal-elb" = 1
  }

  tags = local.tags
}
```

### variables.tfの編集
もともと空ファイルなので変数化したパラメータの定義を追加
```
variable "region" {
  description = "AWS_REGION"
  type        = string
  default     = "ap-northeast-1"
}

variable "vpc_cidr" {
  description = "AWS_VPC_cidr_for_EKS"
  type = string
}

variable "eks_cluster_name" {
  description = "AWS_EKS_Cluster_name"
  type = string
}

variable "cluster_version" {
  description = "AWS_EKS_Cluster_version"
  type = string
}

variable "aws_account_id" {
  description = "AWS_Account_ID"
  type = string
}

variable "aws_iam_role" {
  description = "AWS_IAM_Role_for_EKS"
  type = string
}
```

### terraform.tfvarsの追加
ファイルが存在しないためファイルを追加して、変数化したパラメータの値を定義
```
region     = "ap-northeast-1"

Eks_cluster_name = "ex-man-nodeG-profile"
cluster_version = "1.29"
vpc_cidr = "10.8.0.0/16"

aws_account_id = "<AWSアカウントID>"
aws_iam_role = "ex-man-nodeG-profile-iam-assume-role"

```

## BastionVMとTransit用subnetの追加
BastionVM用subnetとTransit用subnetの追加
### main.tfの編集
修正点は以下の通り
- BastionVM用subnetとtransit用subnetの追加
```
resource "aws_subnet" "bastionvm_subnet" {
  vpc_id = module.vpc.vpc_id

  availability_zone = var.bastionvm_avz
  cidr_block        = var.bastionvm_cidr
  map_public_ip_on_launch = "true"

  tags = {
    Name = "${local.name}-bastionvm-subnet-${var.bastionvm_avz}]"
  }
}

resource "aws_subnet" "transit_subnet" {
  vpc_id = module.vpc.vpc_id

  availability_zone = var.transit_avz
  cidr_block        = var.transit_cidr

  tags = {
    Name = "${local.name}-transit-subnet-${var.transit_avz}]"
  }
}
```

- BastionVMにインターネットから接続できるようにするために、BastionVM用subnetをパブリックroute_tableに紐づける
```
resource "aws_route_table_association" "rta_bastionvm_public_subnet" {
  subnet_id = "${aws_subnet.bastionvm_subnet.id}"
  route_table_id = "${module.vpc.public_route_table_ids[0]}"
}
```

- 同じくtransit用subnetをパブリックroute_tableに紐づける
```
resource "aws_route_table_association" "rta_transit_public_subnet" {
  subnet_id = "${aws_subnet.transit_subnet.id}"
  route_table_id = "${module.vpc.public_route_table_ids[0]}"
} 
```

### variables.tfの編集
subnet追加用tfファイルで使用する変数定義を追加
```
variable "bastionvm_avz" {
  description = "AWS_BastionVM_availabilityzone"
  type = string
}

variable "bastionvm_cidr" {
  description = "AWS_BastionVM_cidr"
  type = string
}

variable "transit_avz" {
  description = "AWS_Transit_availabilityzone"
  type = string
}

variable "transit_cidr" {
  description = "AWS_Transit_cidr"
  type = string
}

```

### terraform.tfvarsの編集
subnet追加用tfファイルで追加したパラメータの値を定義
```
bastionvm_avz = "ap-northeast-1a"
bastionvm_cidr = "10.8.56.0/24"

transit_avz = "ap-northeast-1a"
transit_cidr = "10.8.60.0/28"
```

## BastionVM用instanceとsecurity groupの追加
### user_data.shの作成(`kubectl`と`AWS CLI`と`eksctl`のインストール)
BastionVMに`kubectl`と`AWS CLI`と`eksctl`をインストールする
インスタンス作成時にshellを自動実行するためのファイルを作成する

#### BastionVMにkubectlをインストールする
BastionVM上で以下コマンドを実行する
```
$ sudo apt update
$ sudo apt install -y apt-transport-https ca-certificates curl
$ curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
$ sudo chmod 644 /etc/apt/keyrings/kubernetes-apt-keyring.gpg
$ echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
$ sudo chmod 644 /etc/apt/sources.list.d/kubernetes.list
$ sudo apt update
$ sudo apt install -y kubectl
```

#### BastionVMにAWS CLIをインストールする
BastionVM上で以下コマンドを実行する
```
$ sudo apt install -y unzip
$ curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
$ unzip awscliv2.zip
$ sudo ./aws/install
```

#### BastionVMにeksctlをインストールする
BastionVM上で以下コマンドを実行する
```
$ export ARCH=amd64
$ export PLATFORM=$(uname -s)_$ARCH
$ curl -sLO "https://github.com/eksctl-io/eksctl/releases/latest/download/eksctl_$PLATFORM.tar.gz"
$ tar -xzf eksctl_$PLATFORM.tar.gz -C /tmp && rm eksctl_$PLATFORM.tar.gz
$ sudo mv /tmp/eksctl /usr/local/bin
```

#### すべてを実行するuser_data.shを作成
- Linuxマシン上で実行するため、改行コードを`LF`で保存しておかないと不要なエラーが発生する
```
#!bin/bash
echo "start UserData"
echo "start installation of kubectl"
sudo apt update
sudo apt install -y apt-transport-https ca-certificates curl
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
sudo chmod 644 /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo chmod 644 /etc/apt/sources.list.d/kubernetes.list
sudo apt update
sudo apt install -y kubectl
echo "end installation of kubectl"
echo "start installation of AWS CLI"
sudo apt install -y unzip
curl https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip -o awscliv2.zip
unzip awscliv2.zip
sudo ./aws/install
echo "end installation of AWS CLI"
echo "start installation of eksctl"
curl -sLO https://github.com/eksctl-io/eksctl/releases/latest/download/eksctl_Linux_amd64.tar.gz
tar -xzf eksctl_Linux_amd64.tar.gz -C /tmp && rm eksctl_Linux_amd64.tar.gz
sudo mv /tmp/eksctl /usr/local/bin
echo "end installation of eksctl"
echo "end UserData"
```

### main.tfの編集
修正点は以下の通り
- BastionVM用security groupの追加
- outboundはAny permit, inboundは特定アドレスからのssh/RDPのみを許可する
```
# Security Group for BastionVM
resource "aws_security_group" "for_bastionvm_public_sg" {
  name        = "${local.name}-bastionvm-public-sg"
  description = "Allow BastionVM traffic from public"
  vpc_id      = module.vpc.vpc_id
  tags = {
    Name = "${local.name}-bastionvm-public-sg"
  }
}

# For Outbound Any permit
resource "aws_security_group_rule" "public_outbound_any" {
  description = "Any_permit"
  type        = "egress"
  from_port   = 0
  to_port     = 0
  protocol    = "-1"
  cidr_blocks = ["0.0.0.0/0"] # Any

  # Matching to for_BastionVM_sg
  security_group_id = aws_security_group.for_bastionvm_public_sg.id
}

# For SSH
resource "aws_security_group_rule" "public_inbound_ssh" {
  description = "SSH"
  type        = "ingress"
  from_port   = 22
  to_port     = 22
  protocol    = "tcp"
  cidr_blocks = [var.my_term_ip] # My Terminal IP

  # Matching to for_BastionVM_sg
  security_group_id = aws_security_group.for_bastionvm_public_sg.id
}

# For RDP
resource "aws_security_group_rule" "public_inbound_RDP" {
  description = "RDP"
  type        = "ingress"
  from_port   = 3389
  to_port     = 3389
  protocol    = "tcp"
  cidr_blocks = [var.my_term_ip] # My Terminal IP

  # Matching to for_BastionVM_sg
  security_group_id = aws_security_group.for_bastionvm_public_sg.id
}
```

- BastionVM用インスタンスのデプロイ
- 複数NIC(ENI)をアサインして、一つはpublic subnetへ接続し、もう一つはintra subnetへ接続する
- Intra_nicにはEKSのControl Plane用のSecurity groupをアタッチしておく。Security groupではこのSecurity groupを送信元とするinboundトラフィックが許可されているため
- インスタンス作成時に`user_data.sh`を実行する
- インスタンスに接続するためのパブリックIPはEIPとして作成する
```
################################################################################
# Addtional EC2 instance
################################################################################
# Create EC2 NIC(ENI)
resource "aws_network_interface" "bastionvm_public_nic" {
  subnet_id       = aws_subnet.bastionvm_subnet.id
  security_groups = [aws_security_group.for_bastionvm_public_sg.id]
  tags = {
    Name = "BastionVM_public_nic"
  }
}

resource "aws_network_interface" "bastionvm_intra_nic" {
  subnet_id       = module.vpc.intra_subnets[0]
  security_groups = [module.eks.cluster_primary_security_group_id]
  tags = {
    Name = "BastionVM_intra_nic"
  }
}

# user_data for bastionvm
data "template_file" "user_data" {
  template = "${file("${path.module}/user_data.sh")}"
}

# Deploy EC2 instance
resource "aws_instance" "bastionvm_instance" {
  ami           = var.bastionvm_imageid
  instance_type = var.bastionvm_instance_type
  key_name      = var.my_key_name
  network_interface {
    network_interface_id = aws_network_interface.bastionvm_public_nic.id
    device_index         = 0
  }
  network_interface {
    network_interface_id = aws_network_interface.bastionvm_intra_nic.id
    device_index         = 1
  }
  root_block_device {
    volume_type = "gp2"
    volume_size = "8"
  }
  user_data = "${data.template_file.user_data.rendered}"

  tags = {
    Name = "${local.name}-bastionvm-instance"
  }
}

# Create EIP for BastionVM
resource "aws_eip" "bastionvm_eip" {
  network_interface = aws_network_interface.bastionvm_public_nic.id
  domain = "vpc"
  tags = {
    Name = "BastionVM_eip"
  }
}
```

### variables.tfの編集
WorkVM用tfファイルで使用する変数定義を追加
```
variable "my_term_ip" {
  description = "My Terminal addr"
  type = string
}

variable "my_key_name" {
  description = "EC2_SSH_Key_Pair_Name"
  type        = string
}

variable "bastionvm_imageid" {
  description = "AWS_BastionVM_EC2_Image_ID"
  type        = string
}

variable "bastionvm_instance_type" {
  description = "AWS_BastionVM_instance_type"
  type = string
}
```

### terraform.tfvarsの修正
BastionVM用tfファイルで追加したパラメータの値を定義
- BastionVMイメージはUbuntu Server 24.04 LTSとした
- `my_term_ip`でBastionVMに接続するIPアドレスを指定する
- `my_key_name`はBastionVMに接続するユーザのSSHキーペアを指定する。AWS IAMであらかじめ作成して、秘密鍵ファイル(pemファイル)をダウンロードしておく
```
bastionvm_imageid = "ami-01bef798938b7644d" # Ubuntu Server 24.04 LTS (HVM), SSD Volume Type
bastionvm_instance_type = "t2.micro"
my_term_ip = "<踏み台サーバへの接続を許可するIPアドレス>/xx"
my_key_name = "<BastionVM用キーペア名>"
```

### output.tfの修正
Transit_subnetとBastionVMに関して出力したい情報を追加しておく
```
################################################################################
# Transit subnet
################################################################################
output "transit_subnets" {
  description = "Subnet for Transit"
  value = aws_subnet.transit_subnet
}

################################################################################
# BastionVM
################################################################################
output "bastionvm_subnets" {
  description = "Subnet for bastionVM"
  value = aws_subnet.bastionvm_subnet
}

output "bastionvm_public_ip" {
  description = "Public IP address of bastionVM"
  value = aws_eip.bastionvm_eip.public_ip
}
```

# terraformの確認と実行
以下コマンドで初期化する
backendの変更やmoduleを追加したときは実行が必要
```
$ terraform init
```

適用前にSyntax errorを確認するには以下のコマンドを実行
```
$ terraform plan
```

実際の適用は以下コマンドを実行
```
$ terraform apply
```
EKSクラスタの作成には20分くらいかかる

# BastionVMへの接続
AWSコンソールでBastionVM用インスタンスのパブリックIPアドレスを確認する


TeratermでパブリックIPアドレスを宛先としてSSH接続する
- ユーザ名は`ubuntu`
- RSA秘密鍵の使用を選択して秘密鍵ファイルを選択する

## static routeの追加
EKS Clusterのcontrol_plane subnetにアクセスするためには、enX1の先にあるNext hop経由で通信する必要がある
今回availability zoneを3つに分けて/24のセグメントが3つできるので、そちらへの通信のためのstatic routeを永続的に追加する
`/etc/netplan/50-cloud-init.yaml`に`+`の部分を追加する
```
$ sudo vi /etc/netplan/50-cloud-init.yaml

network:
    ethernets:
        enX0:
            dhcp4: true
            dhcp4-overrides:
                route-metric: 100
            dhcp6: false
            match:
                macaddress: 06:f1:42:b3:4d:87
            set-name: enX0
        enX1:
            dhcp4: true
            dhcp4-overrides:
                route-metric: 200
                use-routes: true
            dhcp6: false
            match:
                macaddress: 06:69:d2:de:fc:9f
            routes:
+            -   to: 10.8.52.0/22
+                via: 10.8.52.1
            -   table: 101
                to: 0.0.0.0/0
                via: 10.8.52.1
            -   table: 101
                to: 10.8.52.0/24
            routing-policy:
            -   from: 10.8.52.187
                table: 101
            set-name: enX1
    version: 2
```
設定を適用するには以下コマンドを実行する
```
$ sudo netplan apply
```
static routeが追加されたことを確認
```
$ ip route show

...
10.8.52.0/22 via 10.8.52.1 dev enX1 proto static onlink
...
```

## AWS CLIの認証情報設定
BationVM上で以下コマンドでAWS認証情報を設定
```
$ aws configure
AWS Access Key ID [None]: <AWS IAMユーザのAccess Key ID>
AWS Secret Access Key [None]: <AWS IAMユーザのSecret Access Key>
Default region name [None]: ap-northeast-1
Default output format [None]: json
```

## kubectlからEKSへの接続
### kubeconfigファイルの作成
接続にはkubeconfigファイルが必要となる
BastionVM上のAWS CLIで作成する
```
$ aws eks update-kubeconfig --name <クラスタ名>
```
```
$ aws eks update-kubeconfig --name ex-man-nodeG-profile
```
kubeconfigファイルは以下に作成される
`/home/<ユーザ名>/.kube/config`

### kubectlによる接続確認
BastionVMのkubectlで以下を実行してみる
稼働しているpod情報が出力されれば接続は成功している
```
$ kubectl get pods -A
NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE
kube-system   aws-node-5wbdl             2/2     Running   0          22m
kube-system   coredns-6d86494944-2lsrw   1/1     Running   0          22m
kube-system   coredns-6d86494944-m8fqv   1/1     Running   0          22m
kube-system   kube-proxy-9sc4v           1/1     Running   0          22m
```

# まとめ
以上の手順でEKSクラスタとcontrol_planeに接続する踏み台サーバのデプロイが成功した
`terraform.tfvars`に定義している値を変更することで様々な環境のデプロイが行える
[terraform-aws-eks/examples](https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/examples)にはサンプルのマニフェストファイルがいくつか提供されているため、こちらを参考とすることで必要な構成のEKSクラスターをデプロイすることができる

# マニフェストファイル一覧
今回作成したマニフェストファイルは以下
一部情報はマスクしているためそのままでは利用できない
```
$ tree /f

│  main.tf
│  outputs.tf
│  terraform.tfvars
│  user_data.sh
│  variables.tf
│  versions.tf
│
```
## main.tf
```
# Setting of AWS Provider
provider "aws" {
  region     = var.region
}

terraform {
  backend "s3" {
    bucket = "tfstate-bucket-2024"
    key = "backend/terraform.tfstate"
    region = "ap-northeast-1"
    dynamodb_table = "tfstate-lockdb-2024"
    encrypt = true
  }
}

data "aws_availability_zones" "available" {}

locals {
  name            = var.eks_cluster_name
  cluster_version = var.cluster_version
  region          = var.region

  vpc_cidr = var.vpc_cidr
  azs      = slice(data.aws_availability_zones.available.names, 0, 3)

  tags = {
    Example    = local.name
    GithubRepo = "terraform-aws-eks"
    GithubOrg  = "terraform-aws-modules"
  }
}

################################################################################
# EKS Module
################################################################################

module "eks" {
  source = "terraform-aws-modules/eks/aws"

  cluster_name                   = local.name
  cluster_version                = local.cluster_version
  cluster_endpoint_public_access = true

  cluster_addons = {
    coredns = {
      most_recent = true
    }
    kube-proxy = {
      most_recent = true
    }
    vpc-cni = {
      most_recent = true
    }
  }

  vpc_id                   = module.vpc.vpc_id
  subnet_ids               = module.vpc.private_subnets
  control_plane_subnet_ids = module.vpc.intra_subnets

  # EKS Managed Node Group(s)
  eks_managed_node_group_defaults = {
    instance_types = ["m6i.large", "m5.large", "m5n.large", "m5zn.large"]
  }

  eks_managed_node_groups = {
    example = {
      min_size     = 1
      max_size     = 10
      desired_size = 1

      instance_types = ["t3.large"]
      capacity_type  = "SPOT"
    }
  }

  # Cluster access entry
  # To add the current caller identity as an administrator
  enable_cluster_creator_admin_permissions = true

  access_entries = {
    # One access entry with a policy associated
    example = {
      kubernetes_groups = []
      principal_arn     = "arn:aws:iam::${var.aws_account_id}:role/${var.aws_iam_role}"

      policy_associations = {
        example = {
          policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy"
          access_scope = {
            namespaces = ["default"]
            type       = "namespace"
          }
        }
      }
    }
  }

  tags = {
    Environment = "dev"
    Terraform   = "true"
  }
}

################################################################################
# Supporting Resources
################################################################################

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = local.name
  cidr = local.vpc_cidr

  azs             = local.azs
  private_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 4, k)]
  public_subnets  = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k + 48)]
  intra_subnets   = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k + 52)]

  enable_nat_gateway = true
  single_nat_gateway = true

  public_subnet_tags = {
    "kubernetes.io/role/elb" = 1
  }

  private_subnet_tags = {
    "kubernetes.io/role/internal-elb" = 1
  }

  tags = local.tags
}

################################################################################
# Addtional subnet
################################################################################

resource "aws_subnet" "bastionvm_subnet" {
  vpc_id = module.vpc.vpc_id

  availability_zone = var.bastionvm_avz
  cidr_block        = var.bastionvm_cidr
  map_public_ip_on_launch = "true"

  tags = {
    Name = "${local.name}-bastionvm-subnet-${var.bastionvm_avz}]"
  }
}

resource "aws_subnet" "transit_subnet" {
  vpc_id = module.vpc.vpc_id

  availability_zone = var.transit_avz
  cidr_block        = var.transit_cidr

  tags = {
    Name = "${local.name}-transit-subnet-${var.transit_avz}]"
  }
}

resource "aws_route_table_association" "rta_bastionvm_public_subnet" {
  subnet_id = "${aws_subnet.bastionvm_subnet.id}"
  route_table_id = "${module.vpc.public_route_table_ids[0]}"
} 

resource "aws_route_table_association" "rta_transit_public_subnet" {
  subnet_id = "${aws_subnet.transit_subnet.id}"
  route_table_id = "${module.vpc.public_route_table_ids[0]}"
} 

################################################################################
# Addtional security group
################################################################################

# Security Group for BastionVM
resource "aws_security_group" "for_bastionvm_public_sg" {
  name        = "${local.name}-bastionvm-public-sg"
  description = "Allow BastionVM traffic from public"
  vpc_id      = module.vpc.vpc_id
  tags = {
    Name = "${local.name}-bastionvm-public-sg"
  }
}

# For Outbound Any permit
resource "aws_security_group_rule" "public_outbound_any" {
  description = "Any_permit"
  type        = "egress"
  from_port   = 0
  to_port     = 0
  protocol    = "-1"
  cidr_blocks = ["0.0.0.0/0"] # Any

  # Matching to for_BastionVM_sg
  security_group_id = aws_security_group.for_bastionvm_public_sg.id
}

# For SSH
resource "aws_security_group_rule" "public_inbound_ssh" {
  description = "SSH"
  type        = "ingress"
  from_port   = 22
  to_port     = 22
  protocol    = "tcp"
  cidr_blocks = [var.my_term_ip] # My Terminal IP

  # Matching to for_BastionVM_sg
  security_group_id = aws_security_group.for_bastionvm_public_sg.id
}

# For RDP
resource "aws_security_group_rule" "public_inbound_RDP" {
  description = "RDP"
  type        = "ingress"
  from_port   = 3389
  to_port     = 3389
  protocol    = "tcp"
  cidr_blocks = [var.my_term_ip] # My Terminal IP

  # Matching to for_BastionVM_sg
  security_group_id = aws_security_group.for_bastionvm_public_sg.id
}

################################################################################
# Addtional EC2 instance
################################################################################
# Create EC2 NIC(ENI)
resource "aws_network_interface" "bastionvm_public_nic" {
  subnet_id       = aws_subnet.bastionvm_subnet.id
  security_groups = [aws_security_group.for_bastionvm_public_sg.id]
  tags = {
    Name = "BastionVM_public_nic"
  }
}

resource "aws_network_interface" "bastionvm_intra_nic" {
  subnet_id       = module.vpc.intra_subnets[0]
  security_groups = [module.eks.cluster_primary_security_group_id]
  tags = {
    Name = "BastionVM_intra_nic"
  }
}

# user_data for bastionvm
data "template_file" "user_data" {
  template = "${file("${path.module}/user_data.sh")}"
}

# Deploy EC2 instance
resource "aws_instance" "bastionvm_instance" {
  ami           = var.bastionvm_imageid
  instance_type = var.bastionvm_instance_type
  key_name      = var.my_key_name
  network_interface {
    network_interface_id = aws_network_interface.bastionvm_public_nic.id
    device_index         = 0
  }
  network_interface {
    network_interface_id = aws_network_interface.bastionvm_intra_nic.id
    device_index         = 1
  }
  root_block_device {
    volume_type = "gp2"
    volume_size = "8"
  }
  user_data = "${data.template_file.user_data.rendered}"

  tags = {
    Name = "${local.name}-bastionvm-instance"
  }
}

# Create EIP for BastionVM
resource "aws_eip" "bastionvm_eip" {
  network_interface = aws_network_interface.bastionvm_public_nic.id
  domain = "vpc"
  tags = {
    Name = "BastionVM_eip"
  }
}
```
## outputs.tf
```
################################################################################
# Cluster
################################################################################

output "cluster_arn" {
  description = "The Amazon Resource Name (ARN) of the cluster"
  value       = module.eks.cluster_arn
}

output "cluster_certificate_authority_data" {
  description = "Base64 encoded certificate data required to communicate with the cluster"
  value       = module.eks.cluster_certificate_authority_data
}

output "cluster_endpoint" {
  description = "Endpoint for your Kubernetes API server"
  value       = module.eks.cluster_endpoint
}

output "cluster_id" {
  description = "The ID of the EKS cluster. Note: currently a value is returned only for local EKS clusters created on Outposts"
  value       = module.eks.cluster_id
}

output "cluster_name" {
  description = "The name of the EKS cluster"
  value       = module.eks.cluster_name
}

output "cluster_oidc_issuer_url" {
  description = "The URL on the EKS cluster for the OpenID Connect identity provider"
  value       = module.eks.cluster_oidc_issuer_url
}

output "cluster_platform_version" {
  description = "Platform version for the cluster"
  value       = module.eks.cluster_platform_version
}

output "cluster_status" {
  description = "Status of the EKS cluster. One of `CREATING`, `ACTIVE`, `DELETING`, `FAILED`"
  value       = module.eks.cluster_status
}

output "cluster_primary_security_group_id" {
  description = "Cluster security group that was created by Amazon EKS for the cluster. Managed node groups use this security group for control-plane-to-data-plane communication. Referred to as 'Cluster security group' in the EKS console"
  value       = module.eks.cluster_primary_security_group_id
}

################################################################################
# Access Entry
################################################################################

output "access_entries" {
  description = "Map of access entries created and their attributes"
  value       = module.eks.access_entries
}

################################################################################
# KMS Key
################################################################################

output "kms_key_arn" {
  description = "The Amazon Resource Name (ARN) of the key"
  value       = module.eks.kms_key_arn
}

output "kms_key_id" {
  description = "The globally unique identifier for the key"
  value       = module.eks.kms_key_id
}

output "kms_key_policy" {
  description = "The IAM resource policy set on the key"
  value       = module.eks.kms_key_policy
}

################################################################################
# Security Group
################################################################################

output "cluster_security_group_arn" {
  description = "Amazon Resource Name (ARN) of the cluster security group"
  value       = module.eks.cluster_security_group_arn
}

output "cluster_security_group_id" {
  description = "ID of the cluster security group"
  value       = module.eks.cluster_security_group_id
}

################################################################################
# Node Security Group
################################################################################

output "node_security_group_arn" {
  description = "Amazon Resource Name (ARN) of the node shared security group"
  value       = module.eks.node_security_group_arn
}

output "node_security_group_id" {
  description = "ID of the node shared security group"
  value       = module.eks.node_security_group_id
}

################################################################################
# IRSA
################################################################################

output "oidc_provider" {
  description = "The OpenID Connect identity provider (issuer URL without leading `https://`)"
  value       = module.eks.oidc_provider
}

output "oidc_provider_arn" {
  description = "The ARN of the OIDC Provider if `enable_irsa = true`"
  value       = module.eks.oidc_provider_arn
}

output "cluster_tls_certificate_sha1_fingerprint" {
  description = "The SHA1 fingerprint of the public key of the cluster's certificate"
  value       = module.eks.cluster_tls_certificate_sha1_fingerprint
}

################################################################################
# IAM Role
################################################################################

output "cluster_iam_role_name" {
  description = "IAM role name of the EKS cluster"
  value       = module.eks.cluster_iam_role_name
}

output "cluster_iam_role_arn" {
  description = "IAM role ARN of the EKS cluster"
  value       = module.eks.cluster_iam_role_arn
}

output "cluster_iam_role_unique_id" {
  description = "Stable and unique string identifying the IAM role"
  value       = module.eks.cluster_iam_role_unique_id
}

################################################################################
# EKS Addons
################################################################################

output "cluster_addons" {
  description = "Map of attribute maps for all EKS cluster addons enabled"
  value       = module.eks.cluster_addons
}

################################################################################
# EKS Identity Provider
################################################################################

output "cluster_identity_providers" {
  description = "Map of attribute maps for all EKS identity providers enabled"
  value       = module.eks.cluster_identity_providers
}

################################################################################
# CloudWatch Log Group
################################################################################

output "cloudwatch_log_group_name" {
  description = "Name of cloudwatch log group created"
  value       = module.eks.cloudwatch_log_group_name
}

output "cloudwatch_log_group_arn" {
  description = "Arn of cloudwatch log group created"
  value       = module.eks.cloudwatch_log_group_arn
}

################################################################################
# Fargate Profile
################################################################################

output "fargate_profiles" {
  description = "Map of attribute maps for all EKS Fargate Profiles created"
  value       = module.eks.fargate_profiles
}

################################################################################
# EKS Managed Node Group
################################################################################

output "eks_managed_node_groups" {
  description = "Map of attribute maps for all EKS managed node groups created"
  value       = module.eks.eks_managed_node_groups
}

output "eks_managed_node_groups_autoscaling_group_names" {
  description = "List of the autoscaling group names created by EKS managed node groups"
  value       = module.eks.eks_managed_node_groups_autoscaling_group_names
}

################################################################################
# Self Managed Node Group
################################################################################

output "self_managed_node_groups" {
  description = "Map of attribute maps for all self managed node groups created"
  value       = module.eks.self_managed_node_groups
}

output "self_managed_node_groups_autoscaling_group_names" {
  description = "List of the autoscaling group names created by self-managed node groups"
  value       = module.eks.self_managed_node_groups_autoscaling_group_names
}

################################################################################
# Transit subnet
################################################################################
output "transit_subnets" {
  description = "Subnet for Transit"
  value = aws_subnet.transit_subnet
}

################################################################################
# BastionVM
################################################################################
output "bastionvm_subnets" {
  description = "Subnet for bastionVM"
  value = aws_subnet.bastionvm_subnet
}

output "bastionvm_public_ip" {
  description = "Public IP address of bastionVM"
  value = aws_eip.bastionvm_eip.public_ip
}
```
## terraform.tfvars
```
region     = "ap-northeast-1"

eks_cluster_name = "ex-man-nodeG-profile"
cluster_version = "1.29"
vpc_cidr = "10.8.0.0/16"

bastionvm_avz = "ap-northeast-1a"
bastionvm_cidr = "10.8.56.0/24"
bastionvm_imageid = "ami-01bef798938b7644d" # Ubuntu Server 24.04 LTS (HVM), SSD Volume Type
my_term_ip = "<踏み台サーバへの接続を許可するIPアドレス>/32"
my_key_name = "<BastionVM用キーペア名>"

transit_avz = "ap-northeast-1a"
transit_cidr = "10.8.60.0/28"

aws_account_id = "<AWSアカウントID>"
aws_iam_role = "ex-man-nodeG-profile-iam-assume-role"
```
## user_data.sh
```
#!/bin/bash
echo "start installation of kubectl"
sudo apt update
sudo apt install -y apt-transport-https ca-certificates curl
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
sudo chmod 644 /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo chmod 644 /etc/apt/sources.list.d/kubernetes.list
sudo apt update
sudo apt install -y kubectl
echo "end installation of kubectl"
echo "start installation of AWS CLI"
sudo apt install -y unzip
curl https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip -o awscliv2.zip
unzip awscliv2.zip
sudo ./aws/install
echo "end installation of AWS CLI"
echo "start installation of eksctl"
curl -sLO https://github.com/eksctl-io/eksctl/releases/latest/download/eksctl_Linux_amd64.tar.gz
tar -xzf eksctl_Linux_amd64.tar.gz -C /tmp && rm eksctl_Linux_amd64.tar.gz
sudo mv /tmp/eksctl /usr/local/bin
echo "end installation of eksctl"
echo "end UserData"
```
## variables.tf
```
variable "region" {
  description = "AWS_REGION"
  type        = string
  default     = "ap-northeast-1"
}

variable "vpc_cidr" {
  description = "AWS_VPC_cidr_for_EKS"
  type = string
}

variable "eks_cluster_name" {
  description = "AWS_EKS_Cluster_name"
  type = string
}

variable "cluster_version" {
  description = "AWS_EKS_Cluster_version"
  type = string
}

variable "bastionvm_avz" {
  description = "AWS_BastionVM_availabilityzone"
  type = string
}

variable "bastionvm_cidr" {
  description = "AWS_BastionVM_cidr"
  type = string
}

variable "my_term_ip" {
  description = "My Terminal addr"
  type = string
}

variable "my_key_name" {
  description = "EC2_SSH_Key_Pair_Name"
  type        = string
}

variable "bastionvm_imageid" {
  description = "AWS_BastionVM_EC2_Image_ID"
  type        = string
}

variable "bastionvm_instance_type" {
  description = "AWS_BastionVM_instance_type"
  type = string
}

variable "transit_avz" {
  description = "AWS_Transit_availabilityzone"
  type = string
}

variable "transit_cidr" {
  description = "AWS_Transit_cidr"
  type = string
}

variable "aws_account_id" {
  description = "AWS_Account_ID"
  type = string
}

variable "aws_iam_role" {
  description = "AWS_IAM_Role_for_EKS"
  type = string
}
```
## versions.tf
```
terraform {
  required_version = ">= 1.3.2"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.40"
    }
  }
}
```

# 参考

参考: [TerraformのEKS published moduleを使ってEKSクラスターを作成する | DevelopersIO (classmethod.jp)](https://dev.classmethod.jp/articles/create-eks-cluster-with-terraform-module/)
参考: [EKSの構築～接続までをTerraformでやりきる #kubernetes - Qiita](https://qiita.com/kokohei39/items/b63b35e5ee71e397897a)
参考: [Terraformのsubnet用のモジュール (zenn.dev)](https://zenn.dev/uepon/articles/060c62a7af336a)
参考: [Install and Set Up kubectl on Linux | Kubernetes](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#install-using-other-package-management)
参考: [AWS CLIの最新バージョンのインストールまたは更新 - AWS Command Line Interface (amazon.com)](https://docs.aws.amazon.com/ja_jp/cli/latest/userguide/getting-started-install.html)
参考: [terraformでsubnetにroute_tableをマッピングする #AWS - Qiita](https://qiita.com/shnagai/items/e21ae5fa2666faeba201)
参考: [【Windows PowerShell】SSH接続する際にパーミッションエラーが発生した場合の権限変更 #Windows - Qiita](https://qiita.com/eltociear/items/02e8b1f5590b49eb9d87)
参考: [【Terraform】複数nic(ENI)を持つEC2を構築する (royozaki.net)](https://tech.royozaki.net/tech/jp/tfec2withenis.html)
参考: [EC2に複数のENIをアタッチする手順と制約（Public-ip,DNSが割当てられなくなる） #AWS - Qiita](https://qiita.com/kaojiri/items/94bc62c7b003367b5e46)参考: [[Terraform]Module間の値の受け渡しについて | DevelopersIO (classmethod.jp)](https://dev.classmethod.jp/articles/terraform_module_coordination/)
参考: [aws_eip | Resources | hashicorp/aws | Terraform | Terraform Registry](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eip)
参考: [[Terraform]EC2 Linux作成時のユーザデータにfile関数を使用する](https://dev.classmethod.jp/articles/terraform-ec2-linux-userdata-file-function/)