TerraformでぼくのかんがえたさいきょうのSpringboot on AWS環境構築

※ タイトルは釣り気味です

### バージョンアップ情報
2021−06−23

|      名称     |        更新前バージョン       |      更新後バージョン     |
|:------------:|:-------------------------:|:---------------------:|
| OS           | Mac OS 10.15.1 (Catalina) | Mac OS 11.4 (Big Sur) |
| Homebrew     | 2.1.16                    | 3.1.9                 |
| tfenv        | 1.0.2                     | 2.2.1                 |
| Terraform    | 0.12.13                   | 0.15.4                |
| AWS Provider | 2.34.0                    | 3.44.0                |
| AWS CLI      | aws-cli/1.16.260 Python/3.7.5 Darwin/19.0.0 botocore/1.12.250 | aws-cli/2.0.48 Python/3.7.4 Darwin/20.5.0 exe/x86_64 |

# 目的
Spring Boot + GradleのWebアプリケーションをなるはやで検証環境に公開したい。
AWSマネジメントコンソールを操作するのはめんどくさい、という人向けです。

作成する環境のインフラ構成図は以下の通りです。
![terraform-aws-Page-1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/205862/92efc574-c333-de4e-5ac5-100394f990e3.png)

# 前提
- 基本的に東京リージョンでの作業
- AWSで新規IAMユーザー作成および権限付与できるアカウントを保持している
- 上記アカウントでプログラム(AWS CLI)からアクセスできる
- Terraformを触ったことがある

# 環境情報
- OS: `Mac OS 11.4 (Big Sur)`
- Homebrew: `3.1.9`
- AWS CLI: `aws-cli/2.0.48 Python/3.7.4 Darwin/20.5.0 exe/x86_64`
- tfenv: `2.2.1`
- Terraform: `0.15.4`

# ディレクトリ構成
VPCやリポジトリなどの共通情報は`shared`ディレクトリで管理します。
また、検証環境や本番環境の切り替えは[workspace機能](https://www.terraform.io/docs/language/state/workspaces.html)を使用します。
環境ごとの差分が少ない構成で作成したのでworkspaceを使っていますが、差分が大きい場合は一部ディレクトリ分割が必要になるかも。

```sh
tree
.
├── .terraform/ #　自動生成
├── certs/ #　pem等認証情報（コミットしない）
├── shared/ #　共通リソース
│       ├── .terraform/ #　自動生成
│       ├── .terraform.lock.hcl # 自動生成
│       ├── terraform.tf # terraform設定
│　　　　　　　　　　　　　　├── ・・・・・・ # リソースファイル
│       └── variables.tf # 変数宣言
├── terraform.tfstate.d/ #　自動生成
├── .terraform.lock.hcl # 自動生成
├── terraform.tf # terraform設定
├── ・・・・・・ # リソースファイル
├── terraform.tfvars # 変数設定
└── variables.tf # 変数宣言
```

# 事前準備
## AWS CLI
### インストール
https://docs.aws.amazon.com/ja_jp/cli/latest/userguide/install-macos.html

### Terraformユーザー作成
```sh
$ aws iam create-user \
  --user-name terraform-sample
{
    "User": {
        "Path": "/",
        "UserName": "terraform-sample",
        "UserId": "XXXXXXXXXXXXXXXXXXXX",
        "Arn": "arn:aws:iam::XXXXXXXXXXXX:user/terraform-sample",
        "CreateDate": "YYYY-MM-DDTHH:mm:ssZ"
    }
}
```

### 権限付与
管理者権限を付与します。付与する権限については、環境に応じて適宜変更してください。
とはいえ、必要な権限が多岐に渡るので、作成時のみ`AdministratorAccess`を付与し、作業が終了したら外す形が良さそうです。

```sh
$ aws iam attach-user-policy \
  --user-name terraform-sample \
  --policy-arn arn:aws:iam::aws:policy/AdministratorAccess
```

### 認証情報作成・保存
```sh
$ aws iam create-access-key \
  --user-name terraform-sample
{
    "AccessKey": {
        "UserName": "terraform-sample",
        "AccessKeyId": "AAAAAAAAAAAAAAAAAAA",
        "Status": "Active",
        "SecretAccessKey": "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB",
        "CreateDate": "YYYY-MM-DDTHH:mm:ssZ"
    }
}
$ cat - << EOS >> ~/.aws/credentials
# プロファイル名
[terraform-sample]
region = ap-northeast-1 # 東京リージョン
aws_access_key_id = AAAAAAAAAAAAAAAAAAA
aws_secret_access_key = BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
EOS
```

## Terraform管理外のリソース作成
アプリケーション/静的コンテンツを格納するCodeCommitリポジトリおよびTerraformの状態を保存するS3バケットは`terraform destroy`などで削除されないよう、AWS CLIで作成します。

### Terraform用S3バケット
以下の部分を作成します。
![terraform-aws-Terraform用S3バケット.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/205862/8065495e-2b4c-615a-69d2-c1a27c9fe95f.png)

```sh
$ aws s3api create-bucket \
  --bucket terraform-sample-tfstate \
  --acl private \
  --region ap-northeast-1 \
  --create-bucket-configuration LocationConstraint=ap-northeast-1 \
  --profile terraform-sample
{
    "Location": "http://terraform-sample-tfstate.s3.amazonaws.com/"
}
```

#### バージョニングを有効化
```sh
$ aws s3api put-bucket-versioning \
  --bucket terraform-sample-tfstate \
  --versioning-configuration Status=Enabled \
  --profile terraform-sample
$ aws s3api get-bucket-versioning \
  --bucket terraform-sample-tfstate \
  --profile terraform-sample
{
    "Status": "Enabled"
}
```

#### バケットの暗号化
```sh
$ aws s3api put-bucket-encryption \
  --bucket terraform-sample-tfstate \
  --server-side-encryption-configuration '{"Rules": [{"ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "AES256"}}]}' \
  --profile terraform-sample
$ aws s3api get-bucket-encryption \
  --bucket terraform-sample-tfstate \
  --profile terraform-sample
{
    "ServerSideEncryptionConfiguration": {
        "Rules": [
            {
                "ApplyServerSideEncryptionByDefault": {
                    "SSEAlgorithm": "AES256"
                }
            }
        ]
    }
}
```

## Terraformセットアップ
### tfenvインストール
普段Terraformのバージョンで悩まされることがあるので、「tfenv」というツールを使ってバージョン管理しています。
[tfenvでTerraformのバージョン管理をする - Qiita](https://qiita.com/kamatama_41/items/ba59a070d8389aab7694)

```sh
$ brew install tfenv
$ which tfenv
/usr/local/bin/tfenv
$ tfenv --version
tfenv 2.2.1
```

### Terraform最新版をインストール/利用
```sh
$ tfenv install latest
$ tfenv use latest
[INFO] Switching to v0.15.4
[INFO] Switching completed
```

### Terraform初期化
#### 基本情報を記述
AWS providerの最新版については以下のページで確認できます。
[Terraform AWS Provider CHANGELOG - GitHub](https://github.com/terraform-providers/terraform-provider-aws/blob/master/CHANGELOG.md)
今回は執筆時点で最新版の3.44.0を使用します。

```tf:terraform.tf(新規作成)
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.44.0"
    }
  }
  backend "s3" {
    bucket  = "terraform-sample-tfstate" # 上記作成したTerraform用S3バケット名
    region  = "ap-northeast-1" # 作業対象のリージョン情報
    profile = "terraform-sample" # ~/.aws/credentialsに保存した認証情報のプロファイル名
    key     = "terraform.tfstate" # tfstateファイルパス
    encrypt = true
  }
}

provider "aws" {
  region                  = "ap-northeast-1"
  shared_credentials_file = "/Users/exotic-toybox/.aws/credentials"　# ~/.aws/credentials
  profile                 = "terraform-sample"
}
```

`shared`ディレクトリに配下にコピーします。

```sh
$ cp terraform.tf shared/
```

tfstateのファイルパスだけ変更しましょう。

```diff_ruby:shared/terraform.tf(修正)
terraform {
  required_providers {
    #-- 中略 --#
  }
  backend "s3" {
    #-- 中略 --#
-   key     = "terraform.tfstate" # tfstateファイルパス
+   key     = "shared/terraform.tfstate" # tfstateファイルパス
  }
}
#-- 中略 --#
```

#### 初期化コマンド実行
```sh
$ terraform init

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "aws" (hashicorp/aws) 3.44.0...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
```
`Terraform has been successfully initialized!`と表示されていれば成功です。
同様に`shared`ディレクトリ以下でも行ってください。

##### 変数使用について
providerやterraformブロックでは変数を扱うことができません。
以下のような記述はエラーとなります。

```tf:terraform.tf
variable "region" {
  default = "ap-northeast-1"
}

data "aws_s3_bucket" "tfstate" {
  bucket        = "terraform-sample-tfstate"
  #-- 中略 --#
}

terraform {
  #-- 中略 --#
  backend "s3" {
    bucket  = aws_s3_bucket.tfstate.bucket # これとか
    region  = var.region # これも
    #-- 中略 --#
  }
}
```
```sh
$ terraform init
Initializing the backend...
Error: Variables not allowed
  on terraform.tf line 18, in terraform:
  18:     bucket  = aws_s3_bucket.tfstate.bucket
  on terraform.tf line 19, in terraform:
  19:     region  = var.region
Variables may not be used here.
```

### module読み込み
`shared`ディレクトリ配下のリソースを読み込みます。

```tf:load_modules.tf(新規作成)
data terraform_remote_state shared {
  backend = "s3"
 
  config = {
    bucket  = "terraform-sample-tfstate"
    key     = "shared/terraform.tfstate"
    region  = "ap-northeast-1"
    profile = "terraform-sample"
  }
}```


# 検証(Stage)環境構築
```tf:terraform.tfvars(新規作成)
app_name = "terraform-sample"
```
```tf:variables.tf(新規作成)
variable app_name {}
```
`shared/variables.tf`にも同様に記述ください。
以降、`terraform.tfvars`へ変数を追加した時は`variables.tf`と`shared/variables.tf`に変数宣言しているものとします。

##　terraform管理外のCodeCommitリポジトリ
### 静的コンテンツ用CodeCommitリポジトリ
以下の部分を作成します。
![static-content (1).png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/205862/8b2d1a9d-c7d5-e434-599a-f3451ca1a93d.png)

```sh
$ aws codecommit create-repository \
  --repository-name terraform-sample-static-contents \
  --repository-description "static contents repository" \
  --profile terraform-sample
{
    "repositoryMetadata": {
        "accountId": "XXXXXXXXXXXX",
        "repositoryId": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
        "repositoryName": "terraform-sample-static-contents",
        "repositoryDescription": "static contents repository",
        "lastModifiedDate": XXXXXXXXXX.XX,
        "creationDate": XXXXXXXXXX.XX,
        "cloneUrlHttp": "https://git-codecommit.ap-northeast-1.amazonaws.com/v1/repos/terraform-sample-static-contents",
        "cloneUrlSsh": "ssh://git-codecommit.ap-northeast-1.amazonaws.com/v1/repos/terraform-sample-static-contents",
        "Arn": "arn:aws:codecommit:ap-northeast-1:XXXXXXXXXXXX:terraform-sample-static-contents"
    }
}
```

#### Terraformで変数化
```tf:shared/codecommit.tf(新規作成)
data aws_codecommit_repository static_contents {
  repository_name = "terraform-sample-static-contents"
}

output codecommit_repository_static_contents {
  value = data.aws_codecommit_repository.static_contents
}
```

#### 中身
`index.html`などの静的コンテンツを配置します。

```sh
tree
.
├── error.html
├── favicon.png
└── index.html
```
`stage`ブランチを作成しておきましょう。

### アプリケーション用CodeCommitリポジトリ
以下の部分を作成します。
![application.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/205862/0f09523c-9672-1ba8-21c6-312ec33270a6.png)

```sh
$ aws codecommit create-repository \
  --repository-name terraform-sample-application-sources \
  --repository-description "application sources repository" \
  --profile terraform-sample
{
    "repositoryMetadata": {
        "accountId": "XXXXXXXXXXXX",
        "repositoryId": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
        "repositoryName": "terraform-sample-application-sources",
        "repositoryDescription": "application sources repository",
        "lastModifiedDate": XXXXXXXXXX.XX,
        "creationDate": XXXXXXXXXX.XX,
        "cloneUrlHttp": "https://git-codecommit.ap-northeast-1.amazonaws.com/v1/repos/terraform-sample-application-sources",
        "cloneUrlSsh": "ssh://git-codecommit.ap-northeast-1.amazonaws.com/v1/repos/terraform-sample-application-sources",
        "Arn": "arn:aws:codecommit:ap-northeast-1:XXXXXXXXXXXX:terraform-sample-application-sources"
    }
}
```

#### Terraformで変数化
```tf:shared/codecommit.tf(追記)
#-- 中略 --#
data aws_codecommit_repository application_sources {
  repository_name = "terraform-sample-application-sources"
}

output codecommit_repository_application_sources {
  value = data.aws_codecommit_repository.application_sources
}
```

#### 中身
`admin`と`user`2つのgradleプロジェクト内に`@SpringBootApplication`が付いたメインクラスを配置します。

```sh
tree
.
├── admin
│   ├── bin
│   ├── build
│   ├── build.gradle
│   └── src
│       ├── main
│       │   ├── java
│       │   │   └── com
│       │   │       └── example
│       │   │           └── admin
│       │   │               └── AdminApplication.java
│       │   └── resources
│       │       └── application.yaml
│       └── test
│           ├── java
│           └── resources
├── appspec.yml
├── build.gradle
├── buildspec_admin.yml
├── buildspec_user.yml
├── data
│   └── script
│       ├── after_install.sh
│       ├── application_start.sh
│       ├── application_stop.sh
│       └── before_install.sh
├── gradle
│   └── wrapper
│       ├── gradle-wrapper.jar
│       └── gradle-wrapper.properties
├── gradlew
├── gradlew.bat
├── settings.gradle
└── user
    ├── bin
    ├── build
    ├── build.gradle
    └── src
        ├── main
        │   ├── java
        │   │   └── com
        │   │       └── example
        │   │           └── user
        │   │               └── UserApplication.java
        │   └── resources
        │       └── application.yaml
        └── test
            └── resources
```
`stage`ブランチを作成しておきましょう。

以降、.tfファイルの作成・更新時は`$ terraform apply`コマンドを実行しているものとします。
また、`shared`配下に.tfファイルを作成した際は、`shared$ terraform apply -var-file=../terraform.tfvars`コマンドを実行しているものとします。

## 静的コンテンツの自動デプロイ
以下の部分を作成します。
![terraform-aws-静的コンテンツの自動デプロイ.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/205862/8df49cad-48df-753b-1bac-90ce941b1e56.png)

### デプロイ先のS3バケット
![terraform-aws-静的コンテンツの自動デプロイ-デプロイ先のS3バケット.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/205862/64be52ab-6595-ea15-8bab-527e3c16efaa.png)

```tf:static_contents_s3.tf(新規作成)
resource "aws_s3_bucket" "static_contents" {
  bucket = "${var.app_name}-static-contents-${terraform.workspace}"
  acl    = "private"

  tags = {
    Name = "${var.app_name}-static-contents-${terraform.workspace}"
  }
}

data "aws_iam_policy_document" "s3_static_contents" {
  statement {
    effect = "Allow"
    actions = ["s3:GetObject"]

    resources = [
      aws_s3_bucket.static_contents.arn,
      "${aws_s3_bucket.static_contents.arn}/*"
    ]
  }
}
```

### CodePipelineのアーティファクトを格納するS3バケット
![terraform-aws-静的コンテンツの自動デプロイ-CodePipelineのアーティファクトを格納するS3バケット.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/205862/abe8cf31-c84c-aeb1-bb7b-9f800dceda90.png)

```tf:codepipeline_s3.tf(新規作成)
resource aws_s3_bucket codepipeline_static_contents {
  bucket = "${var.app_name}-codepipeline-static-contents-${terraform.workspace}"
  acl    = "private"

  tags = {
    Name = "${var.app_name}-codepipeline-static-contents-${terraform.workspace}"
  }
}

data aws_iam_policy_document s3_codepipeline_static_contents {
  statement {
    effect = "Allow"
    actions = [
      "s3:GetObject",
      "s3:GetObjectVersion",
      "s3:GetBucketVersioning",
      "s3:PutObject"
    ]

    resources = [
      aws_s3_bucket.codepipeline_static_contents.arn,
      "${aws_s3_bucket.codepipeline_static_contents.arn}/*"
    ]
  }
}
```

### S3暗号化用kms
![terraform-aws-静的コンテンツの自動デプロイ-S3暗号化用kms.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/205862/55948bb7-caa2-5a27-f044-b403ca645da5.png)

```tf:shared/kms.tf(新規作成)
resource aws_kms_key kms {}

resource aws_kms_alias kms {
  name          = "alias/${var.app_name}"
  target_key_id = aws_kms_key.kms.key_id
}

output kms_alias_arn {
  value = aws_kms_alias.kms.arn
}
```
```tf:shared/kms_iam.tf(新規作成)
data aws_iam_policy_document kms {
  statement {
    effect  = "Allow"
    actions = ["kms:*"]

    resources = ["*"]
  }
}

output kms_policy_json {
  value = data.aws_iam_policy_document.kms.json
}
```

### Policy設定
```tf:shared/codepipeline_iam.tf(新規作成)
data aws_iam_policy_document codepipeline_assume_role {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRole"]
    principals {
      identifiers = [
        "codepipeline.amazonaws.com",
        "events.amazonaws.com"
      ]
      type = "Service"
    }
  }
}

output codepipeline_assume_role_policy_json {
  value = data.aws_iam_policy_document.codepipeline_assume_role.json
}
```

```tf:shared/codecommit_iam.tf(新規作成)
data aws_iam_policy_document codecommit_static_contents {
  statement {
    effect = "Allow"
    actions = [
      "codecommit:GitPull",
      "codecommit:GetBranch",
      "codecommit:GetCommit",
      "codecommit:UploadArchive",
      "codecommit:GetUploadArchiveStatus"
    ]

    resources = [
      data.aws_codecommit_repository.static_contents.arn,
      "${data.aws_codecommit_repository.static_contents.arn}/*"
    ]
  }
}

output codecommit_static_contents_policy_json {
  value = data.aws_iam_policy_document.codecommit_static_contents.json
}
```

### CodePipeline
![terraform-aws-静的コンテンツの自動デプロイ-CodePipeline.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/205862/528636cd-93f4-7bea-d200-829c106885a6.png)

```tf:static_contents_codepipeline.tf(新規作成)
resource aws_iam_role codepipeline_static_contents {
  name               = "${var.app_name}-codepipeline-static-contents-${terraform.workspace}"
  assume_role_policy = data.terraform_remote_state.shared.outputs.codepipeline_assume_role_policy_json
}

# CodeCommitからソース取得を許可
resource aws_iam_role_policy codecommit_codepipeline_static_contents {
  role   = aws_iam_role.codepipeline_static_contents.name
  policy = data.aws_iam_policy_document.codecommit_static_contents.json
}

# CodePipelineのアーティファクト用S3を許可
resource aws_iam_role_policy s3_codepipeline_static_contents {
  role   = aws_iam_role.codepipeline_static_contents.name
  policy = data.aws_iam_policy_document.s3_codepipeline_static_contents.json
}

# CodePipelineで使用するS3用kmsを許可
resource aws_iam_role_policy kms_codepipeline_static_contents {
  role   = aws_iam_role.codepipeline_static_contents.name
  policy = data.terraform_remote_state.shared.outputs.kms_policy_json
}

# デプロイ先S3へのアクセス許可
resource aws_iam_role_policy s3_static_contents {
  role   = aws_iam_role.codepipeline_static_contents.name
  policy = data.aws_iam_policy_document.s3_static_contents.json
}

resource aws_codepipeline static_contents {
  name     = "${var.app_name}-static-contents-${terraform.workspace}"
  role_arn = aws_iam_role.codepipeline_static_contents.arn

  artifact_store {
    location = aws_s3_bucket.codepipeline_static_contents.bucket
    type     = "S3"
    encryption_key {
      id   = data.terraform_remote_state.shared.outputs.kms_alias_arn
      type = "KMS"
    }
  }

  # 静的コンテンツ用CodeCommitリポジトリからソースを取得し、CodePipelineのアーティファクトを格納するS3バケットに保存する
  stage {
    name = "${var.app_name}-static-contents-${terraform.workspace}-source"

    action {
      name             = "${var.app_name}-static-contents-${terraform.workspace}-source-action"
      category         = "Source"
      owner            = "AWS"
      provider         = "CodeCommit"
      version          = "1"
      output_artifacts = ["SOURCE"]

      configuration = {
        PollForSourceChanges = "false"
        RepositoryName       = data.terraform_remote_state.shared.outputs.codecommit_repository_static_contents.repository_name
        BranchName           = terraform.workspace
      }
    }
  }

  # CodePipelineのアーティファクトを格納するS3バケットから上記stageの結果を取得し、デプロイ先のS3バケットに展開する
  stage {
    name = "${var.app_name}-static-contents-${terraform.workspace}-deploy"

    action {
      name            = "${var.app_name}-static-contents-${terraform.workspace}-deploy-action"
      category        = "Deploy"
      owner           = "AWS"
      provider        = "S3"
      input_artifacts = ["SOURCE"]
      version         = "1"

      configuration = {
        BucketName = aws_s3_bucket.static_contents.id,
        Extract    = true,
      }
    }
  }
}
```

#### 動作確認
```sh
$ aws codepipeline start-pipeline-execution \
  --name terraform-sample-static-contents-stage \
  --profile terraform-sample
{
    "pipelineExecutionId": "AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE"
}
$ aws codepipeline get-pipeline-execution \
  --pipeline-name terraform-sample-static-contents-stage \
  --pipeline-execution-id AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE \
  --query "pipelineExecution.status" \
  --profile terraform-sample
"Succeeded"
```
`Succeeded`となっていれば正常です。

### CloudWatch
![terraform-aws-静的コンテンツの自動デプロイ-CloudWatch.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/205862/273d4a35-f658-2875-895a-047761f91157.png)

```tf:shared/cloudwatch_iam.tf(新規作成)
data aws_iam_policy_document cloudwatch_assume_role {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRole"]
    principals {
      identifiers = [
        "codepipeline.amazonaws.com",
        "events.amazonaws.com"
      ]
      type = "Service"
    }
  }
}

output cloudwatch_assume_role_policy_json {
  value = data.aws_iam_policy_document.cloudwatch_assume_role.json
}
```
```tf:static_contents_codepipeline_cloudwatch.tf(新規作成)
resource aws_iam_role codepipeline_static_contents_cloudwatch {
  name               = "${var.app_name}-codepipeline-static-contents-cloudwatch-${terraform.workspace}"
  assume_role_policy = data.terraform_remote_state.shared.outputs.cloudwatch_assume_role_policy_json
}

# resourcesに上記で作成したCodePipelineを指定するので、ここで宣言
data aws_iam_policy_document codepipeline_static_contents_cloudwatch {
  statement {
    effect  = "Allow"
    actions = ["codepipeline:StartPipelineExecution"]

    resources = [aws_codepipeline.static_contents.arn]
  }
}

resource aws_iam_role_policy codepipeline_static_contents_cloudwatch {
  role   = aws_iam_role.codepipeline_static_contents_cloudwatch.name
  policy = data.aws_iam_policy_document.codepipeline_static_contents_cloudwatch.json
}

resource aws_cloudwatch_event_rule codepipeline_static_contents {
  name          = "${var.app_name}-codepipeline-static-contents-${terraform.workspace}"
  # var.static_contents_repository_arnのvar.static_contents_target_branchに変更が発生したら発火する
  event_pattern = <<PATTERN
  {
    "source": [
      "aws.codecommit"
    ],
    "detail-type": [
      "CodeCommit Repository State Change"
    ],
    "resources": [
      "${data.terraform_remote_state.shared.outputs.codecommit_repository_static_contents.arn}"
    ],
    "detail": {
      "event": [
        "referenceCreated",
        "referenceUpdated"
      ],
      "referenceType": [
        "branch"
      ],
      "referenceName": [
        "${terraform.workspace}"
      ]
    }
  }
PATTERN
}

resource aws_cloudwatch_event_target codepipeline_static_contents {
  rule      = aws_cloudwatch_event_rule.codepipeline_static_contents.name
  target_id = "${var.app_name}-codepipeline-static-contents-${terraform.workspace}"
  arn       = aws_codepipeline.static_contents.arn
  role_arn  = aws_iam_role.codepipeline_static_contents_cloudwatch.arn
}
```

#### 動作確認
1. `var.static_contents_target_branch`で指定したブランチに変更をpushします。
1. 以下のコマンドで確認します。

```sh
$ aws codepipeline list-pipeline-executions \
  --pipeline-name terraform-sample-static-contents-stage \
  --profile terraform-sample
{
    "pipelineExecutionSummaries": [
        {
            "pipelineExecutionId": "YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY",
            "status": "Succeeded",
            "startTime": BBBBBBBBBB.BBB,
            "lastUpdateTime": BBBBBBBBBB.BBB,
            "sourceRevisions": [
                {
                    "actionName": "terraform-sample-static-contents-stage-source-action",
                    "revisionId": "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB",
                    "revisionSummary": "Test commit",
                    "revisionUrl": "https://ap-northeast-1.console.aws.amazon.com/codecommit/home#/repository/terraform-sample-static-contents/commit/BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB"
                }
            ],
            "trigger": {
                "triggerType": "CloudWatchEvent",
                "triggerDetail": "arn:aws:events:ap-northeast-1:338927112236:rule/terraform-sample-codepipeline-static-contents-stage"
            }
        },
        {
            "pipelineExecutionId": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
            "status": "Succeeded",
            "startTime": AAAAAAAAAA.AAA,
            "lastUpdateTime": AAAAAAAAAA.AAA,
            "sourceRevisions": [
                {
                    "actionName": "terraform-sample-static-contents-stage-source-action",
                    "revisionId": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
                    "revisionSummary": "Initial commit",
                    "revisionUrl": "https://ap-northeast-1.console.aws.amazon.com/codecommit/home#/repository/terraform-sample-static-contents/commit/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
                }
            ],
            "trigger": {
                "triggerType": "StartPipelineExecution",
                "triggerDetail": "arn:aws:iam::XXXXXXXXXXXX:user/terraform-sample"
            }
        }
    ]
}
```
`"triggerType": "CloudWatchEvent"`である実行履歴の`status`が`Succeeded`となっていれば正常です。

## 静的コンテンツのWebサイト化
以下の部分を作成します。
![terraform-aws-静的コンテンツのWebサイト化.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/205862/413a4677-90d0-c779-eacf-19ae1d4321ab.png)

### S3をWebサイト化
```diff_ruby:static_contents_s3.tf(追記)
resource aws_s3_bucket static_contents {
  bucket = "${var.app_name}-static-contents-${terraform.workspace}"
  acl    = "private"

+ website {
+   index_document = "index.html"
+   error_document = "error.html"
+ }

  tags = {
    Name = "${var.app_name}-static-contents-${terraform.workspace}"
  }
}
#-- 中略 --#
```

### CloudFront
#### ドメイン設定
```tf:terraform.tfvars(追記)
#-- 省略 --#
user_domain = "app.example.com"
admin_domain = "admin.app.example.com"
```

#### ドメイン証明書
##### 独自ドメインをAWSで管理している（Route53にホストゾーンがある）場合
ホストゾーン情報をterraformで取得します。

```tf:shared/route53.tf(新規作成)
data aws_route53_zone route53-zone {
  name         = "example.com."
  private_zone = false
}

output route53-zone {
  value = data.aws_route53_zone.route53-zone
}
```

ACMで作成する証明書は登録するリージョンは***バージニア北部(us-east-1)***にする必要があります。
証明書のリソースでリージョンを指定することはできないので、terraformのデフォルト引数であるproviderを使用してリージョンを指定します。

```tf:terraform.tf(追記)
#-- 中略 --#
provider "aws" {
  alias  = "virginia"
  region = "us-east-1"
}
```

```ｔｆ:acm.tf(新規作成)
resource aws_acm_certificate user_cert {
  domain_name       = "${terraform.workspace == "production" ? "" : terraform.workspace}${var.user_domain}"
  validation_method = "DNS"
  provider          = aws.virginia
}

resource aws_route53_record user_cert_validation {
  zone_id = data.terraform_remote_state.shared.outputs.route53-zone.zone_id
  name    = tolist(aws_acm_certificate.user_cert.domain_validation_options)[0].resource_record_name
  type    = tolist(aws_acm_certificate.user_cert.domain_validation_options)[0].resource_record_type
  records = [tolist(aws_acm_certificate.user_cert.domain_validation_options)[0].resource_record_value]
  ttl     = 60
}

resource aws_acm_certificate_validation user_cert {
  certificate_arn         = aws_acm_certificate.user_cert.arn
  validation_record_fqdns = [aws_route53_record.user_cert_validation.fqdn]
  provider                = aws.virginia
}

resource aws_acm_certificate admin_cert {
  domain_name       = "${terraform.workspace == "production" ? "" : terraform.workspace}${var.admin_domain}"
  validation_method = "DNS"
  provider          = aws.virginia
}

resource aws_route53_record admin_cert_validation {
  zone_id = data.terraform_remote_state.shared.outputs.route53-zone.zone_id
  name    = tolist(aws_acm_certificate.admin_cert.domain_validation_options)[0].resource_record_name
  type    = tolist(aws_acm_certificate.admin_cert.domain_validation_options)[0].resource_record_type
  records = [tolist(aws_acm_certificate.admin_cert.domain_validation_options)[0].resource_record_value]
  ttl     = 60
}

resource aws_acm_certificate_validation admin_cert {
  certificate_arn         = aws_acm_certificate.admin_cert.arn
  validation_record_fqdns = [aws_route53_record.admin_cert_validation.fqdn]
  provider                = aws.virginia
}
```

##### 独自ドメインをAWSで管理していない(オレオレ証明書を作成する)場合
[AWS Certificate Managerでオレオレ証明書をインポートする - Qiita](https://qiita.com/Raccoon-san/items/f342c4f41c8d6ce465cb)を参考に、証明書を作成します。
実際は自動で割り当てられた`XXXXXXXXXXXXXXXXXX.cloudfront.net`にHTTPSアクセスするので、証明するドメインはなんでも良いです。

```sh
$ mkdir certs
$ cd certs

# ルート証明書
certs$ openssl genrsa -out root.key -des3 2048
Enter pass phrase for root.key:パスフレーズ
Verifying - Enter pass phrase for root.key:パスフレーズ
certs$ openssl req -new -x509 -key root.key -sha256 -days 3650 -out root.pem -subj "/C=JP/ST=Tokyo/O=example corp./CN=example root 2020"
Enter pass phrase for root.key:パスフレーズ

# 中間CA証明書
certs$ openssl genrsa -out intermediate-ca.key -des3 2048
Enter pass phrase for intermediate-ca.key:パスフレーズ
Verifying - Enter pass phrase for intermediate-ca.key:パスフレーズ
certs$ openssl req -new -key intermediate-ca.key -sha256 -outform PEM -keyform PEM -out intermediate-ca.csr -subj "/C=JP/ST=Tokyo/O=example corp./CN=example Inter CA 2020"
Enter pass phrase for intermediate-ca.key:パスフレーズ
certs$ cat - << EOS >> openssl-sign-intermediate-ca.conf
[ v3_ca ]
basicConstraints = CA:true, pathlen:0
keyUsage = cRLSign, keyCertSign
nsCertType = sslCA, emailCA
EOS
certs$ openssl x509 -extfile openssl-sign-intermediate-ca.conf -req -in intermediate-ca.csr -sha256 -CA root.pem -CAkey root.key -set_serial 01 -extensions v3_ca -days 3650 -out intermediate-ca.pem
Enter pass phrase for root.key:パスフレーズ

# サーバ証明書
certs$ openssl genrsa 2048 > server.key
certs$ openssl req -new -key server.key -outform PEM -keyform PEM -sha256 -out server.csr -subj "/C=JP/ST=Tokyo/O=example corp./CN=vpn.example.com"
certs$ openssl x509 -req -in server.csr -sha256 -CA intermediate-ca.pem -CAkey intermediate-ca.key -set_serial 01 -days 3650 -out server.pem
Enter pass phrase for intermediate-ca.key:パスフレーズ
```

###### AWS CLIで登録する
[AWS CLI を使用してインポートする - AWS Certificate Manager ](https://docs.aws.amazon.com/ja_jp/acm/latest/userguide/import-certificate-api-cli.html#import-certificate-cli)の通り、AWS CLIで登録します。
この時、登録するリージョンは***バージニア北部(us-east-1)***にする必要があります。

```sh
$ aws acm import-certificate \
  --profile terraform-sample \
  --region us-east-1 \
  --certificate fileb://server.pem \
  --certificate-chain fileb://intermediate-ca.pem \
  --private-key fileb://server.key
{
    "CertificateArn": "arn:aws:acm:us-east-1:XXXXXXXXXXXX:certificate/abcdefghijklmnopqrstuvwxyz"
}
```

###### Terraformで変数化
```tf:terraform.tfvars(追記)
#-- 中略 --#
acm_certificate_arn = "arn:aws:acm:us-east-1:XXXXXXXXXXXX:certificate/abcdefghijklmnopqrstuvwxyz"
```

#### Origin Access Identity
```tf:shared/origin_access_identity.tf(新規作成)
resource aws_cloudfront_origin_access_identity oai {
  comment = var.app_name
}

output cloudfront_origin_access_identity {
  value = aws_cloudfront_origin_access_identity.oai
}
```

#### ログ用S3
```tf:modules/s3/logs.tf(新規作成)
variable "app_name" {}
variable "cloudfront_origin_access_identity_iam_arn" {}

resource "aws_s3_bucket" "logs" {
  bucket = "${var.app_name}-logs"
  acl    = "private"

  tags = {
    Name = "${var.app_name}-logs"
  }
}

output "logs" {
  value = aws_s3_bucket.logs
}

data "aws_iam_policy_document" "s3_logs" {
  statement {
    effect    = "Allow"
    actions   = ["s3:PutObject"]
    resources = ["${aws_s3_bucket.logs.arn}/*"]

    principals {
      type        = "AWS"
      identifiers = [var.cloudfront_origin_access_identity_iam_arn]
    }
  }
}

resource "aws_s3_bucket_policy" "logs" {
  bucket = aws_s3_bucket.logs.id
  policy = data.aws_iam_policy_document.s3_logs.json
}
```
```tf:stage/load_modules.tf(追記)
#-- 中略 --#
module "s3" {
  source                                    = "../modules/s3"
  app_name                                  = var.app_name
  cloudfront_origin_access_identity_iam_arn = module.cloudfront.origin_access_identity.iam_arn
}
```

#### Administrators
```tf:stage/admin_cloudfront.tf(新規作成)
resource "aws_cloudfront_distribution" "admin" {
  enabled             = true
  comment             = var.admin_domain
  default_root_object = "index.html"

  origin {
    origin_id   = "s3-${var.admin_domain}"
    domain_name = aws_s3_bucket.static_contents.bucket_domain_name

    s3_origin_config {
      origin_access_identity = module.cloudfront.origin_access_identity.cloudfront_access_identity_path
    }
  }

  default_cache_behavior {
    target_origin_id       = "s3-${var.admin_domain}"
    allowed_methods        = ["GET", "HEAD"]
    cached_methods         = ["GET", "HEAD"]
    compress               = true
    viewer_protocol_policy = "redirect-to-https"
    default_ttl            = 3600
    min_ttl                = 0
    max_ttl                = 86400

    forwarded_values {
      query_string = false

      cookies {
        forward = "none"
      }
    }
  }

  restrictions {
    geo_restriction {
      restriction_type = "none"
    }
  }

  viewer_certificate {
    acm_certificate_arn      = var.acm_certificate_arn
    ssl_support_method       = "sni-only"
    minimum_protocol_version = "TLSv1"
  }

  logging_config {
    bucket          = module.s3.logs.bucket_domain_name
    prefix          = "${terraform.workspace}/cloudfront/admin"
    include_cookies = false
  }

  tags = {
    Name = var.admin_domain
  }
}
```

#### CloudFrontから静的コンテンツS3へのアクセス許可
```tf:stage/static_contents_s3.tf(追記)
#-- 中略 --#
data "aws_iam_policy_document" "cloudfront_s3_static_contents" {
  statement {
    effect    = "Allow"
    actions   = ["s3:GetObject"]
    resources = ["${aws_s3_bucket.static_contents.arn}/*"]

    principals {
      type        = "AWS"
      identifiers = [module.cloudfront.origin_access_identity.iam_arn]
    }
  }
}

resource "aws_s3_bucket_policy" "static_contents" {
  bucket = aws_s3_bucket.static_contents.id
  policy = data.aws_iam_policy_document.cloudfront_s3_static_contents.json
}
```

## アプリケーション環境
### VPC
```tf:terraform.tfvars(追記)
#-- 中略 --#
vpc_cidr_block = "10.1.0.0/16"
```
```tf:stage/variables.tf(追記)
#-- 中略 --#
variable "vpc_cidr_block" {}
```
```tf:modules/vpc/main.tf(新規作成)
variable "app_name" {}
variable "vpc_cidr_block" {}

resource "aws_vpc" "vpc" {
  cidr_block           = var.vpc_cidr_block
  instance_tenancy     = "default"
  enable_dns_support   = "true"
  enable_dns_hostnames = "true"

  tags = {
    Name = var.app_name
  }
}

output "id" {
  value = aws_vpc.vpc.id
}
```
```tf:stage/load_modules.tf(追記)
#-- 中略 --#
module "vpc" {
  source         = "../modules/vpc"
  app_name       = var.app_name
  vpc_cidr_block = var.vpc_cidr_block
}
```

### 踏み台サーバ
アプリケーションサーバやDBにアクセスする場合に経由する踏み台サーバを作成します。

#### Subnet
インターネットからパブリックアクセスできるよう、関連リソースも合わせて作成します。

```tf:terraform.tfvars(追記)
#-- 中略 --#
availability_zone_a = "ap-northeast-1a"

public_cidr_block_a = "10.1.1.0/24"
```
```tf:stage/variables.tf(追記)
#-- 中略 --#
variable "availability_zone_a" {}

variable "public_cidr_block_a" {}
```
```tf:modules/subnet/gateway.tf(新規作成)
resource "aws_internet_gateway" "igw" {
  vpc_id = aws_vpc.vpc.id

  tags = {
    Name = var.app_name
  }
}
```
```tf:modules/subnet/public.tf(新規作成)
variable "app_name" {}
variable "vpc_id" {}
variable "public_cidr_block_a" {}
variable "availability_zone_a" {}

resource "aws_subnet" "public_a" {
  vpc_id            = var.vpc_id
  cidr_block        = var.public_cidr_block_a
  availability_zone = var.availability_zone_a

  tags = {
    Name = "${var.app_name}-public-a"
  }
}

output "public_a_id" {
  value = aws_subnet.public_a.id
}
```
```tf:modules/subnet/route_table.tf(新規作成)
resource "aws_route_table" "public" {
  vpc_id = aws_vpc.vpc.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.igw.id
  }

  tags = {
    Name = "${var.app_name}-public"
  }
}

resource "aws_route_table_association" "public_a" {
  subnet_id      = aws_subnet.public_a.id
  route_table_id = aws_route_table.public.id
}
```
```tf:stage/load_modules.tf(追記)
#-- 中略 --#
module "subnet" {
  source              = "../modules/subnet"
  app_name            = var.app_name
  vpc_id              = module.vpc.id
  availability_zone_a = var.availability_zone_a
  public_cidr_block_a = var.public_cidr_block_a
}
```

#### セキュリティグループ
踏み台サーバにSSHアクセスできるよう、22番ポートを開放します。
自宅や社内からのみアクセスできるように、固定IPからのSSHアクセスのみ許可することも可能です。

```tf:modules/security_group/jump.tf(新規作成)
variable "app_name" {}
variable "vpc_id" {}

resource "aws_security_group" "jump" {
  name   = "${var.app_name}-jump"
  vpc_id = var.vpc_id

  tags = {
    Name = "${var.app_name}-jump"
  }
}

resource "aws_security_group_rule" "jump_ssh" {
  security_group_id = aws_security_group.jump.id
  type              = "ingress"
  from_port         = 22
  to_port           = 22
  protocol          = "tcp"
  cidr_blocks       = ["0.0.0.0/0"] # 固定IPからアクセスする場合はここで指定
}

output "jump_id" {
  value = aws_security_group.jump.id
}
```
```tf:stage/load_modules.tf(追記)
#-- 中略 --#
module "security_group" {
  source   = "../modules/security_group"
  app_name = var.app_name
  vpc_id   = module.vpc.id
}
```

[Amazon Linux 2 で SSH を保護する - Qiita](https://qiita.com/ooxif/items/fb8f6a0bd1c516a92b66)のように、SSHで使用するポート番号を変更した場合はTerraformも更新しましょう。(例：51921番に変更)

```diff:modules/security_group/jump.tf(修正)
#-- 省略 --#
resource "aws_security_group_rule" "jump_ssh" {
  security_group_id = aws_security_group.jump.id
  type              = "ingress"
- from_port         = 22
+ from_port         = 51921
- to_port           = 22
+ to_port           = 51921
  protocol          = "tcp"
  cidr_blocks       = ["0.0.0.0/0"]
}
```

#### EC2
##### Amazon linux 2の最新AMI ID取得
[CloudFormationで最新のAmazon Linux 2のAMI IDを取得してEC2を構築する - DevelopersIO](https://dev.classmethod.jp/cloud/aws/get-the-latest-amazon-linux-2-ami-id-with-cloudformation/)を参考に取得します。

```sh
$ aws ssm get-parameter \
  --name /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2 \
  --region ap-northeast-1 \
  --query "Parameter.Value" \
  --profile terraform-sample
"ami-0064e711cbc7a825e"
```

##### EC2にアクセスするためのkey pair作成
誤操作による削除を防ぐため、[Amazon EC2 キーペアの作成、表示、および削除 - AWS Identity and Access Management](https://docs.aws.amazon.com/ja_jp/cli/latest/userguide/cli-services-ec2-keypairs.html#creating-a-key-pair)を参考にTerraform管理外で作成します。

```sh
$ aws ec2 create-key-pair \
  --key-name terraform-sample-jump \
  --query 'KeyMaterial' \
  --output text \
  --profile terraform-sample > certs/terraform-sample-jump.pem
$ chmod 400 certs/terraform-sample-jump.pem
```

##### EC2インスタンス
```tf:modules/ec2/jump.tf(新規作成)
variable "app_name" {}
variable "availability_zone_a" {}
variable "jump_key_name" {}
variable "subnet_id" {}
variable "jump_security_group_id" {}

resource "aws_instance" "jump" {
  ami                    = "ami-011facbea5ec0363b" # 最新AMI ID
  instance_type          = "t2.micro"
  availability_zone      = var.availability_zone_a
  key_name               = var.jump_key_name
  monitoring             = "false"
  subnet_id              = var.subnet_id
  vpc_security_group_ids = [var.jump_security_group_id]

  tags = {
    Name = "${var.app_name}-jump"
  }
}
```
```tf:stage/load_modules.tf(追記)
#-- 中略 --#
module "ec2" {
  source                 = "../modules/ec2"
  app_name               = var.app_name
  availability_zone_a    = var.availability_zone_a
  jump_key_name          = var.jump_key_name
  subnet_id              = module.subnet.public_a_id
  jump_security_group_id = module.security_group.jump_id
}
```

##### Elastic IP
```tf:modules/ec2/jump.tf(追記)
#-- 中略 --#
resource "aws_eip" "jump" {
  vpc = true

  tags = {
    Name = "${var.app_name}-jump"
  }
}

resource "aws_eip_association" "jump" {
  allocation_id = aws_eip.jump.id
  instance_id   = aws_instance.jump.id
}
```

##### 踏み台サーバのパブリックIPアドレスを取得
```sh
$ aws ec2 describe-instances \
  --filter "Name=tag:Name,Values=terraform-sample-jump" \
  --query "Reservations[0].Instances[0].PublicIpAddress" \
  --profile terraform-sample
"XXX.XXX.XXX.XXX"
```

##### 接続確認
以下のように接続できれば正しい状態となります。

```sh
$ ssh -i certs/terraform-sample-jump.pem ec2-user@XXX.XXX.XXX.XXX
The authenticity of host 'XXX.XXX.XXX.XXX (XXX.XXX.XXX.XXX)' can't be established.
ECDSA key fingerprint is SHA256:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'XXX.XXX.XXX.XXX' (ECDSA) to the list of known hosts.

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/
[ec2-user@ip-XXX-XXX-XXX-XXX ~]$ 
```
`Are you sure you want to continue connecting (yes/no)?`のところで`yes`と入力しましょう。

### CodePipeline & CodeBuildによるCI
#### CodePipelineのアーティファクトを格納するS3バケット
```tf:stage/codepipeline_s3.tf(追記)
#-- 中略 --#
resource "aws_s3_bucket" "codepipeline_application_sources" {
  bucket = "${var.app_name}-codepipeline-application-sources-${terraform.workspace}"
  acl    = "private"

  tags = {
    Name = "${var.app_name}-codepipeline-application-sources-${terraform.workspace}"
  }
}

data "aws_iam_policy_document" "s3_codepipeline_application_sources" {
  statement {
    effect = "Allow"
    actions = [
      "s3:GetObject",
      "s3:GetObjectVersion",
      "s3:GetBucketVersioning",
      "s3:PutObject"
    ]

    resources = [
      aws_s3_bucket.codepipeline_application_sources.arn,
      "${aws_s3_bucket.codepipeline_application_sources.arn}/*"
    ]
  }
}
```

#### CodeCommitのアクセス権限
```tf:modules/iam/codecommit.tf(追記)
#-- 中略 --#
variable "application_sources_repository_arn" {}

data "aws_iam_policy_document" "codecommit_application_sources" {
  statement {
    effect = "Allow"
    actions = [
      "codecommit:GitPull",
      "codecommit:GetBranch",
      "codecommit:GetCommit",
      "codecommit:UploadArchive",
      "codecommit:GetUploadArchiveStatus"
    ]

    resources = [
      var.application_sources_repository_arn,
      "${var.application_sources_repository_arn}/*"
    ]
  }
}

output "codecommit_application_sources_policy_json" {
  value = data.aws_iam_policy_document.codecommit_application_sources.json
}
```
```diff:stage/load_modules.tf(追記)
#-- 中略 --#
module "iam" {
  source                             = "../modules/iam"
  static_contents_repository_arn     = var.static_contents_repository_arn
+ application_sources_repository_arn = var.application_sources_repository_arn
}
#-- 中略 --#
```

#### CodeBuild
```tf:modules/iam/codebuild.tf(新規作成)
variable "app_name" {}

data "aws_iam_policy_document" "codebuild_assume_role" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRole"]
    principals {
      identifiers = ["codebuild.amazonaws.com"]
      type        = "Service"
    }
  }
}

output "codebuild_assume_role_policy_json" {
  value = data.aws_iam_policy_document.codebuild_assume_role.json
}

data "aws_iam_policy_document" "codebuild" {
  statement {
    effect = "Allow"
    actions = [
      "logs:CreateLogGroup",
      "logs:CreateLogStream",
      "logs:PutLogEvents"
    ]
    resources = ["*"]
  }

  statement {
    effect = "Allow"
    actions = [
      "codebuild:StopBuild",
      "ec2:*"
    ]

    resources = ["*"]
  }
}

output "codebuild_policy_json" {
  value = data.aws_iam_policy_document.codebuild.json
}
```
```diff:stage/load_modules.tf(追記)
#-- 中略 --#
module "iam" {
  source                             = "../modules/iam"
  static_contents_repository_arn     = var.static_contents_repository_arn
  application_sources_repository_arn = var.application_sources_repository_arn
+ app_name                           = var.app_name
}
#-- 中略 --#
```
```tf:stage/admin_codebuild.tf(新規作成)
resource "aws_iam_role" "codebuild_admin" {
  name               = "${var.app_name}-codebuild-admin"
  assume_role_policy = module.iam.codebuild_assume_role_policy_json
}

resource "aws_iam_role_policy" "codebuild_admin" {
  role   = aws_iam_role.codebuild_admin.name
  policy = module.iam.codebuild_policy_json
}

resource "aws_iam_role_policy" "s3_codebuild_admin" {
  role   = aws_iam_role.codebuild_admin.name
  policy = data.aws_iam_policy_document.s3_codepipeline_application_sources.json
}

resource "aws_iam_role_policy" "kms_codebuild_admin" {
  role   = aws_iam_role.codebuild_admin.name
  policy = module.iam.kms_policy_json
}

resource "aws_codebuild_project" "admin" {
  name          = "${var.app_name}-admin-${terraform.workspace}"
  description   = "${var.app_name}-admin-${terraform.workspace}"
  build_timeout = "15"
  service_role  = aws_iam_role.codebuild_admin.arn

  artifacts {
    type = "NO_ARTIFACTS"
  }

  cache {
    type = "LOCAL"
    modes = [
      "LOCAL_SOURCE_CACHE",
      "LOCAL_CUSTOM_CACHE"
    ]
  }

  environment {
    compute_type                = "BUILD_GENERAL1_SMALL"
    image                       = "aws/codebuild/standard:2.0"
    type                        = "LINUX_CONTAINER"
    image_pull_credentials_type = "CODEBUILD"
    privileged_mode             = true
  }

  logs_config {
    cloudwatch_logs {
      status      = "ENABLED"
      group_name  = "${var.app_name}-admin-${terraform.workspace}"
      stream_name = "${var.app_name}-admin-${terraform.workspace}"
    }
  }

  source {
    type            = "CODECOMMIT"
    buildspec       = "buildspec_admin.yml"
    git_clone_depth = 1
    location        = var.application_sources_repository_name
  }

  tags = {
    Name = "${var.app_name}-admin-${terraform.workspace}"
  }
}

data "aws_iam_policy_document" "codebuild_admin" {
  statement {
    effect = "Allow"
    actions = [
      "codebuild:BatchGetBuilds",
      "codebuild:StartBuild"
    ]

    resources = [aws_codebuild_project.admin.arn]
  }
}
```

##### ビルド設定
アプリケーション用CodeCommitリポジトリ内に作成します。

```yaml:buildspec_admin.yml(新規作成)
version: 0.2
phases:
  install:
    runtime-versions:
      docker: 18
    commands:
      - echo Install started on `date`
    finally:
      - echo Install completed on `date`
  pre_build:
    commands:
      - echo PreBuild started on `date`
      - ./gradlew clean test --info
    finally:
      - echo PreBuild completed on `date`
  build:
    commands:
      - echo Build started on `date`
    finally:
      - echo Build completed on `date`
  post_build:
    commands:
      - echo PostBuild started on `date`
    finally:
      - echo PostBuild completed on `date`
```

#### CodePipelineからCodeBuildを起動
```tf:stage/variables.tf(追記)
#-- 中略 --#
variable "application_sources_target_branch" {
  default = "stage"
}
```
```tf:stage/application_sources_codepipeline.tf(新規作成)
resource "aws_iam_role" "codepipeline_application_sources" {
  name               = "${var.app_name}-codepipeline-application-sources-${terraform.workspace}"
  assume_role_policy = module.iam.codepipeline_assume_role_policy_json
}

resource "aws_iam_role_policy" "codecommit_codepipeline_application_sources" {
  role   = aws_iam_role.codepipeline_application_sources.name
  policy = module.iam.codecommit_application_sources_policy_json
}

resource "aws_iam_role_policy" "s3_codepipeline_application_sources" {
  role   = aws_iam_role.codepipeline_application_sources.name
  policy = data.aws_iam_policy_document.s3_codepipeline_application_sources.json
}

resource "aws_iam_role_policy" "kms_codepipeline_application_sources" {
  role   = aws_iam_role.codepipeline_application_sources.name
  policy = module.iam.kms_policy_json
}

resource "aws_iam_role_policy" "codebuild_admin_codepipeline_application_sources" {
  role   = aws_iam_role.codepipeline_application_sources.name
  policy = data.aws_iam_policy_document.codebuild_admin.json
}

resource "aws_codepipeline" "application_sources" {
  name     = "${var.app_name}-application-sources-${terraform.workspace}"
  role_arn = aws_iam_role.codepipeline_application_sources.arn

  artifact_store {
    location = aws_s3_bucket.codepipeline_application_sources.bucket
    type     = "S3"
    encryption_key {
      id   = module.kms.alias_arn
      type = "KMS"
    }
  }

  stage {
    name = "${var.app_name}-application-sources-${terraform.workspace}-source"

    action {
      name             = "${var.app_name}-application-sources-${terraform.workspace}-source-action"
      category         = "Source"
      owner            = "AWS"
      provider         = "CodeCommit"
      version          = "1"
      output_artifacts = ["SOURCE"]

      configuration = {
        PollForSourceChanges = "false"
        RepositoryName       = var.application_sources_repository_name
        BranchName           = var.application_sources_target_branch
      }
    }
  }

  stage {
    name = "${var.app_name}-application-sources-${terraform.workspace}-build-admin"

    action {
      name             = "${var.app_name}-application-sources-${terraform.workspace}-build-admin-action"
      category         = "Build"
      owner            = "AWS"
      provider         = "CodeBuild"
      input_artifacts  = ["SOURCE"]
      output_artifacts = ["ADMIN_BUILD"]
      version          = "1"

      configuration = {
        ProjectName = aws_codebuild_project.admin.name
      }
    }
  }
}
```

##### 動作確認
```sh
$ aws codepipeline start-pipeline-execution \
  --name terraform-sample-application-sources-stage \
  --profile terraform-sample
{
    "pipelineExecutionId": "AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE"
}
$ aws codepipeline get-pipeline-execution \
  --pipeline-name terraform-sample-application-sources-stage \
  --pipeline-execution-id AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE
  --query "pipelineExecution.status" \
  --profile terraform-sample
"Succeeded"
```
`Succeeded`となっていれば正常です。

#### CodeCommitリポジトリ対象ブランチへのプッシュを検知
```tf:stage/application_sources_codepipeline_cloudwatch.tf(新規作成)
resource "aws_iam_role" "codepipeline_application_sources_cloudwatch" {
  name               = "${var.app_name}-codepipeline-application-sources-cloudwatch-${terraform.workspace}"
  assume_role_policy = module.iam.cloudwatch_assume_role_policy_json
}

data "aws_iam_policy_document" "codepipeline_application_sources_cloudwatch" {
  statement {
    effect  = "Allow"
    actions = ["codepipeline:StartPipelineExecution"]

    resources = [aws_codepipeline.application_sources.arn]
  }
}

resource "aws_iam_role_policy" "codepipeline_application_sources_cloudwatch" {
  role   = aws_iam_role.codepipeline_application_sources_cloudwatch.name
  policy = data.aws_iam_policy_document.codepipeline_application_sources_cloudwatch.json
}

resource "aws_cloudwatch_event_rule" "codepipeline_application_sources" {
  name          = "${var.app_name}-codepipeline-application-sources-${terraform.workspace}"
  event_pattern = <<PATTERN
  {
    "source": [
      "aws.codecommit"
    ],
    "detail-type": [
      "CodeCommit Repository State Change"
    ],
    "resources": [
      "${var.application_sources_repository_arn}"
    ],
    "detail": {
      "event": [
        "referenceCreated",
        "referenceUpdated"
      ],
      "referenceType": [
        "branch"
      ],
      "referenceName": [
        "${var.application_sources_target_branch}"
      ]
    }
  }
PATTERN
}

resource "aws_cloudwatch_event_target" "codepipeline_application_sources" {
  rule      = aws_cloudwatch_event_rule.codepipeline_application_sources.name
  target_id = "${var.app_name}-codepipeline-application-sources-${terraform.workspace}"
  arn       = aws_codepipeline.application_sources.arn
  role_arn  = aws_iam_role.codepipeline_application_sources_cloudwatch.arn
}
```

##### 動作確認
1. `var.static_contents_target_branch`で指定したブランチに変更をpushします。
1. 以下のコマンドで確認します。

```sh
$ aws codepipeline list-pipeline-executions \
  --pipeline-name terraform-sample-application-sources-stage \
  --profile terraform-sample
{
    "pipelineExecutionSummaries": [
        {
            "pipelineExecutionId": "YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY",
            "status": "Succeeded",
            "startTime": BBBBBBBBBB.BBB,
            "lastUpdateTime": BBBBBBBBBB.BBB,
            "sourceRevisions": [
                {
                    "actionName": "terraform-sample-application-sources-stage-source-action",
                    "revisionId": "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB",
                    "revisionSummary": "Test commit",
                    "revisionUrl": "https://ap-northeast-1.console.aws.amazon.com/codecommit/home#/repository/terraform-sample-application-sources/commit/BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB"
                }
            ],
            "trigger": {
                "triggerType": "CloudWatchEvent",
                "triggerDetail": "arn:aws:events:ap-northeast-1:338927112236:rule/terraform-sample-codepipeline-application-sources-stage"
            }
        },
        {
            "pipelineExecutionId": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
            "status": "Succeeded",
            "startTime": AAAAAAAAAA.AAA,
            "lastUpdateTime": AAAAAAAAAA.AAA,
            "sourceRevisions": [
                {
                    "actionName": "terraform-sample-application-sources-stage-source-action",
                    "revisionId": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
                    "revisionSummary": "Initial commit",
                    "revisionUrl": "https://ap-northeast-1.console.aws.amazon.com/codecommit/home#/repository/terraform-sample-static-contents/commit/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
                }
            ],
            "trigger": {
                "triggerType": "StartPipelineExecution",
                "triggerDetail": "arn:aws:iam::XXXXXXXXXXXX:user/terraform-sample"
            }
        }
    ]
}
```
`"triggerType": "CloudWatchEvent"`である実行履歴の`status`が`Succeeded`となっていれば正常です。

### アプリケーションサーバ
#### サブネット
```tf:terraform.tfvars(追記)
#-- 中略 --#
availability_zone_c  = "ap-northeast-1c"
private_cidr_block_a = "10.1.2.0/24"
private_cidr_block_c = "10.1.3.0/24"
```
```tf:stage/variables.tf(追記)
#-- 中略 --#
variable "availability_zone_c" {}

variable "private_cidr_block_a" {}
variable "private_cidr_block_c" {}
```
```tf:modules/subnet/private.tf(新規作成)
variable "availability_zone_c" {}
variable "private_cidr_block_a" {}
variable "private_cidr_block_c" {}

resource "aws_subnet" "private_a" {
  vpc_id            = var.vpc_id
  cidr_block        = var.private_cidr_block_a
  availability_zone = var.availability_zone_a

  tags = {
    Name = "${var.app_name}-private-a"
  }
}

output "private_a_id" {
  value = aws_subnet.private_a.id
}

resource "aws_subnet" "private_c" {
  vpc_id            = var.vpc_id
  cidr_block        = var.private_cidr_block_c
  availability_zone = var.availability_zone_c

  tags = {
    Name = "${var.app_name}-private-c"
  }
}

output "private_c_id" {
  value = aws_subnet.private_c.id
}
```
```diff:stage/load_modules.tf(追記)
#-- 中略 --#
module "subnet" {
  source               = "../modules/subnet"
  app_name             = var.app_name
  vpc_id               = module.vpc.id
  availability_zone_a  = var.availability_zone_a
+ availability_zone_c  = var.availability_zone_c
  public_cidr_block_a  = var.public_cidr_block_a
+ private_cidr_block_a = var.private_cidr_block_a
+ private_cidr_block_c = var.private_cidr_block_c
}
#-- 中略 --#
```

#### EC2
##### EC2にアクセスするためのkey pair作成
```sh
$ aws ec2 create-key-pair \
  --key-name terraform-sample-stage \
  --query 'KeyMaterial' \
  --output text \
  --profile terraform-sample > certs/terraform-sample-stage.pem
$ chmod 400 certs/terraform-sample-stage.pem
```


###### 踏み台サーバに設置し、ローカルからは削除
```sh
$ ssh -i certs/terraform-sample-jump.pem ec2-user@XXX.XXX.XXX.XXX "mkdir ~/.ssh/pem"
$ scp -i certs/terraform-sample-jump.pem certs/terraform-sample-stage.pem ec2-user@XXX.XXX.XXX.XXX:~/.ssh/pem
$ ssh -i certs/terraform-sample-jump.pem ec2-user@XXX.XXX.XXX.XXX "chmod 600 ~/.ssh/pem/terraform-sample-stage.pem"
$ rm -f certs/terraform-sample-stage.pem
```
```tf:stage/variables.tf(追記)
#-- 中略 --#
variable "key_name" {
  default = "terraform-sample-stage"
}
```

##### セキュリティグループ
踏み台サーバーからのみSSHアクセスを許可するセキュリティグループを作成します。
また、踏み台サーバーからStageインスタンスへSSHできるよう、アウトバウンドルールを追加します。

```tf:modules/security_group/jump.tf(追記)
#-- 中略 --#
variable "vpc_cidr_block" {}

resource "aws_security_group_rule" "jump_ssh_out" {
  security_group_id = aws_security_group.jump.id
  type              = "egress"
  from_port         = 22
  to_port           = 22
  protocol          = "tcp"
  cidr_blocks       = [var.vpc_cidr_block]
}

resource "aws_security_group" "from_jump" {
  name   = "${var.app_name}-from-jump"
  vpc_id = var.vpc_id

  tags = {
    Name = "${var.app_name}-from-jump"
  }
}

resource "aws_security_group_rule" "from_jump_ssh" {
  security_group_id        = aws_security_group.from_jump.id
  type                     = "ingress"
  from_port                = 22
  to_port                  = 22
  protocol                 = "tcp"
  source_security_group_id = aws_security_group.jump.id
}

output "from_jump_id" {
  value = aws_security_group.from_jump.id
}
```
```diff:stage/load_modules.tf(追記)
#-- 中略 --#
module "security_group" {
  source         = "../modules/security_group"
  app_name       = var.app_name
  vpc_id         = module.vpc.id
+ vpc_cidr_block = var.vpc_cidr_block
}
#-- 中略 --#
```

##### EC2インスタンス
後ほどApplicationLoadBalancerのAutoScalingGroupに追加する雛形のインスタンスを生成し、AMIを作成します。

```tf:stage/admin_ec2.tf(新規作成)
resource "aws_instance" "admin" {
  ami                    = "ami-011facbea5ec0363b"
  instance_type          = "t2.small"
  availability_zone      = var.availability_zone_a
  key_name               = var.key_name
  monitoring             = "false"
  subnet_id              = module.subnet.private_a_id
  vpc_security_group_ids = [module.security_group.from_jump_id]

  tags = {
    Name = "${var.app_name}-admin-${terraform.workspace}"
  }
}
```

##### StageサーバのプライベートIPアドレスを取得
```sh
$ aws ec2 describe-instances \
  --filter "Name=tag:Name,Values=terraform-sample-admin-stage" \
  --query "Reservations[0].Instances[0].PrivateIpAddress" \
  --profile terraform-sample
"YYY.YYY.YYY.YYY"
```

##### 接続確認
以下のように接続できれば正しい状態となります。

```sh
$ ssh -i certs/terraform-sample-stage.pem ec2-user@XXX.XXX.XXX.XXX
[ec2-user@ip-XXX-XXX-XXX-XXX ~]$ ssh -i ~/.ssh/pem/terraform-sample-stage.pem ec2-user@YYY.YYY.YYY.YYY
The authenticity of host 'YYY.YYY.YYY.YYY (YYY.YYY.YYY.YYY)' can't be established.
ECDSA key fingerprint is SHA256:XXXXXXXXXXXXXXXXXXXXXXXXXX
ECDSA key fingerprint is MD5:XXXXXXXXXXXXXXXXXXXXXXXXXX
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'YYY.YYY.YYY.YYY' (ECDSA) to the list of known hosts.

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/
[ec2-user@ip-YYY-YYY-YYY-YYY ~]$
```
先ほどと同様に、`Are you sure you want to continue connecting (yes/no)?`のところで`yes`と入力しましょう。

#### CodeDeploy agent
CodeDeployで自動デプロイできるように、agentをyumとwgetでインストールします。

##### アプリケーションサーバーからインターネットに接続可能にする
public subnetにnat gatewayを設定し、private subnetのデフォルトルートに設定することで、private subnet内のEC2からインターネットにアクセスできるようになります。
参考：
[Terraform を使ってパブリックサブネットを構築する · mzumi's blog](https://blog.mzumi.com/post/2016/08/26/terraform-subnet/)
[Terraform を使ってプライベートサブネットを構築する · mzumi's blog](https://blog.mzumi.com/post/2016/09/01/terraform-private-subnet/)

```tf:modules/subnet/gateway.tf(追記)
#-- 中略 --#
resource "aws_eip" "nat" {
  vpc = true

  tags = {
    Name = "${var.app_name}-nat"
  }
}

resource "aws_nat_gateway" "ngw" {
  allocation_id = aws_eip.nat.id
  subnet_id     = aws_subnet.public_a.id

  tags = {
    Name = var.app_name
  }
}
```
```tf:modules/subnet/route_table.tf(追記)
#-- 中略 --#
resource "aws_route_table" "private" {
  vpc_id = var.vpc_id

  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.ngw.id
  }

  tags = {
    Name = "${var.app_name}-private"
  }
}

resource "aws_route_table_association" "private_a" {
  subnet_id      = aws_subnet.private_a.id
  route_table_id = aws_route_table.private.id
}

resource "aws_route_table_association" "private_c" {
  subnet_id      = aws_subnet.private_c.id
  route_table_id = aws_route_table.private.id
}
```

###### yumとwget用のアウトバウンドルールを設定したセキュリティグループを作成
http, https, dnsのリクエストをEC2からインターネットへ送信できるようにします。

```tf:modules/security_group/allow_internet.tf(新規作成)
resource "aws_security_group" "allow_internet" {
  name   = "${var.app_name}-allow-internet"
  vpc_id = var.vpc_id

  tags = {
    Name = "${var.app_name}-allow-internet"
  }
}

resource "aws_security_group_rule" "allow_internet_http" {
  security_group_id = aws_security_group.allow_internet.id
  type              = "egress"
  from_port         = 80
  to_port           = 80
  protocol          = "tcp"
  cidr_blocks       = ["0.0.0.0/0"]
}

resource "aws_security_group_rule" "allow_internet_https" {
  security_group_id = aws_security_group.allow_internet.id
  type              = "egress"
  from_port         = 443
  to_port           = 443
  protocol          = "tcp"
  cidr_blocks       = ["0.0.0.0/0"]
}

resource "aws_security_group_rule" "allow_internet_dns" {
  security_group_id = aws_security_group.allow_internet.id
  type              = "egress"
  from_port         = 53
  to_port           = 53
  protocol          = "udp"
  cidr_blocks       = ["0.0.0.0/0"]
}

output "allow_internet_id" {
  value = aws_security_group.allow_internet.id
}
```

###### EC2へセキュリティグループを追加
```diff:stage/admin_ec2.tf(修正)
resource "aws_instance" "admin" {
  ami                    = "ami-011facbea5ec0363b"
  instance_type          = "t2.small"
  availability_zone      = var.availability_zone_a
  key_name               = var.key_name
  monitoring             = "false"
  subnet_id              = module.subnet.private_a_id
- vpc_security_group_ids = [module.security_group.from_jump_id]
+ vpc_security_group_ids = [module.security_group.from_jump_id, module.security_group.allow_internet_id]

  tags = {
    Name = "${var.app_name}-admin-${terraform.workspace}"
  }
}
```

##### yum更新・必要なパッケージの取得
root権限で実行します。

```sh:アプリケーションサーバー
[root@ip-YYY-YYY-YYY-YYY ~]# yum upgrade -y
[root@ip-YYY-YYY-YYY-YYY ~]# yum install -y wget git ruby
[root@ip-YYY-YYY-YYY-YYY ~]# amazon-linux-extras install -y java-openjdk11
```
`complete!`もしくは`完了しました!`と表示されれば成功です。

##### CodeDeploy agentのインストール
```sh:アプリケーションサーバー
[root@ip-YYY-YYY-YYY-YYY ~]# wget https://aws-codedeploy-ap-northeast-1.s3.amazonaws.com/latest/install
[root@ip-YYY-YYY-YYY-YYY ~]# chmod +x ./install
[root@ip-YYY-YYY-YYY-YYY ~]# ./install auto
```
同様に、`complete!`もしくは`完了しました!`と表示されれば成功です。

#### CodeDeploy
##### 起動するSpringBootアプリケーションをSystemctlに登録
[SpringBoot アプリをサービスとして動かす方法 - qiita](https://qiita.com/uenosy/items/19829f0e212ac81e0044)を参考に、`/var/lib/springboot/boot.jar`(CodeDeployでデプロイされる実行可能Jar)をサービス登録します。

```sh:アプリケーションサーバー
[root@ip-YYY-YYY-YYY-YYY ~]# adduser application #　グループも同時に作成されます
[root@ip-YYY-YYY-YYY-YYY ~]# id application
uid=1001(application) gid=1001(application) groups=1001(application)
[root@ip-YYY-YYY-YYY-YYY ~]# mkdir /var/lib/springboot/
[root@ip-YYY-YYY-YYY-YYY ~]# mkdir /var/lib/springboot/logs
[root@ip-YYY-YYY-YYY-YYY ~]# chown -R application:application /var/lib/springboot/
[root@ip-YYY-YYY-YYY-YYY ~]# cat - << EOS >> /var/lib/springboot/boot.conf
export JAVA_OPTS="-Dspring.profiles.active=stage"
export LANG="ja_JP.utf8"
EOS
[root@ip-YYY-YYY-YYY-YYY ~]# cat - << EOS >> /etc/systemd/system/springboot.service
[Unit]
Description = springboot application
[Service]
ExecStart = /bin/sh -c 'java -jar /var/lib/springboot/boot.jar &>> /var/lib/springboot/logs/stage.log'
Restart = always
Type = simple
User = application
Group = application
SuccessExitStatus = 143
[Install]
WantedBy = multi-user.target
EOS
[root@ip-YYY-YYY-YYY-YYY ~]# systemctl enable springboot.service
```

##### スクリプト
アプリケーション用CodeCommitリポジトリ内に作成します。
環境に合わせて、各種コマンドを入れてください。

```sh:data/script/application_stop.sh(新規作成)
echo application stop
```
```sh:data/script/before_install.sh(新規作成)
echo before install
systemctl stop springboot.service
rm -f /var/lib/springboot/boot.jar
```
```sh:data/script/after_install.sh(新規作成)
echo after install
```
```sh:data/script/application_start.sh(新規作成)
echo application start
systemctl start springboot.service
```

##### 設定ファイル
アプリケーション用CodeCommitリポジトリ内に作成します。

```yaml:appspec.yml(新規作成)
version: 0.0
os: linux
files:
  - source: /boot.jar
    destination: /var/lib/springboot/
hooks:
  ApplicationStop:
    - location: /application_stop.sh
      timeout: 300
      runas: root
  BeforeInstall:
    - location: /before_install.sh
      timeout: 300
      runas: root
  AfterInstall:
    - location: /after_install.sh
      timeout: 300
      runas: root
  ApplicationStart:
    - location: /application_start.sh
      timeout: 300
      runas: root
```

##### CodeBuildでビルド結果をS3に保存する
```diff:buildspec_admin.yml(追記)
#-- 省略 --#
  build:
    commands:
      - echo Build started on `date`
+     - ./gradlew admin:build -x test
    finally:
      - echo Build completed on `date`
  post_build:
    commands:
      - echo PostBuild started on `date`
+     - cp -p admin/build/libs/admin-0.0.1-SNAPSHOT.jar boot.jar
    finally:
      - echo PostBuild completed on `date`
+artifacts:
+ files:
+   - 'boot.jar'
+   - 'appspec.yml'
+   - 'data/script/*'
+ discard-paths: yes
```

##### CodeDeployアプリケーション
KEY: `Deploy`, Value: `${var.app_name}-admin-${terraform.workspace}`のタグが付与されているEC2をデプロイ対象とします。

```diff:stage/admin_ec2.tf(修正)
resource "aws_instance" "admin" {
  ami                    = "ami-011facbea5ec0363b"
  instance_type          = "t2.small"
  availability_zone      = var.availability_zone_a
  key_name               = var.key_name
  monitoring             = "false"
  subnet_id              = module.subnet.private_a_id
  vpc_security_group_ids = [module.security_group.from_jump_id, module.security_group.allow_internet_id]

  tags = {
    Name   = "${var.app_name}-admin-${terraform.workspace}"
+   Deploy = "${var.app_name}-admin-${terraform.workspace}"
  }
}
```
```tf:modules/iam/codedeploy.tf(新規作成)
data "aws_iam_policy_document" "codedeploy_assume_role" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRole"]
    principals {
      identifiers = ["codedeploy.amazonaws.com"]
      type        = "Service"
    }
  }
}

resource "aws_iam_role" "codedeploy" {
  name               = var.app_name
  assume_role_policy = data.aws_iam_policy_document.codedeploy_assume_role.json
}

output "codedeploy_role_arn" {
  value = aws_iam_role.codedeploy.arn
}

data "aws_iam_policy_document" "codedeploy" {
  statement {
    effect = "Allow"
    actions = [
      "ec2:DescribeInstances",
      "ec2:DescribeInstanceStatus",
      "tag:GetTags",
      "tag:GetResources"
    ]
    resources = ["*"]
  }
}

resource "aws_iam_role_policy" "codedeploy" {
  role   = aws_iam_role.codedeploy.name
  policy = data.aws_iam_policy_document.codedeploy.json
}
```
```tf:stage/admin_codedeploy.tf(新規作成)
resource "aws_codedeploy_app" "admin" {
  name             = "${var.app_name}-admin-${terraform.workspace}"
  compute_platform = "Server"
}

resource "aws_codedeploy_deployment_group" "admin" {
  deployment_group_name  = "${var.app_name}-admin-${terraform.workspace}"
  app_name               = aws_codedeploy_app.admin.name
  deployment_config_name = "CodeDeployDefault.OneAtATime"
  service_role_arn       = module.iam.codedeploy_role_arn
  ec2_tag_filter {
    key   = "Deploy"
    value = "${var.app_name}-admin-${terraform.workspace}"
    type  = "KEY_AND_VALUE"
  }
}
```

##### CodePipelineと紐付け
```tf:modules/iam/codedeploy.tf(追記)
#-- 省略 --#
data "aws_iam_policy_document" "codedeploy_codepipeline" {
  statement {
    effect = "Allow"
    actions = [
      "codedeploy:CreateDeployment",
      "codedeploy:GetApplication",
      "codedeploy:GetApplicationRevision",
      "codedeploy:GetDeployment",
      "codedeploy:GetDeploymentConfig",
      "codedeploy:RegisterApplicationRevision"
    ]
    resources = ["*"]
  }
}

output "codedeploy_codepipeline_policy_json" {
  value = data.aws_iam_policy_document.codedeploy_codepipeline.json
}
```
```tf:stage/application_sources_codepipeline.tf(追記)
#-- 省略 --#
resource "aws_iam_role_policy" "codedeploy_codepipeline_application_sources" {
  role   = aws_iam_role.codepipeline_application_sources.name
  policy = module.iam.codedeploy_codepipeline_policy_json
}
#-- 省略 --#
resource "aws_codepipeline" "application_sources" {
#-- 省略 --#
  stage {
    name = "${var.app_name}-application-sources-${terraform.workspace}-source"
#-- 省略 --#
  }

  stage {
    name = "${var.app_name}-application-sources-${terraform.workspace}-build-admin"
#-- 省略 --#
  }

  stage {
    name = "${var.app_name}-application-sources-${terraform.workspace}-deploy-admin"

    action {
      name            = "${var.app_name}-application-sources-${terraform.workspace}-deploy-admin-action"
      category        = "Deploy"
      owner           = "AWS"
      provider        = "CodeDeploy"
      input_artifacts = ["ADMIN_BUILD"]
      version         = "1"

      configuration = {
        ApplicationName     = aws_codedeploy_app.admin.name,
        DeploymentGroupName = aws_codedeploy_deployment_group.admin.deployment_group_name
      }
    }
  }
}
```

##### EC2の権限設定
このままではEC2にインストールしたCodeDeploy AgentからCodePipelineのアーティファクト(ビルド結果)を格納したS3にアクセスできないので、各種権限を追加します。

```tf:modules/iam/ec2.tf(新規作成)
data "aws_iam_policy_document" "ec2_assume_role" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRole"]
    principals {
      identifiers = ["ec2.amazonaws.com"]
      type        = "Service"
    }
  }
}

output "ec2_assume_role_policy_json" {
  value = data.aws_iam_policy_document.ec2_assume_role.json
}
```
```tf:stage/codepipeline_s3.tf(追記)
#-- 省略 --#
data "aws_iam_policy_document" "s3_codepipeline_application_sources_codedeploy" {
  statement {
    effect = "Allow"
    actions = [
      "s3:Get*",
      "s3:List*"
    ]
    resources = [
      aws_s3_bucket.codepipeline_application_sources.arn,
      "${aws_s3_bucket.codepipeline_application_sources.arn}/*",
      "arn:aws:s3:::aws-codedeploy-us-east-2/*",
      "arn:aws:s3:::aws-codedeploy-us-east-1/*",
      "arn:aws:s3:::aws-codedeploy-us-west-1/*",
      "arn:aws:s3:::aws-codedeploy-us-west-2/*",
      "arn:aws:s3:::aws-codedeploy-ca-central-1/*",
      "arn:aws:s3:::aws-codedeploy-eu-west-1/*",
      "arn:aws:s3:::aws-codedeploy-eu-west-2/*",
      "arn:aws:s3:::aws-codedeploy-eu-west-3/*",
      "arn:aws:s3:::aws-codedeploy-eu-central-1/*",
      "arn:aws:s3:::aws-codedeploy-ap-east-1/*",
      "arn:aws:s3:::aws-codedeploy-ap-northeast-1/*",
      "arn:aws:s3:::aws-codedeploy-ap-northeast-2/*",
      "arn:aws:s3:::aws-codedeploy-ap-southeast-1/*",
      "arn:aws:s3:::aws-codedeploy-ap-southeast-2/*",
      "arn:aws:s3:::aws-codedeploy-ap-south-1/*",
      "arn:aws:s3:::aws-codedeploy-sa-east-1/*"
    ]
  }
}
```
```tf:stage/ec2_instance_profile.tf(新規作成)
resource "aws_iam_role" "ec2_codedeploy" {
  name               = "${var.app_name}-ec2-codedeploy-${terraform.workspace}"
  assume_role_policy = module.iam.ec2_assume_role_policy_json
}

resource "aws_iam_role_policy" "ec2_codedeploy" {
  role   = aws_iam_role.ec2_codedeploy.name
  policy = data.aws_iam_policy_document.s3_codepipeline_application_sources_codedeploy.json
}

resource "aws_iam_role_policy" "ec2_codedeploy_kms" {
  role   = aws_iam_role.ec2_codedeploy.name
  policy = module.iam.kms_policy_json
}

resource "aws_iam_instance_profile" "ec2_codedeploy" {
  name = "${var.app_name}-ec2-codedeploy-${terraform.workspace}"
  role = aws_iam_role.ec2_codedeploy.name
}
```
```diff:stage/admin_ec2.tf(修正)
resource "aws_instance" "admin" {
  ami                    = "ami-011facbea5ec0363b"
  instance_type          = "t2.small"
  availability_zone      = var.availability_zone_a
  key_name               = var.key_name
  monitoring             = "false"
  subnet_id              = module.subnet.private_a_id
  vpc_security_group_ids = [module.security_group.from_jump_id, module.security_group.allow_internet_id]
+ iam_instance_profile   = aws_iam_instance_profile.ec2_codedeploy.name

  tags = {
    Name   = "${var.app_name}-admin-${terraform.workspace}"
    Deploy = "${var.app_name}-admin-${terraform.workspace}"
  }
}
```
```tf:stage/codepipeline_s3.tf(追記)
#-- 省略 --#
data "aws_iam_policy_document" "codepipeline_application_sources_bucket_policy" {
  statement {
    effect = "Allow"
    principals {
      identifiers = [aws_iam_role.ec2_codedeploy.arn]
      type        = "AWS"
    }
    actions = [
      "s3:Get*",
      "s3:List*"
    ]
    resources = [
      aws_s3_bucket.codepipeline_application_sources.arn,
      "${aws_s3_bucket.codepipeline_application_sources.arn}/*"
    ]
  }
}

resource "aws_s3_bucket_policy" "codepipeline-bucket" {
  bucket = aws_s3_bucket.codepipeline_application_sources.id
  policy = data.aws_iam_policy_document.codepipeline_application_sources_bucket_policy.json
}
```

###### 再起動
EC2にIAMインスタンスプロファイルを設定したので、再起動します。

```sh
$ aws ec2 describe-instances \
  --filter "Name=tag:Name,Values=terraform-sample-admin-stage" \
  --query "Reservations[0].Instances[0].InstanceId" \
  --profile terraform-sample
"i-XXXXXXXXXXXXXXXX"
$ aws ec2 reboot-instances \
  --instance-ids i-XXXXXXXXXXXXXXXX \
  --profile terraform-sample
```

##### 動作確認
1. `var.static_contents_target_branch`で指定したブランチに変更をpushします。
1. 以下のコマンドで確認します。

```sh
$ aws deploy list-deployments \
  --application-name terraform-sample-admin-stage \
  --deployment-group-name terraform-sample-admin-stage \
  --query "deployments[0]" \
  --profile terraform-sample
"d-XXXXXXXXX"
$ aws deploy get-deployment \
  --deployment-id d-XXXXXXXXX \
  --query "deploymentInfo.status" \
  --profile terraform-sample
"Succeeded"
```
`Succeeded`となっていれば正常です。
失敗していた場合は、アプリケーションサーバ内の`/var/log/aws/codedeploy-agent/codedeploy-agent.log`か`/var/log/aws/codedeploy-agent/codedeploy-agent.YYYYMMDD.log`を確認しましょう。

デプロイしたSpringBootアプリケーションのステータスを確認します。

```sh:アプリケーションサーバ
[root@ip-YYY-YYY-YYY-YYY ~]# systemctl status springboot.service 
● springboot.service - springboot application
   Loaded: loaded (/etc/systemd/system/springboot.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri YYYY-MM-DD HH:mm:SS UTC; XXs ago
 Main PID: XXXX (sh)
   CGroup: /system.slice/springboot.service
           ├─XXXX /bin/sh -c /var/lib/springboot/boot.jar &>> /var/lib/springboot/logs/stage.log
           ├─XXXX /bin/bash /var/lib/springboot/boot.jar
           └─XXXX /usr/bin/java -Dsun.misc.URLClassPath.disableJarChecking=true -Dspring.profiles.active=stage -jar /var/lib/springboot/boot.jar
```
`Active: active (running)`となっていれば正常です。
`Caused by: java.net.SocketException: Permission denied`が発生している場合は、以下のコマンドを試してください。

```sh:アプリケーションサーバ
[root@ip-YYY-YYY-YYY-YYY ~]# echo 'net.ipv4.ip_unprivileged_port_start=0' >> /etc/sysctl.conf
```

http経由でアクセス確認をします。

```sh:アプリケーションサーバ
[root@ip-YYY-YYY-YYY-YYY ~]# curl -o /dev/null -w '%{http_code}\n' -s http://localhost/login
200
```
`200`となっていれば正常です。

##### アプリケーションサーバ外からのHTTP(HTTPS)リクエストの許可
今回のインフラ構成ではHTTPS認証をCloudFrontおよびApplicationLoadBalancer(ALB)で行うので、ALB-EC2インスタンス間はHTTPでアクセスします。
ALBはまだ作成していないので、暫定で踏み台サーバからのリクエストを許可します。

```tf:modules/security_group/jump.tf(追記)
#-- 中略 --#
resource "aws_security_group_rule" "jump_http_out" {
  security_group_id = aws_security_group.jump.id
  type              = "egress"
  from_port         = 80
  to_port           = 80
  protocol          = "tcp"
  cidr_blocks       = [var.vpc_cidr_block]
}
```
```tf:modules/security_group/application_server.tf(新規作成)
resource "aws_security_group" "application_server" {
  name   = "${var.app_name}-application-server"
  vpc_id = var.vpc_id

  tags = {
    Name = "${var.app_name}-application-server"
  }
}

resource "aws_security_group_rule" "application_server_http" {
  security_group_id        = aws_security_group.application_server.id
  type                     = "ingress"
  from_port                = 80
  to_port                  = 80
  protocol                 = "tcp"
  source_security_group_id = aws_security_group.jump.id
}

output "application_server_id" {
  value = aws_security_group.application_server.id
}
```
```diff:stage/admin_ec2.tf(修正)
resource "aws_instance" "admin" {
  ami                    = "ami-011facbea5ec0363b"
  instance_type          = "t2.small"
  availability_zone      = var.availability_zone_a
  key_name               = var.key_name
  monitoring             = "false"
  subnet_id              = module.subnet.private_a_id
- vpc_security_group_ids = [module.security_group.from_jump_id, module.security_group.allow_internet_id]
+ vpc_security_group_ids = [module.security_group.from_jump_id, module.security_group.allow_internet_id, module.security_group.application_server_id]
  iam_instance_profile   = aws_iam_instance_profile.ec2_codedeploy.name

  tags = {
    Name   = "${var.app_name}-admin-${terraform.workspace}"
    Deploy = "${var.app_name}-admin-${terraform.workspace}"
  }
}
```

###### 動作確認
```sh:踏み台サーバ
$ curl -o /dev/null -w '%{http_code}\n' -s http://YYY.YYY.YYY.YYY/login
200
```
`200`となっていれば正常です。

#### ApplicationLoadBalancer(ALB)
##### セキュリティグループ
```tf:modules/security_group/alb.tf(新規作成)
resource "aws_security_group" "alb" {
  name   = "${var.app_name}-alb"
  vpc_id = var.vpc_id

  tags = {
    Name = "${var.app_name}-alb"
  }
}

resource "aws_security_group_rule" "alb_http_in" {
  security_group_id = aws_security_group.alb.id
  type              = "ingress"
  from_port         = 80
  to_port           = 80
  protocol          = "tcp"
  cidr_blocks       = ["0.0.0.0/0"]
}

resource "aws_security_group_rule" "alb_http_out" {
  security_group_id        = aws_security_group.alb.id
  type                     = "egress"
  from_port                = 80
  to_port                  = 80
  protocol                 = "tcp"
  source_security_group_id = aws_security_group.application_server.id
}

output "security_group_alb_id" {
  value = aws_security_group.alb.id
}
```

##### ログ用S3
[Application Load Balancer のアクセスログ - Elastic Load Balancing](https://docs.aws.amazon.com/ja_jp/elasticloadbalancing/latest/application/load-balancer-access-logs.html#access-logging-bucket-permissions)の通り、リージョンによって設定するAWSアカウントIDが異なります。
今回は東京リージョンのものを使用します。

```diff:modules/s3/logs.tf(追記)
#-- 中略 --#
data "aws_iam_policy_document" "s3_logs" {
  statement {
    effect    = "Allow"
    actions   = ["s3:PutObject"]
    resources = ["${aws_s3_bucket.logs.arn}/*"]

    principals {
      type        = "AWS"
-     identifiers = [var.cloudfront_origin_access_identity_iam_arn]
+     identifiers = [
+       var.cloudfront_origin_access_identity_iam_arn,
+       "arn:aws:iam::582318560864:root"
+     ]
    }
  }
}
```

##### public subnet
```tf:terraform.tfvars(追記)
#-- 中略 --#
public_cidr_block_c  = "10.1.4.0/24"
```
```tf:stage/variables.tf(追記)
#-- 中略 --#
variable "public_cidr_block_c" {}
```
```tf:modules/subnet/public.tf(追記)
variable "availability_zone_c" {}
variable "public_cidr_block_c" {}

resource "aws_subnet" "public_c" {
  vpc_id            = var.vpc_id
  cidr_block        = var.public_cidr_block_c
  availability_zone = var.availability_zone_c

  tags = {
    Name = "${var.app_name}-public-c"
  }
}

output "public_c_id" {
  value = aws_subnet.public_c.id
}
```
```diff:modules/subnet/private.tf(修正)
-variable "availability_zone_c" {}
```
```tf:modules/subnet/route_table.tf(追記)
#-- 中略 --#
resource "aws_route_table_association" "public_c" {
  subnet_id      = aws_subnet.public_c.id
  route_table_id = aws_route_table.public.id
}
```
```diff:stage/load_modules.tf(追記)
#-- 中略 --#
module "subnet" {
  source               = "../modules/subnet"
  app_name             = var.app_name
  vpc_id               = module.vpc.id
  availability_zone_a  = var.availability_zone_a
  availability_zone_c  = var.availability_zone_c
  public_cidr_block_a  = var.public_cidr_block_a
+ public_cidr_block_c  = var.public_cidr_block_c
  private_cidr_block_a = var.private_cidr_block_a
  private_cidr_block_c = var.private_cidr_block_c
}
#-- 中略 --#
```

##### ALB
```tf:stage/admin_alb.tf(新規作成)
resource "aws_lb" "admin" {
  name                       = "${var.app_name}-admin-${terraform.workspace}"
  internal                   = false
  load_balancer_type         = "application"
  security_groups            = [module.security_group.alb_id]
  subnets                    = [module.subnet.public_a_id, module.subnet.public_c_id]
  enable_http2               = true
  enable_deletion_protection = false

  access_logs {
    bucket  = module.s3.logs.bucket
    prefix  = "${terraform.workspace}/alb/admin"
    enabled = true
  }
}

resource "aws_alb_target_group" "admin" {
  name     = "${var.app_name}-admin-${terraform.workspace}"
  port     = 80
  protocol = "HTTP"
  vpc_id   = module.vpc.id

  health_check {
    interval            = 60
    path                = "/login"
    port                = 80
    protocol            = "HTTP"
    timeout             = 5
    unhealthy_threshold = 2
    matcher             = 200
  }
}

resource "aws_alb_listener" "admin" {
  load_balancer_arn = aws_lb.admin.arn
  port              = "80"
  protocol          = "HTTP"

  default_action {
    type             = "forward"
    target_group_arn = aws_alb_target_group.admin.arn
  }
}
```

##### ALBからアプリケーションサーバへHTTPリクエストの許可
```diff:modules/security_group/jump.tf(削除)
#-- 中略 --#
-resource "aws_security_group_rule" "jump_http_out" {
-  security_group_id = aws_security_group.jump.id
-  type              = "egress"
-  from_port         = 80
-  to_port           = 80
-  protocol          = "tcp"
-  cidr_blocks       = [var.vpc_cidr_block]
-}
```
```diff:modules/security_group/application_server.tf(修正)
#-- 中略 --#
resource "aws_security_group_rule" "application_server_http" {
  security_group_id        = aws_security_group.application_server.id
  type                     = "ingress"
  from_port                = 80
  to_port                  = 80
  protocol                 = "tcp"
- source_security_group_id = aws_security_group.jump.id
+ source_security_group_id = aws_security_group.alb.id
}
```

##### ALBのtargetに追加
```tf:stage/admin_ec2.tf(追記)
#-- 省略 --#
resource "aws_alb_target_group_attachment" "admin" {
  target_group_arn = aws_alb_target_group.admin.arn
  target_id        = aws_instance.admin.id
  port             = 80
}
```

##### 動作確認
```sh
$ aws elbv2 describe-load-balancers \
  --names terraform-sample-admin-stage \
  --query "LoadBalancers[0].DNSName" \
  --profile terraform-sample
"terraform-sample-admin-stage-XXXXXXXX.ap-northeast-1.elb.amazonaws.com"
$ curl -o /dev/null -w '%{http_code}\n' -s http://terraform-sample-admin-stage-XXXXXXXX.ap-northeast-1.elb.amazonaws.com/login
200
```
`200`となっていれば正常です。

##### ALBとCloudFrontの関連付け
先ほどはデフォルトでS3を参照するよう設定しましたが、ALBをデフォルトに設定し、静的コンテンツのみS3を参照するように変更します。

```tf:stage/admin_cloudfront.tf(変更後)
resource "aws_cloudfront_distribution" "admin" {
  enabled             = true
  comment             = var.admin_domain
  default_root_object = "index.html"

  origin {
    origin_id   = "s3-${var.admin_domain}"
    domain_name = aws_s3_bucket.static_contents.bucket_domain_name

    s3_origin_config {
      origin_access_identity = module.cloudfront.origin_access_identity.cloudfront_access_identity_path
    }
  }

  origin {
    origin_id   = "alb-${var.admin_domain}-${terraform.workspace}"
    domain_name = aws_lb.admin.dns_name

    custom_origin_config {
      http_port                = 80
      https_port               = 443
      origin_protocol_policy   = "http-only"
      origin_ssl_protocols     = ["TLSv1", "TLSv1.1", "TLSv1.2"]
      origin_keepalive_timeout = 60
      origin_read_timeout      = 60
    }
  }

  ordered_cache_behavior {
    path_pattern           = "/js/*"
    target_origin_id       = "s3-${var.admin_domain}"
    allowed_methods        = ["GET", "HEAD"]
    cached_methods         = ["GET", "HEAD"]
    compress               = true
    viewer_protocol_policy = "redirect-to-https"
    default_ttl            = 3600
    min_ttl                = 0
    max_ttl                = 86400

    forwarded_values {
      query_string = false

      cookies {
        forward = "none"
      }
    }
  }

  ordered_cache_behavior {
    path_pattern           = "/css/*"
    target_origin_id       = "s3-${var.admin_domain}"
    allowed_methods        = ["GET", "HEAD"]
    cached_methods         = ["GET", "HEAD"]
    compress               = true
    viewer_protocol_policy = "redirect-to-https"
    default_ttl            = 3600
    min_ttl                = 0
    max_ttl                = 86400

    forwarded_values {
      query_string = false

      cookies {
        forward = "none"
      }
    }
  }

  ordered_cache_behavior {
    path_pattern           = "/img/*"
    target_origin_id       = "s3-${var.admin_domain}"
    allowed_methods        = ["GET", "HEAD"]
    cached_methods         = ["GET", "HEAD"]
    compress               = true
    viewer_protocol_policy = "redirect-to-https"
    default_ttl            = 3600
    min_ttl                = 0
    max_ttl                = 86400

    forwarded_values {
      query_string = false

      cookies {
        forward = "none"
      }
    }
  }

  ordered_cache_behavior {
    path_pattern           = "*.html"
    target_origin_id       = "s3-${var.admin_domain}"
    allowed_methods        = ["GET", "HEAD"]
    cached_methods         = ["GET", "HEAD"]
    compress               = true
    viewer_protocol_policy = "redirect-to-https"
    default_ttl            = 3600
    min_ttl                = 0
    max_ttl                = 86400

    forwarded_values {
      query_string = false

      cookies {
        forward = "none"
      }
    }
  }

  ordered_cache_behavior {
    path_pattern           = "favicon*"
    target_origin_id       = "s3-${var.admin_domain}"
    allowed_methods        = ["GET", "HEAD"]
    cached_methods         = ["GET", "HEAD"]
    compress               = true
    viewer_protocol_policy = "redirect-to-https"
    default_ttl            = 3600
    min_ttl                = 0
    max_ttl                = 86400

    forwarded_values {
      query_string = false

      cookies {
        forward = "none"
      }
    }
  }

  ordered_cache_behavior {
    path_pattern           = "/"
    target_origin_id       = "s3-${var.admin_domain}"
    allowed_methods        = ["GET", "HEAD"]
    cached_methods         = ["GET", "HEAD"]
    compress               = true
    viewer_protocol_policy = "redirect-to-https"
    default_ttl            = 3600
    min_ttl                = 0
    max_ttl                = 86400

    forwarded_values {
      query_string = false

      cookies {
        forward = "none"
      }
    }
  }

  default_cache_behavior {
    target_origin_id       = "alb-${var.admin_domain}-${terraform.workspace}"
    allowed_methods        = ["HEAD", "DELETE", "POST", "GET", "OPTIONS", "PUT", "PATCH"]
    cached_methods         = ["HEAD", "GET"]
    compress               = false
    viewer_protocol_policy = "redirect-to-https"
    default_ttl            = 0
    min_ttl                = 0
    max_ttl                = 0

    forwarded_values {
      query_string = true
      headers      = ["*"]

      cookies {
        forward = "all"
      }
    }
  }

  restrictions {
    geo_restriction {
      restriction_type = "none"
    }
  }

  viewer_certificate {
    acm_certificate_arn      = var.acm_certificate_arn
    ssl_support_method       = "sni-only"
    minimum_protocol_version = "TLSv1"
  }

  logging_config {
    bucket          = module.s3.logs.bucket_domain_name
    prefix          = "${terraform.workspace}/cloudfront/admin"
    include_cookies = false
  }

  tags = {
    Name = var.admin_domain
  }
}
```

##### 動作確認
```sh
$ aws cloudfront list-distributions \
  --query "DistributionList.Items[0].DomainName" \
  --profile terraform-sample
"XXXXXXXXXXXXXX.cloudfront.net"
$ curl -o /dev/null -w '%{http_code}\n' -s https://XXXXXXXXXXXXXX.cloudfront.net/login
200
```
`200`となっていれば正常です。

#### AutoScalingGroup(ASG)
CodeDeployでEC2にデプロイできるようになったので、ASGでの運用を想定して構築します。

##### AMI取得
```sh
$ aws ec2 describe-instances \
  --filter "Name=tag:Name,Values=terraform-sample-admin-stage" \
  --query "Reservations[0].Instances[0].InstanceId" \
  --profile terraform-sample \
"i-XXXXXXXXXXXXX"
$ aws ec2 create-image \
  --instance-id i-XXXXXXXXXXXXX \
  --reboot \
  --name "任意のAMI名" \
  --query "ImageId" \
  --profile terraform-sample
"ami-XXXXXXXXXXXXXXXXXX"
```

##### 起動テンプレート
`stage/admin_ec2.tf`のネットワーク部分以外を記述するイメージです。

```tf:stage/admin_autoscaling_group.tf(新規作成)
resource "aws_launch_template" "admin" {
  name        = "${var.app_name}-admin-${terraform.workspace}"
  description = "${var.app_name}-admin-${terraform.workspace}"
  image_id    = "ami-096ca23b0da9b4e9d"
  iam_instance_profile {
    arn = aws_iam_instance_profile.ec2_codedeploy.arn
  }
  instance_type           = "t2.small"
  key_name                = var.key_name
  vpc_security_group_ids  = [module.security_group.from_jump_id, module.security_group.allow_internet_id, module.security_group.application_server_id]
  disable_api_termination = false
  ebs_optimized           = false
  monitoring {
    enabled = false
  }
  tag_specifications {
    resource_type = "instance"
    tags = {
      Name   = "${var.app_name}-admin-${terraform.workspace}"
      Deploy = "${var.app_name}-admin-${terraform.workspace}"
    }
  }
}
```

##### ASG
`stage/admin_ec2.tf`のネットワーク部分を記述するイメージです。
これを作成したら`stage_ec2.tf`は削除します。

```tf:stage/admin_autoscaling_group.tf(追記)
#-- 省略 --#
resource "aws_autoscaling_group" "admin" {
  name                      = "${var.app_name}-admin-${terraform.workspace}"
  max_size                  = 10
  min_size                  = 1
  desired_capacity          = 1
  vpc_zone_identifier       = [module.subnet.private_a_id, module.subnet.private_c_id]
  default_cooldown          = 300
  health_check_grace_period = 300
  health_check_type         = "ELB"
  force_delete              = false
  target_group_arns         = [aws_alb_target_group.admin.arn]
  termination_policies      = ["Default"]
  protect_from_scale_in     = false
  launch_template {
    id      = aws_launch_template.admin.id
    version = "$Latest"
  }
}
```

##### スケーリング設定
以下のルールを作成します。

1. 直近３分間のうち2分においてASGグループ内全インスタンスのCPU使用率平均が50-70%の場合、インスタンス数を50%増加する
1. 直近３分間のうち2分においてASGグループ内全インスタンスのCPU使用率平均が70%以上の場合、インスタンス数を100%増加する
1. 直近３分間のうち2分においてASGグループ内全インスタンスのCPU使用率平均が30-40%の場合、インスタンス数を25%減少する
1. 直近３分間のうち2分においてASGグループ内全インスタンスのCPU使用率平均が30%以下の場合、インスタンス数を50%減少する

※ インスタンス数の増減の最小・最大値(10台稼働中で①の場合通常は5台増加するが、最低7台以上増加させたいなど)を設定する方法はわかりませんでした。
どなたかご存知の方がいらっしゃいましたらご教示いただけると幸いです。

```tf:stage/admin_autoscaling_group.tf(追記)
#-- 省略 --#
resource "aws_autoscaling_policy" "admin_scaleout" {
  name                      = "${var.app_name}-admin-${terraform.workspace}-scaleout"
  autoscaling_group_name    = aws_autoscaling_group.admin.name
  adjustment_type           = "PercentChangeInCapacity"
  policy_type               = "StepScaling"
  estimated_instance_warmup = 300
  metric_aggregation_type   = "Average"
  step_adjustment {
    scaling_adjustment          = 50
    metric_interval_lower_bound = 0
    metric_interval_upper_bound = 20
  }
  step_adjustment {
    scaling_adjustment          = 100
    metric_interval_lower_bound = 20
  }
}

resource "aws_cloudwatch_metric_alarm" "admin_scaleout" {
  alarm_name          = "${var.app_name}-admin-${terraform.workspace}-scaleout"
  alarm_description   = "This metric monitors ec2 cpu utilization"
  comparison_operator = "GreaterThanOrEqualToThreshold"
  period              = 60
  evaluation_periods  = 3
  datapoints_to_alarm = 2
  metric_name         = "CPUUtilization"
  namespace           = "AWS/EC2"
  statistic           = "Average"
  threshold           = 50
  actions_enabled     = true
  alarm_actions       = [aws_autoscaling_policy.admin_scaleout.arn]

  dimensions = {
    AutoScalingGroupName = aws_autoscaling_group.admin.name
  }
  tags = {
    "Name" = "${var.app_name}-admin-${terraform.workspace}-scaleout"
  }
}

resource "aws_autoscaling_policy" "admin_scalein" {
  name                      = "${var.app_name}-admin-${terraform.workspace}-scalein"
  autoscaling_group_name    = aws_autoscaling_group.admin.name
  adjustment_type           = "PercentChangeInCapacity"
  policy_type               = "StepScaling"
  estimated_instance_warmup = 300
  metric_aggregation_type   = "Average"
  step_adjustment {
    scaling_adjustment          = -25
    metric_interval_lower_bound = -10
    metric_interval_upper_bound = 0
  }
  step_adjustment {
    scaling_adjustment          = -50
    metric_interval_upper_bound = -10
  }
}

resource "aws_cloudwatch_metric_alarm" "admin_scalein" {
  alarm_name          = "${var.app_name}-admin-${terraform.workspace}-scalein"
  alarm_description   = "This metric monitors ec2 cpu utilization"
  comparison_operator = "LessThanOrEqualToThreshold"
  period              = 60
  evaluation_periods  = 3
  datapoints_to_alarm = 2
  metric_name         = "CPUUtilization"
  namespace           = "AWS/EC2"
  statistic           = "Average"
  threshold           = 40
  actions_enabled     = true
  alarm_actions       = [aws_autoscaling_policy.admin_scalein.arn]

  dimensions = {
    AutoScalingGroupName = aws_autoscaling_group.admin.name
  }
  tags = {
    "Name" = "${var.app_name}-admin-${terraform.workspace}-scalein"
  }
}
```

##### 動作確認
ストレステストを行い、インスタンスが増加するかどうかを確認します。
[stress-ngコマンドの使い方 - Qiita](https://qiita.com/hana_shin/items/0a3a615274717c89c0a4)を参考にします。

```sh:新規作成アプリケーションサーバ
[root@ip-ZZZ-ZZZ-ZZZ-ZZZ ~]# amazon-linux-extras install -y epel
[root@ip-ZZZ-ZZZ-ZZZ-ZZZ ~]# yum install -y stress-ng
[root@ip-ZZZ-ZZZ-ZZZ-ZZZ ~]# stress-ng -V
stress-ng, version 0.07.29
[root@ip-ZZZ-ZZZ-ZZZ-ZZZ ~]# stress-ng -c 1 -l 80 -q &
```
```sh
$ aws autoscaling describe-scaling-activities \
  --auto-scaling-group-name terraform-sample-admin-stage \
  --query "Activities[0].Cause" \
  --profile terraform-sample
"At YYYY-MM-DDTHH:mm:ssZ a monitor alarm terraform-sample-admin-stage-scaleout in state ALARM triggered policy terraform-sample-admin-stage-scaleout changing the desired capacity from 1 to 2.  At YYYY-MM-DDTHH:mm:ssZ an instance was started in response to a difference between desired and actual capacity, increasing the capacity from 1 to 2."
#-- 省略 --#
```
上記のように、スケールアウトの旨があれば成功です。
ストレステストのプロセスは忘れずにkillしましょう。

```sh:新規作成アプリケーションサーバ
[root@ip-ZZZ-ZZZ-ZZZ-ZZZ ~]# jobs
[1]+  Running                 stress-ng -c 1 -l 80 -q &
[root@ip-ZZZ-ZZZ-ZZZ-ZZZ ~]# kill %1
[1]+  Done                    stress-ng -c 1 -l 80 -q““
```

## Users環境作成
ここまでの設定が問題なければ以下のコマンドでUsers側のCloudFrontやALB等を作成します。
適宜設定を変える必要があれば`apply`前に変更しましょう。

```sh
stage$ for file in admin*; do cat "${file}" | sed -e 's/admin/user/g' > `echo "${file}" | sed -e 's/admin/user/g'`; done
```
```tf:stage/application_sources_codepipeline.tf(追記)
#-- 省略 --#
resource "aws_iam_role_policy" "codebuild_user_codepipeline_application_sources" {
  role   = aws_iam_role.codepipeline_application_sources.name
  policy = data.aws_iam_policy_document.codebuild_user.json
}
#-- 省略 --#
resource "aws_codepipeline" "application_sources" {
#-- 省略 --#
  stage {
    name = "${var.app_name}-application-sources-${terraform.workspace}-source"
#-- 省略 --#
  }

  stage {
    name = "${var.app_name}-application-sources-${terraform.workspace}-build-admin"
#-- 省略 --#
  }

  stage {
#-- 省略 --#
  }

  stage {
    name = "${var.app_name}-application-sources-${terraform.workspace}-build-user"

    action {
      name             = "${var.app_name}-application-sources-${terraform.workspace}-build-user-action"
      category         = "Build"
      owner            = "AWS"
      provider         = "CodeBuild"
      input_artifacts  = ["SOURCE"]
      output_artifacts = ["USER_BUILD"]
      version          = "1"

      configuration = {
        ProjectName = aws_codebuild_project.user.name
      }
    }
  }

  stage {
    name = "${var.app_name}-application-sources-${terraform.workspace}-deploy-user"

    action {
      name            = "${var.app_name}-application-sources-${terraform.workspace}-deploy-user-action"
      category        = "Deploy"
      owner           = "AWS"
      provider        = "CodeDeploy"
      input_artifacts = ["USER_BUILD"]
      version         = "1"

      configuration = {
        ApplicationName     = aws_codedeploy_app.user.name,
        DeploymentGroupName = aws_codedeploy_deployment_group.user.deployment_group_name
      }
    }
  }
}
```

## RDS
```tf:modules/security_group/database.tf(新規作成)
resource "aws_security_group" "database" {
  name   = "${var.app_name}-database"
  vpc_id = var.vpc_id

  tags = {
    Name = "${var.app_name}-database"
  }
}

resource "aws_security_group_rule" "database_in" {
  security_group_id        = aws_security_group.database.id
  type                     = "ingress"
  from_port                = 5432
  to_port                  = 5432
  protocol                 = "tcp"
  source_security_group_id = aws_security_group.application_server.id
}

output "database_id" {
  value = aws_security_group.database.id
}
```
```tf:modules/security_group/application_server.tf(追記)
#-- 省略 --#
resource "aws_security_group_rule" "application_server_postgres" {
  security_group_id        = aws_security_group.application_server.id
  type                     = "egress"
  from_port                = 5432
  to_port                  = 5432
  protocol                 = "tcp"
  source_security_group_id = aws_security_group.database.id
}
```
```tf:stage/rds.tf(新規作成)
resource "aws_db_parameter_group" "postgres" {
  name        = "${var.app_name}-${terraform.workspace}"
  description = "${var.app_name}-${terraform.workspace}"
  family      = "postgres10"

  parameter {
    name  = "timezone"
    value = "Asia/Tokyo"
  }
  parameter {
    name  = "client_encoding"
    value = "UTF8"
  }
}

resource "aws_db_subnet_group" "postgres" {
  name        = "${var.app_name}-${terraform.workspace}"
  description = "${var.app_name}-${terraform.workspace}"
  subnet_ids  = [module.subnet.private_a_id, module.subnet.private_c_id]

  tags = {
    Name = "${var.app_name}-${terraform.workspace}"
  }
}

resource "aws_db_instance" "postgres" {
  allocated_storage               = 20
  max_allocated_storage           = 30
  allow_major_version_upgrade     = false
  auto_minor_version_upgrade      = true
  apply_immediately               = true
  db_subnet_group_name            = aws_db_subnet_group.postgres.name
  parameter_group_name            = aws_db_parameter_group.postgres.name
  identifier                      = "${var.app_name}-${terraform.workspace}"
  instance_class                  = "db.t2.small"
  multi_az                        = false
  deletion_protection             = true
  enabled_cloudwatch_logs_exports = ["postgresql", "upgrade"]
  engine                          = "postgres"
  engine_version                  = "10.10"
  skip_final_snapshot             = false
  final_snapshot_identifier       = "${var.app_name}-${terraform.workspace}-final"
  storage_type                    = "gp2"
  port                            = 5432
  username                        = "postgres"
  password                        = "postgres"
  publicly_accessible             = false
  backup_retention_period         = 1
  vpc_security_group_ids          = [module.security_group.database_id]

  tags = {
    Name = "${var.app_name}-${terraform.workspace}"
  }
}
```
このままではパスワードが平文で残ってしまうので、接続確認と同時にパスワードを変更します。

### 接続確認
```sh
$ aws rds describe-db-instances \
  --db-instance-identifier terraform-sample-stage \
  --query "DBInstances[0].Endpoint.Address" \
  --profile terraform-sample
"terraform-sample-stage.XXXXXXXXXXXXXX.ap-northeast-1.rds.amazonaws.com"
```
```sh:アプリケーションサーバ
[root@ip-ZZZ-ZZZ-ZZZ-ZZZ ~]# amazon-linux-extras install -y postgresql10
[root@ip-ZZZ-ZZZ-ZZZ-ZZZ ~]# psql -U postgres -d postgres -h terraform-sample-stage.XXXXXXXXXXXXXX.ap-northeast-1.rds.amazonaws.com
Password for user postgres: 
psql (10.4, server 10.10)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.

postgres=> 
```

### パスワード変更
```sh:アプリケーションサーバ
postgres=> ALTER USER postgres WITH PASSWORD '任意のパスワード';
```

# 本番(Production)環境構築
本来は`stage`ディレクトリを丸ごとコピーして作成するつもりだったのですが、`modules`ディレクトリ以下のリソースを共有できませんでした。
理想としては、`modules`ディレクトリ以下のリソースは環境間で共有し、その他のディレクトリ以下のリソースは個別に管理したいです。
こちらについてもどなたか知見のある方のコメントをお待ちしています。

# 後処理
##　Terraformユーザー権限削除
```sh
$ aws iam detach-user-policy \
  --user-name terraform-sample \
  --policy-arn arn:aws:iam::aws:policy/AdministratorAccess
```

# 最後に
画像編集で力つきました。
後日やります