【2023年11月版】AmazonLinux 2023 で CloudWatch Agentを使用して EC2インスタンスからログを収集【メトリクス、ログ、トレース】


# はじめに

ec2 インスタンスと、アプリのログを cloudwatch でみたいな。。。
そうだ、cloudwatch agent でログ出力しよう！

（旧版のエージェントによる収集の記事が豊富だけど、AmazonLinux 2023 での構築方法）

# 環境

- AmazonLinux 2023
- 対象インスタンスには、cloudwatch が書き込めるアクセス許可を持った `IAMロール` を割り当てておく
- 読み込むアプリのログは、`/var/log/test/test.log`
- 書き込むロググループは、`/dev/test/test.log`

[CloudWatch エージェントで使用する IAM ロールとユーザーを作成する](https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/monitoring/create-iam-roles-for-cloudwatch-agent-commandline.html)


```terminal
$ uname -a
Linux ip-***-***-***-***.ap-northeast-1.compute.internal 6.1.56-82.125.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Oct 10 17:03:53 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
```

```terminal
$ cat /etc/os-release 
NAME="Amazon Linux"
VERSION="2023"
ID="amzn"
ID_LIKE="fedora"
VERSION_ID="2023"
PLATFORM_ID="platform:al2023"
PRETTY_NAME="Amazon Linux 2023"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2023"
HOME_URL="https://aws.amazon.com/linux/"
BUG_REPORT_URL="https://github.com/amazonlinux/amazon-linux-2023"
SUPPORT_END="2028-03-15"
```

# 手順

## aws cli をインストール

```terminal
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
```

## cloudwatch agent をインストールする + collectd もインストール

[CloudWatch エージェントのインストール](https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Agent-on-EC2-Instance.html)

```terminal
sudo yum install amazon-cloudwatch-agent
sudo yum install collectd
```

:::note
カスタムメトリクス等で `collectd` を利用したログを取得しないのであれば collectd のインストールは不要。ウィザードで設定を作成して起動時にエラーが出るなら、開発環境なら、とりあえず入れておくのも有り。

[collectd を使用してカスタムメトリクスを取得する](https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-custom-metrics-collectd.html)
:::
## ウィザードで設定を作成する

[ウィザードを使用して CloudWatch エージェント設定ファイルを作成する](https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/monitoring/create-cloudwatch-agent-configuration-file-wizard.html)

```terminal
$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard
```

設定例、以下の質問以外はデフォルト

* Do you want to monitor any log files?
* Do you want to store the config in the SSM parameter store?

```terminal
================================================================
= Welcome to the Amazon CloudWatch Agent Configuration Manager =
=                                                              =
= CloudWatch Agent allows you to collect metrics and logs from =
= your host and send them to CloudWatch. Additional CloudWatch =
= charges may apply.                                           =
================================================================
On which OS are you planning to use the agent?
1. linux
2. windows
3. darwin
default choice: [1]:

Trying to fetch the default region based on ec2 metadata...
2023/11/20 20:29:15 I! imds retry client will retry 1 times
Are you using EC2 or On-Premises hosts?
1. EC2
2. On-Premises
default choice: [1]:

Which user are you planning to run the agent?
1. root
2. cwagent
3. others
default choice: [1]:

Do you want to turn on StatsD daemon?
1. yes
2. no
default choice: [1]:

Which port do you want StatsD daemon to listen to?
default choice: [8125]

What is the collect interval for StatsD daemon?
1. 10s
2. 30s
3. 60s
default choice: [1]:

What is the aggregation interval for metrics collected by StatsD daemon?
1. Do not aggregate
2. 10s
3. 30s
4. 60s
default choice: [4]:

Do you want to monitor metrics from CollectD? WARNING: CollectD must be installed or the Agent will fail to start
1. yes
2. no
default choice: [1]:

Do you want to monitor any host metrics? e.g. CPU, memory, etc.
1. yes
2. no
default choice: [1]:

Do you want to monitor cpu metrics per core?
1. yes
2. no
default choice: [1]:

Do you want to add ec2 dimensions (ImageId, InstanceId, InstanceType, AutoScalingGroupName) into all of your metrics if the info is available?
1. yes
2. no
default choice: [1]:

Do you want to aggregate ec2 dimensions (InstanceId)?
1. yes
2. no
default choice: [1]:

Would you like to collect your metrics at high resolution (sub-minute resolution)? This enables sub-minute resolution for all metrics, but you can customize for specific metrics in the output json file.
1. 1s
2. 10s
3. 30s
4. 60s
default choice: [4]:

Which default metrics config do you want?
1. Basic
2. Standard
3. Advanced
4. None
default choice: [1]:

Current config as follows:
{
	"agent": {
		"metrics_collection_interval": 60,
		"run_as_user": "root"
	},
	"metrics": {
		"aggregation_dimensions": [
			[
				"InstanceId"
			]
		],
		"append_dimensions": {
			"AutoScalingGroupName": "${aws:AutoScalingGroupName}",
			"ImageId": "${aws:ImageId}",
			"InstanceId": "${aws:InstanceId}",
			"InstanceType": "${aws:InstanceType}"
		},
		"metrics_collected": {
			"collectd": {
				"metrics_aggregation_interval": 60
			},
			"disk": {
				"measurement": [
					"used_percent"
				],
				"metrics_collection_interval": 60,
				"resources": [
					"*"
				]
			},
			"mem": {
				"measurement": [
					"mem_used_percent"
				],
				"metrics_collection_interval": 60
			},
			"statsd": {
				"metrics_aggregation_interval": 60,
				"metrics_collection_interval": 10,
				"service_address": ":8125"
			}
		}
	}
}
Are you satisfied with the above config? Note: it can be manually customized after the wizard completes to add additional items.
1. yes
2. no
default choice: [1]:

Do you have any existing CloudWatch Log Agent (http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html) configuration file to import for migration?
1. yes
2. no
default choice: [2]:

Do you want to monitor any log files?
1. yes
2. no
default choice: [1]:

Log file path:
/var/log/test/test.log
Log group name:
default choice: [test.log]
/dev/test/test.log
Log stream name:
default choice: [{instance_id}]

Log Group Retention in days
1. -1
2. 1
3. 3
4. 5
5. 7
6. 14
7. 30
8. 60
9. 90
10. 120
11. 150
12. 180
13. 365
14. 400
15. 545
16. 731
17. 1096
18. 1827
19. 2192
20. 2557
21. 2922
22. 3288
23. 3653
default choice: [1]:

Do you want to specify any additional log files to monitor?
1. yes
2. no
default choice: [1]:
2
Do you want the CloudWatch agent to also retrieve X-ray traces?
1. yes
2. no
default choice: [1]:

Do you have an existing X-Ray Daemon configuration file to import for migration?
1. yes
2. no
default choice: [1]:

What is the file path for the existing X-Ray Daemon configuration file?

There was an error reading X-Ray Daemon config file. Using default traces configurations
Current Traces Configurations:
{
	"traces": {
		"traces_collected": {
			"xray": {
				"bind_address": "127.0.0.1:2000",
				"tcp_proxy": {
					"bind_address": "127.0.0.1:2000"
				}
			}
		},
		"concurrency": 8,
		"buffer_size_mb": 3,
		"insecure": false
	}
}
Enter a number of the field you would like to update (or 0 to exit)
0: Keep this configuration and exit
1: UDP BindAddress
2: TCP BindAddress
3: concurrency
4: buffer_size_mb
5: resource_arn
6: local_mode
7: insecure
8: role_arn
9: endpoint_override
10: region_override
11: proxy_override


Existing config JSON identified and copied to:  /opt/aws/amazon-cloudwatch-agent/etc/backup-configs
Saved config file to /opt/aws/amazon-cloudwatch-agent/bin/config.json successfully.
Current config as follows:
{
	"agent": {
		"metrics_collection_interval": 60,
		"run_as_user": "root"
	},
	"logs": {
		"logs_collected": {
			"files": {
				"collect_list": [
					{
						"file_path": "/var/log/test/test.log",
						"log_group_name": "/dev/test/test.log",
						"log_stream_name": "{instance_id}",
						"retention_in_days": -1
					}
				]
			}
		}
	},
	"metrics": {
		"aggregation_dimensions": [
			[
				"InstanceId"
			]
		],
		"append_dimensions": {
			"AutoScalingGroupName": "${aws:AutoScalingGroupName}",
			"ImageId": "${aws:ImageId}",
			"InstanceId": "${aws:InstanceId}",
			"InstanceType": "${aws:InstanceType}"
		},
		"metrics_collected": {
			"collectd": {
				"metrics_aggregation_interval": 60
			},
			"disk": {
				"measurement": [
					"used_percent"
				],
				"metrics_collection_interval": 60,
				"resources": [
					"*"
				]
			},
			"mem": {
				"measurement": [
					"mem_used_percent"
				],
				"metrics_collection_interval": 60
			},
			"statsd": {
				"metrics_aggregation_interval": 60,
				"metrics_collection_interval": 10,
				"service_address": ":8125"
			}
		}
	},
	"traces": {
		"buffer_size_mb": 3,
		"concurrency": 8,
		"insecure": false,
		"traces_collected": {
			"xray": {
				"bind_address": "127.0.0.1:2000",
				"tcp_proxy": {
					"bind_address": "127.0.0.1:2000"
				}
			}
		}
	}
}
Please check the above content of the config.
The config file is also located at /opt/aws/amazon-cloudwatch-agent/bin/config.json.
Edit it manually if needed.
Do you want to store the config in the SSM parameter store?
1. yes
2. no
default choice: [1]:
2
Program exits now.
```

設定は、`/opt/aws/amazon-cloudwatch-agent/bin/config.json` に出力されている。

## 設定を読み込ませる

```terminal
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json
```

設定読み込み時にエラーが出たら、collectd が入っているか確認する

```
Configuration validation second phase failed
======== Error Log ========
2023-11-20T11:05:05Z E! [telegraf] Error running agent: Error loading config file /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml: error parsing socket_listener, open /usr/share/collectd/types.db: no such file or directory
```

## 手動で設定を作成する

CloudWatch Agent の設定は `agent`、 `metrics`、 `logs` の3つのセクションで、jsonを作成する。
以下は、Linux システム上のメモリ使用量とディスク使用量のメトリクスを報告する基本的なエージェント設定ファイルの例。

```
{
  "metrics": {
    "metrics_collected": {
      "mem": {
        "measurement": [
          "mem_used_percent"
        ]
      },
      "disk": {
        "measurement": [
          "used_percent"
        ],
        "resources": [
          "*"
        ]
      }
    },
    "append_dimensions": {
      "InstanceId": "${aws:InstanceId}"
    }
  }
}
```

[EC2 インスタンスから CloudWatch にメモリとディスクのメトリクスを送信するにはどうすればよいですか?](https://repost.aws/ja/knowledge-center/cloudwatch-memory-metrics-ec2)

## サービスを開始する

```
sudo systemctl start amazon-cloudwatch-agent.service
```

## cloudwatch で該当の loggroup にログが出力されているか確認する

```terminal
date >> /var/log/test/test.log
```

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/143127/3bc3ddf2-257b-d979-b8bf-fd66fbed38b4.png)

# その他

## 起動

```terminal
sudo systemctl start amazon-cloudwatch-agent.service
```

## 状態確認

```terminal
sudo systemctl status amazon-cloudwatch-agent.service
```

```terminal
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status
```

## 停止

```terminal
sudo systemctl stop amazon-cloudwatch-agent.service
```

## cloudwatach agent のログを確認

```terminal
sudo journalctl -u amazon-cloudwatch-agent.service 
```

# 参考リンク

- [CloudWatch エージェントを使用して Amazon EC2 インスタンスとオンプレミスサーバーからメトリクス、ログ、トレースを収集する](https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html)
- [統合された CloudWatch エージェントを使用して CloudWatch Logs を使用する](https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/logs/UseCloudWatchUnifiedAgent.html)
- [CloudWatch エージェントのインストール](https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Agent-on-EC2-Instance.html)
- [コマンドラインを使用して CloudWatch エージェントをダウンロードおよび設定する](https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/monitoring/download-cloudwatch-agent-commandline.html)
- [CloudWatch エージェントで使用する IAM ロールとユーザーを作成する](https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/monitoring/create-iam-roles-for-cloudwatch-agent-commandline.html)
- [EC2 インスタンスから CloudWatch にメモリとディスクのメトリクスを送信するにはどうすればよいですか?](https://repost.aws/ja/knowledge-center/cloudwatch-memory-metrics-ec2)
- [CloudWatch エージェントにより収集されるメトリクス](https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/monitoring/metrics-collected-by-CloudWatch-agent.html)

# おわりに

かんたんでしたね