AWS CloudFormationでEC2インスタンスのログがAmazon CloudWatch logsで確認できるようにする

AWS CloudFormation(CFn)でEC2インスタンスを作成・管理する際にユーザーデータや`AWS::CloudFormation::Init` タイプを利用したメタデータで環境構築するのが便利なのですが、スタック作成・更新時のログはEC2インスタンス内にあるため、確認が面倒です。

EC2のログをAmazon CloudWatch Logsへ送信する仕組みとして「CloudWatch Logs エージェント」があると知ったのでCFnでEC2インスタンスを管理する際に利用できるテンプレートを作成してみました。

CloudWatch Logs エージェントのリファレンス - Amazon CloudWatch Logs
https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/logs/AgentReference.html


## 前提

- AWSアカウントがある
- AWS CLIがインストール済みで利用可能
- CFn、EC2関連の権限がある


## テンプレート

下記で作成したテンプレートに`awslogs` を利用する定義を追加しました。

AWS::CloudFormation::Init タイプを使ってEC2インスタンスの環境構築ができるようにしてみた - Qiita
https://qiita.com/kai_kou/items/e1ef9fa6fb0375dcf15f


EC2インスタンスを起動するVPCやサブネットは既存のリソースを利用する前提となります。
SSHアクセスする際のキーペアは事前に作成し、セキュリティグループはインスタンスとあわせて作成しています。
作成するリージョンは検証なので、`us-east-1` のみとしています。

```yaml:cfn-template.yaml
Parameters:
  VpcId:
    Type: AWS::EC2::VPC::Id
  SubnetId:
    Type: AWS::EC2::Subnet::Id
  EC2KeyPairName:
    Type: AWS::EC2::KeyPair::KeyName
  InstanceType:
    Type: String
    Default: t3.small
  MyInstanceSSHCidrIp:
    Type: String
    Default: '0.0.0.0/0'

Mappings:
  AWSRegionToAMI:
    us-east-1:
      HVM64: ami-0080e4c5bc078760e

Resources:
  MyInstanceSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Join
        - " "
        - - !Ref AWS::StackName
          - "SecurityGroup"
      GroupDescription: "MyInstance SecurityGroup"
      VpcId: !Ref VpcId
      SecurityGroupEgress:
        -
          CidrIp: "0.0.0.0/0"
          IpProtocol: "-1"
      SecurityGroupIngress:
        -
          CidrIp: !Ref MyInstanceSSHCidrIp
          Description: !Join
            - " "
            - - !Ref AWS::StackName
              - "SSH Port"
          IpProtocol: "tcp"
          FromPort: 22
          ToPort: 22
      Tags:
        -
          Key: "Name"
          Value: !Join
            - " "
            - - !Ref AWS::StackName
              - "SecurityGroup"

  MyInstanceIAMRole:
    Type: "AWS::IAM::Role"
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          -
            Effect: "Allow"
            Principal:
              Service:
                - "ec2.amazonaws.com"
            Action:
              - "sts:AssumeRole"
      Path: "/"
      Policies:
        -
          PolicyName: "root"
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              -
                Effect: "Allow"
                Action:
                  - "logs:CreateLogGroup"
                  - "logs:CreateLogStream"
                  - "logs:PutLogEvents"
                  - "logs:DescribeLogStreams"
                Resource: "arn:aws:logs:*:*:*"

  MyInstanceIAMInstanceProfile:
    Type: "AWS::IAM::InstanceProfile"
    Properties:
      Path: "/"
      Roles:
        - Ref: MyInstanceIAMRole
    DependsOn: MyInstanceIAMRole

  MyInstance:
    Type: AWS::EC2::Instance
    Metadata:
      AWS::CloudFormation::Init:
        config:
          packages:
            yum:
              awslogs: []
          files:
            /etc/awslogs/awslogs.conf:
              content: !Sub |
                [general]
                state_file = /var/lib/awslogs/agent-state

                [/var/log/cloud-init-output.log]
                datetime_format = %Y-%m-%d %H:%M:%S.%f
                file = /var/log/cloud-init-output.log
                buffer_duration = 5000
                log_stream_name = {instance_id}/cloud-init-output.log
                initial_position = start_of_file
                log_group_name = /aws/ec2/${AWS::StackName}

                [/var/log/cfn-init.log]
                datetime_format = %Y-%m-%d %H:%M:%S.%f
                file = /var/log/cfn-init.log
                buffer_duration = 5000
                log_stream_name = {instance_id}/cfn-init.log
                initial_position = start_of_file
                log_group_name = /aws/ec2/${AWS::StackName}

                [/var/log/cfn-init-cmd.log]
                datetime_format = %Y-%m-%d %H:%M:%S.%f
                file = /var/log/cfn-init-cmd.log
                buffer_duration = 5000
                log_stream_name = {instance_id}/cfn-init-cmd.log
                initial_position = start_of_file
                log_group_name = /aws/ec2/${AWS::StackName}

                [/var/log/cfn-hup.log]
                datetime_format = %Y-%m-%d %H:%M:%S.%f
                file = /var/log/cfn-hup.log
                buffer_duration = 5000
                log_stream_name = {instance_id}/cfn-hup.log
                initial_position = start_of_file
                log_group_name = /aws/ec2/${AWS::StackName}
              mode: "000644"
              owner: "root"
              group: "root"
            /etc/cfn/cfn-hup.conf:
              content: !Sub |
                [main]
                stack = ${AWS::StackName}
                region = ${AWS::Region}
              mode: "000400"
              owner: "root"
              group: "root"
            /etc/cfn/hooks.d/cfn-auto-reloader.conf:
              content: !Sub |
                [cfn-auto-reloader-hook]
                triggers = post.update
                path = Resources.MyInstance.Metadata.AWS::CloudFormation::Init
                action = /opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource MyInstance --region ${AWS::Region}
                runas = root
              mode: "000400"
              owner: "root"
              group: "root"
          commands:
            test:
              command: "echo $STACK_NAME test"
              env:
                STACK_NAME: !Ref AWS::StackName
          services:
            sysvinit:
              cfn-hup:
                enabled: "true"
                files:
                  - /etc/cfn/cfn-hup.conf
                  - /etc/cfn/hooks.d/cfn-auto-reloader.conf
              awslogs:
                enabled: "true"
                packages:
                  - awslogs
                files:
                  - /etc/awslogs/awslogs.conf
    Properties:
      InstanceType: !Ref InstanceType
      KeyName: !Ref EC2KeyPairName
      ImageId: !FindInMap [ AWSRegionToAMI, !Ref "AWS::Region", HVM64 ]
      IamInstanceProfile: !Ref MyInstanceIAMInstanceProfile
      NetworkInterfaces:
        - AssociatePublicIpAddress: True
          DeviceIndex: 0
          GroupSet:
            - !Ref MyInstanceSecurityGroup
          SubnetId: !Ref SubnetId
      Tags:
        - Key: 'Name'
          Value: !Ref AWS::StackName
      UserData:
        Fn::Base64:
          !Sub |
            #!/bin/bash
            echo "start UserData"
            /opt/aws/bin/cfn-init --stack ${AWS::StackName} --resource MyInstance --region ${AWS::Region}
            /opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackName} --resource MyInstance --region ${AWS::Region}
            echo "finish UserData"
    CreationPolicy:
      ResourceSignal:
        Timeout: PT5M

Outputs:
  ClientInstanceId:
    Value: !Ref MyInstance
  ClientPublicIp:
    Value: !GetAtt MyInstance.PublicIp
```

## ポイント

### IAMロールを作成する

EC2インスタンスからAmazon CloudWatch Logsへログを送るのに権限が必要となるためIAMロールを作成します。
必要となる権限については下記を参考にしました。

クイックスタート: 実行中の EC2 Linux インスタンスに CloudWatch Logs エージェントをインストールして設定する - Amazon CloudWatch Logs
https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/logs/QuickStartEC2Instance.html


```yaml:cfn-template.yaml_一部抜粋
  MyInstanceIAMRole:
    Type: "AWS::IAM::Role"
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          -
            Effect: "Allow"
            Principal:
              Service:
                - "ec2.amazonaws.com"
            Action:
              - "sts:AssumeRole"
      Path: "/"
      Policies:
        -
          PolicyName: "root"
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              -
                Effect: "Allow"
                Action:
                  - "logs:CreateLogGroup"
                  - "logs:CreateLogStream"
                  - "logs:PutLogEvents"
                  - "logs:DescribeLogStreams"
                Resource: "arn:aws:logs:*:*:*"

  MyInstanceIAMInstanceProfile:
    Type: "AWS::IAM::InstanceProfile"
    Properties:
      Path: "/"
      Roles:
        - Ref: MyInstanceIAMRole
    DependsOn: MyInstanceIAMRole

  MyInstance:
    Type: AWS::EC2::Instance
    (略)
    Properties:
      InstanceType: !Ref InstanceType
      KeyName: !Ref EC2KeyPairName
      ImageId: !FindInMap [ AWSRegionToAMI, !Ref "AWS::Region", HVM64 ]
      IamInstanceProfile: !Ref MyInstanceIAMInstanceProfile
      (略)
```


### `awslogs` パッケージを利用する

CloudWatch Logs エージェントは`awslogs` パッケージとして提供されています。`awslogs` パッケージはAmazon LinuxとUbuntuに対応しています。

パッケージのインストール、設定ファイル、サービス起動はメタデータで定義して設定変更できるようにしました。
`/etc/awslogs/awslogs.conf` ファイルでAmazon CloudWatch Logsに出力するログファイルを定義します。`log_stream_name` や`log_group_name` は任意で指定が可能です。
ここでは`/var/log/` 内に出力されるメタデータやユーザーデータによる処理のログを対象としました。

```yaml:cfn-template.yaml_一部抜粋
  MyInstance:
    Type: AWS::EC2::Instance
    Metadata:
      AWS::CloudFormation::Init:
        config:
          packages:
            yum:
              awslogs: []
          files:
            /etc/awslogs/awslogs.conf:
              content: !Sub |
                [general]
                state_file = /var/lib/awslogs/agent-state

                [/var/log/cloud-init-output.log]
                datetime_format = %Y-%m-%d %H:%M:%S.%f
                file = /var/log/cloud-init-output.log
                buffer_duration = 5000
                log_stream_name = {instance_id}/cloud-init-output.log
                initial_position = start_of_file
                log_group_name = /aws/ec2/${AWS::StackName}

                [/var/log/cfn-init.log]
                datetime_format = %Y-%m-%d %H:%M:%S.%f
                file = /var/log/cfn-init.log
                buffer_duration = 5000
                log_stream_name = {instance_id}/cfn-init.log
                initial_position = start_of_file
                log_group_name = /aws/ec2/${AWS::StackName}

                [/var/log/cfn-init-cmd.log]
                datetime_format = %Y-%m-%d %H:%M:%S.%f
                file = /var/log/cfn-init-cmd.log
                buffer_duration = 5000
                log_stream_name = {instance_id}/cfn-init-cmd.log
                initial_position = start_of_file
                log_group_name = /aws/ec2/${AWS::StackName}

                [/var/log/cfn-hup.log]
                datetime_format = %Y-%m-%d %H:%M:%S.%f
                file = /var/log/cfn-hup.log
                buffer_duration = 5000
                log_stream_name = {instance_id}/cfn-hup.log
                initial_position = start_of_file
                log_group_name = /aws/ec2/${AWS::StackName}
              mode: "000644"
              owner: "root"
              group: "root"
            (略)
          services:
            sysvinit:
              (略)
              awslogs:
                enabled: "true"
                packages:
                  - awslogs
                files:
                  - /etc/awslogs/awslogs.conf
```

## 利用方法

### パラメータ値を取得

#### キーペア

EC2インスタンスへSSHログインするのに利用するキーペアを作成します。
既存のキーペアを利用する場合は作成不要です。

```console
# Create KeyPair
> aws ec2 create-key-pair \
  --key-name cfn-awslogs-test-ec2-key \
  --query "KeyMaterial" \
  --output text > cfn-awslogs-test-ec2-key.pem

> chmod 400 cfn-awslogs-test-ec2-key.pem
```

#### VPC、サブネット

既存のVPC、サブネット配下にEC2インスタンスを作成する前提ですので各IDを取得します。

```console
# VpcId
> aws ec2 describe-vpcs \
  --query "Vpcs"

[
    {
        "CidrBlock": "172.31.0.0/16",
        "DhcpOptionsId": "dopt-b06bd8c8",
        "State": "available",
        "VpcId": "vpc-xxxxxxxx",
        "OwnerId": "xxxxxxxxxxxx",
        "InstanceTenancy": "default",
        "CidrBlockAssociationSet": [
            {
                "AssociationId": "vpc-cidr-assoc-2b23e646",
                "CidrBlock": "172.31.0.0/16",
                "CidrBlockState": {
                    "State": "associated"
                }
            }
        ],
        "IsDefault": true
    },
]


# SubnetId
> aws ec2 describe-subnets \
  --filters '{"Name": "vpc-id", "Values": ["vpc-xxxxxxxx"]}' \
  --query "Subnets"

[
    {
        "AvailabilityZone": "us-east-1a",
        "AvailabilityZoneId": "use1-az2",
        "AvailableIpAddressCount": 4089,
        "CidrBlock": "172.31.80.0/20",
        "DefaultForAz": true,
        "MapPublicIpOnLaunch": true,
        "State": "available",
        "SubnetId": "subnet-xxxxxxxx",
        "VpcId": "vpc-xxxxxxxx",
        "OwnerId": "xxxxxxxxxxxx",
        "AssignIpv6AddressOnCreation": false,
        "Ipv6CidrBlockAssociationSet": [],
        "SubnetArn": "arn:aws:ec2:us-east-1:xxxxxxxxxxxx:subnet/subnet-xxxxxxxx"
    },
]
```

#### 自身のグローバルIP

SSHアクセスするためのセキュリティグループに指定するIPアドレスを取得します。
ここでは`ifconfig.io` を利用していますが他の手段でもOKです。

```console
## GlobalIP
> curl ifconfig.io

xxx.xxx.xxx.xxx
```


### スタック作成

`aws cloudformation create-stack` コマンドを利用してスタック作成します。
AWSマネジメントコンソールから作成してもOKです。

```console
> aws cloudformation create-stack \
  --stack-name cfn-awslogs-test \
  --template-body file://cfn-template.yaml \
  --capabilities CAPABILITY_IAM \
  --region us-east-1 \
  --parameters '[
    {
      "ParameterKey": "VpcId",
      "ParameterValue": "vpc-xxxxxxxx"
    },
    {
      "ParameterKey": "SubnetId",
      "ParameterValue": "subnet-xxxxxxxx"
    },
    {
      "ParameterKey": "EC2KeyPairName",
      "ParameterValue": "cfn-awslogs-test-ec2-key"
    },
    {
      "ParameterKey": "MyInstanceSSHCidrIp",
      "ParameterValue": "xxx.xxx.xxx.xxx/32"
    }
  ]'
```


### スタック作成結果を確認

スタック作成ができたか確認します。

```console
 > aws cloudformation describe-stacks \
  --stack-name cfn-awslogs-test

{
    "Stacks": [
        {
            "StackId": "arn:aws:cloudformation:us-east-1:xxxxxxxxxxxx:stack/cfn-awslogs-test/f1e864b0-a9fd-11e9-aa93-12ccfe651680",
            "StackName": "cfn-awslogs-test",
            "Parameters": [
                {
                    "ParameterKey": "MyInstanceSSHCidrIp",
                    "ParameterValue": "xxx.xxx.xxx.xxx/32"
                },
                {
                    "ParameterKey": "VpcId",
                    "ParameterValue": "vpc-xxxxxxxx"
                },
                {
                    "ParameterKey": "EC2KeyPairName",
                    "ParameterValue": "cfn-init-test-ec2-key"
                },
                {
                    "ParameterKey": "SubnetId",
                    "ParameterValue": "subnet-xxxxxxxx"
                },
                {
                    "ParameterKey": "InstanceType",
                    "ParameterValue": "t3.small"
                }
            ],
            "CreationTime": "2019-07-19T08:19:36.199Z",
            "RollbackConfiguration": {},
            "StackStatus": "CREATE_COMPLETE",
            "DisableRollback": false,
            "NotificationARNs": [],
            "Capabilities": [
                "CAPABILITY_IAM"
            ],
            "Outputs": [
                {
                    "OutputKey": "ClientPublicIp",
                    "OutputValue": "xxx.xxx.xxx.xxx"
                },
                {
                    "OutputKey": "ClientInstanceId",
                    "OutputValue": "i-xxxxxxxxxxxxxxxxx"
                }
            ],
            "Tags": [],
            "EnableTerminationProtection": false,
            "DriftInformation": {
                "StackDriftStatus": "NOT_CHECKED"
            }
        }
    ]
}
```

### Amazon CloudWatch logsでログを確認する

```console
> aws logs describe-log-streams \
  --log-group-name /aws/ec2/cfn-awslogs-test \
  --query "logGroups[*].logGroupName"

[
    "i-xxxxxxxxxxxxxxxxx/cfn-hup.log",
    "i-xxxxxxxxxxxxxxxxx/cfn-init-cmd.log",
    "i-xxxxxxxxxxxxxxxxx/cfn-init.log",
    "i-xxxxxxxxxxxxxxxxx/cloud-init-output.log"
]


> aws logs get-log-events \
  --log-group-name /aws/ec2/cfn-awslogs-test \
  --log-stream-name i-xxxxxxxxxxxxxxxxx/cfn-hup.log \
  --query "events[*].message"

[
    "2019-07-19 08:22:57,542 [DEBUG] CloudFormation client initialized with endpoint https://cloudformation.us-east-1.amazonaws.com",
    "2019-07-19 08:22:57,543 [DEBUG] Creating /var/lib/cfn-hup/data",
    "2019-07-19 08:22:57,551 [INFO] No umask value specified in config file. Using the default one: 022",
    "2019-07-19 08:37:57,714 [INFO] cfn-hup processing is alive."
]



> aws logs get-log-events \
  --log-group-name /aws/ec2/cfn-awslogs-test \
  --log-stream-name i-xxxxxxxxxxxxxxxxx/cfn-init-cmd.log \
  --query "events[*].message" \

[
    "2019-07-19 08:22:49,803 P2190 [INFO] ************************************************************",
    "2019-07-19 08:22:49,803 P2190 [INFO] ConfigSet default",
    "2019-07-19 08:22:49,804 P2190 [INFO] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++",
    "2019-07-19 08:22:49,804 P2190 [INFO] Config config",
    "2019-07-19 08:22:55,092 P2190 [INFO] ============================================================",
    "2019-07-19 08:22:55,092 P2190 [INFO] yum install awslogs",
    "2019-07-19 08:22:56,958 P2190 [INFO] -----------------------Command Output-----------------------",
    "2019-07-19 08:22:56,958 P2190 [INFO] \tLoaded plugins: priorities, update-motd, upgrade-helper",
    "2019-07-19 08:22:56,958 P2190 [INFO] \tResolving Dependencies",
    "2019-07-19 08:22:56,958 P2190 [INFO] \t--> Running transaction check",
    "2019-07-19 08:22:56,958 P2190 [INFO] \t---> Package awslogs.noarch 0:1.1.4-1.12.amzn1 will be installed",
    "2019-07-19 08:22:56,958 P2190 [INFO] \t--> Processing Dependency: aws-cli-plugin-cloudwatch-logs(python27) for package: awslogs-1.1.4-1.12.amzn1.noarch",
    "2019-07-19 08:22:56,959 P2190 [INFO] \t--> Running transaction check",
    "2019-07-19 08:22:56,959 P2190 [INFO] \t---> Package aws-cli-plugin-cloudwatch-logs.noarch 0:1.4.4-1.16.amzn1 will be installed",
    "2019-07-19 08:22:56,959 P2190 [INFO] \t--> Finished Dependency Resolution",
    "2019-07-19 08:22:56,959 P2190 [INFO] \t",
    "2019-07-19 08:22:56,959 P2190 [INFO] \tDependencies Resolved",
    "2019-07-19 08:22:56,959 P2190 [INFO] \t",
    "2019-07-19 08:22:56,959 P2190 [INFO] \t================================================================================",
    "2019-07-19 08:22:56,959 P2190 [INFO] \t Package                          Arch     Version            Repository   Size",
    "2019-07-19 08:22:56,959 P2190 [INFO] \t================================================================================",
    "2019-07-19 08:22:56,959 P2190 [INFO] \tInstalling:",
    "2019-07-19 08:22:56,959 P2190 [INFO] \t awslogs                          noarch   1.1.4-1.12.amzn1   amzn-main   9.2 k",
    "2019-07-19 08:22:56,959 P2190 [INFO] \tInstalling for dependencies:",
    "2019-07-19 08:22:56,960 P2190 [INFO] \t aws-cli-plugin-cloudwatch-logs   noarch   1.4.4-1.16.amzn1   amzn-main    71 k",
    "2019-07-19 08:22:56,960 P2190 [INFO] \t",
    "2019-07-19 08:22:56,960 P2190 [INFO] \tTransaction Summary",
    "2019-07-19 08:22:56,960 P2190 [INFO] \t================================================================================",
    "2019-07-19 08:22:56,960 P2190 [INFO] \tInstall  1 Package (+1 Dependent package)",
    "2019-07-19 08:22:56,960 P2190 [INFO] \t",
    "2019-07-19 08:22:56,960 P2190 [INFO] \tTotal download size: 81 k",
    "2019-07-19 08:22:56,960 P2190 [INFO] \tInstalled size: 246 k",
    "2019-07-19 08:22:56,960 P2190 [INFO] \tDownloading packages:",
    "2019-07-19 08:22:56,960 P2190 [INFO] \t--------------------------------------------------------------------------------",
    "2019-07-19 08:22:56,960 P2190 [INFO] \tTotal                                              250 kB/s |  81 kB  00:00     ",
    "2019-07-19 08:22:56,960 P2190 [INFO] \tRunning transaction check",
    "2019-07-19 08:22:56,961 P2190 [INFO] \tRunning transaction test",
    "2019-07-19 08:22:56,961 P2190 [INFO] \tTransaction test succeeded",
    "2019-07-19 08:22:56,961 P2190 [INFO] \tRunning transaction",
    "2019-07-19 08:22:56,961 P2190 [INFO] \t  Installing : aws-cli-plugin-cloudwatch-logs-1.4.4-1.16.amzn1.noarch       1/2 ",
    "2019-07-19 08:22:56,961 P2190 [INFO] \t  Installing : awslogs-1.1.4-1.12.amzn1.noarch                              2/2 ",
    "2019-07-19 08:22:56,961 P2190 [INFO] \t  Verifying  : awslogs-1.1.4-1.12.amzn1.noarch                              1/2 ",
    "2019-07-19 08:22:56,961 P2190 [INFO] \t  Verifying  : aws-cli-plugin-cloudwatch-logs-1.4.4-1.16.amzn1.noarch       2/2 ",
    "2019-07-19 08:22:56,961 P2190 [INFO] \t",
    "2019-07-19 08:22:56,962 P2190 [INFO] \tInstalled:",
    "2019-07-19 08:22:56,962 P2190 [INFO] \t  awslogs.noarch 0:1.1.4-1.12.amzn1                                             ",
    "2019-07-19 08:22:56,962 P2190 [INFO] \t",
    "2019-07-19 08:22:56,962 P2190 [INFO] \tDependency Installed:",
    "2019-07-19 08:22:56,962 P2190 [INFO] \t  aws-cli-plugin-cloudwatch-logs.noarch 0:1.4.4-1.16.amzn1                      ",
    "2019-07-19 08:22:56,962 P2190 [INFO] \t",
    "2019-07-19 08:22:56,962 P2190 [INFO] \tComplete!",
    "2019-07-19 08:22:56,962 P2190 [INFO] ------------------------------------------------------------",
    "2019-07-19 08:22:56,962 P2190 [INFO] Completed successfully.",
    "2019-07-19 08:22:56,965 P2190 [INFO] ============================================================",
    "2019-07-19 08:22:56,965 P2190 [INFO] Command test",
    "2019-07-19 08:22:56,969 P2190 [INFO] -----------------------Command Output-----------------------",
    "2019-07-19 08:22:56,969 P2190 [INFO] \tcfn-awslogs-test test",
    "2019-07-19 08:22:56,969 P2190 [INFO] ------------------------------------------------------------",
    "2019-07-19 08:22:56,970 P2190 [INFO] Completed successfully."
]



> aws logs get-log-events \
  --log-group-name /aws/ec2/cfn-awslogs-test \
  --log-stream-name i-xxxxxxxxxxxxxxxxx/cfn-init.log \
  --query "events[*].message"

[
    "2019-07-19 08:22:49,801 [INFO] -----------------------Starting build-----------------------",
    "2019-07-19 08:22:49,802 [INFO] Running configSets: default",
    "2019-07-19 08:22:49,803 [INFO] Running configSet default",
    "2019-07-19 08:22:49,804 [INFO] Running config config",
    "2019-07-19 08:22:56,962 [INFO] Yum installed [u'awslogs']",
    "2019-07-19 08:22:56,970 [INFO] Command test succeeded",
    "2019-07-19 08:22:56,975 [INFO] enabled service cfn-hup",
    "2019-07-19 08:22:57,568 [INFO] Restarted cfn-hup successfully",
    "2019-07-19 08:22:57,607 [INFO] enabled service awslogs",
    "2019-07-19 08:22:59,648 [INFO] Restarted awslogs successfully",
    "2019-07-19 08:22:59,649 [INFO] ConfigSets completed",
    "2019-07-19 08:22:59,650 [INFO] -----------------------Build complete-----------------------",
    "2019-07-19 08:23:00,249 [DEBUG] CloudFormation client initialized with endpoint https://cloudformation.us-east-1.amazonaws.com",
    "2019-07-19 08:23:00,250 [DEBUG] Signaling resource MyInstance in stack cfn-awslogs-test with unique ID i-xxxxxxxxxxxxxxxxx and status SUCCESS"
]



> aws logs get-log-events \
  --log-group-name /aws/ec2/cfn-awslogs-test \
  --log-stream-name i-xxxxxxxxxxxxxxxxx/cloud-init-output.log \
  --query "events[*].message"

(略)
    "Updated:\n  java-1.7.0-openjdk.x86_64 1:1.7.0.211-2.6.17.1.79.amzn1                       \n  kernel-tools.x86_64 0:4.14.128-87.105.amzn1                                   \n  perl.x86_64 4:5.16.3-294.43.amzn1                                             \n  perl-Pod-Escapes.noarch 1:1.04-294.43.amzn1                                   \n  perl-libs.x86_64 4:5.16.3-294.43.amzn1                                        \n  perl-macros.x86_64 4:5.16.3-294.43.amzn1                                      \n  python27-jinja2.noarch 0:2.7.2-3.16.amzn1                                     \n  wget.x86_64 0:1.18-5.30.amzn1                                                 \n",
    "Complete!",
    "Cloud-init v. 0.7.6 running 'modules:final' at Fri, 19 Jul 2019 08:22:49 +0000. Up 20.37 seconds.",
    "start UserData",
    "finish UserData",
    "Cloud-init v. 0.7.6 finished at Fri, 19 Jul 2019 08:23:00 +0000. Datasource DataSourceEc2.  Up 31.51 seconds"
]

```



## まとめ

CloudWatch Logs エージェントを利用することでCFnでEC2インスタンス作成・更新時のログをAmazon CloudWatch Logsへ出力することができました。
個人的にはこれでCFnで実行エラーによるロールバックでEC2インスタンスが削除されてログが確認できない悲しみがなくなって満足です^^


## 参考

CloudWatch Logs エージェントのリファレンス - Amazon CloudWatch Logs
https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/logs/AgentReference.html

AWS::CloudFormation::Init タイプを使ってEC2インスタンスの環境構築ができるようにしてみた - Qiita
https://qiita.com/kai_kou/items/e1ef9fa6fb0375dcf15f

クイックスタート: 実行中の EC2 Linux インスタンスに CloudWatch Logs エージェントをインストールして設定する - Amazon CloudWatch Logs
https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/logs/QuickStartEC2Instance.html