[JAWS-UG CLI] Amazon Kinesis Firehose re:入門 (1) 事前準備（Source（Kinesis Agent）およびDestination（S3）の作成）

# この記事について

[JAWS-UG CLI専門支部 #90 Kinesis Firehose 復習編](https://jawsug-cli.doorkeeper.jp/events/59236)で実施するハンズオン用の手順書です。


# 前提条件

## 必要な権限

作業にあたっては、以下の権限を有したIAMユーザもしくはIAMロールを利用してください。

- 以下のサービスに対するフルコントロール権限
    - Kinesis Firehose
    - IAM
    - EC2
    - S3
    - CloudWatch Logs
    - STS
    - (Lambda)
        - データの変換を行う場合
    - (KMS)
        - データの暗号化を行う場合


# 0. 準備

## 0.1. リージョンを指定

オレゴンリージョンで実施します。（東京マダー？）

```bash:コマンド
export AWS_DEFAULT_REGION="us-west-2"
```

## 0.2. 資格情報を確認


```bash:コマンド
aws configure list
```

インスタンスプロファイルを設定したEC2インスタンスでアクセスキーを設定せずに実行した場合、以下のようになります。

```text:結果
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                <not set>             None    None
access_key     ****************QSAA         iam-role
secret_key     ****************c1xY         iam-role
    region                us-west-2              env    AWS_DEFAULT_REGION
```

## 0.3. バージョン確認

```bash:コマンド
aws --version
```

```text:結果
aws-cli/1.11.129 Python/2.7.12 Linux/4.9.38-16.33.amzn1.x86_64 botocore/1.5.92
```

## 0.4. バージョンアップ（必要に応じて）

```bash:コマンド
sudo pip install -U awscli
```



# 1. 管理対象の構築

CloudFormationを利用して、Source（Kinesis AgentをインストールしたEC2インスタンス）とDestination（S3バケット）を作成します。

## 1.1. KeyPairの作成

EC2インスタンス用にKeyPairを作成します。

### KeyPairの名前を指定

```bash:コマンド
AWS_ID=$(aws sts get-caller-identity \
    --query "Account" \
    --output text) \
    && echo ${AWS_ID}
```

```bash:コマンド
KEY_PAIR_NAME="${AWS_ID}_firehose_jawsug_cli"
KEY_MATERIAL_FILE_NAME=${KEY_PAIR_NAME}.pem
```

### 同名KeyPairの不存在を確認

```bash:コマンド
aws ec2 describe-key-pairs \
    --query "KeyPairs[?KeyName==\`${KEY_PAIR_NAME}\`]"
```

```json:結果
[]
```

### KeyPairの作成

```bash:コマンド
aws ec2 create-key-pair \
    --key-name ${KEY_PAIR_NAME} \
    --query "KeyMaterial" \
    --output text \
    > ~/.ssh/${KEY_MATERIAL_FILE_NAME} \
    && cat ~/.ssh/${KEY_MATERIAL_FILE_NAME}
```

### KeyPairの存在を確認

```bash:コマンド
aws ec2 describe-key-pairs \
    --query "KeyPairs[?KeyName==\`${KEY_PAIR_NAME}\`]"
```

```json:結果
[
    {
        "KeyName": "XXXXXXXXXXXX_firehose_jawsug_cli",
        "KeyFingerprint": "xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx"
    }
]
```

### 秘密鍵のPermissionを変更

```bash:コマンド
chmod 600 ~/.ssh/${KEY_MATERIAL_FILE_NAME}
ls -al ~/.ssh/${KEY_MATERIAL_FILE_NAME}
```

```text:結果
-rw------- 1 ec2-user ec2-user 1671 Aug  5 18:33 /home/ec2-user/.ssh/788063364413_firehose_jawsug_cli.pem
```


## 1.2. CloudFormation テンプレートの生成

### テンプレートの作成

```bash:コマンド
CF_TEMPLATE_FILE_NAME="firehose_jawsug_cli.yml"
```

```bash:コマンド
cat << EOF > ${CF_TEMPLATE_FILE_NAME}
AWSTemplateFormatVersion: "2010-09-09"
Description: JAWS-UG CLI Kinesis Firehose Hands-on

Parameters: 
  VPCNetworkAddress: 
    Type: String
    Description: "Network Address on AWS"
    MinLength: 9
    MaxLength: 18
    # AllowedPattern: "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})"
    Default: "10.0.0.0/16"
  PublicSubnetAddr: 
    Type: String
    Description: "Network Address on AWS"
    MinLength: 9
    MaxLength: 18
    # AllowedPattern: "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})"
    Default: "10.0.0.0/24"
  KeyPairName:
    Type: AWS::EC2::KeyPair::KeyName


Resources:
  S3Bucket: 
    Type: "AWS::S3::Bucket"
  IAMRole: 
    Type: "AWS::IAM::Role"
    Properties: 
      RoleName: "service-role-firehose"
      AssumeRolePolicyDocument: 
        Version: "2012-10-17"
        Statement: 
          - 
            Effect: "Allow"
            Principal: 
              Service: 
                - "firehose.amazonaws.com"
            Action: 
              - "sts:AssumeRole"
      Path: "/"
  IAMPolicy:
    Type: "AWS::IAM::Policy"
    Properties: 
      PolicyName: "service-policy-firehose"
      PolicyDocument: 
        Version: "2012-10-17"
        Statement: 
          - 
            Effect: "Allow"
            Action: 
              - "s3:AbortMultipartUpload"
              - "s3:GetBucketLocation"
              - "s3:GetObject"
              - "s3:ListBucket"
              - "s3:ListBucketMultipartUploads"
              - "s3:PutObject"
            Resource:
              - !GetAtt S3Bucket.Arn
              - Fn::Join: 
                - "/"
                - 
                  - !GetAtt S3Bucket.Arn
                  - "*"
          - 
            Effect: "Allow"
            Action: 
              - "logs:PutLogEvents"
            Resource: "*"
      Roles:
        - Ref: IAMRole

  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: !Ref VPCNetworkAddress
      Tags: 
        - 
          Key: "Name"
          Value: "KinesisFirehoseClient"
  IGW:
    Type: AWS::EC2::InternetGateway
    Properties:
      Tags: 
        - 
          Key: "Name"
          Value: "KinesisFirehoseClient"
  AttachIGW:
      Type: AWS::EC2::VPCGatewayAttachment
      Properties:
          VpcId:
              Ref: VPC
          InternetGatewayId:
              Ref: IGW
  PublicSubnet:
    Type: AWS::EC2::Subnet
    Properties: 
      AvailabilityZone: 
        Fn::Select: 
          - 0
          - Fn::GetAZs: ""
      CidrBlock: !Ref PublicSubnetAddr
      MapPublicIpOnLaunch: true
      VpcId: 
        Ref: VPC
      Tags: 
        - 
          Key: "Name"
          Value: "Public"
  PublicRT:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId:
        Ref: VPC
      Tags: 
        - 
          Key: Name
          Value: Public
  PublicDefaultRoute:
    Type: AWS::EC2::Route
    Properties: 
      DestinationCidrBlock: "0.0.0.0/0"
      GatewayId: 
        Ref: IGW
      RouteTableId: 
        Ref: PublicRT
  PublicSubnetARouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId:
        Ref: PublicSubnet
      RouteTableId:
        Ref: PublicRT
  SecurityGroup:
    Type: "AWS::EC2::SecurityGroup"
    Properties: 
      GroupDescription: WebServer
      SecurityGroupEgress:
        - 
          IpProtocol: "-1"
          CidrIp: "0.0.0.0/0"
      SecurityGroupIngress:
        - 
          IpProtocol: "tcp"
          FromPort: 80
          ToPort: 80
          CidrIp: "0.0.0.0/0"        
        - 
          IpProtocol: "tcp"
          FromPort: 22
          ToPort: 22
          CidrIp: "0.0.0.0/0"
      VpcId:
        Ref: VPC    
  InstanceRole: 
    Type: "AWS::IAM::Role"
    Properties: 
      RoleName: "instance-role"
      AssumeRolePolicyDocument: 
        Version: "2012-10-17"
        Statement: 
          - 
            Effect: "Allow"
            Principal: 
              Service: 
                - "ec2.amazonaws.com"
                - "ssm.amazonaws.com"
            Action: 
              - "sts:AssumeRole"
      Path: "/"
      ManagedPolicyArns: 
        - "arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM"
        - "arn:aws:iam::aws:policy/AmazonKinesisFirehoseFullAccess"
  InstanceProfile: 
    Type: "AWS::IAM::InstanceProfile"
    Properties: 
      Path: "/"
      Roles: 
        - !Ref InstanceRole
  Instance:
    Type: "AWS::EC2::Instance"
    Properties:
      KeyName: 
        Ref: KeyPairName
      ImageId: ami-6df1e514
      InstanceType: t2.micro
      SecurityGroupIds: 
        - Ref: SecurityGroup
      SubnetId:
        Ref: PublicSubnet
      IamInstanceProfile:
        Ref: InstanceProfile
      BlockDeviceMappings: 
        - DeviceName: "/dev/sdm"
          Ebs: 
            VolumeType: "gp2"
            DeleteOnTermination: "true"
            VolumeSize: "8"
      UserData:
        Fn::Base64: |
          #!/bin/bash
          cd /tmp
          sudo yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm
          sudo yum install -y aws-kinesis-agent
          sudo chkconfig aws-kinesis-agent on
          sudo service aws-kinesis-agent start
          sudo yum install -y httpd
          sudo chkconfig httpd on
          sudo service httpd start
          sudo yum install npm --enablerepo=epel -y
          sudo npm install -g jsonlint

Outputs:
  S3BucketName:
    Value:
      Ref: S3Bucket
  IAMRoleARN:
    Value: !GetAtt IAMRole.Arn
  PublicIP:
    Value: !GetAtt Instance.PublicIp
EOF

cat ${CF_TEMPLATE_FILE_NAME}
```


### CloudFormation テンプレートの検証

```bash:コマンド
aws cloudformation validate-template \
    --template-body file://${CF_TEMPLATE_FILE_NAME}
```

```json:結果
{
    "CapabilitiesReason": "The following resource(s) require capabilities: [AWS::IAM::InstanceProfile, AWS::IAM::Role]",
    "Description": "JAWS-UG CLI Kinesis Firehose Hands-on",
    "Parameters": [
        {
            "NoEcho": false,
            "ParameterKey": "KeyPairName"
        },
        {
            "DefaultValue": "10.0.0.0/16",
            "NoEcho": false,
            "Description": "Network Address on AWS",
            "ParameterKey": "VPCNetworkAddress"
        },
        {
            "DefaultValue": "10.0.0.0/24",
            "NoEcho": false,
            "Description": "Network Address on AWS",
            "ParameterKey": "PublicSubnetAddr"
        }
    ],
    "Capabilities": [
        "CAPABILITY_NAMED_IAM"
    ]
}
```


## 1.3. CloudFormation Stackの作成

### CloudFormation Stack名の指定

```bash:コマンド
CF_STACK_NAME="firehose-jawsug-cli"
```

### 同名CloudFormation Stackの不存在を確認

```bash:コマンド
aws cloudformation describe-stacks \
    --query "Stacks[?StackName==\`${CF_STACK_NAME}\`]"
```

```json:結果
[]
```

### CloudFormation Stackの作成

```bash:コマンド
aws cloudformation create-stack \
    --stack-name ${CF_STACK_NAME} \
    --template-body file://${CF_TEMPLATE_FILE_NAME} \
    --capabilities "CAPABILITY_NAMED_IAM" \
    --parameters ParameterKey=KeyPairName,ParameterValue=${KEY_PAIR_NAME},UsePreviousValue=false
```

```json:結果
{
    "StackId": "arn:aws:cloudformation:us-west-2:XXXXXXXXXXXX:stack/firehose-jawsug-cli/8812e540-7a0e-11e7-aac3-50a68d01a68d"
}
```

### CloudFormation Stackの作成完了を待機

5分程度で作成が完了すると思います。

```bash:コマンド
aws cloudformation wait stack-create-complete \
    --stack-name ${CF_STACK_NAME}
```

```text:結果
（返値無し）
```


### CloudFormation Stackの存在を確認

"StackStatus"が"CREATE_COMPLETE"になっていることを確認します。

```bash:コマンド
aws cloudformation describe-stacks \
    --stack-name ${CF_STACK_NAME}
```

```json:結果
{
    "Stacks": [
        {
            "StackId": "arn:aws:cloudformation:us-west-2:XXXXXXXXXXXX:stack/firehose-jawsug-cli/4043bde0-7a16-11e7-8701-50a686be73ba",
            "Description": "JAWS-UG CLI Kinesis Firehose Hands-on",
            "Parameters": [
                {
                    "ParameterValue": "XXXXXXXXXXXX_firehose_jawsug_cli",
                    "ParameterKey": "KeyPairName"
                },
                {
                    "ParameterValue": "10.0.0.0/16",
                    "ParameterKey": "VPCNetworkAddress"
                },
                {
                    "ParameterValue": "10.0.0.0/24",
                    "ParameterKey": "PublicSubnetAddr"
                }
            ],
            "Tags": [],
            "Outputs": [
                {
                    "OutputKey": "PublicIP",
                    "OutputValue": "54.191.102.113"
                },
                {
                    "OutputKey": "IAMRoleARN",
                    "OutputValue": "arn:aws:iam::XXXXXXXXXXXX:role/service-role-firehose"
                },
                {
                    "OutputKey": "S3BucketName",
                    "OutputValue": "firehose-jawsug-cli-s3bucket-134czh3hcofqz"
                }
            ],
            "CreationTime": "2017-08-05T19:42:44.440Z",
            "Capabilities": [
                "CAPABILITY_NAMED_IAM"
            ],
            "StackName": "firehose-jawsug-cli",
            "NotificationARNs": [],
            "StackStatus": "CREATE_COMPLETE",
            "DisableRollback": false
        }
    ]
}
```


## 1.4. パラメータの確認

以降の手順で必要になるパラメータを抽出します。

- IAMロールARN
- S3バケット名
- パブリックIPアドレス


```bash:コマンド
OUTPUTKEY_ROLE_ARN="IAMRoleARN"
OUTPUTKEY_BUCKET_NAME="S3BucketName"
OUTPUTKEY_PUBLIC_IP_ADDRESS="PublicIP"
```

```bash:コマンド
ROLE_ARN=$(aws cloudformation describe-stacks \
    --stack-name ${CF_STACK_NAME} \
    --query "Stacks[].Outputs[?OutputKey==\`${OUTPUTKEY_ROLE_ARN}\`].OutputValue[]" \
    --output text) \
    && echo ${ROLE_ARN}
```

```text:結果
arn:aws:iam::XXXXXXXXXXXX:role/service-role-firehose
```


```bash:コマンド
BUCKET_NAME=$(aws cloudformation describe-stacks \
    --stack-name ${CF_STACK_NAME} \
    --query "Stacks[].Outputs[?OutputKey==\`${OUTPUTKEY_BUCKET_NAME}\`].OutputValue[]" \
    --output text) \
    && echo ${BUCKET_NAME}
```

```text:結果
firehose-jawsug-cli-s3bucket-134czh3hcofqz
```


```bash:コマンド
PUBLIC_IP_ADDRESS=$(aws cloudformation describe-stacks \
    --stack-name ${CF_STACK_NAME} \
    --query "Stacks[].Outputs[?OutputKey==\`${OUTPUTKEY_PUBLIC_IP_ADDRESS}\`].OutputValue[]" \
    --output text) \
    && echo ${PUBLIC_IP_ADDRESS}
```

```text:結果
54.191.***.***
```

### 動作確認

パブリックIPアドレスにブラウザでアクセスします。


以上