10分でわかる「kube-aws」

本スライドは[「Google Cloud Community fes @ Google Cloud Next'17 Tokyo」](https://www.meetup.com/ja-JP/GDG-Cloud-Tokyo/events/240052509/)の登壇資料です。

---

## Kubernetesとは?

**オープンソースのコンテナオーケストレーションシステム**

似た用途で使われることがあるもの:

Docker Swarm, Amazon ECS(EC2 Container Service), Kontena, Mesos, Rancher, ...

---

## kube-awsとは?

https://github.com/kubernetes-incubator/kube-aws

**Production-Ready(本番運用を目的とした)KubernetesクラスタをAWS上に構築するためのオープンソースのツール**

Kubernetes Incubatorの１プロジェクト

似た用途で使われることがあるもの:

ツール: kops, kargo, Tectonic
マネージドサービス: GKE(Google Container Engine), Giant Swarm, Stackpoint(Control-planeのみ)

AWSの向けならkopsかkube-awsがおすすめ。

---

## kube-aws for whom?

質問: 本番環境でAWSを使ってる人?

---

## No: 本番環境でAWSを使っていない

GKEを使いましょう:smile:

---

## Yes: 本番環境でAWSを使っている

kube-awsは中でも以下のような人におすすめ

* 様々なアプリケーションをセルフホストしたい
    * 例: [kubernetes/charts](https://github.com/kubernetes/charts)に登録されているアプリは確実に動く
* 運用は楽にしたい
    * アプリ開発者がセルフサービスで本番デプロイ
* ベンダーロックインはイヤ
    * どのクラウドプロバイダーにデプロイしても、K8SのAPIは一緒
    * →将来的にGKE/GCPに移行できる :heart: 
* Dev-Prod Parityを実現したい
    * 開発機でもAWSでもK8S
* インフラコスト削減したい
    * 性能や運用性を犠牲にせずにEC2インスタンスにできるだけ沢山のコンテナを相乗りさせたい
    * スポットインスタンス使いたいけどAvailabilityも犠牲にしたくない
* 上記を少人数で達成したい
* AWSチョットワカル

~~GoogleさんがGKEのAWS版を作ってくれるのを待ちましょう~~
→kube-awsを使いましょう:smile:

---

## Kubernetesで動くもの

一般的には...

* nginxのようなWebサーバ
* JenkinsやConcourseのようなCI
* fluentdのようなログフォワーダ
* Tensorflowなどを使ったマシンラーニング用システム
* 皆さんのWebアプリ、バッチなど
* MySQLのようなDB
* Prometheusのような監視システム
* Redisのようなキャッシュストアキューなど
* RabbitMQのようなメッセージキューなど
* Sparkのような分散コンピューティングエンジン

---

## AWSユーザがKubernetes on AWSで動かしたいもの

* **nginxのようなWebサーバ**
* **JenkinsやConcourseのようなCI**
* **fluentdのようなログフォワーダ**
* **Tensorflowなどを使ったマシンラーニング用システム**
* **皆さんのWebアプリ、バッチなど**
* MySQLのようなDB(ただし本番環境ではAurora推奨。AWSだし)
* Prometheusのような監視システム(予算や規模によってはDatadog推奨)
* Redisのようなキャッシュストア(本番環境ではElasticache推奨)
* RabbitMQのようなメッセージキューなど(タイミングによってはKinesis。Kafkaコネクターも来てベンダーロックイン感も薄れますし・・
* Sparkのような分散コンピューティングエンジン(場合によってはEMR推奨

**AWSのマネージドサービスで賄えないところをKubernetes on AWSでカバー!**
**AWSとKubernetesのいいとこどり!**

---

## 反論: Dev-Prod Parity?

* AWSのメッセージキューサービスのSQSとか、Key-Value StoreのDynamoDBとか使いたいんです。
* Kubernetes使ったとしてもそこはDev・Parity達成できないですよね

いえいえ、そんなことはありません。

[atlassian/localstack](https://github.com/atlassian/localstack)をK8Sにデプロイすれば、K8SクラスタにAWSのfakeをつくれます

※localstackみたいなものがGCPにもあるといいなぁ

---

## kube-awsのデザインゴール

* Production-Readyなクラスタをつくる
* コード行数を抑える=AWSサービスを最大限活用する
* カスタマイズ性

---

## デザインゴール#1: Production-Readyなクラスタをつくる

* H/A
    * Multi-AZ、K8S MasterとEtcdノードの独立性
* 既存VPC/SubnetにKubernetesをデプロイ
    * K8S内外のサービス間で簡単に通信できる!
* Private、Publicネットワーク混在可能な「ネットワーク・トポロジー」
    * 利用例: 個人情報を扱うコンテナはPrivateネットワーク内のK8Sのノードだけで実行
* 様々なスペックのサーバを一元管理できる「ノードプール」
    * 利用例: TensorflowはGPUのあるノードだけで実行
    * 利用例: Webアプリのコンテナをオンデマンド、リザーブド、スポットインスタンスにだいたい1:1:1でデプロイして低コストと高可用性を両立
* Etcd自動復旧
    * そもそも: 例えば3台のEtcdクラスタを組んだら、１台までは突然死してもサービス影響なし
    * Etcdのノードが突然死したら、新しいノードが数分でディスクを引き継いで稼働開始
    * Etcdノードのディスク上のデータが壊れたら、データを消去してまだ壊れていないリーダーノードのデータから自動復旧
    * 全Etcdノードが完全に壊れたら、バッグアップから自動復旧
* クラスタオートスケーリング
    * [NEW] v0.9.7-rc.1から
    * CPU・メモリ・GPUなどのリソース利用状況に応じて自動的にノード追加・削除

---

## kube-aws cluster

a.k.a kube-awsが皆さんのために何をやってくれるか

<img src="https://qiita-image-store.s3.amazonaws.com/0/5912/a3baadb1-7966-ad0b-0f87-0b3719213b25.png" width=640>

* 全コンポーネントMulti-AZでHigh Availabilityを実現
    * ただし日本では3 AZ中2 AZしか使えない＼(^o^)／
* Worker Node Pool
    * CA(cluster-autoscaler)によりオートスケール
    * Pool毎にインスタンスタイプ（GCEでいうマシンタイプ)などを変えられる
* EC2インスタンスにばらまくTLS用の秘密鍵、証明書などはKMSで暗号化
    * kube-awsは暗号化キーを管理しなくていい。kube-awsが原因で暗号化キーが漏れることはない。セキュア。
* EC2インスタンスにばらまく全ファイルはS3を経由させる
    * 各EC2インスタンスが必要なファイルをS3から自律的に取得してセットアップ実行
    * 利点: 各ノードのセットアップをやるだけの管理ノードみたいなもの(ansibleならansible実行元ノードとか)が不要

---

## デザインゴール#2: コード行数を抑える=AWSサービスを最大限活用する

kube-awsはAWS特化

* kops 20k LOCくらい
    * サポートしているクラウド: GCP, AWS, Azure, ...
    * サポートされているホストOS: Ubuntu, CentOS, RHEL7, CoreOS(Container Linux)
    * クラウドのプロビジョニング: Terraform, CloudFormation
    * ノードのプロビジョニング: goベースの独自ツール(nodeup/[protokube](https://github.com/kubernetes/kops/tree/master/protokube))
* kube-aws 5k LOCくらい
    * サポートしているクラウド: AWS
    * サポートされているホストOS: CoreOS(Container Linux)
    * クラウドのプロビジョニング: CloudFormation
    * ノードのプロビジョニング: cloud-init

---

## kube-awsのCloudFormationスタックテンプレート

CloudFormation = AWSのTerraformみたいなもの
スタックテンプレート = CloudFormationに作って欲しいインフラを記述したもの(Infrastructure as Code)

```json
{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "kube-aws Kubernetes cluster {{.ClusterName}}",
  "Parameters": {
    "NotificationTargetARN": {
      "Type": "String",
      "Description": "ASG LifecycleHook notification target ARN"
    },
    "NotificationRoleARN": {
      "Type": "String",
      "Description": "Role to be used by ASG LifecycleHook to publish notification"
    }
  },
  "Resources": {
    "{{.Controller.LogicalName}}": {
      "Type": "AWS::AutoScaling::AutoScalingGroup",
      "Properties": {
        "HealthCheckGracePeriod": 600,
        "HealthCheckType": "EC2",
        "LaunchConfigurationName": {
          "Ref": "{{.Controller.LogicalName}}LC"
        },
        "MaxSize": "{{.MaxControllerCount}}",
        "MetricsCollection": [
          {
            "Granularity": "1Minute"
          }
        ],
        "MinSize": "{{.MinControllerCount}}",
        "Tags": [
          {
            "Key": "kubernetes.io/cluster/{{.ClusterName}}",
            "PropagateAtLaunch": "true",
            "Value": "true"
          },
          {
            "Key": "Name",
            "PropagateAtLaunch": "true",
            "Value": "{{.ClusterName}}-{{.StackName}}-kube-aws-controller"
          },
          {
            "Key": "kubernetes.io/role/master",
            "PropagateAtLaunch": "true",
            "Value": ""
          }
        ],
        "VPCZoneIdentifier": [
          {{range $index, $subnet := .Controller.Subnets}}
          {{if gt $index 0}},{{end}}
          {{$subnet.Ref}}
          {{end}}
        ],
        "LoadBalancerNames" : [
          {{range $i, $ref := .APIEndpoints.ELBRefs -}}
          {{- if gt $i 0}},{{end}}
          {{ $ref }}
          {{- end}}
        ]
      },
      {{if .WaitSignal.Enabled}}
      "CreationPolicy" : {
        "ResourceSignal" : {
          "Count" : "{{.MinControllerCount}}",
          "Timeout" : "{{.Controller.CreateTimeout}}"
        }
      },
      {{end}}
      "UpdatePolicy" : {
        "AutoScalingRollingUpdate" : {
          "MinInstancesInService" : "{{.ControllerRollingUpdateMinInstancesInService}}",
          "MaxBatchSize" : "1",
          {{if .WaitSignal.Enabled}}
          "WaitOnResourceSignals" : "true",
          "PauseTime": "{{.Controller.CreateTimeout}}"
          {{else}}
          "PauseTime": "PT2M"
          {{end}}
        }
      },
      "Metadata" : {
        "AWS::CloudFormation::Init" : {
          "configSets" : {
              "etcd-client": [ "etcd-client-env" ]{{if .Experimental.AwsEnvironment.Enabled}},
              "aws-environment": [ "aws-environment-env" ]{{end}}
              {{ if .SharedPersistentVolume }},
                "load-efs-pv": [ "load-efs-pv-env" ]
              {{end}}
          },
          {{ if .Experimental.AwsEnvironment.Enabled }}
          "aws-environment-env" : {
            "commands": {
              "write-environment": {
                "command": { "Fn::Join" : ["", [ "echo '",
                  {{range $variable, $function := .Experimental.AwsEnvironment.Environment}}
                  "{{$variable}}=", {{$function}} , "\n",
                  {{end}}
                  "' > /etc/aws-environment" ] ] }
              }
            }
          },
          {{ end }}
          {{ if .SharedPersistentVolume }}
          "load-efs-pv-env" : {
            "files" : {
              "/etc/kubernetes/efs-pv.yaml": {
                "content": { "Fn::Join" : [ "", [
                  "apiVersion: v1\n",
                  "kind: PersistentVolume\n",
                  "metadata:\n",
                  "  name: shared-efs\n",
                  "spec:\n",
                  "  accessModes:\n",
                  "  - ReadWriteMany\n",
                  "  capacity:\n",
                  "    storage: 500Gi\n",
                  "  nfs:\n",
                  "    path: /\n",
                  "    server: ", {"Ref": "FileSystemCustom"}, ".efs.{{ $.Region }}.amazonaws.com", "\n",
                  "  persistentVolumeReclaimPolicy: Recycle\n"
                ]]}
              }
            }
          },
          {{ end }}
          "etcd-client-env": {
            "files" : {
              "/var/run/coreos/etcd-environment": {
                "content": { "Fn::Join" : [ "", [
                  "ETCD_ENDPOINTS='",
                  {{range $index, $etcdInstance := $.EtcdNodes}}
                  {{if $index}}",", {{end}} "https://",
                  {{$etcdInstance.AdvertisedFQDNRef}}, ":2379",
                  {{end}}
                  "'\n"
                ]]}
              }
            }
          }
        }
      },
      "DependsOn": ["{{$.Etcd.LogicalName}}{{sub $.Etcd.Count 1}}"]
    },
    {{ if not .Controller.IAMConfig.InstanceProfile.Arn }}
    "IAMInstanceProfileController": {
      "Properties": {
        "Path": "/",
        "Roles": [
          {
            "Ref": "IAMRoleController"
          }
        ]
      },
      "Type": "AWS::IAM::InstanceProfile"
    },
    {{end}}
    "IAMInstanceProfileEtcd": {
      "Properties": {
        "Path": "/",
        "Roles": [
          {
            "Ref": "IAMRoleEtcd"
          }
        ]
      },
      "Type": "AWS::IAM::InstanceProfile"
    },
    {{ if not .Controller.IAMConfig.InstanceProfile.Arn }}
    "IAMManagedPolicyController" : {
      "Type" : "AWS::IAM::ManagedPolicy",
      "Properties" : {
        "Description" : "Policy for managing kube-aws k8s controllers",
        "Path" : "/",
        "PolicyDocument" :   {
          "Version":"2012-10-17",
          "Statement": [
                {
                  "Action": "ec2:*",
                  "Effect": "Allow",
                  "Resource": "*"
                },
                {
                  "Action": "elasticloadbalancing:*",
                  "Effect": "Allow",
                  "Resource": "*"
                },
		            {
                  "Effect": "Allow",
                  "Action": [
                  "s3:GetObject"
                  ],
                  "Resource": "arn:{{.Region.Partition}}:s3:::{{$.UserDataControllerS3Prefix}}*"
		            },
                {{if .WaitSignal.Enabled}}
                {
                  "Action": "cloudformation:SignalResource",
                  "Effect": "Allow",
                  "Resource":
                    { "Fn::Join": [ "", [
                      "arn:{{.Region.Partition}}:cloudformation:",
                      { "Ref": "AWS::Region" },
                      ":",
                      { "Ref": "AWS::AccountId" },
                      ":stack/",
                      { "Ref": "AWS::StackName" },
                      "/*" ]
                    ] }
                },
                {{end}}
                {{if .Experimental.AwsNodeLabels.Enabled}}
                {
                  "Action": "autoscaling:Describe*",
                  "Effect": "Allow",
                  "Resource": [ "*" ]
                },
                {{end}}
                {{if .Addons.ClusterAutoscaler.Enabled}}
                {
                "Effect": "Allow",
                  "Action": [
                    "autoscaling:DescribeAutoScalingGroups",
                    "autoscaling:DescribeAutoScalingInstances",
                    "autoscaling:DescribeTags",
                    "autoscaling:SetDesiredCapacity",
                    "autoscaling:TerminateInstanceInAutoScalingGroup"
                  ],
                  "Resource": "*"
                },
                {{end}}
                {{if .Experimental.Kube2IamSupport.Enabled }}
                {
                  "Action": "sts:AssumeRole",
                  "Effect":"Allow",
                  "Resource":"*"
                },
                {{end}}
                {{if .AssetsEncryptionEnabled}}
                {
                  "Action" : "kms:Decrypt",
                  "Effect" : "Allow",
                  "Resource" : "{{.KMSKeyARN}}"
                },
                {{end}}
                {{if .Experimental.NodeDrainer.Enabled }}
                {
                  "Action": [
                    "autoscaling:DescribeAutoScalingInstances",
                    "autoscaling:DescribeLifecycleHooks"
                  ],
                  "Effect": "Allow",
                  "Resource": "*"
                },
                {
                  "Action": [
                    "autoscaling:CompleteLifecycleAction"
                  ],
                  "Effect": "Allow",
                  "Condition": {
                    "Null": { "autoscaling:ResourceTag/kubernetes.io/cluster/{{.ClusterName}}": "false" }
                  },
                  "Resource": "*"
                },
                {{end}}
                {
                  "Action": [
                    "ecr:GetAuthorizationToken",
                    "ecr:BatchCheckLayerAvailability",
                    "ecr:GetDownloadUrlForLayer",
                    "ecr:GetRepositoryPolicy",
                    "ecr:DescribeRepositories",
                    "ecr:ListImages",
                    "ecr:BatchGetImage"
                  ],
                  "Resource": "*",
                  "Effect": "Allow"
                }
          ]
        }
      }
    },
    "IAMRoleController": {
      "Properties": {
        "AssumeRolePolicyDocument": {
          "Statement": [
            {
              "Action": [
                "sts:AssumeRole"
              ],
              "Effect": "Allow",
              "Principal": {
                "Service": [
                  "ec2.{{.Region.PublicDomainName}}"
                ]
              }
            }
          ],
          "Version": "2012-10-17"
        },
        "Path": "/",
        {{if .Controller.IAMConfig.Role.Name }}
        "RoleName":  {"Fn::Join": ["",[{"Ref": "AWS::Region"},"-","{{.Controller.IAMConfig.Role.Name}}"]]},
        {{end}}
        "ManagedPolicyArns": [ 
          {{range $policyIndex, $policyArn := .Controller.IAMConfig.Role.ManagedPolicies }}
            "{{$policyArn.Arn}}",
          {{end}}
          {"Ref": "IAMManagedPolicyController"}
        ]
      },
      "Type": "AWS::IAM::Role"
    },
    {{end}}
    "IAMRoleEtcd": {
      "Properties": {
        "AssumeRolePolicyDocument": {
          "Statement": [
            {
              "Action": [
                "sts:AssumeRole"
              ],
              "Effect": "Allow",
              "Principal": {
                "Service": [
                  "ec2.{{.Region.PublicDomainName}}"
                ]
              }
            }
          ],
          "Version": "2012-10-17"
        },
        "Path": "/",
        "Policies": [
          {
            "PolicyDocument": {
              "Statement": [{{if .AssetsEncryptionEnabled}}
                {
                  "Action" : "kms:Decrypt",
                  "Effect" : "Allow",
                  "Resource" : "{{.KMSKeyARN}}"
                },{{end}}
                {{if $.Etcd.KMSKeyARN -}}
                {{/* Required for mounting encrypted data volume */}}
                {
                "Action": [
                  "kms:CreateGrant",
                  "kms:Decrypt",
                  "kms:Describe*",
                  "kms:Encrypt",
                  "kms:GenerateDataKey*",
                  "kms:ReEncrypt*"
                ],
                "Effect": "Allow",
                "Resource": "{{ $.Etcd.KMSKeyARN }}"
                },
                {{end -}}
                {
                  "Action": "ec2:DescribeTags",
                  "Effect": "Allow",
                  "Resource": "*"
                },
                {{/* Required for cfn-etcd-environment.service to discover the volume */}}
                {
                  "Action": "ec2:DescribeVolumes",
                  "Effect": "Allow",
                  "Resource": "*"
                },
                {{/* Required for cfn-etcd-environment.service to start attaching the volume */}}
                {
                  "Action": "ec2:AttachVolume",
                  "Effect": "Allow",
                  "Resource": "*"
                },
                {{/* Required for cfn-etcd-environment.service to wait until the volume is attached */}}
                {
                  "Action": "ec2:DescribeVolumeStatus",
                  "Effect": "Allow",
                  "Resource": "*"
                },
                {{if $.Etcd.NodeShouldHaveEIP -}}
                {{/* Required for cfn-etcd-environment.service to associate an EIP */}}
                {
                  "Action": "ec2:AssociateAddress",
                  "Effect": "Allow",
                  "Resource": "*"
                },
                {{end -}}
                {{if $.Etcd.NodeShouldHaveSecondaryENI -}}
                {{/* Required for cfn-etcd-environment.service to associate a network interface */}}
                {
                  "Action": "ec2:AttachNetworkInterface",
                  "Effect": "Allow",
                  "Resource": "*"
                },
                {
                  "Action": "ec2:DescribeNetworkInterfaces",
                  "Effect": "Allow",
                  "Resource": "*"
                },
                {
                  "Action": "ec2:DescribeNetworkInterfaceAttribute",
                  "Effect": "Allow",
                  "Resource": "*"
                },
                {{end -}}
                {
                  "Effect": "Allow",
                  "Action": [
                    "s3:GetObject"
                  ],
                  "Resource": "arn:{{.Region.Partition}}:s3:::{{$.UserDataEtcdS3Prefix}}*"
                },
                {{/* Required for `etcdadm reconfigure` to check existence of an etcd snapshot in S3 */}}
                {
                  "Effect": "Allow",
                  "Action": [
                    "s3:ListBucket"
                  ],
                  "Resource": "arn:{{.Region.Partition}}:s3:::{{$.EtcdSnapshotsS3Bucket}}"
                },
                {
                  "Effect": "Allow",
                  "Action": [
                    "s3:List*",
                    "s3:GetObject*"
                  ],
                  "Resource": "arn:{{.Region.Partition}}:s3:::{{$.EtcdSnapshotsS3Bucket}}",
                  "Condition": {
                    "StringLike": {
                      "s3:prefix": { "Fn::Join" : [ "", [{{$.EtcdSnapshotsS3PrefixRef}}, "/*" ]]}
                    }
                  }
                },
                {{/* Required for `etcdadm save` to save etcd snapshots in S3 */}}
                {
                  "Effect": "Allow",
                  "Action": [
                    "s3:*"
                  ],
                  "Resource": { "Fn::Join" : [ "", ["arn:{{.Region.Partition}}:s3:::", {{$.EtcdSnapshotsS3PathRef}}, "/*" ]]}
                },
                {{/* Required for `etcdadm reconfigure` to determine the number of active etcd nodes */}}
                {
                  "Action": "ec2:DescribeInstances",
                  "Resource": "*",
                  "Effect": "Allow"
                }
              ],
              "Version": "2012-10-17"
            },
            "PolicyName": "root"
          }
        ]
      },

      "Type": "AWS::IAM::Role"
    },
    {{if $.Etcd.HostedZoneManaged}}
    "{{$.Etcd.HostedZoneLogicalName}}": {
      "Type": "AWS::Route53::HostedZone",
      "Properties": {
        "HostedZoneConfig": {
          "Comment": "My hosted zone for {{$.Etcd.InternalDomainName}}"
        },
        "Name": "{{$.Etcd.InternalDomainName}}",
        "VPCs": [{
          "VPCId": {{$.VPCRef}},
          "VPCRegion": { "Ref": "AWS::Region" }
        }],
        "HostedZoneTags" : [{
          "Key": "kubernetes.io/cluster/{{$.ClusterName}}",
          "Value": "true"
        }]
      }
    },
    {{end}}
    {{range $etcdIndex, $etcdInstance := .EtcdNodes}}
    {{if $etcdInstance.RecordSetManaged}}
    "{{$etcdInstance.RecordSetLogicalName}}" : {
      "Type" : "AWS::Route53::RecordSet",
        "Properties" : {
        "HostedZoneId": {{$.Etcd.HostedZoneRef}},
        "Name": {{$etcdInstance.AdvertisedFQDNRef}},
        "Comment" : "A record for the private IP address of Etcd node named {{$etcdInstance.Name}} at index {{$etcdIndex}}",
        "Type" : "A",
        "TTL" : "300",
        "ResourceRecords" : [
          {{$etcdInstance.NetworkInterfacePrivateIPRef}}
        ]
      }
    },
    {{end}}
    {{if $etcdInstance.NetworkInterfaceManaged}}
    "{{$etcdInstance.NetworkInterfaceLogicalName}}": {
      "Properties": {
        "SubnetId": {{$etcdInstance.SubnetRef}},
        "GroupSet": [
          {{range $sgIndex, $sgRef := $.Etcd.SecurityGroupRefs}}
          {{if gt $sgIndex 0}},{{end}}
          {{$sgRef}}
          {{end}}
        ]
      },
      "Type": "AWS::EC2::NetworkInterface"
    },
    {{end}}
    {{if $etcdInstance.EIPManaged}}
    "{{$etcdInstance.EIPLogicalName}}": {
      "Properties": {
        "Domain": "vpc"
      },
      "Type": "AWS::EC2::EIP"
    },
    {{end}}
    {{if not $.Etcd.DataVolume.Ephemeral}}
    "{{$etcdInstance.EBSLogicalName}}": {
      "Properties": {
          "AvailabilityZone": "{{$etcdInstance.SubnetAvailabilityZone}}",
          "Size": "{{$.Etcd.DataVolume.Size}}",
          {{if gt $.Etcd.DataVolume.IOPS 0}}
          "Iops": "{{$.Etcd.DataVolume.IOPS}}",
          {{end}}
          {{if $.Etcd.DataVolume.Encrypted}}
          "Encrypted": {{$.Etcd.DataVolume.Encrypted}},
          {{if $.Etcd.KMSKeyARN}}
          "KmsKeyId": "{{$.Etcd.KMSKeyARN}}",
          {{end}}
          {{end}}
          "VolumeType": "{{$.Etcd.DataVolume.Type}}",
          "Tags": [
            {
              "Key": "kube-aws:etcd:index",
              "Value": "{{$etcdIndex}}"
            },
            {{if $etcdInstance.EIPManaged}}{
              "Key": "{{$.Etcd.EIPAllocationIDTagKey}}",
              "Value": {{$etcdInstance.EIPAllocationIDRef}}
            },{{end}}
            {{if $etcdInstance.NetworkInterfaceManaged}}{
              "Key": "{{$.Etcd.NetworkInterfaceIDTagKey}}",
              "Value": {{$etcdInstance.NetworkInterfaceIDRef}}
            },{{end}}
            {
              "Key": "{{$.Etcd.AdvertisedFQDNTagKey}}",
              "Value": {{$etcdInstance.AdvertisedFQDNRef}}
            },
            {
              "Key": "{{$.Etcd.NameTagKey}}",
              "Value": "{{$etcdInstance.Name}}"
            }
          ]
      },
      "Type": "AWS::EC2::Volume"
    },
    {{end}}
    "{{$etcdInstance.LogicalName}}": {
      "Type": "AWS::AutoScaling::AutoScalingGroup",
      "Properties": {
        "HealthCheckGracePeriod": 600,
        "HealthCheckType": "EC2",
        "LaunchConfigurationName": {
          "Ref": "{{$etcdInstance.LaunchConfigurationLogicalName}}"
        },
        "MaxSize": "1",
        "MetricsCollection": [
          {
            "Granularity": "1Minute"
          }
        ],
        "MinSize": "1",
        "Tags": [
          {
            "Key": "kubernetes.io/cluster/{{$.ClusterName}}",
            "PropagateAtLaunch": "true",
            "Value": "true"
          },
          {
            "Key": "Name",
            "PropagateAtLaunch": "true",
            "Value": "{{$.ClusterName}}-{{$.StackName}}-kube-aws-etcd-{{$etcdIndex}}"
          },
          {
            "Key": "kube-aws:role",
            "PropagateAtLaunch": "true",
            "Value": "etcd"
          }
        ],
        "VPCZoneIdentifier": [
          {{$etcdInstance.SubnetRef}}
        ]
      },
      {{if $.WaitSignal.Enabled}}
      "CreationPolicy" : {
        "ResourceSignal" : {
          "Count" : "1",
          "Timeout" : "{{$.Controller.CreateTimeout}}"
        }
      },
      {{end}}
      "UpdatePolicy" : {
        "AutoScalingRollingUpdate" : {
          "MinInstancesInService" : "0",
          "MaxBatchSize" : "1",
          {{if $.WaitSignal.Enabled}}
          "WaitOnResourceSignals" : "true",
          "PauseTime": "{{$.Controller.CreateTimeout}}"
          {{else}}
          "PauseTime": "PT2M"
          {{end}}
        }
      },
      "Metadata" : {
        "AWS::CloudFormation::Init" : {
          "configSets" : {
              "etcd-server": [ "etcd-server-env" ]
          },
          "etcd-server-env": {
            "files" : {
              "/var/run/coreos/etcd-environment": {
                "content": { "Fn::Join" : [ "", [
                  "ETCD_INITIAL_CLUSTER='",
                    {{range $etcdIndex, $etcdInstance := $.EtcdNodes}}
                    {{if $etcdIndex}}",", {{end}}
                    "{{$etcdInstance.Name}}",
                    "=https://",
                    {{$etcdInstance.AdvertisedFQDNRef}},
                    ":2380",
                    {{end}}
                  "'\n"
                ]]}
              },
              "/var/run/coreos/etcdadm-environment": {
                "content": { "Fn::Join" : [ "", [
                  "ETCD_ENDPOINTS='",
                    {{range $index, $etcdInstance := $.EtcdNodes}}
                    {{if $index}}",", {{end}}
                    "https://",
                    {{$etcdInstance.AdvertisedFQDNRef}},
                    ":2379",
                    {{end}}
                  "'\n",
                  "AWS_DEFAULT_REGION='",
                    "{{$.Region}}",
                  "'\n",
                  "KUBERNETES_CLUSTER='",
                    "{{$.ClusterName}}",
                  "'\n",
                  "ETCDCTL_CACERT='",
                    "/etc/ssl/certs/ca.pem",
                  "'\n",
                  "ETCDCTL_CERT='",
                    "/etc/ssl/certs/etcd-client.pem",
                  "'\n",
                  "ETCDCTL_KEY='",
                    "/etc/ssl/certs/etcd-client-key.pem",
                  "'\n",
                  "ETCDCTL_CA_FILE='",
                    "/etc/ssl/certs/ca.pem",
                  "'\n",
                  "ETCDCTL_CERT_FILE='",
                    "/etc/ssl/certs/etcd-client.pem",
                  "'\n",
                  "ETCDCTL_KEY_FILE='",
                    "/etc/ssl/certs/etcd-client-key.pem",
                  "'\n",
                  "ETCDADM_MEMBER_SYSTEMD_SERVICE_NAME='",
                    "etcd-member",
                  "'\n",
                  "ETCDADM_CLUSTER_SNAPSHOTS_S3_URI='",
                    { "Fn::Join" : [ "", ["s3://", {{$.EtcdSnapshotsS3PathRef}} ]] },
                  "'\n",
                  "ETCDADM_STATE_FILES_DIR='",
                    "/var/run/coreos/etcdadm",
                  "'\n",
                  "ETCDADM_MEMBER_COUNT='",
                    "{{$.Etcd.Count}}",
                  "'\n",
                  "ETCDADM_MEMBER_INDEX='",
                    "{{$etcdIndex}}",
                  "'\n"
                ]]}
              }
            }
          }
        }
      },
      "DependsOn": [
        {{if $etcdInstance.DependencyExists}}{{$etcdInstance.DependencyRef}},{{end}}
        {{if $etcdIndex}}"{{$.Etcd.LogicalName}}{{sub $etcdIndex 1}}",{{end}}
        {{if $etcdInstance.EIPManaged}}
        "{{$etcdInstance.EIPLogicalName}}",
        {{end}}
        {{if $etcdInstance.NetworkInterfaceManaged}}
        "{{$etcdInstance.NetworkInterfaceLogicalName}}",
        {{end}}
        {{if $etcdInstance.RecordSetManaged}}
        "{{$etcdInstance.RecordSetLogicalName}}",
        {{end}}
        "{{$etcdInstance.EBSLogicalName}}"
      ]
    },
    "{{$etcdInstance.LaunchConfigurationLogicalName}}": {
      "Properties": {
        "BlockDeviceMappings": [
          {
            "DeviceName": "/dev/xvda",
            "Ebs": {
              "VolumeSize": "{{$.Etcd.RootVolume.Size}}",
              {{if gt $.Etcd.RootVolume.IOPS 0}}
              "Iops": "{{$.Etcd.RootVolume.IOPS}}",
              {{end}}
              "VolumeType": "{{$.Etcd.RootVolume.Type}}"
            }
          }
          {{if $.Etcd.DataVolume.Ephemeral}}
          ,
          {
            "DeviceName": "/dev/xvdf",
            "VirtualName" : "ephemeral0"
          }
          {{end}}
        ],
        "IamInstanceProfile": {
          "Ref": "IAMInstanceProfileEtcd"
        },
        "ImageId": "{{$.AMI}}",
        "InstanceType": "{{$.Etcd.InstanceType}}",
        {{if $.KeyName}}"KeyName": "{{$.KeyName}}",{{end}}
        "SecurityGroups": [
          {{range $sgIndex, $sgRef := $.Etcd.SecurityGroupRefs}}
          {{if gt $sgIndex 0}},{{end}}
          {{$sgRef}}
          {{end}}
        ],
        "PlacementTenancy": "{{$.Etcd.Tenancy}}",
        "UserData": { "Fn::Base64": { "Fn::Join" : ["\n", [
          "#!/bin/bash -xe",
          {"Fn::Join":["",[ "echo '{{$.StackNameEnvVarName}}=", { "Ref": "AWS::StackName" }, "' >> {{$.EtcdNodeEnvFileName}}" ]]},
          "echo '{{$.EtcdIndexEnvVarName}}={{$etcdIndex}}' >> {{$.EtcdNodeEnvFileName}}",
          " . /etc/environment",
          "export COREOS_PRIVATE_IPV4 COREOS_PRIVATE_IPV6 COREOS_PUBLIC_IPV4 COREOS_PUBLIC_IPV6",
          "REGION=$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r '.region')",
          "USERDATA_FILE={{$.UserDataEtcdFileName}}",
          "while ! /usr/bin/rkt run \\",
          "   --net=host \\",
          "   --volume=dns,kind=host,source=/etc/resolv.conf,readOnly=true --mount volume=dns,target=/etc/resolv.conf  \\",
          "   --volume=awsenv,kind=host,source=/var/run/coreos,readOnly=false --mount volume=awsenv,target=/var/run/coreos \\",
          "   --trust-keys-from-https \\",
          "   {{$.AWSCliImage.Options}}{{$.AWSCliImage.RktRepo}} --exec=aws -- s3 --region $REGION  cp {{ $.UserDataEtcdS3URI }} /var/run/coreos/$USERDATA_FILE; do",
          "sleep 1",
          "done",
          "exec /usr/bin/coreos-cloudinit --from-file /var/run/coreos/$USERDATA_FILE"
        ]]}}
      },
      "Type": "AWS::AutoScaling::LaunchConfiguration"
    },
    {{end}}
    {{if .Experimental.NodeDrainer.Enabled }}
    "{{.Controller.LogicalName}}NodeDrainerLH" : {
      "Properties" : {
        "AutoScalingGroupName" : {
          "Ref": "{{.Controller.LogicalName}}"
        },
        "DefaultResult" : "CONTINUE",
        "HeartbeatTimeout" : "{{.Experimental.NodeDrainer.DrainTimeoutInSeconds}}",
        "LifecycleTransition" : "autoscaling:EC2_INSTANCE_TERMINATING",
        "NotificationTargetARN" : { "Ref": "NotificationTargetARN" },
        "RoleARN" : { "Ref": "NotificationRoleARN" }
      },
      "Type" : "AWS::AutoScaling::LifecycleHook"
    },
    {{end}}
    "{{.Controller.LogicalName}}LC": {
      "Properties": {
        "BlockDeviceMappings": [
          {
            "DeviceName": "/dev/xvda",
            "Ebs": {
              "VolumeSize": "{{.Controller.RootVolume.Size}}",
              {{if gt .Controller.RootVolume.IOPS 0}}
              "Iops": "{{.Controller.RootVolume.IOPS}}",
              {{end}}
              "VolumeType": "{{.Controller.RootVolume.Type}}"
            }
          }
        ],
        {{if .Controller.IAMConfig.InstanceProfile.Arn }}
        "IamInstanceProfile": "{{.Controller.IAMConfig.InstanceProfile.Arn}}",
        {{else}}
        "IamInstanceProfile": {
          "Ref": "IAMInstanceProfileController"
        },
        {{end}}
        "ImageId": "{{.AMI}}",
        "InstanceType": "{{.Controller.InstanceType}}",
        {{if .KeyName}}"KeyName": "{{.KeyName}}",{{end}}
        "SecurityGroups": [
          {{range $sgIndex, $sgRef := $.Controller.SecurityGroupRefs}}
          {{if gt $sgIndex 0}},{{end}}
          {{$sgRef}}
          {{end}}
        ],
        "PlacementTenancy": "{{ .Controller.Tenancy }}",
        "UserData": { "Fn::Base64": { "Fn::Join" : ["\n", [
		"#!/bin/bash -xe",
		" . /etc/environment",
                "export COREOS_PRIVATE_IPV4 COREOS_PRIVATE_IPV6 COREOS_PUBLIC_IPV4 COREOS_PUBLIC_IPV6",
	        "REGION=$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r '.region')",
		"USERDATA_FILE={{.UserDataControllerFileName}}",
		"while ! /usr/bin/rkt run \\",
		"   --net=host \\",
		"   --volume=dns,kind=host,source=/etc/resolv.conf,readOnly=true --mount volume=dns,target=/etc/resolv.conf  \\",
		"   --volume=awsenv,kind=host,source=/var/run/coreos,readOnly=false --mount volume=awsenv,target=/var/run/coreos \\",
   		"   --trust-keys-from-https \\",
		"   {{.AWSCliImage.Options}}{{.AWSCliImage.RktRepo}} --exec=aws -- s3 --region $REGION  cp {{ .UserDataControllerS3URI }} /var/run/coreos/$USERDATA_FILE; do",
                "sleep 1",
                "done",
		"exec /usr/bin/coreos-cloudinit --from-file /var/run/coreos/$USERDATA_FILE"
			]]}}
      },
  {{ if .Experimental.AwsEnvironment.Enabled }}
      "Metadata" : {
        "AWS::CloudFormation::Init" : {
          "config" : {
            "commands": {
              "write-environment": {
                "command": { "Fn::Join" : ["", [ "echo '",
{{range $variable, $function := .Experimental.AwsEnvironment.Environment}}
"{{$variable}}=", {{$function}} , "\n",
{{end}}
"' > /etc/aws-environment" ] ] }
              }
            }
          }
        }
      },
  {{end}}
      "Type": "AWS::AutoScaling::LaunchConfiguration"
    },
    {{range $i, $apiEndpoint := $.APIEndpoints -}}
    {{if .LoadBalancer.ManageELB -}}
    {{if .LoadBalancer.ManageELBRecordSet -}}
    "{{.LoadBalancer.RecordSetLogicalName}}": {
      "Type": "AWS::Route53::RecordSet",
      "Properties": {
        "HostedZoneId": "{{.LoadBalancer.HostedZoneRef}}",
        "Name": "{{$apiEndpoint.DNSName}}",
        "TTL": {{.LoadBalancer.RecordSetTTL}},
        "ResourceRecords": [{{.LoadBalancer.DNSNameRef}}],
        "Type": "CNAME"
      }
    },
    {{ end -}}
    "{{.LoadBalancer.LogicalName}}" : {
      "Type" : "AWS::ElasticLoadBalancing::LoadBalancer",
      "Properties" : {
        "CrossZone" : true,
        "HealthCheck" : {
          "HealthyThreshold" : "3",
          "Interval" : "10",
          "Target" : "SSL:443",
          "Timeout" : "8",
          "UnhealthyThreshold" : "3"
        },
        "ConnectionSettings" : {
          "IdleTimeout" : "3600"
        },
        "Subnets" : [
          {{range $index, $subnet := .LoadBalancer.Subnets}}
          {{if gt $index 0}},{{end}}
          {{$subnet.Ref}}
          {{end}}
        ],
        "Listeners" : [
          {
            "InstancePort" : "443",
            "InstanceProtocol" : "TCP",
            "LoadBalancerPort" : "443",
            "Protocol" : "TCP"
          }
        ],
        {{if .LoadBalancer.Private}}
        "Scheme": "internal",
        {{else}}
        "Scheme": "internet-facing",
        {{end}}
        "SecurityGroups": [
          {{range $sgIndex, $sgRef := .LoadBalancer.SecurityGroupRefs}}
          {{if gt $sgIndex 0}},{{end}}
          {{$sgRef}}
          {{end}}
        ]
      }
    },
    {{if .LoadBalancer.ManageSecurityGroup -}}
    "{{.LoadBalancer.SecurityGroupLogicalName}}" : {
      "Properties": {
        "GroupDescription": {
          "Ref": "AWS::StackName"
        },
        "SecurityGroupIngress": [
          {{ range $j, $r := .LoadBalancer.APIAccessAllowedSourceCIDRs -}}
          {{if gt $j 0}},{{end}}
          {
            "CidrIp": "{{$r}}",
            "FromPort": 443,
            "IpProtocol": "tcp",
            "ToPort": 443
          }
          {{end}}
        ],
        "Tags": [
          {
            "Key": "Name",
            "Value": "{{$.ClusterName}}-sg-api-endpoint-{{$i}}"
          }
        ],
        "VpcId": {{$.VPCRef}}
      },
      "Type": "AWS::EC2::SecurityGroup"
    },
    {{end -}}
    {{end -}}
    {{end -}}
    "SecurityGroupElbAPIServer" : {
      "Properties": {
        "GroupDescription": {
          "Ref": "AWS::StackName"
        },
        "SecurityGroupIngress": [
          {
             "CidrIp": "0.0.0.0/0",
             "FromPort": -1,
             "IpProtocol": "icmp",
             "ToPort": -1
          }
        ],
        "Tags": [
          {
            "Key": "Name",
            "Value": "{{$.ClusterName}}-sg-elb-api-server"
          }
        ],
        "VpcId": {{$.VPCRef}}
      },
      "Type": "AWS::EC2::SecurityGroup"
    },
    "SecurityGroupController": {
      "Properties": {
        "GroupDescription": {
          "Ref": "AWS::StackName"
        },
        "SecurityGroupEgress": [
          {
            "CidrIp": "0.0.0.0/0",
            "FromPort": -1,
            "IpProtocol": "icmp",
            "ToPort": -1
          },
          {
            "CidrIp": "0.0.0.0/0",
            "FromPort": 0,
            "IpProtocol": "tcp",
            "ToPort": 65535
          },
          {
            "CidrIp": "0.0.0.0/0",
            "FromPort": 0,
            "IpProtocol": "udp",
            "ToPort": 65535
          }
        ],
        "SecurityGroupIngress": [
          {
            "CidrIp": "0.0.0.0/0",
            "FromPort": -1,
            "IpProtocol": "icmp",
            "ToPort": -1
          },
          {{ range $_, $r := $.SSHAccessAllowedSourceCIDRs -}}
          {
            "CidrIp": "{{$r}}",
            "FromPort": 22,
            "IpProtocol": "tcp",
            "ToPort": 22
          },
          {{end -}}
          {
            "SourceSecurityGroupId" : { "Ref" : "SecurityGroupElbAPIServer" },
            "FromPort": 443,
            "IpProtocol": "tcp",
            "ToPort": 443
          },
          {
            "SourceSecurityGroupId" : { "Ref" : "SecurityGroupWorker" },
            "FromPort": 443,
            "IpProtocol": "tcp",
            "ToPort": 443
          }
        ],
        "Tags": [
          {
            "Key": "Name",
            "Value": "{{$.ClusterName}}-sg-controller"
          }
        ],
        "VpcId": {{.VPCRef}}
      },
      "Type": "AWS::EC2::SecurityGroup"
    },
    {{/* 443 ingress from controller to controller is required by:
      - API server endpoint with HA controllers
      - calico-policy-controller when calico is enabled
      See https://github.com/kubernetes-incubator/kube-aws/issues/494#issuecomment-291687137 
      and https://github.com/kubernetes-incubator/kube-aws/issues/512 */}}
    "SecurityGroupControllerIngressFromControllerToController": {
      "Properties": {
        "FromPort": 443,
        "GroupId": {
          "Ref": "SecurityGroupController"
        },
        "IpProtocol": "tcp",
        "SourceSecurityGroupId": {
          "Ref": "SecurityGroupController"
        },
        "ToPort": 443
      },
      "Type": "AWS::EC2::SecurityGroupIngress"
    },
    "SecurityGroupControllerIngressFromControllerToKubelet": {
      "Properties": {
        "FromPort": 10250,
        "GroupId": {
          "Ref": "SecurityGroupController"
        },
        "IpProtocol": "tcp",
        "SourceSecurityGroupId": {
          "Ref": "SecurityGroupController"
        },
        "ToPort": 10250
      },
      "Type": "AWS::EC2::SecurityGroupIngress"
    },
    "SecurityGroupControllerIngressFromWorkerToEtcd": {
      "Properties": {
        "FromPort": 2379,
        "GroupId": {
          "Ref": "SecurityGroupController"
        },
        "IpProtocol": "tcp",
        "SourceSecurityGroupId": {
          "Ref": "SecurityGroupWorker"
        },
        "ToPort": 2379
      },
      "Type": "AWS::EC2::SecurityGroupIngress"
    },
    "SecurityGroupWorker": {
      "Properties": {
        "GroupDescription": {
          "Ref": "AWS::StackName"
        },
        "SecurityGroupEgress": [
          {
            "CidrIp": "0.0.0.0/0",
            "FromPort": -1,
            "IpProtocol": "icmp",
            "ToPort": -1
          },
          {
            "CidrIp": "0.0.0.0/0",
            "FromPort": 0,
            "IpProtocol": "tcp",
            "ToPort": 65535
          },
          {
            "CidrIp": "0.0.0.0/0",
            "FromPort": 0,
            "IpProtocol": "udp",
            "ToPort": 65535
          }
        ],
        "SecurityGroupIngress": [
          {{ range $_, $r := $.SSHAccessAllowedSourceCIDRs -}}
          {
            "CidrIp": "{{$r}}",
            "FromPort": 22,
            "IpProtocol": "tcp",
            "ToPort": 22
          },
          {{end -}}
          {
            "CidrIp": "0.0.0.0/0",
            "FromPort": -1,
            "IpProtocol": "icmp",
            "ToPort": -1
          }
        ],
        "Tags": [
          {
            "Key": "Name",
            "Value": "{{$.ClusterName}}-sg-worker"
          }
        ],
        "VpcId": {{.VPCRef}}
      },
      "Type": "AWS::EC2::SecurityGroup"
    },
    "SecurityGroupWorkerIngressFromControllerToFlannel": {
      "Properties": {
        "FromPort": 8472,
        "GroupId": {
          "Ref": "SecurityGroupWorker"
        },
        "IpProtocol": "udp",
        "SourceSecurityGroupId": {
          "Ref": "SecurityGroupController"
        },
        "ToPort": 8472
      },
      "Type": "AWS::EC2::SecurityGroupIngress"
    },
    "SecurityGroupWorkerIngressFromFlannelToController": {
      "Properties": {
        "FromPort": 8472,
        "GroupId": {
          "Ref": "SecurityGroupController"
        },
        "IpProtocol": "udp",
        "SourceSecurityGroupId": {
          "Ref": "SecurityGroupWorker"
        },
        "ToPort": 8472
      },
      "Type": "AWS::EC2::SecurityGroupIngress"
    },
    "SecurityGroupWorkerIngressFromControllerToKubelet": {
      "Properties": {
        "FromPort": 10250,
        "GroupId": {
          "Ref": "SecurityGroupWorker"
        },
        "IpProtocol": "tcp",
        "SourceSecurityGroupId": {
          "Ref": "SecurityGroupController"
        },
        "ToPort": 10250
      },
      "Type": "AWS::EC2::SecurityGroupIngress"
    },
    "SecurityGroupWorkerIngressFromControllerTocAdvisor": {
      "Properties": {
        "FromPort": 4194,
        "GroupId": {
          "Ref": "SecurityGroupWorker"
        },
        "IpProtocol": "tcp",
        "SourceSecurityGroupId": {
          "Ref": "SecurityGroupController"
        },
        "ToPort": 4194
      },
      "Type": "AWS::EC2::SecurityGroupIngress"
    },
    "SecurityGroupEtcdIngressFromControllerToEtcd": {
      "Properties": {
        "FromPort": 2379,
        "GroupId": {
          "Ref": "SecurityGroupEtcd"
        },
        "IpProtocol": "tcp",
        "SourceSecurityGroupId": {
          "Ref": "SecurityGroupController"
        },
        "ToPort": 2379
      },
      "Type": "AWS::EC2::SecurityGroupIngress"
    },
    "SecurityGroupEtcdIngressFromWorkerToEtcd": {
      "Properties": {
        "FromPort": 2379,
        "GroupId": {
          "Ref": "SecurityGroupEtcd"
        },
        "IpProtocol": "tcp",
        "SourceSecurityGroupId": {
          "Ref": "SecurityGroupWorker"
        },
        "ToPort": 2379
      },
      "Type": "AWS::EC2::SecurityGroupIngress"
    },
    "SecurityGroupWorkerIngressFromWorkerToFlannel": {
      "Properties": {
        "FromPort": 8472,
        "GroupId": {
          "Ref": "SecurityGroupWorker"
        },
        "IpProtocol": "udp",
        "SourceSecurityGroupId": {
          "Ref": "SecurityGroupWorker"
        },
        "ToPort": 8472
      },
      "Type": "AWS::EC2::SecurityGroupIngress"
    },
    "SecurityGroupWorkerIngressFromWorkerToWorkerKubeletReadOnly": {
      "Properties": {
        "FromPort": 10255,
        "GroupId": {
          "Ref": "SecurityGroupWorker"
        },
        "IpProtocol": "tcp",
        "SourceSecurityGroupId": {
          "Ref": "SecurityGroupWorker"
        },
        "ToPort": 10255
      },
      "Type": "AWS::EC2::SecurityGroupIngress"
    },
    "SecurityGroupWorkerIngressFromWorkerToControllerKubeletReadOnly": {
      "Properties": {
        "FromPort": 10255,
        "GroupId": {
          "Ref": "SecurityGroupController"
        },
        "IpProtocol": "tcp",
        "SourceSecurityGroupId": {
          "Ref": "SecurityGroupWorker"
        },
        "ToPort": 10255
      },
      "Type": "AWS::EC2::SecurityGroupIngress"
    },
    "SecurityGroupEtcd": {
      "Properties": {
        "GroupDescription": {
          "Ref": "AWS::StackName"
        },
        "SecurityGroupEgress": [
          {
            "CidrIp": "0.0.0.0/0",
            "FromPort": 0,
            "IpProtocol": "tcp",
            "ToPort": 65535
          },
          {
            "CidrIp": "0.0.0.0/0",
            "FromPort": 0,
            "IpProtocol": "udp",
            "ToPort": 65535
          }
        ],
        "SecurityGroupIngress": [
          {{ range $_, $r := $.SSHAccessAllowedSourceCIDRs -}}
          {
            "CidrIp": "{{$r}}",
            "FromPort": 22,
            "IpProtocol": "tcp",
            "ToPort": 22
          },
          {{end -}}
          {
            "CidrIp": "0.0.0.0/0",
            "FromPort": 3,
            "IpProtocol": "icmp",
            "ToPort": -1
          }
        ],
        "Tags": [
          {
            "Key": "Name",
            "Value": "{{$.ClusterName}}-sg-etcd"
          }
        ],
        "VpcId": {{.VPCRef}}
      },
      "Type": "AWS::EC2::SecurityGroup"
    },
    "SecurityGroupEtcdPeerHealthCheckIngress": {
      "Properties": {
        "FromPort": 2379,
        "GroupId": {
          "Ref": "SecurityGroupEtcd"
        },
        "IpProtocol": "tcp",
        "SourceSecurityGroupId": {
          "Ref": "SecurityGroupEtcd"
        },
        "ToPort": 2379
      },
      "Type": "AWS::EC2::SecurityGroupIngress"
    },
    "SecurityGroupEtcdPeerIngress": {
      "Properties": {
        "FromPort": 2380,
        "GroupId": {
          "Ref": "SecurityGroupEtcd"
        },
        "IpProtocol": "tcp",
        "SourceSecurityGroupId": {
          "Ref": "SecurityGroupEtcd"
        },
        "ToPort": 2380
      },
      "Type": "AWS::EC2::SecurityGroupIngress"
    }
    {{if or $.ElasticFileSystemID .SharedPersistentVolume}}
    ,
    "SecurityGroupMountTarget": {
      "Properties": {
        "GroupDescription": {
          "Ref": "AWS::StackName"
        },
        "SecurityGroupIngress": [
          {
            "SourceSecurityGroupId": { "Ref": "SecurityGroupWorker" },
            "FromPort": 2049,
            "IpProtocol": "tcp",
            "ToPort": 2049
          },
          {
            "SourceSecurityGroupId": { "Ref": "SecurityGroupController" },
            "FromPort": 2049,
            "IpProtocol": "tcp",
            "ToPort": 2049
          }
        ],
        "Tags": [
          {
            "Key": "Name",
            "Value": "{{$.ClusterName}}-sg-mount-target"
          }
        ],
        "VpcId": {{.VPCRef}}
      },
      "Type": "AWS::EC2::SecurityGroup"
    }
    {{ if .SharedPersistentVolume }}
    ,
      "FileSystemCustom": {
        "Type": "AWS::EFS::FileSystem",
        "Properties": {
          "PerformanceMode": "maxIO",
          "FileSystemTags": [
            {
              "Key": "Name",
              "Value": "SharedData"
            }
          ]
        }
      }
      {{range $index, $subnet := .Subnets}}
      ,
      "{{$subnet.LogicalName}}MountTargetCustom": {
        "Properties" : {
          "FileSystemId": { "Ref": "FileSystemCustom" },
          "SubnetId": {{$subnet.Ref}},
          "SecurityGroups": [ { "Ref": "SecurityGroupMountTarget" } ]
        },
        "Type" : "AWS::EFS::MountTarget"
      }
      {{end}}
    {{end}}
    {{end}}

    {{range $index, $subnet := .Subnets}}
    {{if $subnet.ManageSubnet}}
    ,
    "{{$subnet.LogicalName}}": {
      "Properties": {
        "AvailabilityZone": "{{$subnet.AvailabilityZone}}",
        "CidrBlock": "{{$subnet.InstanceCIDR}}",
        "MapPublicIpOnLaunch": {{$subnet.MapPublicIPs}},
        "Tags": [
          {
            "Key": "Name",
            "Value": "{{$.ClusterName}}-{{$subnet.LogicalName}}"
          }
        ],
        "VpcId": {{$.VPCRef}}
      },
      "Type": "AWS::EC2::Subnet"
    }
    ,
    "{{$subnet.LogicalName}}RouteTableAssociation": {
      "Properties": {
        "RouteTableId": {{$subnet.RouteTableRef}},
        "SubnetId": {{$subnet.Ref}}
      },
      "Type": "AWS::EC2::SubnetRouteTableAssociation"
    }
    {{if $subnet.ManageRouteTable}}
    ,
    "{{$subnet.RouteTableLogicalName}}": {
      "Properties": {
        "Tags": [
          {
            "Key": "Name",
            "Value": "{{$.ClusterName}}-{{$subnet.RouteTableLogicalName}}"
          }
        ],
        "VpcId": {{$.VPCRef}}
      },
      "Type": "AWS::EC2::RouteTable"
    }
    {{end}}
    {{if $.ElasticFileSystemID}}
    ,
    "{{$subnet.LogicalName}}MountTarget": {
      "Properties" : {
        "FileSystemId": "{{$.ElasticFileSystemID}}",
        "SubnetId": {{$subnet.Ref}},
        "SecurityGroups": [ { "Ref": "SecurityGroupMountTarget" } ]
      },
      "Type" : "AWS::EFS::MountTarget"
    }
    {{end}}
    {{if $subnet.ManageRouteToInternet}}
    ,
    "{{$subnet.InternetGatewayRouteLogicalName}}": {
      "Properties": {
        "DestinationCidrBlock": "0.0.0.0/0",
        "GatewayId": {{$.InternetGatewayRef}},
        "RouteTableId": {{$subnet.RouteTableRef}}
      },
      "Type": "AWS::EC2::Route"
    }
    {{end}}
    {{end}}
    {{end}}

    {{range $i, $ngw := .NATGateways}}
    {{if $ngw.ManageEIP}}
    ,
    "{{$ngw.EIPLogicalName}}": {
      "Properties": {
        "Domain": "vpc"
      },
      "Type": "AWS::EC2::EIP"
    }
    {{end}}
    {{if $ngw.ManageNATGateway}}
    ,
    "{{$ngw.LogicalName}}": {
      "Properties": {
        "AllocationId": {{$ngw.EIPAllocationIDRef}},
        "SubnetId": {{$ngw.PublicSubnetRef}}
      },
      "Type": "AWS::EC2::NatGateway"
    }
    {{end}}
    {{range $_, $s := $ngw.PrivateSubnets}}
    {{if $s.ManageRouteToNATGateway}}
    ,
    "{{$s.NATGatewayRouteLogicalName}}": {
      "Properties": {
        "DestinationCidrBlock": "0.0.0.0/0",
        "NatGatewayId": {{$ngw.Ref}},
        "RouteTableId": {{$s.RouteTableRef}}
      },
      "Type": "AWS::EC2::Route"
    }
    {{end}}
    {{end}}
    {{end}}

    {{if not .VPCID}}
    ,
    "{{.InternetGatewayLogicalName}}": {
      "Properties": {
        "Tags": [
          {
            "Key": "Name",
            "Value": "{{$.ClusterName}}-{{.InternetGatewayLogicalName}}"
          }
        ]
      },
      "Type": "AWS::EC2::InternetGateway"
    }
    ,
    "{{.VPCLogicalName}}": {
      "Properties": {
        "CidrBlock": "{{.VPCCIDR}}",
        "EnableDnsHostnames": true,
        "EnableDnsSupport": true,
        "InstanceTenancy": "default",
        "Tags": [
          {
            "Key": "kubernetes.io/cluster/{{.ClusterName}}",
            "Value": "true"
          },
          {
            "Key": "Name",
            "Value": "{{.ClusterName}}-vpc"
          }
        ]
      },
      "Type": "AWS::EC2::VPC"
    },
    "VPCGatewayAttachment": {
      "Properties": {
        "InternetGatewayId": {{.InternetGatewayRef}},
        "VpcId": {{.VPCRef}}
      },
      "Type": "AWS::EC2::VPCGatewayAttachment"
    }
    {{end}}
  },

  "Outputs": {
    {{if not .VPCID}}
    "VPC" : {
      "Description" : "The VPC managed by this stack",
      "Value" :  { "Ref" : "{{.VPCLogicalName}}" },
      "Export" : { "Name" : {"Fn::Sub": "${AWS::StackName}-VPC" }}
    },
    {{end}}
    {{range $index, $subnet := .Subnets}}
    {{if $subnet.ManageRouteTable}}
    "{{$subnet.RouteTableLogicalName}}" : {
      "Description" : "The route table assigned to the subnet {{$subnet.LogicalName}}",
      "Value" :  {{$subnet.RouteTableRef}},
      "Export" : { "Name" : {"Fn::Sub": "${AWS::StackName}-{{$subnet.RouteTableLogicalName}}" }}
    },
    {{end}}
    {{if $subnet.ManageSubnet}}
    "{{$subnet.LogicalName}}" : {
      "Description" : "The subnet id of {{$subnet.LogicalName}}",
      "Value" :  {{$subnet.Ref}},
      "Export" : { "Name" : {"Fn::Sub": "${AWS::StackName}-{{$subnet.LogicalName}}" }}
    },
    {{end}}
    {{end}}
    {{range $index, $etcdInstance := $.EtcdNodes}}
    {{if $etcdInstance.EIPManaged}}
    "{{$etcdInstance.EIPLogicalName}}": {
      "Description": "The EIP for etcd node {{$index}}",
      "Value": {{$etcdInstance.EIPRef}},
      "Export": { "Name" : {"Fn::Sub": "${AWS::StackName}-{{$etcdInstance.EIPLogicalName}}" }}
    },
    {{end}}
    {{if $etcdInstance.NetworkInterfaceManaged}}
    "{{$etcdInstance.NetworkInterfacePrivateIPLogicalName}}": {
      "Description": "The private IP for etcd node {{$index}}",
      "Value": {{$etcdInstance.NetworkInterfacePrivateIPRef}},
      "Export": { "Name" : {"Fn::Sub": "${AWS::StackName}-{{$etcdInstance.NetworkInterfacePrivateIPLogicalName}}" }}
    },
    {{end}}
    {{end}}
    {{ if not .Controller.IAMConfig.InstanceProfile.Arn }}
    "ControllerIAMRoleArn": {
      "Description": "The ARN of the IAM role for Controllers",
      "Value": { "Fn::GetAtt": ["IAMRoleController", "Arn"] },
      "Export": { "Name": { "Fn::Sub": "${AWS::StackName}-ControllerIAMRoleArn" } }
    },
    {{end}}
    "WorkerSecurityGroup" : {
      "Description" : "The security group assigned to worker nodes",
      "Value" :  { "Ref" : "SecurityGroupWorker" },
      "Export" : { "Name" : {"Fn::Sub": "${AWS::StackName}-WorkerSecurityGroup" }}
    },
    "StackName": {
      "Description": "The name of this stack which is used by node pool stacks to import outputs from this stack",
      "Value": { "Ref": "AWS::StackName" }
    }
  }
}
```

---

## デザインゴール#3: カスタマイズ性

* 現在:
    * ノードの設定はyamlやcloud-configで自由にカスタマイズできる
        * 例: workerノードでは特定のsystemd unitをインストールしたい
    * インフラ構成のよくあるカスタマイズはyamlで完結
    * それで完結しないことは、CloudFormationスタックテンプレートを直接書き換えることで可能
        * CloudFormationでできるカスタマイズはなんでもできる!
* 将来
    * kube-awsプラグインで、インフラ構成・ノード設定を自由にカスタマイズ
        * 例: GPUプラグインをONにしたら、NVIDIAのドライバを自動的にビルドしてノードにインストール

---

## まとめ

* ステートレスなアプリケーションをAWS上の本場環境で簡単にホストしたいときにはkube-awsがおすすめ
* AWSのマネージドサービスとKubernetesのいいとこ取りをしよう

twitter/github/slack.k8s.io: @mumoshu (むもしゅ)
Primary maintainer of [kubernetes-incubator/kube-aws](https://github.com/kubernetes-incubator/kube-aws)

Fin :bow: :clap: 

---

## リンク集

* [Highly available and scalable Kubernetes on AWS #builderscon - Qiita](http://qiita.com/mumoshu/items/922a36dec039e8d230f9)
* [Highly available and scalable Kubernetes on AWS for Kubernetes users 日本語版 - Qiita](http://qiita.com/mumoshu/items/a084661b4fe2c496b5f1)