Highly available and scalable Kubernetes on AWS #builderscon

# Table of contents

1. A bit of context
2. About this presentation
3. Overview of Kubernetes on AWS
4. Detailed guide for kube-aws

---
# A bit of context

* 　Kubernetes :point_left: 
* 　AWS
* 　Why Kubernetes on AWS
* 　Why they matter to you

---
## :shell: Kubernetes in a nutshell

> An open-source system for automating deployment, scaling, and management of *containerized applications*.
> :link: http://kubernetes.io/

* Hosted at :link: https://github.com/kubernetes/kubernetes

---
## :shell: Kubernetes does

* Automatically keep your applications running according to your requirements
  * Types of an application: long-running/one-shot, stateful/stateless, standalone/distributed
  * Required resources like CPU, memory, storage, machine, network capability, permissions, etc.

---
## :shell: Kubernetes does

* Provides APIs to CRUD desired states of your applications
* Converges your applications to their desired states

---
### Kubernetes Alternatives

* Docker Swarm
* Hashicorp Nomad
* Kontena
* AWS ECS(EC2 Container Service)
* Mesos
* etc.

---
### 3 catchphrases of Kubernetes

- Planet Scale
- Never Outgrow
- Runs Anywhere

---
#### :globe_with_meridians: "Planet Scale" with Kubernetes

> Designed on the same principles that allows Google to run billions of containers a week
:link: http://kubernetes.io/

* [Tested with 2,000 nodes/60,000 pods each release](http://blog.kubernetes.io/2016/07/kubernetes-updates-to-performance-and-scalability-in-1.3.html)

---
#### :lifter: You "Never Outgrow" Kubernetes

> Whether testing locally or running a global enterprise, Kubernetes flexibility grows with you to deliver your applications consistently and easily no matter how complex your need is
:link: http://kubernetes.io/

---
##### How we can never outgrow?

* Model your application in Kubernetes way:
  * Containers, Pods, Jobs, ReplicaSets, Deployments, StatefulSets, etc.

---
##### Kubernetes Models 1/4
  * Containers = [cgroups + namespaces](http://www.slideshare.net/jpetazzo/anatomy-of-a-container-namespaces-cgroups-some-filesystem-magic-linuxcon) w/ copy-on-write
  * Pods = VMs in the container era w/ shared CPU, memory, volumes, permissions

---
##### Kubernetes Models 2/4
  * [Services](http://kubernetes.io/docs/user-guide/services/) = Named sets of pods load balanced via kubernetes internal DNS

![image](https://qiita-image-store.s3.amazonaws.com/0/5912/02ad5573-9eb1-535b-78c6-6594753fdb81.png)

---
##### Kubernetes Models 3/4

  * [Jobs](http://kubernetes.io/docs/user-guide/jobs/) = One-shot or long-running jobs backed by pod(s), with optional parallelism
  * [ReplicaSets](http://kubernetes.io/docs/user-guide/replicasets/) = A set of identical pods with its number maintained

---
##### Kubernetes Models 4/4
  * [Deployments](http://kubernetes.io/docs/user-guide/deployments/) = Desired state of your ReplicaSet
  * [StatefulSets](http://kubernetes.io/docs/user-guide/petset/) = A set of named, persistent pods to host e.g. stateful distributed system like databases, filesystems

---
#### :airplane: Kubernetes "Runs Anywhere"

> Kubernetes is open source giving you the freedom to take advantage of on-premise, hybrid, or public cloud infrastructure, letting you effortlessly move workloads to where it matters to you.
:link: http://kubernetes.io/

----
##### Anywhere = ?

* Run your Kubernetes locally for development with [minikube](https://github.com/kubernetes/minikube)
* On IaaSes including Baremetal, GCP, AWS, OpenStack, etc.
* On Ubuntu, CentOS, CoreOS, etc.

---
## Context: AWS

* :ballot_box_with_check: Kubernetes
* 　AWS :point_left: 
* 　Why Kubernetes on AWS
* 　Why they matter to you

---
### AWS is a leader in IaaS

:link: [AWS named as a leader in the Infrastructure as a Service (IaaS) Magic Quadrant report for 6th consecutive year](https://aws.amazon.com/resources/gartner-2016-mq-learn-more/?nc1=h_ls)

---
### AWS is IaaS

* VPS
* IaaS :point_left: 
* PaaS
* SaaS

---
#### :computer: VPS

* Virtual Private Server
* Equipped with a common set of services(including installed softwares, public IPs, storages, CPUs, memories, etc.)
* e.g. DigitalOcean, さくらのVPS

---
#### :gear: IaaS

* Infrastructure as a Service
* Serves you a series of resources and APIs to build your own virtual data centers
* e.g. Azure, GCP, AWS, etc

---
#### :station: PaaS

* Platform as a Service
* Serves you a platform to run your application without taking care of the infrastructure
* e.g. Heroku, Google App Engine

---
#### :fork_knife_plate: SaaS

* Software as a Service
* Serves you software(s) or managed applications to assists part of your business(es)

---
## Context: Why Kubernetes on AWS

* :ballot_box_with_check: Kubernetes
* :ballot_box_with_check: AWS
* 　Why Kubernetes on AWS :point_left: 
* 　Why they matter to you

---
### Recap: What is Kubernetes?

* Kubernetes is all about how to run your application in an unified way regardless of where it runs: from home to a datacenter, from a laptop to a rack of blade servers

---
### Kubernetes on What?

* VPS can be used to run Kubernetes
* There're PaaSes to host your Kubernetes clusters
* There're SaaSes hosted by Kubernetes
* There're IaaSes provides foundations to build/host your Kubernetes clusters

---
### So: Why Kubernetes on IaaS?

* IaaS allow us to build our own PaaS based on Kubernetes to host our SaaS according to our requirements

---
## Context: Why they matter to you

* :ballot_box_with_check: Kubernetes
* :ballot_box_with_check: AWS
* :ballot_box_with_check: Why Kubernetes on AWS
* 　Why they matter to you :point_left: 

---
### :ticket: builderscon is

* IMHO [builderscon](https://builderscon.io/builderscon/tokyo/2016) is almost a conference for programmers.
* Why all those things matter to us programmers?

---
### From a programmer's viewpoint: API

* Kubernetes provides a rich set of APIs to manage Kubernetes resources
* AWS provides a rich set of APIs to manage AWS resources

---
### From a programmer's viewpoint: Kubernetes and AWS

* Kubernetes is a just framework to program how my applications are managed
* AWS is a just framework to program how my Kubernetes clusters are managed

---
### :keyboard: I'd like to program apps, not manage them

* Kubernetes allows us to focus on programming
* Also: Kubernetes and its surrounding ecosystem themselves would be interesting to be programmed :wink:

---
## :ok_woman: Now, you understand the context

* :ballot_box_with_check: Kubernetes
* :ballot_box_with_check: AWS
* :ballot_box_with_check: Why they matter to you

Let's keep going :muscle: 

---
# Table of contents

1. A bit of context
2. About this presentation :point_left: 
3. Overview of Kubernetes on AWS
4. Detailed guide for kube-aws

---
# About this presentation

* Topics explained in this presentation
* Questions answered in this presentation

---
## :information_source: Topics explained in this presentation

* Overview of *Kubernetes on AWS*
* Detailed guide for kube-aws
* Related personal experience(s) from the presenter

---
## :question: Questions answered in this presentation

* How to achieve high scalability w/ Kubernetes on AWS?
* How to achieve high availability w/ Kubernetes on AWS?
* Are there any tool(s) to deploy and manage Kubernetes clusters?
* How to select tool(s) for your use?
  * Requirements which arises frequently today
  * What is kube-aws? How to use it?
* What's my use-case?
  * Requirements
  * Tools

---
# Table of contents

1. A bit of context
2. About this presentation
3. Overview of Kubernetes on AWS :point_left: 
4. Detailed guide for kube-aws

---
# Overview of *Kubernetes on AWS*

* How to achieve high availability and scalability
* Tools
* How to select those tools

---
## High availability with Kubernetes on AWS

* High availability in nutshell
* How to achieve it

---
### :shell: High availability in nutshell

Entirely/Almost no downtime when a expected/unexpected shutdown of 1 or more nodes and pods

* Pods = Sets of containers running on EC2 instances
* Nodes = EC2 instances

---
### High availability and Redundancy

* Pods = Sets of containers
  * 2 or more identical pods
  * ReplicaSets in Kubernetes
* Nodes = EC2 instances
  * 2 or more nodes usable to host the same set of pods
  * Auto Scaling Group and/or Spot Fleet in AWS

---
## High scalability with Kubernetes on AWS

* High scalability in nutshell
* How to achieve it

---
### :shell: High scalability in nutshell

No redundant cost + No user visible frustration in less/more workload

* Pods = Sets of containers running on EC2 instances
* Nodes = EC2 instances

Cost = Number of nodes * unit price
Number of nodes = Total resources required by pods

---
### High scalability for pods

* Pods = Sets of containers
  * Less pods when less workload, more pods when more workload
  * Kubernetes's [Horizontal Pod Autoscaler](http://kubernetes.io/docs/user-guide/horizontal-pod-autoscaling/)
  * [kube-sqs-autoscaler](https://github.com/Wattpad/kube-sqs-autoscaler) from Wattpad

---
### High scalability for nodes

* Nodes = EC2 instances
  * Less nodes when less pods, more pods when more workload
  * Kubernetes's [cluster-autoscaler](https://github.com/kubernetes/contrib/tree/master/cluster-autoscaler)
     * Adds nodes to meet capacity enough for scheduling all the pending pods
  * AWS Auto Scaling
     * Adds/removes nodes according to CPU and/or memory usage

---
## :tools: Are there any tool(s) to deploy and manage Kubernetes clusters?

* Yes,
* A lot
  * klondike, kubernetes-anywhere, kops, kube-aws, kope, kube-cluster, kubeadm-aws, halcyon-kubernetes, tack, etc.
* :bookmark: [An overview of those tools](http://qiita.com/mumoshu/items/72ae0f07a58a37626204)

---
## :question: How to select tool(s) for your use?

Selecting a tool is all about:

* knowing your requirements,
* investing in tools, and then
* connecting them

---
## Knowing your requirements

Requirements which aries frequently today:

* Highly available
* Highly scalable
* Highly secure
* Be locked in to technologies give you higher leverage

---
### Lock-in to what gives you higher leverage?

* An IaaS frequently reduce pricing
* An IaaS frequently add new features, APIs, SDKs, etc., which are ready to be used for your business
* // AWS for me

---
# Table of contents

1. A bit of context
2. About this presentation
3. Overview of Kubernetes on AWS
4. Detailed guide for kube-aws :point_left: 

---
# Detailed guide for kube-aws

* For all:
  * kube-aws in a nutshell
* For users:
  * Usage guide for kube-aws
* For developers:
  * Development guide for kube-aws
* For potential contributors:
  * Contribution guide for kube-aws

---
## :shell: kube-aws in a nutshell
* A single binary to create/update/destroy Kubernetes cluster(s) on AWS
* Infrastructure as Code achieved via AWS CloudFormation + Golang
* Originally developed by CoreOS, currently maintained by community
https://github.com/coreos/kube-aws

---
### :hamburger: Under the hood of kube-aws
* Standard Golang application based on Golang 1.7, glide, cobra, etc., coupled w/
* CoreOS, which has official support for running Kubernetes
* Kubernetes 1.4.x, upstream releases followed timely
* Various AWS services
   * VPC, EC2, ELB, Route53, KMS, CloudFormation

---
### :no_entry: kube-aws does not use
* Terraform
* Ansible
* Chef, Puppet, Saltstack, etc.

---
## Usage guide for kube-aws
Create your first Kubernetes cluster on AWS :smile: 
* Install `kube-aws`
* Generate `cluster.yaml`
* Customize your cluster
* Render the contents of the assets directory 
* Validate your configuration
* Updating the cluster without downtime
* Destroying the cluster

---
### Creating your first Kubernetes cluster on AWS
  * Create keys
  * Install the latest version of `kube-aws` command
  * Customizing the cluster

---
#### Prepare keys
* EC2 Keypair
  * Used to authenticate SSH access to your server
* KMS key
  * Used to encrypt/decrypt TLS assets generated by kube-aws

---
#### Install the `kube-aws` command

```bash
$ platform=darwin-amd64
$ version=v0.9.2-rc.1
$ curl -L https://github.com/coreos/kube-aws/releases/download/${version}/kube-aws-${platform}.tar.gz | tar zxv ${platform}/kube-aws && \
  mv ${platform}/kube-aws /usr/local/bin/kube-aws && \
  chmod +x /usr/local/bin/kube-aws
$ kube-aws version
kube-aws version v0.9.2-rc.1
```
:link: https://github.com/coreos/kube-aws/releases

---
### Customizing the cluster

Generate a `cluster.yaml`:

```
$ mkdir my-cluster
$ cd my-cluster

$ kube-aws init \
--cluster-name=my-cluster-name \
--external-dns-name=my-cluster-endpoint \
--region=us-west-1 \
--availability-zone=us-west-1c \
--key-name=key-pair-name \
--kms-key-arn="arn:aws:kms:us-west-1:xxxxxxxxxx:key/xxxxxxxxxxxxxxxxxxx"
```

---
### Customizing the cluster
cluster.yaml:

```yaml
clusterName: my-cluster-name
externalDNSName: my-cluster-name-api.my.domain
releaseChannel: stable
createRecordSet: true
recordSetTTL: 300
hostedZoneId: "XASDASDASD" # DEV private only route53 zone
keyName: my-ssh-keypair
region: ap-northeast-1
availabilityZone: ap-northeast-1c
kmsKeyArn: "arn:aws:kms:ap-northeast-1:0w123456789:key/d345fcd1-c77c-4fca-acdc-asdasdf3234232"
controllerCount: 1
controllerInstanceType: m3.medium
controllerRootVolumeSize: 30
controllerRootVolumeType: gp2
workerCount: 1
workerInstanceType: m3.medium
workerRootVolumeSize: 30
workerRootVolumeType: gp2
etcdCount: 1
etcdInstanceType: m3.medium
etcdRootVolumeSize: 30
etcdDataVolumeSize: 30
vpcId: vpc-xxxccc45
routeTableId: "rtb-aaabbb12" # main external no NAT uses internet gateway
vpcCIDR: "10.1.0.0/16"
instanceCIDR: "10.1.10.0/24"
serviceCIDR: "10.3.0.0/24"
podCIDR: "10.2.0.0/16"
dnsServiceIP: 10.3.0.10
stackTags:
  Name: "my-cluster-name"
  Environment: "development"
```

---
### Render the contents of the assets directory
```
$ kube-aws render
$ tree
.
├── cluster.yaml
├── credentials
│   ├── admin-key.pem
│   ├── admin.pem
│   ├── apiserver-key.pem
│   ├── apiserver.pem
│   ├── ca-key.pem
│   ├── ca.pem
│   ├── worker-key.pem
│   └── worker.pem
│   ├── etcd-key.pem
│   └── etcd.pem
│   ├── etcd-client-key.pem
│   └── etcd-client.pem
├── kubeconfig
├── stack-template.json
└── userdata
    ├── cloud-config-controller
    └── cloud-config-worker
```

---
#### Credential files
* `admin*.pem`, used on your local machine and/or build servers
* `worker*.pem`, used in worker nodes
* `apiserver*.pem`, used in controller nodes
* `etcd-client*pem`, used in worker and controller nodes
* `etcd*pem`, used in etcd nodes

---
#### Credential file naming

* `foo-key.pem`
  * The private key for `foo` access
* `foo.pem`
  * THe public key for `foo` access

---
#### That's a lot!

* Every little mistake in configuration would result in non-functional clusters
* That's why we need `kube-aws validate` :smile: 

---
### Validate your configuration
```
$ kube-aws validate
```
* Fails only when there're syntax and/or semantic errors in your configuration files
  * Especially in:
     * cloud-config
     * CloudFormation templates
     * No need to wait for minutes until you find easy mistakes like typos :wink: 

---
#### It is just "validating"

* `kube-aws validate` doesn't(can't) test functional correctness of the resulting Kubernetes cluster
* It is covered by our End-to-end testing

---
### Launch your cluster
```
$ kube-aws up
Creating AWS resources. This should take around 5 minutes.
Success! Your AWS resources have been created:
Cluster Name:  	my-cluster-name

The containers that power your cluster are now being downloaded.

You should be able to access the Kubernetes API once the containers finish downloading.
```

---
### Or just export your cluster
```
$ kube-aws up --export
```
* Generates a CloudFormation stack template
* A template is a JSON file which defines which AWS resources to create and how they're connected.

---
### Destroy your cluster
```
$ kube-aws destroy
```

---
## Development guide for kube-aws
* Understanding how kube-aws is built
* Build your own `kube-aws`
* Make targets
* End-to-end testing in kube-aws

---
### Understanding how kube-aws is built
* golang 1.7.x
* glide for package management
  * Alternatives: Godep, trash, govendor, etc.
* `go gen` + `go run` for code generation
    * To embed the default configuration files into kube-aws binaries
* `go test` for unit testing
* `gofmt` for code formatting and validation

---
### Build your own `kube-aws`

* Install required tools
* git clone
* make build

---
#### Installing required tools

```bash
brew update
```

```bash
brew install go
brew install glide
brew install make
```

----
#### Clone the repository to an appropriate path

```bash
mkdir -p $GOPATH/src/github.com/coreos/kube-aws
```

```
git clone git@github.com:coreos/kube-aws.git \
  $GOPATH/src/github.com/coreos/kube-aws
```
----
#### Build a `kube-aws` binary

```
$ make build
./build
Building kube-aws 102cb5b8c9d71af1a5074b77955c4913e7d0b84d

$ bin/kube-aws version
kube-aws version 102cb5b8c9d71af1a5074b77955c4913e7d0b84d
```

---
#### FYI: Make targets
* `make build` to build `bin/kube-aws`
* `make format` to automatically format all the code of kube-aws
* `make test` to run all the unit tests

---
### End-to-end testing in kube-aws
* Create a main cluster
* Create a node pool
* Update the main cluster
* Run Kubernetes conformance tests (:exclamation:)

----
#### Kubernetes conformance tests

* are a subset of Kubernetes E2E tests
* to confirm that a minimum feature set of functions required for a cluster to be a Kubernetes cluster is properly working

----
#### Kubernetes E2E tests

> End-to-end (e2e) tests for Kubernetes provide a mechanism to test end-to-end behavior of the system, and is the last signal to ensure end user operations match developer specifications.
> :link: [End to end testing in Kubernetes](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/e2e-tests.md)

----
#### Why E2E tests are needed

> Although unit and integration tests provide a good signal, in a distributed system like Kubernetes it is not uncommon that a minor change may pass all unit and integration tests, but cause unforeseen changes at the system level.
> :link: [End to end testing in Kubernetes](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/e2e-tests.md)

---
#### Running End-to-end tests
* `cd e2e && KUBE_AWS_CLUSTER_NAME=my-cluster-name ./run.sh all`

* It takes approximately 1 hour to run each test case :cry: 
* A good news is that we run it against multiple combinations of settings and options
* Before releasing new kube-aws releases
* For you

---
## Contribution guide for kube-aws

* Where to start: GitHub issues
* What are discussed and answered
* Please file your own issue(s)!

---
### Where to start

Our github issues page is the single place to start your contribution: 

https://github.com/coreos/kube-aws/issues

---
### Discussed and answered in github issues

* Feature Request
* Proposal
    * For possible features
    * For possible improvement(s) in the documentation
* Bug reports
* Questions

---
### What a proposal would look like

![image](https://qiita-image-store.s3.amazonaws.com/0/5912/c6a9a76b-378c-57f0-d5a6-c8896af18556.png)

---
### Please file your own issue(s)!

Every little feedback is welcomed to shape our goals and plans toward production-ready Kubernetes clusters on AWS

---
# Fin.

Thanks for your attention and interest on Kubernetes on AWS,

From @mumoshu, the primary maintainer of [kube-aws](https://github.com/coreos/kube-aws)