Kubernetes and Beyond Democratizing Workflow Automation

※本記事は[CloudNative Dyas Fukuoka 2019](https://cloudnativedays.jp/cndf2019/)の登壇スライドです

---


<img src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/5912/99c39fec-c27b-ccfe-edb4-b8555b3bd002.png" width="100%"/>


---

# Intro.

> ![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/5912/d22489f3-9fce-acf1-c943-2dc19c720e8e.png)
> 「クラウドは人間を強化する」
> "Cloud reinforces people"
>
> "飛び込もう、Cloud Nativeの世界", CloudNative Days Fukuoka 2019

---

- Life is too short to waste
- I don't want to write a **lot** of code to get things done

---

# Master Kelsey says...

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/5912/baeff70e-38df-3dba-f377-d4d4891a37f0.png)

https://twitter.com/kelseyhightower/status/935252923721793536

---

# Platform?

- Cloud?
- Serverless?
- PaaS?

---

# Platform?

- PaaS is abbrev of **Platform** as a Service
- Google Cloud **Platform** ?
- Cloud Foundry - Cloud Application **Platform**
- OpenShift - Container Application **Platform**
- Knative is often called as Serverless **Platform**  ?

Is **Platform** something to do with **Cloud**, **Serverless** , **Application**?

---

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/5912/77fd6652-72e0-9bc9-e9ce-d8474433aa44.png)


> [Cloud Programming Simplified: A Berkeley View on Serverless Computing](https://www2.eecs.berkeley.edu/Pubs/TechRpts/2019/EECS-2019-3.pdf)
> February 10, 2019

---

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/5912/969ee71a-1ab6-73b6-1161-e927521cf7a2.png)

---

- **Cloud**: Hardware abstracted away
- **Serverless**: = Cloud abstracted away
  - **Serverless Compute**: Cloud Computing abstracted away(Cloud Run, Lambda
  - **Serverless Storage**: Cloud Storage abstracted away(Spanner, Aurora-serverless
- **Container**: Industry-Standard Application Packaging Format
    - Lambda? Use [img2lambda](https://github.com/awslabs/aws-lambda-container-image-converter)
- **Application Platform**: Serverless abstracted away

And there are a bunch of **Applications**...

---

# Kinds of Applications

- **General Apps**: Project Mangement, Issue Tracking, Chat, E-mail, Spreadsheet, Web Form, Workflow Management
- **Developer Apps**: CI, CD, Observability, Security, (PaaS)

This situation is damaging our lives! (I think)

Why? How?

---

# Ex. Enterprise Application Development & Deployment Workflow 2019

- **Create Issue**: GitHub, Jira
- **Discuss**: GitHub, Jira, Slack
- **Write Design Doc**: GitHub + Markdown, Google Doc, MS Word
- **Write Code** (Editor on local machine or cloud IDE
- **Pull Request** (CLI, GitHub
- **Rewiew** (GitHub
- **Approve** (GitHub
- **Request Approval for Production Op**: Google Form/Your Int. Workflow System
- **Merge to Trigger Automation**: GitOps!
   - Automation happens on Kubernetes(Containerized scripts, workflows, services)
- **Browse Web UI of your CI**: CircleCI, TravisCI, Jenkins
- **Browse Web UI of your Observability Stack**: Kibana, Grafana, Datadog, Prometheus
- **Debug**: Terminal, Debugger, `kubectl`, See Logs
- :beers: 
- Repeat

---

# WHY

Are we forced to switch apps SO many times?

- GitHub
- Slack
- JIRA
- Google Form
- CI
- Observability Stack
- Terminal

---

# Can't we automate it?

---

# Kinds of Automation

- Domain-specific Automations
  - CI/CD, Monitoring, Log Collection&Aggregation
  - Operations for Cluster, Apps, Middlewares, Services, ...
- Kubernetes "Operator" Pattern
  - https://coreos.com/operators/
  - Pattern for building domain-specific automations
     - [postgres-operator](https://github.com/zalando/postgres-operator) and [es-operartor by Zalando](https://github.com/zalando-incubator/es-operator)

---

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/5912/fd036b64-08a5-a1c0-a18e-aec5f942aeae.png)


"Kubernetes Operators" - https://coreos.com/operators/

---

# Is "Operator" the only solution?

Do we need to build an "Enterprise Application Development & Deployment Workflow" operator? In golang?

---

Can't we start *small*, then think **BIG**?
--

---

# History

Back to the good old days of Linux

You've been used to write **shell scripts** to automate all the daunting tasks

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/5912/37c5ce4f-94c4-12ee-f466-6989d3629ce6.png)


---

> Shell scripts remain an optimum choice for enhancing software leverage.
> "Linux and the Unix Philosophy", Mike Gancarz, 2003

---

# The world got more complicated...

---

# Modern Environment(at glance)

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/5912/3c53a564-622d-df61-6461-7f682988fd89.png)


> [Microservice Architecture — Learn, Build, and Deploy Applications - DZone Microservices](https://dzone.com/articles/microservice-architecture-learn-build-and-deploy-a)

---

# Modern Environment(Reality)

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/5912/f44a7901-a898-e1a9-1074-a3ff8edefa4d.png)


> "Amazon"
> [Practicing Microservices](https://medium.com/@LachlanEvenson/practicing-microservices-ac43c8b3b712), Lachlan Evenson

---

- Distributed microservices...
- Inside and outside of your company...
- Connected with various protocols, transports, and network topology
- With varying assumptions on latencies, durability, etc...

Today, it isn't simple as scripting:

```bash
#!/usr/bin/env bash
inputs | map | reduce | present > output
```

---

# Enter Cloud Native

---

> **Cloud native** technologies **empower organizations to build and run scalable applications in modern, dynamic environments** such as public, private, and hybrid clouds. Containers, service meshes, microservices, immutable infrastructure, and declarative APIs exemplify this approach.
> https://github.com/cncf/toc/blob/master/DEFINITION.md

---

## Cloud Native

Does it help **scripting**, too!?

Let's see...

---

# OOT: Baseline

You have no difficulty **deploying** things to Kubernetes:

- `kubectl apply -f manifests/` to install a set of manifsts
- `kustomize build -k env/ | kubectl apply -f` to compose manifests to be applied
- `helm upgrade --install chart/myapp myapp` to upgrade/install a single app
- [helmfile apply](https://github.com/roboll/helmfile) to deploy all the apps

This talk is not about those, but rather about **scripting**.

---

# OOT: Another Stories

- *Continuous Integration*: Use of CircleCI, Concourse CI, Argo CI, Drone, Jenkins, ...
- *Continuous Delivery*: Argo CD, Weaveworks Flux, or whatever to automate builds and deployments
- Observability: Datadog, Prometheus, Grafana, Zipkin, Jaeger, ...

This talk is not about those, but rather about **scripting**.

---

# Hello World, Traditional Style

In old days we've been writing:

```bash:Script
echo hello world
```

```console:Output
hello world
```

---

# Hello World, Modern Style

`kubectl run` to run any command on Kubernetes:

```console:Script
kubectl run \
  # Interactive
  -i \
  # Run only once
  --restart=Never \
  # Run the command in an Alpine Linux container
  --image alpine:3.9 \
  # Gimme an unique ID of my command runtime...
  alpine-cmd-$(date +%s)-$(LC_CTYPE=C tr -dc a-z0-9 < /dev/urandom | fold -w 3 | head -n 1) \
  # This is the command
  -- /bin/sh -c "echo hello world"
```

```console:Output
hello world
```

---

# Hello World, Modern Style(Abstracted)

```console:Script
cat <<'EOS' > ~/bin/kuberun && chmod +x ~/bin/kuberun
#!/usr/bin/env bash
opts="-i"
if tty >/dev/null; then opts="${opts}t"; fi
kubectl run "$opts" --restart=Never --image alpine:3.9 alpine-cmd-$(date +%s)-$(LC_CTYPE=C tr -dc a-z0-9 < /dev/urandom | fold -w 3 | head -n 1) -- /bin/sh -c "$*"
EOS
```

```console:Script
kuberun echo hello world
```

```console:Output
hello world
```

---

# `kubectl run` - Curious How It Works?

See "What happens when ... Kubernetes edition!" for more info:

[github.com/jamiehannaford/what-happens-when-k8s](https://github.com/jamiehannaford/what-happens-when-k8s)

---

# Kubernetes-as-Shell solutions

- [kubectl-repl](https://github.com/Mikulas/kubectl-repl) runs shell with short aliases to various `kubectl` commands. no filc sync
- [kubed-sh](https://github.com/mhausenblas/kubed-sh) runs `ruby`, `node`, `python` in containers, with periodic local-to-remote file sync

---

Good.

We can now run any command on Kubernetes.

---

But how to run multiple commands in parallel?

---

# Run commands on Kubernetes(Parallel)

```console:Script
kuberun echo hello & kuberun echo world & wait
```

---

You can even go further...

```bash:Script
i=0; for c in h e l l o w o r l d; do
  ((i+=1)); bash -c "sleep $i; kuberun echo $c" &
done
```

```console:Output
h
e
l
l
o
w
o
r
l
d
```

---

# Gotcha

- Your containers don't see your local files(of course!)

---

# Problem of "Syncing Files"

Use-cases:

- `Local-to-Container`: Run local code in container(e.g. Webapp dev.)
- `Container-to-Local`: Build binary in container, download the result
- `Container-to-Container` Use-case: Run workflow across containers(e.g. CI/CD pipeline)

---

# "Sync Files" Solutions

- `kubectl cp`
- `rsync`, [kubectl-warp](https://github.com/ernoaapa/kubectl-warp)
- Cloud Object Storage(e.g. AWS S3)
- `ReadWriteOnce` volumes (Block Storage incl. AWS EBS, Ceph Block Device/RBD)
- `ReadWriteMany` volumes (NFS, AzureFS, CephFS, AWS EFS)

---

# Introducing kubectl-warp

- Basically `kubectl run` + initial/periodic `rsync`(but no final rsync)

- Runs container with local-to-remote file sync(no vice versa though)

- `rsync` for point-to-point file sync before and while running the command

---

```console:kubewarp
cat <<'EOS' > ~/bin/kubewarp && chmod +x ~/bin/kubewarp
#!/usr/bin/env bash
inc="$1"
shift
opts="-i"
if tty >/dev/null; then opts="${opts}t"; fi
kubectl warp "$opts" --include "$inc" --image alpine:3.9 alpine-cmd-$(date +%s)-$(LC_CTYPE=C tr -dc a-z0-9 < /dev/urandom | fold -w 3 | head -n 1) -- /bin/sh -c "$*"
EOS
```

```console:Local
$ ls
hello.txt
```

```console:Kubernetes
$ kubewarp ls
hello.txt
```
---

# Gotcha

- Your local machine doesn't see your remote files!

---

# kubectl-biwarp

`kube-warp` + reverse `rsync` before stopping container

https://github.com/mumoshu/kubectl-biwarp

---

# Concurrency + Aggregate Results

```
$ seq 1 5 | xargs -P5 -n1 -I{} kubebiwarp "hello.{}.txt" "echo hello {} > hello.{}.txt" ; wait
```

```
$ cat hello.*.txt
hello 1
hello 2
hello 3
hello 4
hello 5
```

---

Good.

We can now run **simple stateful workflow** on Kubernetes.

---

# Shell Scripting Gotchas

- **No defacto package manager** (I like bpkg. But did you know..?
- **Not so portable**(Windows?
- **Testing**(bats?

---

## Can't we "graduate" to a real programming language?

---

# JavaScript?

**Package Manager**: `npm` or `yarn`

**Portability**: Better than bash! :)

**Testing**: Standard node.js tools

---

## There is...

A CNCF Sandbox Project that helps **Scripting** on Kubernetes

---

# Introducing Brigade

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/5912/0637c12f-176e-c95c-07d3-a9f2917c6170.png)


---

# Brigade

Use-cases:

- Watch for incoming requests from services like GitHub, Docker, and Trello
- Run unit tests, process data, and store results
- Send notifications through services like Slack and Twitter

Anything that can be scripted in node and shell

---

# Automating unit tests on `git push` with Brigade

1. Write some script:

```js
// Run unit tests for a Github push
const { events, Job , Group} = require("brigadier");
const dest = "$GOPATH/src/github.com/technosophos/ulid";

events.on("push", (e, p) => {
  console.log(e.payload)
  var gh = JSON.parse(e.payload)
  var test = new Job("test", "golang:1.9")
  test.tasks = [
    "mkdir -p " + dest,
    "cp -a /src/* " + dest,
    "cd " + dest,
    "go get -u github.com/golang/dep/cmd/dep",
    "dep ensure",
    "make test"
  ];
  test.run()
});
```

---

2. Brigade turns the script into...

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/5912/c462b675-a7c7-9a0a-0274-c2fd4a798c5f.png)


Workflow composed of containers!

---

# Brigade's Abstraction

**Brigade Script**: Dynamic Workflow of Jobs

**Job**: Containerized Shell Scripts

---

# Testing

[technosophos/brigtest](https://github.com/technosophos/brigtest) for testing brigade scripts:

```console
npm run brigtest -c event.json
```

---

# Back to the topic...

- Life is too short to waste. I don't want to write a lot of code to get things done

- Does kubectl, shell and brigade solve our problem as whole?

No (yet)

---

# Re: Enterprise Application Development & Deployment Workflow 2019

- **Create Issue**: GitHub, Jira
- **Discuss**: GitHub, Jira, Slack
- **Write Design Doc**: GitHub + Markdown, Google Doc, MS Word
- **Write Code** (Editor on local machine or cloud IDE
- **Pull Request** (CLI, GitHub
- **Rewiew** (GitHub
- **Approve** (GitHub
- **Request Approval for Production Op**: Google Form/Your Int. Workflow System
- **Merge to Trigger Automation**: GitOps!
   - Automation happens on Kubernetes(Containerized scripts, workflows, services)
- **Browse Web UI of your CI**: CircleCI, TravisCI, Jenkins
- **Browse Web UI of your Observability Stack**: Kibana, Grafana, Datadog, Prometheus
- **Debug**: Terminal, Debugger, `kubectl`, See Logs
- :beers: 
- Repeat

---

# Why are we forced to switch apps so many times?

- GitHub
- Slack
- **JIRA**
- **Google Form**
- ~~CI~~
- **Observability Stack**
- **Terminal**

---

# Can't we script the WHOLE workflow?

```console
read -p "title? " title

create_issue --title=$title > issue.json

echo discussion in the following communication channels...
for url in $(myctl create discussions --issue $(jq .issue.channels[].url issue.json)); do
  open ${url}
done

for ch in $(myctl get channels --discussion $discussion_id -i json | jq .channelId); do
wait_for_settlement $ch
done

echo "All the discussions have been settled! Let's write some code!"

pushd $PROJECT
branch=$(escape-title "$title")
git checkout -b $branch origin/master
code .

read -p "type enter once you finish coding"

git add .
git commit -m "$title"
git push origin $branch
hub pull-request

wait_for_approval $pr_number

verify_ci_builds

# fail if (1)deplyoment failed (2)alert triggered in 10 min.
wait_for_deployment $pr_number

status=$?

if [ $status -ne 0 ]; then
  echo "deployment failed!" 1>&2
  open $datadog_url
fi
```
---

# Going Production...

```console
$ HISTTIMEFORMAT=$'\r\e[K' history > myscript
$ cat <(echo '#!/usr/bin/env bash') myscript > myscript
$ chmod +x myscript
$ $EDITOR $_
```

And finally deploy it as a Brigade script?

```console
$ helm install brigade/brigade-project \
  --set-file defaultScript=brigade.js ...
```

---

# Is that all you need?

No!

---

# Remaining Gotchas

- Make your command **idempotent**
   - Rather than badly paraphrasing here, read: [Designing robust and predictable APIs with idempotency](https://stripe.com/ja-JP/blog/idempotency)
- Multiple Events Involved in Single Workflow

---

# Multiple Events Involved in Single Workflow

- Webhook (Do you own a webhook-to-job-queue server?
- Slack Interaction Message (Internet-facing HTTP server required in order to respond e.g. button clicks!
- And do it without compromising security?

---

**1-to-1**

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/5912/3e17a3e2-7788-8af9-ffe4-d9676bbc491e.png)


**PubSub**

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/5912/73b0acd0-8eef-c15a-c4dd-8719e4f66647.png)

Ref: https://www.knative.dev/docs/eventing/

---

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/5912/8b66d1fb-11e2-6a81-17e7-d88410ac56ce.png)


Compromiesd "Channel Receiver" is horror!

---

Instead of building and running (internet-facing) webhook server per automation,

Use **Event Gateway**

---

# Event Gateway Implementations

- [Serverless Event Gateway](https://serverless.com/event-gateway/): API Gateway + PubSub
- [Knative Eventing](https://www.knative.dev/docs/eventing/) - API Gateway + PubSub
- Brigade Gateway: Kind of API Gateway (1-to-1 Msg Passing

---

# CloudNative Wishlist

**Scripting Platform** that allows me to script the whole workflow:

```bash
if pr-not-created; then
  hub pull-request
fi

# Subscribe to Event Gateway!
await-events \
  -f github-pr-approvals.yaml \
  -f slack-thread-comments.yaml \
  -f circleci-builds-pass.yaml \
  -o result.json

status=$?

if [ $status -ne 0 ];
  exit 1
fi

./do-merge-pr $(./extract_pr_number result.json)

await-events \
  -f argo-cd-deployment-success.yaml \
  -t 300s

status=$?

if [ $status -ne 0 ];
  exit 1
fi
```

---

# Important Point

All those works on top of Kubernetes!

**Kubernetes**: API and Runtime for Various Automations

---

# Ending

---

# Recap

- **WHY**: Life is too short to waste. I don't want to write a **lot** of code to get things done
- **HOW**: I need a **Scripting Platform** that automates all the things(Cloud, Kubernetes, Service Mesh, Dev/Business Process, ...)
- **WHAT**: Shell Script + Brigade + Event Gateway. Kubernetes is just API and Runtime

---

Script on Kubernetes for automating everything.

Bring back your productivity!

---

```console
git add progress/
```

```console
git commit -m 'Finish the talk. Start coding.'
```

Thanks! :bow: :wave: 

---

Any question?
--

@mumoshu / **Yusuke KUOKA** at **Z Lab**, Developing Kubernetes as a Service for YJ

> AWS EC2上にKubernetesクラスタをベンチャー2社で構築・運用し続けて3年、AWS EKSの登場を機にこれから何をしていくべきかを考えてます。

**AWS Container Hero**
**OSSメンテナ**: kube-aws, helmfile, helm-diff, eksctl, ...
**Kubernetesのお仕事** at freee, Quipper, ChatWork, ...

---

Democratizing Workflow Automation
--

**WHY**: 人生は短いので、普段のお仕事を自動化するのに沢山コードを書きたくない！
**HOW**: クラウドもサービスメッシュもPaaSもSaaSも社内システムも、何もかもを自動化するために**Scripting Platform**がほしい
**WHAT**: Shell Script + Brigade + Event Gatewayで作ってます。Kubernetesは実装詳細
](qi)