Fedora CoreOSでAWSにKubernetesを立てる

# はじめに
Fedora CoreOSでAWSにKubernetesを立てる。

- Fedora CoreOS（Stable:34.20210518.3.0）
- Kubernetes v1.21.3（1 master、nodeなしの構成。RBAC）
- CRIランタイム：cri-o
- etcd3（podmanで駆動。tls secured）
- コンテナネットワーク：Flannel
- CoreDNS
- EC2インスタンス：t3a.small（スポットインスタンス）
- zram

# AWSでインスタンスを作成

あらかじめ、Elastic IP アドレスを１つ割り当てておきます。
リージョンはap-northeast-1を前提としています。

次のyamlを編集

- ssh_authorized_keysを置き換える

```yaml:fcos-aws-1master-crio.yaml
variant: fcos
version: 1.3.0
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABA・・・DrOtidE4JlTyG4XOMDLJ core@localhost"
systemd:
  units:
    - name: podman.service
      enabled: true
    - name: etcd3.service
      enabled: true
      contents: |
        [Unit]
        Description=etcd3
        After=network-online.target
        Wants=network-online.target
        [Service]
        ExecStartPre=mkdir -p /var/lib/fcosaws/etcd
        ExecStartPre=-/bin/podman kill etcd3
        ExecStartPre=-/bin/podman rm etcd3
        ExecStart=/bin/podman run \
          -p 2379:2379 \
          -p 2380:2380 \
          -v /var/lib/fcosaws/etcd:/var/lib/fcosaws/etcd:z \
          -v /var/lib/fcosaws/pki:/var/lib/fcosaws/pki:z \
          --name=etcd3 \
          gcr.io/etcd-development/etcd:v3.4.13 \
          /usr/local/bin/etcd \
          --data-dir /var/lib/fcosaws/etcd \
          --name infra1 \
          --client-cert-auth \
          --trusted-ca-file=/var/lib/fcosaws/pki/ca.crt \
          --cert-file=/var/lib/fcosaws/pki/apiserver.crt \
          --key-file=/var/lib/fcosaws/pki/apiserver.key \
          --peer-client-cert-auth \
          --peer-trusted-ca-file=/var/lib/fcosaws/pki/ca.crt \
          --peer-cert-file=/var/lib/fcosaws/pki/worker.crt \
          --peer-key-file=/var/lib/fcosaws/pki/worker.key \
          --advertise-client-urls https://10.10.10.11:2379 \
          --listen-client-urls https://0.0.0.0:2379 \
          --initial-advertise-peer-urls https://10.10.10.11:2380 \
          --listen-peer-urls https://0.0.0.0:2380 \
          --initial-cluster-token etcd-cluster-1 \
          --initial-cluster infra1=https://10.10.10.11:2380 \
          --initial-cluster-state new
        ExecStop=/bin/podman stop etcd3
        Restart=on-failure
        RestartSec=10
        [Install]
        WantedBy=multi-user.target
    - name: kubelet2.service
      enabled: true
      contents: |
        [Unit]
        Description=kubelet
        [Service]
        ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
        ExecStartPre=/bin/mkdir -p /var/lib/rook
        ExecStart=/root/bin/kubelet \
          --config=/etc/kubernetes/kubelet-conf.yaml \
          --network-plugin=cni \
          --cni-conf-dir=/etc/cni/net.d \
          --cni-bin-dir=/opt/cni/bin \
          --node-ip=10.10.10.11 \
          --container-runtime=remote \
          --container-runtime-endpoint=/var/run/crio/crio.sock \
          --register-node=true \
          --hostname-override=ip-10-10-10-11.ap-northeast-1.compute.internal \
          --kubeconfig=/etc/kubernetes/worker-kubeconfig.yaml \
          --v=4 \
          --cloud-provider=external
        Restart=always
        RestartSec=10
        [Install]
        WantedBy=multi-user.target
    - name: set_node.service
      enabled: true
      contents: |
        [Unit]
        Description=set kubernetes node
        After=network-online.target
        [Service]
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/bash /opt/set_node.sh
        [Install]
        WantedBy=multi-user.target
        WantedBy=network-online.target
    - name: fstrim.timer
      mask: true
    - name: fstrim.service
      mask: true
storage:
  files:
    - path: /etc/crictl.yaml
      mode: 0644
      contents:
        inline: |
          runtime-endpoint: unix:///var/run/crio/crio.sock
          timeout: 2
    - path: /etc/systemd/zram-generator.conf
      mode: 0644
      contents:
        inline: |
          [zram0]
    - path: /opt/set_node.sh
      overwrite: true
      mode: 0644
      contents:
        inline: |
          if [ -f /var/home/core/initialized.txt ]; then
          systemctl enable crio.service
          systemctl start crio.service
          else
          sed -i -e "s?pool 2.fedora.pool.ntp.org iburst?server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4?g" /etc/chrony.conf
          sed -i -e "s?#SystemMaxUse=?SystemMaxUse=80M?g" /etc/systemd/journald.conf
          cd /var/home/core
          curl -OL https://github.com/containernetworking/plugins/releases/download/v0.9.1/cni-plugins-linux-amd64-v0.9.1.tgz
          mkdir -p /opt/cni/bin
          tar zxf cni-plugins-linux-amd64-v0.9.1.tgz -C /opt/cni/bin
          curl -OL https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.21.0/crictl-v1.21.0-linux-amd64.tar.gz
          tar zxf crictl-v1.21.0-linux-amd64.tar.gz -C /usr/local/bin
          mkdir /etc/dnf/modules.d
          cat <<EOF > /etc/dnf/modules.d/cri-o.module
          [cri-o]
          name=cri-o
          stream=1.21
          profiles=
          state=enabled
          EOF
          sed -i -z s/enabled=0/enabled=1/ /etc/yum.repos.d/fedora-modular.repo
          sed -i -z s/enabled=0/enabled=1/ /etc/yum.repos.d/fedora-updates-modular.repo
          rpm-ostree install cri-o
          echo "dummy" > /var/home/core/initialized.txt
          shutdown -r now
          fi
    - path: /etc/kubernetes/manifests/kube-apiserver.yaml
      overwrite: true
      mode: 0755
      contents:
        inline: |
          apiVersion: v1
          kind: Pod
          metadata:
            name: kube-apiserver
            namespace: kube-system
          spec:
            hostNetwork: true
            containers:
            - name: kube-apiserver
              image: k8s.gcr.io/kube-apiserver-amd64:v1.21.3
              command:
              - kube-apiserver
              - --bind-address=0.0.0.0
              - --etcd-servers=https://10.10.10.11:2379
              - --etcd-cafile=/var/lib/fcosaws/pki/ca.crt
              - --etcd-certfile=/var/lib/fcosaws/pki/apiserver.crt
              - --etcd-keyfile=/var/lib/fcosaws/pki/apiserver.key
              - --allow-privileged=true
              - --apiserver-count=1
              - --endpoint-reconciler-type=lease
              - --service-cluster-ip-range=10.3.0.0/24
              - --secure-port=6443
              - --advertise-address=10.10.10.11
              - --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota
              - --storage-backend=etcd3
              - --tls-cert-file=/var/lib/fcosaws/pki/apiserver.crt
              - --tls-private-key-file=/var/lib/fcosaws/pki/apiserver.key
              - --client-ca-file=/var/lib/fcosaws/pki/ca.crt
              - --service-account-key-file=/var/lib/fcosaws/pki/apiserver.key
              - --runtime-config=extensions/v1beta1/networkpolicies=true
              - --service-node-port-range=25-32767
              - --authorization-mode=RBAC
              - --cloud-provider=external
              - --service-account-issuer=api
              - --service-account-signing-key-file=/var/lib/fcosaws/pki/apiserver.key
              ports:
              - containerPort: 6443
                hostPort: 6443
                name: https
              - containerPort: 8080
                hostPort: 8080
                name: local
              volumeMounts:
              - mountPath: /var/lib/fcosaws/pki
                name: ssl-certs-kubernetes
                readOnly: true
            volumes:
            - hostPath:
                path: /var/lib/fcosaws/pki
              name: ssl-certs-kubernetes
    - path: /etc/kubernetes/manifests/kube-controller-manager.yaml
      overwrite: true
      mode: 0755
      contents:
        inline: |
          apiVersion: v1
          kind: Pod
          metadata:
            name: kube-controller-manager
            namespace: kube-system
          spec:
            hostNetwork: true
            containers:
            - name: kube-controller-manager
              image: k8s.gcr.io/kube-controller-manager-amd64:v1.21.3
              command:
              - kube-controller-manager
              - --kubeconfig=/etc/kubernetes/worker-kubeconfig.yaml
              - --leader-elect=true
              - --service-account-private-key-file=/var/lib/fcosaws/pki/apiserver.key
              - --root-ca-file=/var/lib/fcosaws/pki/ca.crt
              - --cluster-signing-cert-file=/var/lib/fcosaws/pki/ca.crt
              - --cluster-signing-key-file=/var/lib/fcosaws/pki/ca.key
              - --cluster-cidr=10.244.0.0/16
              - --allocate-node-cidrs=true
              - --cloud-provider=external
              livenessProbe:
                httpGet:
                  host: 127.0.0.1
                  path: /healthz
                  port: 10252
                initialDelaySeconds: 15
                timeoutSeconds: 15
              volumeMounts:
              - mountPath: /etc/kubernetes/worker-kubeconfig.yaml
                name: "kubeconfig"
                readOnly: true
              - mountPath: /var/lib/fcosaws/pki
                name: ssl-certs-kubernetes
                readOnly: true
            volumes:
            - name: "kubeconfig"
              hostPath:
                path: "/etc/kubernetes/worker-kubeconfig.yaml"
            - hostPath:
                path: /var/lib/fcosaws/pki
              name: ssl-certs-kubernetes
    - path: /etc/kubernetes/manifests/kube-scheduler.yaml
      overwrite: true
      mode: 0755
      contents:
        inline: |
          apiVersion: v1
          kind: Pod
          metadata:
            name: kube-scheduler
            namespace: kube-system
          spec:
            hostNetwork: true
            containers:
            - name: kube-scheduler
              image: k8s.gcr.io/kube-scheduler-amd64:v1.21.3
              command:
              - kube-scheduler
              - --kubeconfig=/etc/kubernetes/worker-kubeconfig.yaml
              - --leader-elect=true
              livenessProbe:
                httpGet:
                  host: 127.0.0.1
                  path: /healthz
                  port: 10251
                initialDelaySeconds: 15
                timeoutSeconds: 15
              volumeMounts:
              - mountPath: /etc/kubernetes/worker-kubeconfig.yaml
                name: "kubeconfig"
                readOnly: true
              - mountPath: /var/lib/fcosaws/pki
                name: "etc-kube-ssl"
                readOnly: true
            volumes:
            - name: "kubeconfig"
              hostPath:
                path: "/etc/kubernetes/worker-kubeconfig.yaml"
            - name: "etc-kube-ssl"
              hostPath:
                path: "/var/lib/fcosaws/pki"
    - path: /etc/kubernetes/manifests/kube-proxy.yaml
      overwrite: true
      mode: 0755
      contents:
        inline: |
          apiVersion: v1
          kind: Pod
          metadata:
            name: kube-proxy
            namespace: kube-system
          spec:
            hostNetwork: true
            containers:
              - name: kube-proxy
                image: k8s.gcr.io/kube-proxy-amd64:v1.21.3
                command:
                  - kube-proxy
                  - --hostname-override=ip-10-10-10-11.ap-northeast-1.compute.internal
                  - --kubeconfig=/etc/kubernetes/worker-kubeconfig.yaml
                  - --proxy-mode=iptables
                securityContext:
                  privileged: true
                volumeMounts:
                  - mountPath: /etc/kubernetes/worker-kubeconfig.yaml
                    name: "kubeconfig"
                    readOnly: true
                  - mountPath: /var/lib/fcosaws/pki
                    name: "etc-kube-ssl"
                    readOnly: true
                  - mountPath: /lib/modules
                    name: "lib-modules"
                    readOnly: true
            volumes:
              - name: "kubeconfig"
                hostPath:
                  path: "/etc/kubernetes/worker-kubeconfig.yaml"
              - name: "etc-kube-ssl"
                hostPath:
                  path: "/var/lib/fcosaws/pki"
              - name: "lib-modules"
                hostPath:
                  path: "/lib/modules"
    - path: /etc/kubernetes/worker-kubeconfig.yaml
      overwrite: true
      mode: 0755
      contents:
        inline: |
          apiVersion: v1
          kind: Config
          clusters:
          - name: local
            cluster:
              certificate-authority: /var/lib/fcosaws/pki/ca.crt
              server: https://ip-10-10-10-11.ap-northeast-1.compute.internal:6443
          users:
          - name: kubelet
            user:
              client-certificate: /var/lib/fcosaws/pki/worker.crt
              client-key: /var/lib/fcosaws/pki/worker.key
          contexts:
          - context:
              cluster: local
              user: kubelet
            name: kubelet-context
          current-context: kubelet-context
    - path: /etc/kubernetes/kubelet-conf.yaml
      overwrite: true
      mode: 0755
      contents:
        inline: |
          kind: KubeletConfiguration
          apiVersion: kubelet.config.k8s.io/v1beta1
          cgroupDriver: "systemd"
          volumePluginDir: "/var/lib/kubelet/volumeplugins"
          staticPodPath: "/etc/kubernetes/manifests"
          clusterDNS: ["10.3.0.10"]
          clusterDomain: "cluster.local"
          # Restore default authentication and authorization modes from K8s < 1.9
          authentication:
            anonymous:
              enabled: true # Defaults to false as of 1.10
            webhook:
              enabled: false # Deafults to true as of 1.10
          authorization:
            mode: AlwaysAllow # Deafults to webhook as of 1.10
          readOnlyPort: 10255 # Used by heapster. Defaults to 0 (disabled) as of 1.10. Needed for metrics.
          failSwapOn: false
# kubeletをインストールする
    - path: /root/bin/kubelet
      mode: 0755
      overwrite: true
      contents:
        source: https://storage.googleapis.com/kubernetes-release/release/v1.21.3/bin/linux/amd64/kubelet
```

ButaneでYAMLをJSONに変換（次の例はlinuxコマンドでの例）。

```bash:
$ docker run --interactive --rm -v ${PWD}:/pwd --workdir /pwd quay.io/coreos/butane:release --pretty --strict fcos-aws-1master-crio.yaml > fcos-aws-1master-crio.ign
```

次の内容でspot-options.jsonを作成しておきます。

```json:spot-options.json
{
  "MarketType": "spot",
  "SpotOptions": {
    "MaxPrice": "0.0324",
    "SpotInstanceType": "persistent",
    "InstanceInterruptionBehavior": "stop"
  }
}
```

次のコマンドを実行。

- subnet-idを置き換える
- key-nameを置き換える
- security-group-idsを置き換える
- iam-instance-profileを置き換える

```bash:
$ aws ec2 run-instances \
    --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=master},{Key=kubernetes.io/cluster/ucluster,Value=kubernetes.io/cluster/ucluster},{Key=KubernetesCluster,Value=ucluster}]' \
    --image-id ami-03894a14941c52b30 \
    --count 1 \
    --instance-type t3a.small \
    --block-device-mappings '[{"DeviceName":"/dev/xvda", "Ebs":{"VolumeSize":13}}]' \
    --subnet-id subnet-033e851c267541ce0 \
    --key-name keypair2 \
    --security-group-ids sg-0e16014e82accb264 \
    --instance-market-options file:///path/to/spot-options.json \
    --associate-public-ip-address \
    --private-ip-address 10.10.10.11 \
    --iam-instance-profile Name="demo-profile" \
    --user-data file:///path/to/fcos-aws-1master-crio.ign
```

インスタンスが作成されると、set_node.shに記載のとおり、一度再起動されるので、再起動されるまで待つ。
再起動されたら、

```bash:
$ aws ec2 modify-instance-attribute --instance-id=$(aws ec2 describe-instances --query 'Reservations[].Instances[].{Stt:State.Name,Name:Tags[?Key==`Name`]|[0].Value,InstId:InstanceId}' | jq '.[] | select(.Stt == "running" and .Name == "master")' | jq '.InstId' | sed s/\"//g) --no-source-dest-check
```

また、Elastic IP アドレス（${MASTER_IP}）をこのインスタンスに関連付けておきます。

# KubernetesのためのTLS Assetsを準備
次の手順に従ってRBACに対応するTLS Assetsを生成してください。
（詳しくは次のページを参照。
https://github.com/coreos/coreos-kubernetes/blob/master/Documentation/openssl.md
）

```bash:
$ openssl genrsa -out ca.key 2048
$ openssl req -x509 -new -nodes -key ca.key -days 10950 -out ca.crt -subj "/CN=kube-ca"
```

- ${MASTER_IP}をElastic IP アドレスで置き換える

```bash:
$ echo "[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster.local
DNS.5 = ip-10-10-10-11.ap-northeast-1.compute.internal
IP.1 = 10.3.0.1
IP.2 = 10.10.10.11
IP.3 = ${MASTER_IP}" > openssl.cnf
```

```bash:
$ openssl genrsa -out apiserver.key 2048
$ openssl req -new -key apiserver.key -out apiserver.csr -subj "/CN=kube-apiserver" -config openssl.cnf
$ openssl x509 -req -in apiserver.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out apiserver.crt -days 10950 -extensions v3_req -extfile openssl.cnf
```

```bash:
$ echo "[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
IP.1 = \$ENV::WORKER_IP" > worker-openssl.cnf
```

```bash:
$ openssl genrsa -out kube-worker-1.key 2048
$ WORKER_IP=10.10.10.11 openssl req -new -key kube-worker-1.key -out kube-worker-1.csr -subj "/CN=kube-worker-1/O=system:masters" -config worker-openssl.cnf
$ WORKER_IP=10.10.10.11 openssl x509 -req -in kube-worker-1.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out kube-worker-1.crt -days 10950 -extensions v3_req -extfile worker-openssl.cnf
```

```bash:
$ openssl genrsa -out admin.key 2048
$ openssl req -new -key admin.key -out admin.csr -subj "/CN=kube-admin/O=system:masters"
$ openssl x509 -req -in admin.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out admin.crt -days 10950
```

以降の手順では次の８つのキーがtls-assets-fcos-awsディレクトリに生成されていることを仮定します。

- ca.key
- ca.crt
- apiserver.key
- apiserver.crt
- admin.key
- admin.crt
- kube-worker-1.key
- kube-worker-1.crt

生成したキーをインスタンスにコピーします。

```bash:
$ master=$(aws ec2 describe-instances --query 'Reservations[].Instances[].{Stt:State.Name,Name:Tags[?Key==`Name`]|[0].Value,PublicIp:PublicIpAddress}' | jq '.[] | select(.Stt == "running" and .Name == "master")' | jq '.PublicIp' | sed s/\"//g)
$ scp -r tls-assets-fcos-aws core@$master:/home/core/
```

```bash:
$ ssh -l core $master 'sudo rm -rf /var/lib/fcosaws/pki
sudo mkdir -p /var/lib/fcosaws/pki
sudo cp tls-assets-fcos-aws/ca.crt /var/lib/fcosaws/pki/
sudo cp tls-assets-fcos-aws/ca.key /var/lib/fcosaws/pki/
sudo cp tls-assets-fcos-aws/apiserver.crt /var/lib/fcosaws/pki/
sudo cp tls-assets-fcos-aws/apiserver.key /var/lib/fcosaws/pki/
sudo cp tls-assets-fcos-aws/kube-worker-1.crt /var/lib/fcosaws/pki/
sudo cp tls-assets-fcos-aws/kube-worker-1.key /var/lib/fcosaws/pki/
sudo chmod 600 /var/lib/fcosaws/pki/*.key
sudo chown root:root /var/lib/fcosaws/pki/*.key
sudo ln -s /var/lib/fcosaws/pki/kube-worker-1.crt /var/lib/fcosaws/pki/worker.crt
sudo ln -s /var/lib/fcosaws/pki/kube-worker-1.key /var/lib/fcosaws/pki/worker.key
'
```

```bash:
$ kubectl label nodes ip-10-10-10-11.ap-northeast-1.compute.internal node-role.kubernetes.io/master=""
```

etcd3とkubelet2が動作していることを確認

```bash:
$ ssh -l core ${MASTER_IP}
$ systemctl status etcd3
$ systemctl status kubelet2
```

```bash:
$ exit
```

# kubectlをセットアップする

kubectlをダウンロードする。

```bash:
$ curl -O https://storage.googleapis.com/kubernetes-release/release/v1.21.3/bin/linux/amd64/kubectl
```

ダウンロードしたら、実行権限を確認し、適切なパスに移動する

```bash:
$ chmod +x kubectl
$ mv kubectl /usr/local/bin/kubectl
```

kubectlを設定する。

- ${MASTER_IP}をElastic IP アドレスで置き換える
- Replace ${CA_CERT} with the absolute path to the ca.crt created in previous steps
- Replace ${ADMIN_KEY} with the absolute path to the admin.key created in previous steps
- Replace ${ADMIN_CERT} with the absolute path to the admin.crt created in previous steps

```bash:
$ kubectl config set-cluster default-cluster --server=https://${MASTER_IP}:6443 --certificate-authority=${CA_CERT}
$ kubectl config set-credentials default-admin --certificate-authority=${CA_CERT} --client-key=${ADMIN_KEY} --client-certificate=${ADMIN_CERT}
$ kubectl config set-context default-system --cluster=default-cluster --user=default-admin
$ kubectl config use-context default-system
```

kubectlの設定と接続を確認する。

```bash:
$ kubectl get nodes
NAME                                             STATUS   ROLES    AGE    VERSION
ip-10-10-10-11.ap-northeast-1.compute.internal   Ready    master   159m   v1.21.3
```

# aws-cloud-controller-managerをデプロイする

```bash:
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/cloud-provider-aws/release-1.19/manifests/rbac.yaml
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/cloud-provider-aws/release-1.19/manifests/aws-cloud-controller-manager-daemonset.yaml
```

# Flannelをデプロイする

```bash:
$ curl -O https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml
$ sed -i -e "s?vxlan?aws-vpc?g" kube-flannel.yml
$ kubectl apply -f kube-flannel.yml
```

# CoreDNSをデプロイする

https://github.com/coredns/deployment/tree/master/kubernetes
に記載の手順でCoreDNSをデプロイする。

```bash:
$ sudo apt -y install jq
$ curl -O https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/coredns.yaml.sed
$ curl -O https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/deploy.sh
$ ./deploy.sh -i 10.3.0.10 | kubectl apply -f -
```

次のコマンドを実行すると、次の画面のようになるはずです。

```bash:
$ kubectl get pods -A
```

```bash:
NAMESPACE     NAME                                                                     READY   STATUS    RESTARTS   AGE
kube-system   aws-cloud-controller-manager-x55pm                                       1/1     Running   0          12m
kube-system   coredns-85b78c98f-d6ptr                                                  1/1     Running   0          63m
kube-system   kube-apiserver-ip-10-10-10-11.ap-northeast-1.compute.internal            1/1     Running   0          69m
kube-system   kube-controller-manager-ip-10-10-10-11.ap-northeast-1.compute.internal   1/1     Running   0          68m
kube-system   kube-flannel-ds-zdscm                                                    1/1     Running   0          67m
kube-system   kube-proxy-ip-10-10-10-11.ap-northeast-1.compute.internal                1/1     Running   0          69m
kube-system   kube-scheduler-ip-10-10-10-11.ap-northeast-1.compute.internal            1/1     Running   0          69m
```

# Podがデプロイできることを確認

```yaml:busybox.yaml
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - image: busybox
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
    name: busybox
  restartPolicy: Always
```

```bash:
$ kubectl create -f busybox.yaml
$ kubectl get pods
```

```bash:
NAME      READY   STATUS    RESTARTS   AGE
busybox   1/1     Running   0          3m15s
```