EC2のUbuntu（22.04）上へのKubernetes（v1.28）環境の構築

# 本記事の記載内容

[1. はじめに](#1-はじめに)
[2. 前提情報](#2-前提情報)
[3. ControlPlaneの構築](#3-controlplaneの構築)
[4. WorkerNodeの構築](#4-workernodeの構築)
[5. 参考リンク](#5-参考リンク)
[6. 構築時につまづいた事](#6-構築時につまづいた事)

# 1. はじめに

自己学習用にkubernetesを手元で触れる環境を用意するために構築した際の記録。
本来であれば、各種クラウドベンダーが提供するkubernetes系のマネージドサービスを利用する方が効率的ではあるが、kubernetesに関する知識を習得することが目的である為、あえてEC2上にゼロから構築。


# 2. 前提情報

環境構築時の構成要素と内容（種別／バージョン）は以下の通り。

| 構成要素 | 内容 (種別／バージョン) |
| --- | --- |
| 環境 | AWSのEC2 |
| インスタンスタイプ | t3.small（2 vCPU／2 GiBメモリ） |
| OS | Ubuntu Server 22.04 LTS  |
| AMI名 | ubuntu-jammy-22.04-amd64-server-20230516 |
| AMI-ID | ami-0d52744d6551d851e |
| EBSボリューム | 10GiB |
| コンテナエンジン | containerd v1.7.3 |
| コンテナ低レベルランタイム | runC 1.1.8 |
| コンテナオーケストレーション | kubernetes v1.28.2 |
| 仮想ネットワーク（コンテナ間通信） | Flannel v0.22.3 |


# 3. ControlPlaneの構築

## 3-1. AWSマネジメントコンソールからEC2の構築  

- [2. 前提情報](#2-前提情報) に記載した構成でEC2を構築。
- EC2は、インターネット接続可能なパブリックサブネット上に構築。
- EC2構築の詳細な手順については、本記事では割愛。

## 3-2. 構築したEC2にログインしKubernetes環境構築

- OSにログイン後、rootユーザにて作業を実施。

:::note warn
正式な環境を構築する場合は、セキュリティ要件に応じた権限設計等を考慮する必要がありますが、
今回構築する環境は自己学習用の開発環境であり、すべての作業をrootユーザにて実施しています。
:::

### OSホスト名の変更

```console:実行コマンド
hostnamectl set-hostname <設定するホスト名>
```
```console:コマンド出力結果
root@ip-10-0-0-105:~# hostnamectl set-hostname k8s-control-plane
root@ip-10-0-0-105:~# 
```

### hostsへの自ホストの追加

```console:実行コマンド
vim /etc/hosts
```
```console:コマンド出力結果
root@ip-10-0-0-105:~# vim /etc/hosts
root@ip-10-0-0-105:~# cat /etc/hosts | grep k8s
10.0.0.105 k8s-control-plane
root@ip-10-0-0-105:~# 
```

### OS再起動の実施

```console:実行コマンド
shutdown -r now 
```
```console:コマンド出力結果
root@ip-10-0-0-105:~# shutdown -r now 
Connection to x.x.x.x closed by remote host.
Connection to x.x.x.x closed.
localMac:~ root#
```

### containerdのインストール

```console:実行コマンド（一行ずつ実行）
CONTAINERD_VERSION=1.7.3
mkdir -p /usr/local/src
wget -P /usr/local/src https://github.com/containerd/containerd/releases/download/v${CONTAINERD_VERSION}/containerd-${CONTAINERD_VERSION}-linux-amd64.tar.gz
tar -C /usr/local -xf /usr/local/src/containerd-${CONTAINERD_VERSION}-linux-amd64.tar.gz
wget -P /etc/systemd/system https://raw.githubusercontent.com/containerd/containerd/main/containerd.service
systemctl daemon-reload
systemctl enable --now containerd
```
```console:コマンド出力結果
root@k8s-control-plane:~# CONTAINERD_VERSION=1.7.3
root@k8s-control-plane:~# 
root@k8s-control-plane:~# mkdir -p /usr/local/src
root@k8s-control-plane:~# 
root@k8s-control-plane:~# wget -P /usr/local/src https://github.com/containerd/containerd/releases/download/v${CONTAINERD_VERSION}/containerd-${CONTAINERD_VERSION}-linux-amd64.tar.gz
--2023-09-30 14:07:10--  https://github.com/containerd/containerd/releases/download/v1.7.3/containerd-1.7.3-linux-amd64.tar.gz
Resolving github.com (github.com)... 20.27.177.113
Connecting to github.com (github.com)|20.27.177.113|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/46089560/c5074d7b-7021-4549-b3af-ef5728245812?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230930%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230930T140711Z&X-Amz-Expires=300&X-Amz-Signature=1d57d413832500f7855d0e16d6724d297a90fe97440cd49669f15d7b2d8b2f85&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=46089560&response-content-disposition=attachment%3B%20filename%3Dcontainerd-1.7.3-linux-amd64.tar.gz&response-content-type=application%2Foctet-stream [following]
--2023-09-30 14:07:11--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/46089560/c5074d7b-7021-4549-b3af-ef5728245812?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230930%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230930T140711Z&X-Amz-Expires=300&X-Amz-Signature=1d57d413832500f7855d0e16d6724d297a90fe97440cd49669f15d7b2d8b2f85&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=46089560&response-content-disposition=attachment%3B%20filename%3Dcontainerd-1.7.3-linux-amd64.tar.gz&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.111.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.109.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 46839131 (45M) [application/octet-stream]
Saving to: ‘/usr/local/src/containerd-1.7.3-linux-amd64.tar.gz’

containerd-1.7.3-linux-amd64. 100%[================================================>]  44.67M  26.4MB/s    in 1.7s    

2023-09-30 14:07:13 (26.4 MB/s) - ‘/usr/local/src/containerd-1.7.3-linux-amd64.tar.gz’ saved [46839131/46839131]

root@k8s-control-plane:~# 
root@k8s-control-plane:~# tar -C /usr/local -xf /usr/local/src/containerd-${CONTAINERD_VERSION}-linux-amd64.tar.gz
root@k8s-control-plane:~# 
root@k8s-control-plane:~# 
root@k8s-control-plane:~# wget -P /etc/systemd/system https://raw.githubusercontent.com/containerd/containerd/main/containerd.service
--2023-09-30 14:07:54--  https://raw.githubusercontent.com/containerd/containerd/main/containerd.service
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.111.133, 185.199.108.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1393 (1.4K) [text/plain]
Saving to: ‘/etc/systemd/system/containerd.service’

containerd.service            100%[================================================>]   1.36K  --.-KB/s    in 0s      

2023-09-30 14:07:54 (12.4 MB/s) - ‘/etc/systemd/system/containerd.service’ saved [1393/1393]

root@k8s-control-plane:~# 
root@k8s-control-plane:~# systemctl daemon-reload
root@k8s-control-plane:~# 
root@k8s-control-plane:~# systemctl enable --now containerd
Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /etc/systemd/system/containerd.service.
root@k8s-control-plane:~# 
```

### runCのインストール

```console:実行コマンド（一行ずつ実行）
RUNC_VERSION=1.1.8
wget -O /usr/local/sbin/runc https://github.com/opencontainers/runc/releases/download/v${RUNC_VERSION}/runc.amd64
chmod +x /usr/local/sbin/runc
```
```console:コマンド出力結果
root@k8s-control-plane:~# RUNC_VERSION=1.1.8
root@k8s-control-plane:~# wget -O /usr/local/sbin/runc https://github.com/opencontainers/runc/releases/download/v${RUNC_VERSION}/runc.amd64
--2023-09-30 14:08:20--  https://github.com/opencontainers/runc/releases/download/v1.1.8/runc.amd64
Resolving github.com (github.com)... 20.27.177.113
Connecting to github.com (github.com)|20.27.177.113|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/36960321/789db355-a93d-45b3-af29-d0f5f2196ab9?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230930%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230930T140820Z&X-Amz-Expires=300&X-Amz-Signature=ed74c9a36f3f47ce7c9c961a71b415275ad023578f17ca29728315f104b68030&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=36960321&response-content-disposition=attachment%3B%20filename%3Drunc.amd64&response-content-type=application%2Foctet-stream [following]
--2023-09-30 14:08:20--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/36960321/789db355-a93d-45b3-af29-d0f5f2196ab9?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230930%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230930T140820Z&X-Amz-Expires=300&X-Amz-Signature=ed74c9a36f3f47ce7c9c961a71b415275ad023578f17ca29728315f104b68030&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=36960321&response-content-disposition=attachment%3B%20filename%3Drunc.amd64&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.111.133, 185.199.109.133, 185.199.108.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.111.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10684992 (10M) [application/octet-stream]
Saving to: ‘/usr/local/sbin/runc’

/usr/local/sbin/runc          100%[================================================>]  10.19M  29.4MB/s    in 0.3s    

2023-09-30 14:08:21 (29.4 MB/s) - ‘/usr/local/sbin/runc’ saved [10684992/10684992]

root@k8s-control-plane:~# 
root@k8s-control-plane:~# chmod +x /usr/local/sbin/runc
root@k8s-control-plane:~# 
```

### CNIのインストール

```console:実行コマンド（一行ずつ実行）
CNI_VERSION=1.3.0
wget -P /usr/local/src https://github.com/containernetworking/plugins/releases/download/v${CNI_VERSION}/cni-plugins-linux-amd64-v${CNI_VERSION}.tgz
mkdir -p /opt/cni/bin
tar -C /opt/cni/bin -xf /usr/local/src/cni-plugins-linux-amd64-v${CNI_VERSION}.tgz
```
```console:コマンド出力結果
root@k8s-control-plane:~# CNI_VERSION=1.3.0
root@k8s-control-plane:~# wget -P /usr/local/src https://github.com/containernetworking/plugins/releases/download/v${CNI_VERSION}/cni-plugins-linux-amd64-v${CNI_VERSION}.tgz
--2023-09-30 14:08:34--  https://github.com/containernetworking/plugins/releases/download/v1.3.0/cni-plugins-linux-amd64-v1.3.0.tgz
Resolving github.com (github.com)... 20.27.177.113
Connecting to github.com (github.com)|20.27.177.113|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/84575398/d1ad8456-0aa1-4bb9-84e3-4e03286b4e9f?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230930%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230930T140834Z&X-Amz-Expires=300&X-Amz-Signature=4e1b842bb4a28f885d845f2e207e7420df897b5770eb52e591a2a36692abd475&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=84575398&response-content-disposition=attachment%3B%20filename%3Dcni-plugins-linux-amd64-v1.3.0.tgz&response-content-type=application%2Foctet-stream [following]
--2023-09-30 14:08:34--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/84575398/d1ad8456-0aa1-4bb9-84e3-4e03286b4e9f?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230930%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230930T140834Z&X-Amz-Expires=300&X-Amz-Signature=4e1b842bb4a28f885d845f2e207e7420df897b5770eb52e591a2a36692abd475&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=84575398&response-content-disposition=attachment%3B%20filename%3Dcni-plugins-linux-amd64-v1.3.0.tgz&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.110.133, 185.199.108.133, 185.199.109.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.110.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 45338194 (43M) [application/octet-stream]
Saving to: ‘/usr/local/src/cni-plugins-linux-amd64-v1.3.0.tgz’

cni-plugins-linux-amd64-v1.3. 100%[================================================>]  43.24M   281MB/s    in 0.2s    

2023-09-30 14:08:35 (281 MB/s) - ‘/usr/local/src/cni-plugins-linux-amd64-v1.3.0.tgz’ saved [45338194/45338194]

root@k8s-control-plane:~# 
root@k8s-control-plane:~# mkdir -p /opt/cni/bin
root@k8s-control-plane:~# 
root@k8s-control-plane:~# tar -C /opt/cni/bin -xf /usr/local/src/cni-plugins-linux-amd64-v${CNI_VERSION}.tgz
root@k8s-control-plane:~# 
```

### systemdバックエンドなcgroupを有効にする

```console:実行コマンド（一行ずつ実行）
mkdir /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml > /dev/null
cp -p /etc/containerd/config.toml /etc/containerd/config.toml_`date +%Y%m%d`
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
diff /etc/containerd/config.toml /etc/containerd/config.toml_`date +%Y%m%d`
systemctl restart containerdmkdir /etc/containerd
```
```console:コマンド出力結果
root@k8s-control-plane:~# mkdir /etc/containerd
root@k8s-control-plane:~# containerd config default | sudo tee /etc/containerd/config.toml > /dev/null
root@k8s-control-plane:~# cp -p /etc/containerd/config.toml /etc/containerd/config.toml_`date +%Y%m%d`
root@k8s-control-plane:~# sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
root@k8s-control-plane:~# diff /etc/containerd/config.toml /etc/containerd/config.toml_`date +%Y%m%d`
137c137
<             SystemdCgroup = true
---
>             SystemdCgroup = false
root@k8s-control-plane:~# 
root@k8s-control-plane:~# systemctl restart containerd
root@k8s-control-plane:~# 
```

### カーネルパラメータの設定

```console:実行コマンド（一行ずつ実行）
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

modprobe overlay
modprobe br_netfilter

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF

sysctl --system
```
```console:コマンド出力結果
root@k8s-control-plane:~# cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
overlay
br_netfilter
root@k8s-control-plane:~# 
root@k8s-control-plane:~# modprobe overlay
root@k8s-control-plane:~# modprobe br_netfilter
root@k8s-control-plane:~# 

root@k8s-control-plane:~# cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward　= 1
root@k8s-control-plane:~# 
root@k8s-control-plane:~# sysctl --system
* Applying /etc/sysctl.d/10-console-messages.conf ...
kernel.printk = 4 4 1 7
* Applying /etc/sysctl.d/10-ipv6-privacy.conf ...
net.ipv6.conf.all.use_tempaddr = 2
net.ipv6.conf.default.use_tempaddr = 2
* Applying /etc/sysctl.d/10-kernel-hardening.conf ...
kernel.kptr_restrict = 1
* Applying /etc/sysctl.d/10-magic-sysrq.conf ...
kernel.sysrq = 176
* Applying /etc/sysctl.d/10-network-security.conf ...
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.all.rp_filter = 2
* Applying /etc/sysctl.d/10-ptrace.conf ...
kernel.yama.ptrace_scope = 1
* Applying /etc/sysctl.d/10-zeropage.conf ...
vm.mmap_min_addr = 65536
* Applying /usr/lib/sysctl.d/50-default.conf ...
kernel.core_uses_pid = 1
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.default.accept_source_route = 0
sysctl: setting key "net.ipv4.conf.all.accept_source_route": Invalid argument
net.ipv4.conf.default.promote_secondaries = 1
sysctl: setting key "net.ipv4.conf.all.promote_secondaries": Invalid argument
net.ipv4.ping_group_range = 0 2147483647
net.core.default_qdisc = fq_codel
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
fs.protected_regular = 1
fs.protected_fifos = 1
* Applying /usr/lib/sysctl.d/50-pid-max.conf ...
kernel.pid_max = 4194304
* Applying /etc/sysctl.d/99-cloudimg-ipv6.conf ...
net.ipv6.conf.all.use_tempaddr = 0
net.ipv6.conf.default.use_tempaddr = 0
* Applying /usr/lib/sysctl.d/99-protect-links.conf ...
fs.protected_fifos = 1
fs.protected_hardlinks = 1
fs.protected_regular = 2
fs.protected_symlinks = 1
* Applying /etc/sysctl.d/99-sysctl.conf ...
* Applying /etc/sysctl.d/k8s.conf ...
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
* Applying /etc/sysctl.conf ...
root@k8s-control-plane:~# 
```

:::note warn
カーネルパラメータ設定時、不必要なスペースが含まれると正しく反映されない可能性がある為注意。
カーネルパラメータが正しく設定されていないと、後続のkubejoin実行時にエラーとなる可能性がある。
例：net.ipv4.ip_forward△△△△△△=△1△　・・・（△：スペース）
:::

### swapの無効化

```console:実行コマンド（一行ずつ実行）
swapon -s
swapoff -a
```
```console:コマンド出力結果
root@k8s-control-plane:~# swapon -s
root@k8s-control-plane:~# 
root@k8s-control-plane:~# swapoff -a
root@k8s-control-plane:~# 
```

### kubelet／kubeadm／kubectlのインストール

```console:実行コマンド（一行ずつ実行）
apt-get update
apt-get install -y apt-transport-https ca-certificates curl
curl -fsSL https://dl.k8s.io/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-archive-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl
```
```console:コマンド出力結果
root@k8s-control-plane:~# apt-get update
Hit:1 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy InRelease
Get:2 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
Get:3 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB]
Get:4 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy/universe amd64 Packages [14.1 MB]
Get:5 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Get:6 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy/universe Translation-en [5652 kB]
Get:7 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy/universe amd64 c-n-f Metadata [286 kB]
Get:8 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy/multiverse amd64 Packages [217 kB]
Get:9 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy/multiverse Translation-en [112 kB]
Get:10 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy/multiverse amd64 c-n-f Metadata [8372 B]
Get:11 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1014 kB]
Get:12 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [227 kB]
Get:13 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [15.6 kB]
Get:14 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [905 kB]
Get:15 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [146 kB]
Get:16 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [532 B]
Get:17 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [987 kB]
Get:18 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [215 kB]
Get:19 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [21.9 kB]
Get:20 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB]
Get:21 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9768 B]
Get:22 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B]
Get:23 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [41.7 kB]
Get:24 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB]
Get:25 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B]
Get:26 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy-backports/restricted amd64 c-n-f Metadata [116 B]
Get:27 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [24.3 kB]
Get:28 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB]
Get:29 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [640 B]
Get:30 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy-backports/multiverse amd64 c-n-f Metadata [116 B]
Get:31 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [804 kB]       
Get:32 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [169 kB]
Get:33 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.3 kB]
Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [889 kB]
Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [143 kB]
Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [532 B]
Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [785 kB]
Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [144 kB]
Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.7 kB]
Get:40 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB]
Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7060 B]
Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B]
Fetched 27.4 MB in 5s (5915 kB/s)                
Reading package lists... Done
root@k8s-control-plane:~# 

root@k8s-control-plane:~# apt-get install -y apt-transport-https ca-certificates curl
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  libcurl4
The following NEW packages will be installed:
  apt-transport-https
The following packages will be upgraded:
  ca-certificates curl libcurl4
3 upgraded, 1 newly installed, 0 to remove and 126 not upgraded.
Need to get 641 kB of archives.
After this operation, 193 kB of additional disk space will be used.
Get:1 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 ca-certificates all 20230311ubuntu0.22.04.1 [155 kB]
Get:2 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 apt-transport-https all 2.4.10 [1510 B]
Get:3 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 curl amd64 7.81.0-1ubuntu1.13 [194 kB]
Get:4 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libcurl4 amd64 7.81.0-1ubuntu1.13 [290 kB]
Fetched 641 kB in 0s (16.0 MB/s)  
Preconfiguring packages ...
(Reading database ... 64295 files and directories currently installed.)
Preparing to unpack .../ca-certificates_20230311ubuntu0.22.04.1_all.deb ...
Unpacking ca-certificates (20230311ubuntu0.22.04.1) over (20211016ubuntu0.22.04.1) ...
Selecting previously unselected package apt-transport-https.
Preparing to unpack .../apt-transport-https_2.4.10_all.deb ...
Unpacking apt-transport-https (2.4.10) ...
Preparing to unpack .../curl_7.81.0-1ubuntu1.13_amd64.deb ...
Unpacking curl (7.81.0-1ubuntu1.13) over (7.81.0-1ubuntu1.10) ...
Preparing to unpack .../libcurl4_7.81.0-1ubuntu1.13_amd64.deb ...
Unpacking libcurl4:amd64 (7.81.0-1ubuntu1.13) over (7.81.0-1ubuntu1.10) ...
Setting up apt-transport-https (2.4.10) ...
Setting up ca-certificates (20230311ubuntu0.22.04.1) ...
Updating certificates in /etc/ssl/certs...
rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL
19 added, 6 removed; done.
Setting up libcurl4:amd64 (7.81.0-1ubuntu1.13) ...
Setting up curl (7.81.0-1ubuntu1.13) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for libc-bin (2.35-0ubuntu3.1) ...
Processing triggers for ca-certificates (20230311ubuntu0.22.04.1) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Scanning processes...                                                                                                  
Scanning linux images...                                                                                               

Running kernel seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.
root@k8s-control-plane:~#

root@k8s-control-plane:~# curl -fsSL https://dl.k8s.io/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-archive-keyring.gpg
root@k8s-control-plane:~# 
root@k8s-control-plane:~# echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main
root@k8s-control-plane:~# 

root@k8s-control-plane:~# apt-get update
Hit:1 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy-updates InRelease        
Hit:3 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy-backports InRelease      
Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease                                         
Get:4 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [8993 B]
Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [69.9 kB]
Fetched 78.9 kB in 1s (83.8 kB/s) 
Reading package lists... Done
root@k8s-control-plane:~#

root@k8s-control-plane:~# apt-get install -y kubelet kubeadm kubectl
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  conntrack cri-tools ebtables kubernetes-cni socat
The following NEW packages will be installed:
  conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat
0 upgraded, 8 newly installed, 0 to remove and 126 not upgraded.
Need to get 87.1 MB of archives.
After this operation, 336 MB of additional disk space will be used.
Get:1 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 conntrack amd64 1:1.4.6-2build2 [33.5 kB]
Get:2 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 ebtables amd64 2.0.11-4build2 [84.9 kB]
Get:3 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB]
Get:4 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.26.0-00 [18.9 MB]
Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 1.2.0-00 [27.6 MB]
Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.28.2-00 [19.5 MB]
Get:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.28.2-00 [10.3 MB]
Get:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.28.2-00 [10.3 MB]
Fetched 87.1 MB in 5s (18.0 MB/s) 
Selecting previously unselected package conntrack.
(Reading database ... 64312 files and directories currently installed.)
Preparing to unpack .../0-conntrack_1%3a1.4.6-2build2_amd64.deb ...
Unpacking conntrack (1:1.4.6-2build2) ...
Selecting previously unselected package cri-tools.
Preparing to unpack .../1-cri-tools_1.26.0-00_amd64.deb ...
Unpacking cri-tools (1.26.0-00) ...
Selecting previously unselected package ebtables.
Preparing to unpack .../2-ebtables_2.0.11-4build2_amd64.deb ...
Unpacking ebtables (2.0.11-4build2) ...
Selecting previously unselected package kubernetes-cni.
Preparing to unpack .../3-kubernetes-cni_1.2.0-00_amd64.deb ...
Unpacking kubernetes-cni (1.2.0-00) ...
Selecting previously unselected package socat.
Preparing to unpack .../4-socat_1.7.4.1-3ubuntu4_amd64.deb ...
Unpacking socat (1.7.4.1-3ubuntu4) ...
Selecting previously unselected package kubelet.
Preparing to unpack .../5-kubelet_1.28.2-00_amd64.deb ...
Unpacking kubelet (1.28.2-00) ...
Selecting previously unselected package kubectl.
Preparing to unpack .../6-kubectl_1.28.2-00_amd64.deb ...
Unpacking kubectl (1.28.2-00) ...
Selecting previously unselected package kubeadm.
Preparing to unpack .../7-kubeadm_1.28.2-00_amd64.deb ...
Unpacking kubeadm (1.28.2-00) ...
Setting up conntrack (1:1.4.6-2build2) ...
Setting up kubectl (1.28.2-00) ...
Setting up ebtables (2.0.11-4build2) ...
Setting up socat (1.7.4.1-3ubuntu4) ...
Setting up cri-tools (1.26.0-00) ...
Setting up kubernetes-cni (1.2.0-00) ...
Setting up kubelet (1.28.2-00) ...
Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service → /lib/systemd/system/kubelet.service.
Setting up kubeadm (1.28.2-00) ...
Processing triggers for man-db (2.10.2-1) ...
Scanning processes...                                                                                                  
Scanning linux images...                                                                                               

Running kernel seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.
root@k8s-control-plane:~#

root@k8s-control-plane:~# apt-mark hold kubelet kubeadm kubectl
kubelet set on hold.
kubeadm set on hold.
kubectl set on hold.
root@k8s-control-plane:~# 
```

### Kubernetesの構築

:::note warn
本項目はControl Plane構築時のみ実施する。（Worker Node構築時は実施しない）
:::

```console:実行コマンド（一行ずつ実行）
kubeadm init --pod-network-cidr=10.244.0.0/16

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
```
```console:コマンド出力結果
root@k8s-control-plane:~# kubeadm init --pod-network-cidr=10.244.0.0/16
[init] Using Kubernetes version: v1.28.2
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
W0930 14:27:09.190649    7884 checks.go:835] detected that the sandbox image "registry.k8s.io/pause:3.8" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.k8s.io/pause:3.9" as the CRI sandbox image.
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-control-plane kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.0.0.105]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-control-plane localhost] and IPs [10.0.0.105 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-control-plane localhost] and IPs [10.0.0.105 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 9.504476 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-control-plane as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node k8s-control-plane as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: xxxxxx.xxxxxxxxxxxxxxxx
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.0.0.105:6443 --token xxxxxx.xxxxxxxxxxxxxxxx \
--discovery-token-ca-cert-hash  
sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

root@k8s-control-plane:~# 

root@k8s-control-plane:~# mkdir -p $HOME/.kube
root@k8s-control-plane:~# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
root@k8s-control-plane:~# chown $(id -u):$(id -g) $HOME/.kube/config
root@k8s-control-plane:~# 
```

:::note warn
kubeadm init実行時に最後に出力される以下のコマンドは、Worker Node追加時に必要となるためメモしておく

kubeadm join 10.0.0.105:6443 --token xxxxxx.xxxxxxxxxxxxxxxx \
--discovery-token-ca-cert-hash  
sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
:::

### Flannelを使用したCNIのセットアップ

:::note warn
本項目はControl Plane構築時のみ実施する。（Worker Node構築時は実施しない）
:::

```console:実行コマンド
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
```
```console:コマンド出力結果
root@k8s-control-plane:~# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
namespace/kube-flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created
root@k8s-control-plane:~# 
```

### 構築後の確認

- Nodeの状態の確認

```console:実行コマンド
kubectl get nodes
```
```console:コマンド出力結果
root@k8s-control-plane:~# kubectl get nodes
NAME                STATUS   ROLES           AGE     VERSION
k8s-control-plane   Ready    control-plane   4m37s   v1.28.2
root@k8s-control-plane:~# 
```

- Podの状態の確認

```console:実行コマンド
kubectl get pods -A
```
```console:コマンド出力結果
root@k8s-control-plane:~# kubectl get pods -A
NAMESPACE      NAME                                        READY   STATUS    RESTARTS   AGE
kube-flannel   kube-flannel-ds-gxjw6                       1/1     Running   0          2m13s
kube-system    coredns-5dd5756b68-8xtct                    1/1     Running   0          4m36s
kube-system    coredns-5dd5756b68-kmchv                    1/1     Running   0          4m36s
kube-system    etcd-k8s-control-plane                      1/1     Running   0          4m48s
kube-system    kube-apiserver-k8s-control-plane            1/1     Running   0          4m48s
kube-system    kube-controller-manager-k8s-control-plane   1/1     Running   0          4m48s
kube-system    kube-proxy-n44hz                            1/1     Running   0          4m36s
kube-system    kube-scheduler-k8s-control-plane            1/1     Running   0          4m48s
root@k8s-control-plane:~# 
root@k8s-control-plane:~# kubectl describe nodes
Name:               k8s-control-plane
Roles:              control-plane
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=k8s-control-plane
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=
                    node.kubernetes.io/exclude-from-external-load-balancers=
Annotations:        flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"46:d6:5f:9a:c1:5e"}
                    flannel.alpha.coreos.com/backend-type: vxlan
                    flannel.alpha.coreos.com/kube-subnet-manager: true
                    flannel.alpha.coreos.com/public-ip: 10.0.0.105
                    kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Sat, 30 Sep 2023 14:27:30 +0000
Taints:             node-role.kubernetes.io/control-plane:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  k8s-control-plane
  AcquireTime:     <unset>
  RenewTime:       Sat, 30 Sep 2023 14:33:02 +0000
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Sat, 30 Sep 2023 14:30:22 +0000   Sat, 30 Sep 2023 14:30:22 +0000   FlannelIsUp                  Flannel is running on this node
  MemoryPressure       False   Sat, 30 Sep 2023 14:30:37 +0000   Sat, 30 Sep 2023 14:27:30 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Sat, 30 Sep 2023 14:30:37 +0000   Sat, 30 Sep 2023 14:27:30 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Sat, 30 Sep 2023 14:30:37 +0000   Sat, 30 Sep 2023 14:27:30 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                True    Sat, 30 Sep 2023 14:30:37 +0000   Sat, 30 Sep 2023 14:30:27 +0000   KubeletReady                 kubelet is posting ready status. AppArmor enabled
Addresses:
  InternalIP:  10.0.0.105
  Hostname:    k8s-control-plane
Capacity:
  cpu:                2
  ephemeral-storage:  9974088Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             1983796Ki
  pods:               110
Allocatable:
  cpu:                2
  ephemeral-storage:  9192119486
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             1881396Ki
  pods:               110
System Info:
  Machine ID:                 ec2b3fcc000f6c4f20a5dd789cd2c4cf
  System UUID:                ec2b3fcc-000f-6c4f-20a5-dd789cd2c4cf
  Boot ID:                    541f5670-aeee-424e-b758-d564d98e6c7a
  Kernel Version:             5.19.0-1025-aws
  OS Image:                   Ubuntu 22.04.2 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.7.3
  Kubelet Version:            v1.28.2
  Kube-Proxy Version:         v1.28.2
PodCIDR:                      10.244.0.0/24
PodCIDRs:                     10.244.0.0/24
Non-terminated Pods:          (8 in total)
  Namespace                   Name                                         CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                         ------------  ----------  ---------------  -------------  ---
  kube-flannel                kube-flannel-ds-gxjw6                        100m (5%)     0 (0%)      50Mi (2%)        0 (0%)         3m
  kube-system                 coredns-5dd5756b68-8xtct                     100m (5%)     0 (0%)      70Mi (3%)        170Mi (9%)     5m23s
  kube-system                 coredns-5dd5756b68-kmchv                     100m (5%)     0 (0%)      70Mi (3%)        170Mi (9%)     5m23s
  kube-system                 etcd-k8s-control-plane                       100m (5%)     0 (0%)      100Mi (5%)       0 (0%)         5m35s
  kube-system                 kube-apiserver-k8s-control-plane             250m (12%)    0 (0%)      0 (0%)           0 (0%)         5m35s
  kube-system                 kube-controller-manager-k8s-control-plane    200m (10%)    0 (0%)      0 (0%)           0 (0%)         5m35s
  kube-system                 kube-proxy-n44hz                             0 (0%)        0 (0%)      0 (0%)           0 (0%)         5m23s
  kube-system                 kube-scheduler-k8s-control-plane             100m (5%)     0 (0%)      0 (0%)           0 (0%)         5m35s
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests     Limits
  --------           --------     ------
  cpu                950m (47%)   0 (0%)
  memory             290Mi (15%)  340Mi (18%)
  ephemeral-storage  0 (0%)       0 (0%)
  hugepages-1Gi      0 (0%)       0 (0%)
  hugepages-2Mi      0 (0%)       0 (0%)
Events:
  Type     Reason                   Age                    From             Message
  ----     ------                   ----                   ----             -------
  Normal   Starting                 5m21s                  kube-proxy       
  Normal   Starting                 5m47s                  kubelet          Starting kubelet.
  Warning  InvalidDiskCapacity      5m47s                  kubelet          invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  5m46s                  kubelet          Updated Node Allocatable limit across pods
  Normal   NodeHasNoDiskPressure    5m46s (x7 over 5m47s)  kubelet          Node k8s-control-plane status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     5m46s (x7 over 5m47s)  kubelet          Node k8s-control-plane status is now: NodeHasSufficientPID
  Normal   NodeHasSufficientMemory  5m46s (x8 over 5m47s)  kubelet          Node k8s-control-plane status is now: NodeHasSufficientMemory
  Normal   Starting                 5m36s                  kubelet          Starting kubelet.
  Warning  InvalidDiskCapacity      5m36s                  kubelet          invalid capacity 0 on image filesystem
  Normal   NodeHasSufficientMemory  5m36s                  kubelet          Node k8s-control-plane status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    5m36s                  kubelet          Node k8s-control-plane status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     5m36s                  kubelet          Node k8s-control-plane status is now: NodeHasSufficientPID
  Normal   NodeAllocatableEnforced  5m36s                  kubelet          Updated Node Allocatable limit across pods
  Normal   RegisteredNode           5m23s                  node-controller  Node k8s-control-plane event: Registered Node k8s-control-plane in Controller
  Normal   NodeReady                2m43s                  kubelet          Node k8s-control-plane status is now: NodeReady
root@k8s-control-plane:~# root@k8s-control-plane:~# kubectl get nodes
NAME                STATUS   ROLES           AGE     VERSION
k8s-control-plane   Ready    control-plane   4m37s   v1.28.2
root@k8s-control-plane:~# 
```

### EC2のセキュリティグループの解放

Control PlaneとWorker Node間で通信を行うため、デフォルトでは閉じているEC2のセキュリティグループ（インバウンド通信）を解放する必要がある。
アウトバウンド通信についてはデフォルトで全解放であるため、以下には記載しないが閉じている場合は必要に応じて解放する必要がある。

https://kubernetes.io/ja/docs/reference/networking/ports-and-protocols/

- Control Planeのセキュリティグループ設定

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/142483/d61b9249-8e47-483c-c4ea-6fd86bcf5c4d.png)


- Worker Nodeのセキュリティグループ設定

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/142483/e4eb3983-d488-efd9-15db-c3b9f8b38b75.png)

# 4. WorkerNodeの構築

Control Plane構築時と以下の同じステップを実施する。

- [3-1. AWSマネジメントコンソールからEC2の構築](#3-1-awsマネジメントコンソールからec2の構築)
- [3-2. 構築したEC2にログインしKubernetes環境構築](#3-2-構築したec2にログインしkubernetes環境構築)
  - [OSホスト名の変更](#osホスト名の変更)
  - [hostsへの自ホストの追加](#hostsへの自ホストの追加)
  - [OS再起動の実施](#os再起動の実施)
  - [containerdのインストール](#containerdのインストール)
  - [runCのインストール](#runcのインストール)
  - [CNIのインストール](#cniのインストール)
  - [systemdバックエンドなcgroupを有効にする](#systemdバックエンドなcgroupを有効にする)
  - [カーネルパラメータの設定](#カーネルパラメータの設定)
  - [swapの無効化](#swapの無効化)
  - [kubelet／kubeadm／kubectlのインストール](#kubeletkubeadmkubectlのインストール)

### kubernetesクラスタへのWorker Nodeの追加

:::note info
以下手順はKubernetesクラスタに追加するWorker Node上で実施する
:::

```console:実行コマンド（一行ずつ実行）
kubeadm join <Control PlaneのIPアドレス>:6443 --token xxxxxx.xxxxxxxxxxxxxxxx --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
```
```console:コマンド出力結果
root@k8s-worker-node1:~# kubeadm join 10.0.0.105:6443 --token xxxxxx.xxxxxxxxxxxxxxxx --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

root@k8s-worker-node1:~# 
```

### Worker Node追加後の確認（Control Plane側で実施）

:::note info
以下手順はKubernetesクラスタのControl Plane側で実施する
また、Control PlaneのhostsにWorker Nodeのホスト名を追記しておく
:::

- hosts追加後の確認

```console:実行コマンド
cat /etc/hosts |grep k8s
```
```console:コマンド出力結果
root@k8s-control-plane:~# cat /etc/hosts |grep k8s
10.0.0.105 k8s-control-plane
10.0.0.101 k8s-worker-node1
root@k8s-control-plane:~# 
```

- Nodeの常態の確認

```console:実行コマンド
kubectl get nodes
```
```console:コマンド出力結果
root@k8s-control-plane:~# kubectl get nodes
NAME                STATUS   ROLES           AGE     VERSION
k8s-control-plane   Ready    control-plane   68m     v1.28.2
k8s-worker-node1    Ready    <none>          8m22s   v1.28.2
root@k8s-control-plane:~# 
```

- Podの状態の確認

```console:実行コマンド
kubectl get pods -A
```
```console:コマンド出力結果
root@k8s-control-plane:~# kubectl get pods -A
NAMESPACE      NAME                                        READY   STATUS    RESTARTS   AGE
kube-flannel   kube-flannel-ds-d5kfv                       1/1     Running   0          9m30s
kube-flannel   kube-flannel-ds-gxjw6                       1/1     Running   0          66m
kube-system    coredns-5dd5756b68-8xtct                    1/1     Running   0          68m
kube-system    coredns-5dd5756b68-kmchv                    1/1     Running   0          68m
kube-system    etcd-k8s-control-plane                      1/1     Running   0          69m
kube-system    kube-apiserver-k8s-control-plane            1/1     Running   0          69m
kube-system    kube-controller-manager-k8s-control-plane   1/1     Running   0          69m
kube-system    kube-proxy-n44hz                            1/1     Running   0          68m
kube-system    kube-proxy-zwqt9                            1/1     Running   0          9m30s
kube-system    kube-scheduler-k8s-control-plane            1/1     Running   0          69m
root@k8s-control-plane:~# 
```

# 5. 参考リンク

https://tech.buty4649.net/entry/2023/08/02/125931

https://kubernetes.io/ja/docs/home/

# 6. 構築時につまづいた事

## kubernetesクラスタ構築後、Control Planeの各種PodがCrashLoopBackOffを繰り返す

### 事象

kubernetes構築完了後、問題なくPodが上がっているように見えたが、時間が経つとControl Planeの各種Pod（kube-apiserver／kube-controller-manager／kube-proxy・・・etc）がCrashLoopBackOffとなりkubectlコマンドが通らなくなった。

### 原因

不明・・・（ログ等をもとに調査したものの、分からず。）
おそらく、containerd／runC／Kubernetes／Flannelのバージョンの互換性に起因していた模様。

### 対応

containerdのバージョンを変更して再構築することで解決。