kubernetesのWORKERノードをWindows Serverで構築

何となく、構築してみたかっただけです。
適当です。

~~2020/03/30 00:50現在~~ 
~~なんかnssmのサイト落ちてる？~~
 
------
#0 構築環境 

・ノード構成

| ノード種類 | ホスト名 | OS | IPアドレス | サブネットマスク | 備考 |
| --- | --- | --- | --- | --- | --- |
| MASTER | VM200001 |CentOS 7.7 | 192.168.1.221 | 24 | |
| WORKER | VM200002 | Windows Server 2019 Standard | 192.168.1.222 | 24 | ServerCore、英語版、評価版 |

・kubernetesネットワーク

| ネットワーク種類 | ネットワーク | サブネットマスク |
| --- | --- | ---|
| POD CIDER | 10.244.0.0 | 16 |
| SERVICE CIDER | 10.0.0.0 | 16 |

・hosts
/etc/hosts,C:\Windows\System32\drivers\etc\hostsに以下を登録

~~~
192.168.1.221 VM200001
192.168.1.222 VM200002
~~~

#1 MASTERノードの構築

## 1.1 SELinux無効化
~~~
# sed -i -e 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
# setenforce 0
~~~

## 1.2 SWAP無効化
~~~
# sed -i -e '/swap/d' /etc/fstab
# swapoff -a
~~~

## 1.3 カーネルパラメタ設定
~~~
# cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

# sysctl --system
~~~

## 1.4 FireWall無効化

~~~
# systemctl disable firewalld
# systemctl stop firewalld
~~~


## 1.5 yumリポジトリ登録
~~~
$ cat > /etc/yum.repos.d/add.repo << EOF
[docker-ce-stable]
name=Docker CE Stable
baseurl=https://download.docker.com/linux/centos/7/x86_64/stable
enabled=1
gpgcheck=1
gpgkey=https://download.docker.com/linux/centos/gpg

[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
~~~

## 1.6 Docker、kubernetesのインストール
~~~
# yum install yum-utils device-mapper-persistent-data lvm2 nfs-utils rpcbind wget -y
# yum install docker-ce kubelet kubeadm kubectl -y
~~~

## 1.7 DockerのCgroup設定
~~~
# mkdir /etc/docker/
# echo {\"exec-opts\":[\"native.cgroupdriver=systemd\"]} > /etc/docker/daemon.json
~~~

## 1.8 各種サービス起動設定
~~~
# systemctl enable docker
# systemctl restart docker

# systemctl enable kubelet
# systemctl restart kubelet
~~~

## 1.9 初期化
~~~
# kubeadm init --pod-network-cidr=10.244.0.0/16 --service-cidr=10.0.0.0/16 
~~~

## 1.10 MASTERノードでもPODが動くようにする。

***※今回はLinuxのWORKERノードは作らない***

~~~
# kubectl taint nodes --all node-role.kubernetes.io/master-
~~~


## 1.11 CNI(flannel)のマニュフェストの修正
~~~
# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
# vi kube-flannel.yml
~~~

・修正箇所1

~~~
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }

      	↓

  cni-conf.json: |
    {
      "name": "vxlan0", ★変更箇所
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
~~~

・修正箇所2

~~~
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }

      	↓

  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan", ★変更箇所
        "VNI" : 4096,    ★変更箇所
        "Port": 4789     ★変更箇所
      }
    }

~~~

・修正箇所3

~~~
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-amd64
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:


      	↓

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-amd64
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      nodeSelector:                 ★追加箇所
        kubernetes.io/os: linux     ★追加箇所
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
~~~

## 1.12 CNI(flannel)のインストール

~~~
# kubectl apply -f kube-flannel.yml
~~~

## 1.13 CNI(flannel)の不要なDaemonSetを削除

~~~
# kubectl -n kube-system delete ds \
kube-flannel-ds-arm \
kube-flannel-ds-arm64 \
kube-flannel-ds-ppc64le \
kube-flannel-ds-s390x
~~~
#2 Windows WORKERノードの構築

**※PowerShellで実施する。**

## 2.1 インストールディレクトリ作成

~~~
mkdir c:\k
~~~

## 2.2 バイナリプログラムおよびインストールスクリプトを設置

~~~
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
wget https://storage.googleapis.com/kubernetes-release/release/v1.18.0/bin/windows/amd64/kubectl.exe -o c:\k\kubectl.exe
wget https://storage.googleapis.com/kubernetes-release/release/v1.18.0/bin/windows/amd64/kubelet.exe -o c:\k\kubelet.exe
wget https://storage.googleapis.com/kubernetes-release/release/v1.18.0/bin/windows/amd64/kube-proxy.exe -o c:\k\kube-proxy.exe
wget https://github.com/coreos/flannel/releases/download/v0.12.0/flanneld.exe -o c:\k\flanneld.exe
wget https://github.com/Microsoft/SDN/raw/master/Kubernetes/flannel/start.ps1 -o c:\k\start.ps1
wget https://raw.githubusercontent.com/microsoft/SDN/master/Kubernetes/flannel/register-svc.ps1 -o c:\k\register-svc.ps1
wget https://nssm.cc/release/nssm-2.24.zip -o c:\k\nssm-2.24.zip
Expand-Archive -Path c:\k\nssm-2.24.zip -DestinationPath c:\k
copy c:\k\nssm-2.24\win64\nssm.exe c:\k
~~~


## 2.3 configの設置

MASTERノードの/etc/kubernetes/admin.confをc:\k\の下にconfigという名前でコピーする。


## 2.4 環境変数(PATHとKUBECONFIG)の設定

~~~
[Environment]::SetEnvironmentVariable("Path", $env:Path + ";C:\k", [EnvironmentVariableTarget]::Machine)
[Environment]::SetEnvironmentVariable("KUBECONFIG","C:\k\config", [EnvironmentVariableTarget]::User)
~~~
## 2.5 Dockerのインストール

~~~
Install-Module -Name DockerMsftProvider -Repository PSGallery -Force
Install-Package -Name Docker -ProviderName DockerMsftProvider
~~~

## 2.6 再起動

~~~
Restart-Computer -Force
~~~


## 2.7 WORKERとして参加

~~~
cd c:\k
.\start.ps1 -ManagementIP <WindowsノードIP>  -NetworkMode overlay -ClusterCIDR <POD CIDR> -ServiceCIDR <Service Cider> -KubeDnsServiceIP <kube-DNSのサービスIP> -LogDir c:\k
~~~

・例

~~~
cd c:\k
.\start.ps1 -ManagementIP 192.168.1.222  -NetworkMode overlay -ClusterCIDR 10.244.0.0/16 -ServiceCIDR 10.0.0.0/16 -KubeDnsServiceIP 10.0.0.10 -LogDir c:\k
~~~

## 2.8 サービス自動起動設定

~~~
C:\k\register-svc.ps1 -NetworkMode overlay -ManagementIP <WindowsノードIP> -ClusterCIDR <POD CIDR> -KubeDnsServiceIP <kube-DNSのサービスIP> -LogDir c:\k
~~~

・例

~~~
C:\k\register-svc.ps1 -NetworkMode overlay -ManagementIP 192.168.1.222 -ClusterCIDR 10.244.0.0/16 -KubeDnsServiceIP 10.0.0.10 -LogDir c:\k
~~~
# おまけ
~~~
[root@VM200001 ~]#
[root@VM200001 ~]#
[root@VM200001 ~]# kubectl  get node -o wide
NAME       STATUS   ROLES    AGE     VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                                  KERNEL-VERSION                CONTAINER-RUNTIME
vm200001   Ready    master   19m     v1.18.0   192.168.1.221   <none>        CentOS Linux 7 (Core)                     3.10.0-1062.18.1.el7.x86_64   docker://19.3.8
vm200002   Ready    <none>   2m49s   v1.18.0   192.168.1.222   <none>        Windows Server 2019 Standard Evaluation   10.0.17763.1098               docker://19.3.5
[root@VM200001 ~]#
[root@VM200001 ~]#
[root@VM200001 ~]#
[root@VM200001 ~]#
[root@VM200001 ~]# kubectl  get node --show-labels
NAME       STATUS   ROLES    AGE     VERSION   LABELS
vm200001   Ready    master   20m     v1.18.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=vm200001,kubernetes.io/os=linux,node-role.kubernetes.io/master=
vm200002   Ready    <none>   2m56s   v1.18.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=windows,kubernetes.io/arch=amd64,kubernetes.io/hostname=vm200002,kubernetes.io/os=windows,node.kubernetes.io/windows-build=10.0.17763
[root@VM200001 ~]#
[root@VM200001 ~]#
[root@VM200001 ~]#
[root@VM200001 ~]#
[root@VM200001 ~]#
[root@VM200001 ~]#
[root@VM200001 ~]# kubectl  get all -o wide -A
NAMESPACE     NAME                                   READY   STATUS    RESTARTS   AGE   IP              NODE       NOMINATED NODE   READINESS GATES
kube-system   pod/coredns-66bff467f8-svj5g           1/1     Running   0          19m   10.244.0.3      vm200001   <none>           <none>
kube-system   pod/coredns-66bff467f8-vznrl           1/1     Running   0          19m   10.244.0.2      vm200001   <none>           <none>
kube-system   pod/etcd-vm200001                      1/1     Running   0          20m   192.168.1.221   vm200001   <none>           <none>
kube-system   pod/kube-apiserver-vm200001            1/1     Running   0          20m   192.168.1.221   vm200001   <none>           <none>
kube-system   pod/kube-controller-manager-vm200001   1/1     Running   0          20m   192.168.1.221   vm200001   <none>           <none>
kube-system   pod/kube-flannel-ds-amd64-cd7rq        1/1     Running   0          17m   192.168.1.221   vm200001   <none>           <none>
kube-system   pod/kube-proxy-m9qn2                   1/1     Running   0          19m   192.168.1.221   vm200001   <none>           <none>
kube-system   pod/kube-scheduler-vm200001            1/1     Running   0          20m   192.168.1.221   vm200001   <none>           <none>

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE   SELECTOR
default       service/kubernetes   ClusterIP   10.0.0.1     <none>        443/TCP                  20m   <none>
kube-system   service/kube-dns     ClusterIP   10.0.0.10    <none>        53/UDP,53/TCP,9153/TCP   20m   k8s-app=kube-dns

NAMESPACE     NAME                                   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE   CONTAINERS     IMAGES                                 SELECTOR
kube-system   daemonset.apps/kube-flannel-ds-amd64   1         1         1       1            1           kubernetes.io/os=linux   17m   kube-flannel   quay.io/coreos/flannel:v0.12.0-amd64   app=flannel
kube-system   daemonset.apps/kube-proxy              1         1         1       1            1           kubernetes.io/os=linux   20m   kube-proxy     k8s.gcr.io/kube-proxy:v1.18.0          k8s-app=kube-proxy

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                     SELECTOR
kube-system   deployment.apps/coredns   2/2     2            2           20m   coredns      k8s.gcr.io/coredns:1.6.7   k8s-app=kube-dns

NAMESPACE     NAME                                 DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES                     SELECTOR
kube-system   replicaset.apps/coredns-66bff467f8   2         2         2       19m   coredns      k8s.gcr.io/coredns:1.6.7   k8s-app=kube-dns,pod-template-hash=66bff467f8
[root@VM200001 ~]#
[root@VM200001 ~]#
[root@VM200001 ~]#
[root@VM200001 ~]#
[root@VM200001 ~]#
[root@VM200001 ~]#
[root@VM200001 ~]#
[root@VM200001 ~]#
~~~

~~~
[root@VM200001 ~]#
[root@VM200001 ~]# cat iis.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: iis
  labels:
    app: iis
spec:
  replicas: 2
  selector:
    matchLabels:
      app: iis
  template:
    metadata:
      labels:
        app: iis
    spec:
      nodeSelector:
        kubernetes.io/os: windows
      containers:
      - name: iis
        image: mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2019
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: iis
  labels:
    app: iis
spec:
  ports:
  - port: 80
    targetPort: 80
    nodePort: 30080
  selector:
    app: iis
  type: NodePort
[root@VM200001 ~]#
[root@VM200001 ~]#
[root@VM200001 ~]#
[root@VM200001 ~]#
[root@VM200001 ~]#
[root@VM200001 ~]#
[root@VM200001 ~]#
[root@VM200001 ~]# kubectl apply -f iis.yaml
deployment.apps/iis created
service/iis created
[root@VM200001 ~]#
[root@VM200001 ~]#
[root@VM200001 ~]#
[root@VM200001 ~]#
[root@VM200001 ~]#
[root@VM200001 ~]# kubectl  get deploy,pod,svc -o wide
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                                                                SELECTOR
deployment.apps/iis   2/2     2            2           56s   iis          mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2019   app=iis

NAME                       READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
pod/iis-6975cd4f89-28v94   1/1     Running   0          56s   10.244.1.3   vm200002   <none>           <none>
pod/iis-6975cd4f89-dq6jc   1/1     Running   0          56s   10.244.1.4   vm200002   <none>           <none>

NAME                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE   SELECTOR
service/iis          NodePort    10.0.119.23   <none>        80:30080/TCP   56s   app=iis
service/kubernetes   ClusterIP   10.0.0.1      <none>        443/TCP        37m   <none>
[root@VM200001 ~]#
[root@VM200001 ~]#
[root@VM200001 ~]#
~~~

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/509257/5c8c8379-e5a4-6598-a6cd-d2609662932f.png)

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/509257/89a10daf-1036-5b3d-266a-f77856e5796e.png)

-----
# 参考
Windows 上のコンテナーに関するドキュメント
https://docs.microsoft.com/ja-jp/virtualization/windowscontainers/