GitLab Helm ChartをAWS EKSにインストールする

GitLab v11.0.0からCloud Native GitLab Helm ChartがGitLab Helm Chartに改名され、Beta版になりました。

- https://docs.gitlab.com/ee/install/kubernetes/gitlab_chart.html

[EKSへのインストール公式手順](https://gitlab.com/charts/gitlab/blob/master/doc/cloud/eks.md)がイマイチだったのでこちらでまとめてみました。

![raccoon-dog-tanuki_2015_05_29.jpg](https://qiita-image-store.s3.amazonaws.com/0/260796/097874de-a6b9-9fa9-819f-8b8abfe2f37b.jpeg)

# 事前準備

Mac OSで作業しています。事前に以下を揃えておく必要があります。

* [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-with-homebrew-on-macos)
* [Helm](https://docs.helm.sh/using_helm/#from-homebrew-macos)
* [eksctl](https://eksctl.io/)

# EKSクラスタの構築

[以前ご紹介したEKSCTLツール](https://qiita.com/jb-vasseur/items/6f7d5f8eef1913da211b)でクラスタを作成します。

```shell-session
> eksctl create cluster \
                                            --cluster-name $NAME \
                                            --nodes 3 \
                                            --nodes-min 3 \
                                            --nodes-max 5 \
                                            --node-type t2.medium \
                                            --region us-west-2
2018-07-18T13:33:01+09:00 [ℹ]  importing SSH public key "/Users/jb/.ssh/id_rsa.pub" as "EKS-bakeneco"
2018-07-18T13:33:02+09:00 [ℹ]  creating EKS cluster "bakeneco" in "us-west-2" region
2018-07-18T13:33:02+09:00 [ℹ]  creating VPC stack "EKS-bakeneco-VPC"
2018-07-18T13:33:02+09:00 [ℹ]  creating ServiceRole stack "EKS-bakeneco-ServiceRole"
2018-07-18T13:33:22+09:00 [✔]  created ServiceRole stack "EKS-bakeneco-ServiceRole"
2018-07-18T13:34:03+09:00 [✔]  created VPC stack "EKS-bakeneco-VPC"
2018-07-18T13:34:03+09:00 [ℹ]  creating control plane "bakeneco"
2018-07-18T13:46:05+09:00 [✔]  created control plane "bakeneco"
2018-07-18T13:46:05+09:00 [ℹ]  creating DefaultNodeGroup stack "EKS-bakeneco-DefaultNodeGroup"
2018-07-18T13:49:47+09:00 [✔]  created DefaultNodeGroup stack "EKS-bakeneco-DefaultNodeGroup"
2018-07-18T13:49:47+09:00 [✔]  all EKS cluster "bakeneco" resources has been created
2018-07-18T13:49:47+09:00 [ℹ]  wrote "kubeconfig"
2018-07-18T13:49:48+09:00 [ℹ]  the cluster has 0 nodes
2018-07-18T13:49:48+09:00 [ℹ]  waiting for at least 3 nodes to become ready
2018-07-18T13:50:15+09:00 [ℹ]  the cluster has 4 nodes
2018-07-18T13:50:15+09:00 [ℹ]  node "ip-192-168-100-194.us-west-2.compute.internal" is ready
2018-07-18T13:50:15+09:00 [ℹ]  node "ip-192-168-128-38.us-west-2.compute.internal" is ready
2018-07-18T13:50:15+09:00 [ℹ]  node "ip-192-168-177-212.us-west-2.compute.internal" is ready
2018-07-18T13:50:15+09:00 [ℹ]  node "ip-192-168-209-209.us-west-2.compute.internal" is ready
2018-07-18T13:50:17+09:00 [ℹ]  all command should work, try '/usr/local/bin/kubectl --kubeconfig kubeconfig get nodes'
2018-07-18T13:50:17+09:00 [ℹ]  EKS cluster "bakeneco" in "us-west-2" region is ready
```

kubectlで接続できるように`$KUBECONFIG`環境変数を設定のうえ、確認します。

```shell-session
> kubectl get nodes
NAME                                            STATUS    ROLES     AGE       VERSION
ip-192-168-100-194.us-west-2.compute.internal   Ready     <none>    14m       v1.10.3
ip-192-168-106-191.us-west-2.compute.internal   Ready     <none>    13m       v1.10.3
ip-192-168-128-38.us-west-2.compute.internal    Ready     <none>    14m       v1.10.3
ip-192-168-177-212.us-west-2.compute.internal   Ready     <none>    14m       v1.10.3
ip-192-168-209-209.us-west-2.compute.internal   Ready     <none>    14m       v1.10.3

> kubectl get services
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.100.0.1   <none>        443/TCP   20m

> kubectl get deployments --all-namespaces
NAMESPACE     NAME       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
kube-system   kube-dns   1         1         1            1           24m

> kubectl get pods --all-namespaces
NAMESPACE     NAME                       READY     STATUS    RESTARTS   AGE
kube-system   aws-node-d2v4m             1/1       Running   1          14m
kube-system   aws-node-fbcr6             1/1       Running   0          13m
kube-system   aws-node-ggnps             1/1       Running   1          14m
kube-system   aws-node-s2mnr             1/1       Running   0          14m
kube-system   aws-node-z274r             1/1       Running   0          14m
kube-system   kube-dns-7cc87d595-kdkcv   3/3       Running   0          19m
kube-system   kube-proxy-5m9gf           1/1       Running   0          13m
kube-system   kube-proxy-8fh9v           1/1       Running   0          14m
kube-system   kube-proxy-ggk7j           1/1       Running   0          14m
kube-system   kube-proxy-qngg2           1/1       Running   0          14m
kube-system   kube-proxy-rxhjn           1/1       Running   0          14m
```

# クラスタ環境の設定

### [ネットワーク設定](https://docs.gitlab.com/ee/install/kubernetes/preparation/networking.html)

> Amazon EKS utilizes Elastic Load Balancers, which are addressed by DNS name and cannot be known ahead of time. Skip this section.

はい、スキップします。

### [ストレージ設定](https://gitlab.com/charts/gitlab/blob/master/doc/installation/storage.md)

Persistent volumeはdynamic provisioningを採用します。特定のゾーンに縛られてしまいますが現時点で仕方がないと思います。
各ストレージが分けられますが今回シンプルに1ストレージクラスにします。

```yaml:gp2-storage-class.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: gp2
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
  zone: us-west-2a
reclaimPolicy: Retain
mountOptions:
  - debug
```

```shell-session
> kubectl create -f gp2-storage-class.yaml
storageclass.storage.k8s.io "gp2" created

> kubectl patch storageclass gp2 -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
storageclass.storage.k8s.io "gp2" patched

> kubectl get storageclass
NAME            PROVISIONER             AGE
gp2 (default)   kubernetes.io/aws-ebs   37s
```

### [Tiller](https://docs.gitlab.com/ee/install/kubernetes/preparation/tiller.html)

> Some clusters require authentication to use kubectl to create the Tiller roles.
> For clusters like Amazon EKS, you can directly upload the RBAC configuration.

ということで、Tiller用のServiceAccountを作成しcluster-adminロールを付与します。

```yaml:rbac-config.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system
```

```shell-session
> kubectl create -f rbac-config.yaml
serviceaccount "tiller" created
clusterrolebinding.rbac.authorization.k8s.io "tiller" created
```

そしてTillerをインストールし初期化します。

```shell-session
> helm init --service-account tiller
$HELM_HOME has been configured at /Users/jb/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!
```

# GitLab Chartインストール

GitLab chart repositoryを追加します。

```shell-session
> helm repo add gitlab https://charts.gitlab.io/
"gitlab" has been added to your repositories
```

HelmでGitLab Chartをインストールします。

```shell-session
> helm upgrade --install gitlab gitlab/gitlab \
                                   --timeout 600 \
                                   --set global.hosts.domain=bakeneco.io \
                                   --set gitlab.migrations.initialRootPassword="XXX" \
                                   --set certmanager-issuer.email=XX@YYY.com 
Release "gitlab" does not exist. Installing it now.
NAME:   gitlab
LAST DEPLOYED: Wed Jul 18 17:50:08 2018
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/ServiceAccount
NAME                                  SECRETS  AGE
gitlab-certmanager-issuer             1        11s
certmanager-gitlab                    1        11s
gitlab-gitlab-runner                  1        11s
gitlab-nginx-ingress                  1        11s
gitlab-prometheus-alertmanager        1        11s
gitlab-prometheus-kube-state-metrics  1        11s
gitlab-prometheus-node-exporter       1        11s
gitlab-prometheus-server              1        11s

==> v1/RoleBinding
NAME                       AGE
gitlab-certmanager-issuer  3s
gitlab-nginx-ingress       3s

==> v1beta1/Ingress
NAME             HOSTS                 ADDRESS  PORTS  AGE
gitlab-unicorn   gitlab.bakeneco.io    80, 443  2s
gitlab-minio     minio.bakeneco.io     80, 443  2s
gitlab-registry  registry.bakeneco.io  80, 443  2s

==> v2beta1/HorizontalPodAutoscaler
NAME                     REFERENCE                           TARGETS        MINPODS  MAXPODS  REPLICAS  AGE
gitlab-gitlab-shell      Deployment/gitlab-gitlab-shell      <unknown>/75%  2        10       0         2s
gitlab-sidekiq-all-in-1  Deployment/gitlab-sidekiq-all-in-1  <unknown>/75%  1        10       0         2s
gitlab-unicorn           Deployment/gitlab-unicorn           <unknown>/75%  2        10       0         2s
gitlab-registry          Deployment/gitlab-registry          <unknown>/75%  2        10       0         2s

==> v1/Pod(related)
NAME                                                   READY  STATUS             RESTARTS  AGE
certmanager-gitlab-54467869c4-c7q6t                    0/2    ContainerCreating  0         2s
gitlab-gitlab-runner-845c5b46d8-9vwjm                  0/1    Init:0/1           0         2s
gitlab-gitlab-shell-7d8cd44948-lccrx                   0/1    Init:0/1           0         2s
gitlab-sidekiq-all-in-1-58c996c9fb-mrcd5               0/1    Init:0/2           0         2s
gitlab-task-runner-866bc87864-tgg4p                    0/1    Init:0/1           0         2s
gitlab-unicorn-5c69b9487c-v4pnq                        0/1    Init:0/2           0         2s
gitlab-minio-99bff897b-6cplp                           0/1    Pending            0         2s
gitlab-nginx-ingress-controller-65d58cbf4d-5zxlm       0/1    ContainerCreating  0         2s
gitlab-nginx-ingress-controller-65d58cbf4d-kp9fg       0/1    ContainerCreating  0         2s
gitlab-nginx-ingress-controller-65d58cbf4d-xttvr       0/1    ContainerCreating  0         2s
gitlab-nginx-ingress-default-backend-699b9476dd-5kjjt  0/1    ContainerCreating  0         1s
gitlab-nginx-ingress-default-backend-699b9476dd-jx86w  0/1    Pending            0         1s
gitlab-postgresql-5578b89f58-4jp9k                     0/2    ContainerCreating  0         1s
gitlab-prometheus-server-847c8bb76-9jqm6               0/2    Pending            0         1s
gitlab-redis-6b8b6dbfd9-bs24z                          0/2    Init:0/1           0         1s
gitlab-registry-7f4b9ccfc8-k4zsj                       0/1    Pending            0         1s
gitlab-gitaly-0                                        0/1    Pending            0         2s
gitlab-issuer.1-2xqhq                                  0/1    ContainerCreating  0         2s
gitlab-migrations.1-zs7tj                              0/1    Init:0/1           0         2s
gitlab-minio-create-buckets.1-97wvg                    0/1    ContainerCreating  0         2s

==> v1beta1/CustomResourceDefinition
NAME                               AGE
certificates.certmanager.k8s.io    7s
clusterissuers.certmanager.k8s.io  3s
issuers.certmanager.k8s.io         3s

==> v1beta1/ClusterRole
certmanager-gitlab                    3s
gitlab-prometheus-kube-state-metrics  3s
gitlab-prometheus-server              3s

==> v1beta1/ClusterRoleBinding
NAME                                  AGE
certmanager-gitlab                    3s
gitlab-prometheus-alertmanager        3s
gitlab-prometheus-kube-state-metrics  3s
gitlab-prometheus-node-exporter       3s
gitlab-prometheus-server              3s

==> v1/Role
NAME                       AGE
gitlab-certmanager-issuer  3s
gitlab-nginx-ingress       3s

==> v1beta2/Deployment
NAME                                  DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
gitlab-gitlab-shell                   1        1        1           0          2s
gitlab-sidekiq-all-in-1               1        1        1           0          2s
gitlab-task-runner                    1        1        1           0          2s
gitlab-unicorn                        1        1        1           0          2s
gitlab-minio                          1        1        1           0          2s
gitlab-nginx-ingress-controller       3        0        0           0          2s
gitlab-nginx-ingress-default-backend  2        0        0           0          2s
gitlab-redis                          1        0        0           0          2s
gitlab-registry                       1        0        0           0          2s

==> v1beta2/StatefulSet
NAME           DESIRED  CURRENT  AGE
gitlab-gitaly  1        1        2s

==> v1beta1/PodDisruptionBudget
NAME                                  MIN AVAILABLE  MAX UNAVAILABLE  ALLOWED DISRUPTIONS  AGE
gitlab-gitaly                         N/A            1                0                    2s
gitlab-gitlab-shell                   N/A            1                0                    2s
gitlab-sidekiq                        N/A            1                0                    2s
gitlab-unicorn                        N/A            1                0                    2s
gitlab-minio-v1                       N/A            1                0                    2s
gitlab-nginx-ingress-controller       2              N/A              0                    2s
gitlab-nginx-ingress-default-backend  1              N/A              0                    2s
gitlab-redis-v1                       N/A            1                0                    2s
gitlab-registry-v1                    N/A            1                0                    2s

==> v1/Service
NAME                                  TYPE          CLUSTER-IP      EXTERNAL-IP  PORT(S)                                  AGE
gitlab-gitaly                         ClusterIP     None            <none>       8075/TCP,9236/TCP                        3s
gitlab-gitlab-shell                   ClusterIP     10.100.96.36    <none>       22/TCP                                   3s
gitlab-unicorn                        ClusterIP     10.100.170.239  <none>       8080/TCP,8181/TCP                        3s
gitlab-minio-svc                      ClusterIP     10.100.14.232   <none>       9000/TCP                                 3s
gitlab-nginx-ingress-controller       LoadBalancer  10.100.234.221  <pending>    80:30364/TCP,443:30849/TCP,22:31508/TCP  3s
gitlab-nginx-ingress-default-backend  ClusterIP     10.100.243.113  <none>       80/TCP                                   3s
gitlab-postgresql                     ClusterIP     10.100.10.190   <none>       5432/TCP                                 3s
gitlab-prometheus-server              ClusterIP     10.100.191.234  <none>       80/TCP                                   3s
gitlab-redis                          ClusterIP     10.100.110.213  <none>       6379/TCP,9121/TCP                        3s
gitlab-registry                       ClusterIP     10.100.51.212   <none>       5000/TCP                                 2s

==> v1/Job
NAME                           DESIRED  SUCCESSFUL  AGE
gitlab-issuer.1                1        0           2s
gitlab-migrations.1            1        0           2s
gitlab-minio-create-buckets.1  1        0           2s

==> v1/ConfigMap
NAME                                   DATA  AGE
gitlab-certmanager-issuer-certmanager  2     11s
gitlab-gitlab-runner                   3     11s
gitlab-gitaly                          3     11s
gitlab-gitlab-shell                    2     11s
gitlab-nginx-ingress-tcp               1     11s
gitlab-migrations                      4     11s
gitlab-sidekiq-all-in-1                1     11s
gitlab-sidekiq                         6     11s
gitlab-task-runner                     4     11s
gitlab-unicorn                         8     11s
gitlab-unicorn-tests                   1     11s
gitlab-minio-config-cm                 3     11s
gitlab-nginx-ingress-controller        7     11s
gitlab-postgresql                      0     11s
gitlab-prometheus-server               3     11s
gitlab-redis                           2     11s
gitlab-registry                        2     11s

==> v1/PersistentVolumeClaim
NAME                      STATUS   VOLUME                                    CAPACITY  ACCESS MODES  STORAGECLASS  AGE
gitlab-minio              Pending  gp2                                       11s
gitlab-postgresql         Bound    pvc-9b3e012b-8a67-11e8-8b92-02b5389a29ae  8Gi  RWO  gp2  11s
gitlab-prometheus-server  Bound    pvc-9b3f2af5-8a67-11e8-8b92-02b5389a29ae  8Gi  RWO  gp2  11s
gitlab-redis              Bound    pvc-9b400dee-8a67-11e8-8b92-02b5389a29ae  5Gi  RWO  gp2  11s

==> v1beta1/Role
NAME                  AGE
gitlab-gitlab-runner  3s

==> v1beta1/RoleBinding
NAME                  AGE
gitlab-gitlab-runner  3s

==> v1beta1/Deployment
NAME                      DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
certmanager-gitlab        1        1        1           0          2s
gitlab-gitlab-runner      1        1        1           0          2s
gitlab-postgresql         1        1        1           0          2s
gitlab-prometheus-server  1        1        1           0          2s

```

暫く見てみたら、GitLab Runnerがうまく立ち上がらないことに気づきました。

```shell-session
> kubectl logs gitlab-gitlab-runner-845c5b46d8-9vwjm
WARNING: Running in user-mode.                     
WARNING: The user-mode requires you to manually start builds processing: 
WARNING: $ gitlab-runner run                       
WARNING: Use sudo for system-mode:                 
WARNING: $ sudo gitlab-runner...                   
                                                   
ERROR: Registering runner... failed                 runner=3clGLmm7 status=couldn't execute POST against https://gitlab.bakeneco.io/api/v4/runners: Post https://gitlab.bakeneco.io/api/v4/runners: dial tcp: lookup gitlab.bakeneco.io on 10.100.0.10:53: no such host
PANIC: Failed to register this runner. Perhaps you are having network problems 
```

どうやらドメインのDNSレコードにIngress LoadBalancerのアドレスを登録しておく必要があるようです。
これはHelm installからでないと分からないのでこのタイミングで確認して登録します。

```shell-session
> kubectl describe service gitlab-nginx-ingress-controller | grep Ingress
LoadBalancer Ingress:     aa03b57e68a6711e88b9202b5389a29a-974622501.us-west-2.elb.amazonaws.com
```

DNSレコード登録後、暫く待つと立ち上がります。

```shell-session
> kubectl get pods --all-namespaces
NAMESPACE     NAME                                                    READY     STATUS      RESTARTS   AGE
default       certmanager-gitlab-54467869c4-c7q6t                     2/2       Running     0          44m
default       gitlab-gitaly-0                                         1/1       Running     0          44m
default       gitlab-gitlab-runner-845c5b46d8-9vwjm                   1/1       Running     13         44m
default       gitlab-gitlab-shell-7d8cd44948-fxxls                    1/1       Running     0          44m
default       gitlab-gitlab-shell-7d8cd44948-lccrx                    1/1       Running     0          44m
default       gitlab-issuer.1-2xqhq                                   0/1       Completed   0          44m
default       gitlab-migrations.1-zs7tj                               0/1       Completed   0          44m
default       gitlab-minio-99bff897b-6cplp                            1/1       Running     0          44m
default       gitlab-minio-create-buckets.1-97wvg                     0/1       Completed   0          44m
default       gitlab-nginx-ingress-controller-65d58cbf4d-5zxlm        1/1       Running     0          44m
default       gitlab-nginx-ingress-controller-65d58cbf4d-kp9fg        1/1       Running     0          44m
default       gitlab-nginx-ingress-controller-65d58cbf4d-xttvr        1/1       Running     0          44m
default       gitlab-nginx-ingress-default-backend-699b9476dd-5kjjt   1/1       Running     0          44m
default       gitlab-nginx-ingress-default-backend-699b9476dd-jx86w   1/1       Running     0          44m
default       gitlab-postgresql-5578b89f58-4jp9k                      2/2       Running     0          44m
default       gitlab-prometheus-server-847c8bb76-9jqm6                2/2       Running     0          44m
default       gitlab-redis-6b8b6dbfd9-bs24z                           2/2       Running     0          44m
default       gitlab-registry-7f4b9ccfc8-k4zsj                        1/1       Running     0          44m
default       gitlab-registry-7f4b9ccfc8-kf6sm                        1/1       Running     0          44m
default       gitlab-sidekiq-all-in-1-58c996c9fb-mrcd5                1/1       Running     0          44m
default       gitlab-task-runner-866bc87864-tgg4p                     1/1       Running     0          44m
default       gitlab-unicorn-5c69b9487c-j5klw                         1/1       Running     0          44m
default       gitlab-unicorn-5c69b9487c-v4pnq                         1/1       Running     0          44m
kube-system   aws-node-d2v4m                                          1/1       Running     1          4h
kube-system   aws-node-fbcr6                                          1/1       Running     0          4h
kube-system   aws-node-ggnps                                          1/1       Running     1          4h
kube-system   aws-node-s2mnr                                          1/1       Running     0          4h
kube-system   aws-node-z274r                                          1/1       Running     0          4h
kube-system   kube-dns-7cc87d595-kdkcv                                3/3       Running     0          4h
kube-system   kube-proxy-5m9gf                                        1/1       Running     0          4h
kube-system   kube-proxy-8fh9v                                        1/1       Running     0          4h
kube-system   kube-proxy-ggk7j                                        1/1       Running     0          4h
kube-system   kube-proxy-qngg2                                        1/1       Running     0          4h
kube-system   kube-proxy-rxhjn                                        1/1       Running     0          4h
kube-system   tiller-deploy-f5597467b-b2c5c                           1/1       Running     0          4h
```

GitLabインスタンスにログインしてみたところパラメータで渡したパスワードが効かないようです。
以下で設定されたパスワードを確認しログインします。

```
> kubectl get secret gitlab-gitlab-initial-root-password -ojsonpath='{.data.password}' | base64 --decode
nyanyanya
```

<img width="1342" alt="Screen Shot 2018-07-21 at 11.57.18.png" src="https://qiita-image-store.s3.amazonaws.com/0/260796/40011906-7d80-c404-1f03-2a6a245ebece.png">


デフォルトでHTTPS化されていて, GitLab Shared Runners, Container Registryが入っています。