ICPにELK Stackを追加でデプロイする

ICPにはデフォルトでELKスタックがデプロイされているが、追加でデプロイする場合のメモ。

ICPバージョン: v2.1.0.3
ICPのエディション: CE
ICPの構成: シングル構成
ibm-icplogging: v1.0.0

## チャートのデフォルト値

GitHubやICPのカタログからvaluesのデフォルト値が確認できる。

https://github.com/IBM/charts/tree/master/stable/ibm-icplogging

チャート自体からもvaluesのデフォルト値が確認できる。

```
helm inspect values ibm-charts/ibm-icplogging --version 1.0.0
```

```yaml
# Licensed Materials - Property of IBM
# 5737-E67
# @ Copyright IBM Corporation 2016, 2018. All Rights Reserved.
# US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

image:
  pullPolicy: IfNotPresent

security:
  # Security can be flagged on for any platform, but will only work on x86 nodes.
  enabled: false
  provider: searchguard-tls
  ca:
    keystore:
      password: changeme
    truststore:
      password: changeme
    # set to `external` to use existing CA stored in Kubernetes secret to generate certs
    origin: internal
    external:
      # the secret need to be in the same namespace as the chart release
      secretName: cluster-ca-cert
      # the Kubenetes field name (key) within the specified secret that stores CA cert
      certFieldName: tls.crt
      # the Kubenets field name (key) within the specified secret that stores CA private key
      keyFieldName: tls.key
  app:
    keystore:
      password: changeme

logstash:
  replicas: 1
  name: logstash
  heapSize: "512m"
  memoryLimit: "1024Mi"
  port: 5000
  image:
    repository: "ibmcom/logstash"
    tag: "5.5.1"
  probe:
    enabled: false
    periodSeconds: 60
    minEventsPerPeriod: 1
    maxUnavailablePeriod: 5
    image:
      repository: "ibmcom/logstash-liveness-probe"
      tag: "0.1.8"

kibana:
  install: true
  replicas: 1
  name: kibana
  internal: 5601
  external: 31601
  # maximum old space size (in MB) of the V8 Javascript engine
  maxOldSpaceSize: "1024"
  memoryLimit: "1280Mi"
  image:
    repository: ibmcom/kibana
    tag: 5.5.1
  initImage:
    repository: "ibmcom/curl"
    tag: "3.6"
  routerImage:
    repository: "ibmcom/icp-router"
    tag: "2.2.0"

filebeat:
  name: filebeat-ds
  image:
    repository: "ibmcom/filebeat"
    tag: 5.5.1
  scope:
    nodes:
    namespaces:

elasticsearch:
  name: "elasticsearch"
  internalPort: 9300
  image:
    repository: "ibmcom/elasticsearch"
    tag: "5.5.1"
  pluginImage:
    repository: "ibmcom/elasticsearch-plugin-searchguard"
    tag: "1.0.0"
  pluginInitImage:
    repository: "ibmcom/searchguard-init"
    tag: "1.0.0"
  pkiInitImage:
    repository: "ibmcom/logging-pki-init"
    tag: "1.3.0"
  initImage:
    repository: "ibmcom/icp-initcontainer"
    tag: "1.0.0"

  client:
    name: client
    replicas: 1
    heapSize: "1024m"
    memoryLimit: "1536Mi"
    restPort: 9200
    antiAffinity: soft

  master:
    name: master
    # Set to the # of management (or master, if no mgmt) nodes
    replicas: 1
    heapSize: "1024m"
    memoryLimit: "1536Mi"
    antiAffinity: soft

  data:
    name: data
    # Set to the # of management (or master, if no mgmt) nodes
    replicas: 2
    heapSize: "1024m"
    memoryLimit: "2048M"
    antiAffinity: hard
    storage:
      # When true will expect a PersistentVolume
      persistent: true
      # Set to true if you are using GlusterFS or other dynamic provisioner
      useDynamicProvisioning: false
      # If not using dynamic provisioning, you can use selectors to refine the binding process.
      # These are IGNORED if using dynamic provisioning.
      selector:
        label: ""
        value: ""
      # 10Gi is not the recommended size, but rather a small default.
      # It will match much larger drives as well.
      size: 10Gi
      accessModes:
        - ReadWriteOnce
      ## Specify the storageClass name you want to use
      ## If you don't specify a storageClass name it will use the default
      storageClass: ""

curator:
  name: log-curator
  image:
    repository: "ibmcom/indices-cleaner"
    tag: 0.2
  # runs at 23:30 UTC daily
  schedule: "30 23 * * *"
  app:
    unit: days
    count: 1
  monitoring:
    unit: days
    count: 1
  watcher:
    unit: days
    count: 1

mode: standard

xpack:
  # Must be false for PPC, otherwise the log is spammed with errors
  monitoring: false
  graph: false
  reporting: false
  ml: false
  watcher: false

```

## デフォルトELKのvalues

デフォルトでデプロイされているチャートのvaluesを確認する。

```
helm get values logging --tls
```

```yaml
clusterDomain: cluster.local
curator:
  app:
    count: 1
  image:
    repository: ibmcom/indices-cleaner
  monitoring:
    count: 1
elasticsearch:
  client:
    heapSize: 512m
    memoryLimit: 1024Mi
    restPort: 9200
  data:
    heapSize: 1024m
    memoryLimit: 2048M
    replicas: "1"
    storage:
      persistent: true
      size: 20Gi
      storageClass: logging-storage-datanode
      useDynamicProvisioning: false
  image:
    repository: ibmcom/elasticsearch
  initImage:
    repository: ibmcom/icp-initcontainer
  internalPort: 9300
  master:
    heapSize: 512m
    memoryLimit: 1024Mi
    replicas: "1"
  name: elasticsearch
  pkiInitImage:
    repository: ibmcom/logging-pki-init
  pluginImage:
    repository: ibmcom/elasticsearch-plugin-searchguard
  pluginInitImage:
    repository: ibmcom/searchguard-init
filebeat:
  image:
    repository: ibmcom/filebeat
kibana:
  external: 31601
  image:
    repository: ibmcom/kibana
  install: true
  internal: 5601
logstash:
  heapSize: 512m
  image:
    repository: ibmcom/logstash
  memoryLimit: 1024Mi
  port: 5044
  probe:
    image:
      repository: ibmcom/logstash-liveness-probe
mode: managed
nameOverride: elk
security:
  enabled: false

```

なお、`helm get values`コマンドに`-a`オプションをつけると、明示的に指定していないデフォルト値も含めて表示される。

## 追加デプロイ

### Namespaceの作成

追加のELKスタックをデプロイするNamespaceを作成する。

```
kubectl create ns kiban
```

### 特権の付与

default以外のNamespaceにibm-icploggingをデプロイする場合、特権が必要。特権を付与しないとElasticsearchのPodがスケジュールされない。

[非デフォルト名前空間での高い特権を必要とする Helm チャートのデプロイ](https://www.ibm.com/support/knowledgecenter/ja/SSBS6K_2.1.0.3/app_center/nd_helm.html)

PodSecurityPolicyを作成する。

```yaml:icplogging-psp.yaml
apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
  name: icplogging
spec:
  privileged: true
  allowedCapabilities:
  - IPC_LOCK
  fsGroup:
    rule: RunAsAny
  runAsUser:
    rule: RunAsAny
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  volumes:
  - "*"
```

```
kubectl apply -f icplogging-psp.yaml
```

ClusterRoleを作成する。

```yaml:icplogging-clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: icplogging-role
rules:
- apiGroups:
  - extensions
  resourceNames:
  - icplogging
  resources:
  - podsecuritypolicies
  verbs:
  - use
```

```
kubectl apply -f icplogging-clusterrole.yaml
```

ClusterRoleBindingを作成する。

```yaml:logging-clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: logging-psp-users
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: icplogging-role
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:serviceaccounts:kiban
```

`system:serviceaccounts:kiban`の部分を作成したNamespaceにあわせること。

```
kubectl apply -f logging-clusterrolebinding.yaml
```

### valuesを指定してデプロイ

デフォルトELKを参考にして、ファイルでvaluesを渡してみる。
Elasticsearchのデータ保管場所はデフォルトELKとは別にする必要がある。

デフォルトELKのvaluesを`values.yaml`ファイルに保存する。

```
helm get values logging --tls > values.yaml
```

`values.yaml`を修正する。
storageClass名を変更しておく。その他、kibanaのインストールの有無や、Filebeatがログ収集する対象のNamespaceなど、カスタマイズを行う。

```yaml:values.yaml
clusterDomain: cluster.local
curator:
  app:
    count: 1
  image:
    repository: ibmcom/indices-cleaner
  monitoring:
    count: 1
elasticsearch:
  client:
    heapSize: 512m
    memoryLimit: 1024Mi
    restPort: 9200
  data:
    heapSize: 1024m
    memoryLimit: 2048M
    replicas: "1"
    storage:
      persistent: true
      size: 20Gi
      storageClass: logging-kiban-storage-datanode
      useDynamicProvisioning: false
  image:
    repository: ibmcom/elasticsearch
  initImage:
    repository: ibmcom/icp-initcontainer
  internalPort: 9300
  master:
    heapSize: 512m
    memoryLimit: 1024Mi
    replicas: "1"
  name: elasticsearch
  pkiInitImage:
    repository: ibmcom/logging-pki-init
  pluginImage:
    repository: ibmcom/elasticsearch-plugin-searchguard
  pluginInitImage:
    repository: ibmcom/searchguard-init
filebeat:
  image:
    repository: ibmcom/filebeat
kibana:
  install: false
logstash:
  heapSize: 512m
  image:
    repository: ibmcom/logstash
  memoryLimit: 1024Mi
  port: 5044
  probe:
    image:
      repository: ibmcom/logstash-liveness-probe
mode: managed
nameOverride: elk
security:
  enabled: false
```

ファイルを指定してチャートをデプロイ。

```
helm install --name logging-kiban ibm-charts/ibm-icplogging \
  --version 1.0.0 \
  --namespace kiban \
  --tls \
  --values values.yaml
```

PVCがPendingとなっているのを確認。

```shell-session
# kubectl get pvc -n kiban
NAME                            STATUS    VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS                     AGE
data-logging-kiban-elk-data-0   Pending                                       kiban-logging-storage-datanode   1m
#
```

### PVの作成

デフォルトELKのPVを参考に、PVCに対応するPVを作成する。

```
mkdir -p /var/lib/icp/logging-kiban/elk-data
chmod 777 /var/lib/icp/logging-kiban/elk-data
```

```yaml:logging-kiban-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
  name: logging-kiban-datanode-172.30.1.224
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 20Gi
  local:
    path: /var/lib/icp/logging-kiban/elk-data
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - 172.30.1.224
  persistentVolumeReclaimPolicy: Retain
  storageClassName: logging-kiban-storage-datanode
```

```
kubectl apply -f logging-kiban-pv.yaml
```

PendingであったPVCがBoundになることを確認。

```shell-session
# kubectl get pvc -n kiban
NAME                            STATUS    VOLUME                                CAPACITY   ACCESS MODES   STORAGECLASS                     AGE
data-logging-kiban-elk-data-0   Bound     logging-kiban-datanode-172.30.1.224   20Gi       RWO            logging-kiban-storage-datanode   2m
#
```

### 稼働確認

しばらく待ってPodが作られたことを確認。

```shell-session
# kubectl get po -n kiban
NAME                                          READY     STATUS    RESTARTS   AGE
logging-kiban-elk-client-796fb8cb47-zrwh8     2/2       Running   0          3m
logging-kiban-elk-data-0                      1/1       Running   0          4m
logging-kiban-elk-filebeat-ds-wjn57           1/1       Running   0          12m
logging-kiban-elk-logstash-6dcc6f8cc9-rsd4f   1/1       Running   0          12m
logging-kiban-elk-master-7f79644f55-fvfhv     1/1       Running   0          3m
#
```

## 参考リンク

[IBM Cloud PrivateでPodをmanagementノードで動かす](https://qiita.com/teruq/items/ff0a8f76824fa4bb4ea5)