Prometheus + Grafana + 自作Exporter on Kubernetes で部屋の温湿度監視を行った

タイトルの通り、Prometheus, Grafana, 自作Exporter をKubernetes上に展開し、自宅(3部屋)の温湿度監視を行ったのでメモ。
Prometheus, Grafana, Kubernetes についての説明は省略させてもらいます。

## 前準備
- Raspberry Piと温湿度センサの接続  
温湿度取得に使用するセンサをラズパイに予め接続しておく必要があります。
使用したセンサーは[AM2320](http://akizukidenshi.com/catalog/g/gM-08663/)です。  
[にゃみかんてっくろぐ | Raspberry Piでダッシュボードを作る(5) -温度・湿度(センサ)](https://typewriter.hatenablog.jp/entry/2017/11/07/221836)
- Kubernetesのセットアップ  
kubeadmを使用してクラスタを作成しておきましょう。  
[Raspberry PiでおうちKubernetes構築【物理編】](https://qiita.com/go_vargo/items/d1271ab60f2bba375dcc)  
[Raspberry PiでおうちKubernetes構築【論理編】](https://qiita.com/go_vargo/items/29f6d832ea0a289b4778)

## 構成
- RX200S6 - ubuntu server 18.04(Master) on KVM
    - CPU: 4Core
    - RAM: 2GB
    - Disk: 50GB
- RX200S6 - ubuntu server 18.04(Worker) on KVM × 3
    - CPU: 4Core
    - RAM: 4GB
    - Disk: 50GB
- Raspberry Pi 3 Model B - raspbian 10.1(Worker) × 3
    - CPU: 4Core
    - RAM: 1GB
    - Disk: 30GB  

**前準備にも記載していますがKubeadmでKubernetesのセットアップを完了していることが前提です。**
CNIにはFlannelを使用しています。

```bash
~$ kubectl get nodes
NAME                  STATUS   ROLES    AGE   VERSION
kubernetes-master     Ready    master   20d   v1.15.3
kubernetes-worker-1   Ready    worker   20d   v1.15.3
kubernetes-worker-2   Ready    worker   20d   v1.15.3
kubernetes-worker-3   Ready    worker   20d   v1.15.3
kubernetes-worker-4   Ready    worker   20d   v1.15.3
kubernetes-worker-5   Ready    worker   20d   v1.15.3
kubernetes-worker-6   Ready    worker   20d   v1.15.3
```

### Master
```bash
~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:11:18Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
```
```bash
~$ kubelet --version
Kubernetes v1.15.3
```
```bash
~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.4", GitCommit:"67d2fcf276fcd9cf743ad4be9a9ef5828adc082f", GitTreeState:"clean", BuildDate:"2019-09-18T14:41:55Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
```
```bash
~$ docker version
Client:
 Version:           18.09.9
 API version:       1.39
 Go version:        go1.11.13
 Git commit:        039a7df9ba
 Built:             Wed Sep  4 16:57:28 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.9
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.11.13
  Git commit:       039a7df
  Built:            Wed Sep  4 16:19:38 2019
  OS/Arch:          linux/amd64
  Experimental:     false
```
### Worker
#### kubernetes-worker-[1~3]
```bash
~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:11:18Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
```
```bash
~$ kubelet --version
Kubernetes v1.15.3
```
```bash
~$ docker version
Client:
 Version:           18.09.9
 API version:       1.39
 Go version:        go1.11.13
 Git commit:        039a7df9ba
 Built:             Wed Sep  4 16:57:28 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.9
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.11.13
  Git commit:       039a7df
  Built:            Wed Sep  4 16:19:38 2019
  OS/Arch:          linux/amd64
  Experimental:     false
```

#### kubernetes-worker-[4~6]
```bash
~ $ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:11:18Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/arm"}
```
```bash
~ $ kubelet --version
Kubernetes v1.15.3
```
```bash
~ $ docker version
Client:
 Version:           18.09.9
 API version:       1.39
 Go version:        go1.11.13
 Git commit:        039a7df
 Built:             Wed Sep  4 17:02:31 2019
 OS/Arch:           linux/arm
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.9
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.11.13
  Git commit:       039a7df
  Built:            Wed Sep  4 16:21:03 2019
  OS/Arch:          linux/arm
  Experimental:     false
```

## Namespace作成
監視用のリソースを展開するNamespaceを作成しておきます。

```yml:namespace.yml
apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
  labels:
    name: monitoring
```
```bash
~ $ kubectl create -f namespace.yml
```

## ClusterRole, ClusterRoleBinding
ここではClusterRole, ClusterRoleBindingを作成し、
monitoringというNamespace作成時のdefaultというService Accountに対して
クラスタのリソースに対する操作権限を適用します。

```yml:cluster-role.yml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: default
  namespace: monitoring
```
```bash
~ $ kubectl create -f cluster-role.yml
```

## Prometheusのデプロイ
```yml:prometheus-deployment.yml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: prometheus-server
  namespace: monitoring
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: prometheus-server
    spec:
      volumes:
        - name: prom-config
          configMap:
            name: prometheus-config
      containers:
      - name: prometheus
        image: prom/prometheus:latest
        imagePullPolicy: IfNotPresent
        volumeMounts:
          - name: prom-config
            mountPath: /etc/prometheus
        ports:
        - containerPort: 9090
```
ConfigMapにて[Kubernetes用のService Discovery](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config)を設定する。

```yml:prometheus-configmap.yml
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  labels:
    name: prometheus-config
  namespace: monitoring
data:
  prometheus.yml: |-
    # my global config
    global:
      scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
      evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
      # scrape_timeout is set to the global default (10s).

    # Alertmanager configuration
    alerting:
      alertmanagers:
      - static_configs:
        - targets:
          # - alertmanager:9093

    # Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
    rule_files:
      # - "first_rules.yml"
      # - "second_rules.yml"

    # A scrape configuration containing exactly one endpoint to scrape:
    # Here it's Prometheus itself.
    scrape_configs:
      # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
      - job_name: 'prometheus'

        # metrics_path defaults to '/metrics'
        # scheme defaults to 'http'.

        static_configs:
        - targets: ['localhost:9090']

      - job_name: kubernetes-apiservers
        kubernetes_sd_configs:
        - role: endpoints
        relabel_configs:
        - action: keep
          regex: default;kubernetes;https
          source_labels:
          - __meta_kubernetes_namespace
          - __meta_kubernetes_service_name
          - __meta_kubernetes_endpoint_port_name
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      
      - job_name: kubernetes-service-endpoints
        kubernetes_sd_configs:
        - role: endpoints
        relabel_configs:
        - action: keep
          regex: true
          source_labels:
          - __meta_kubernetes_service_annotation_prometheus_io_scrape
        - action: replace
          regex: (https?)
          source_labels:
          - __meta_kubernetes_service_annotation_prometheus_io_scheme
          target_label: __scheme__
        - action: replace
          regex: (.+)
          source_labels:
          - __meta_kubernetes_service_annotation_prometheus_io_path
          target_label: __metrics_path__
        - action: replace
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
          source_labels:
          - __address__
          - __meta_kubernetes_service_annotation_prometheus_io_port
          target_label: __address__
        - action: labelmap
          regex: __meta_kubernetes_service_label_(.+)
        - action: replace
          source_labels:
          - __meta_kubernetes_namespace
          target_label: kubernetes_namespace
        - action: replace
          source_labels:
          - __meta_kubernetes_service_name
          target_label: kubernetes_name
      
      - job_name: kubernetes-services
        kubernetes_sd_configs:
        - role: service
        metrics_path: /probe
        params:
          module:
          - http_2xx
        relabel_configs:
        - action: keep
          regex: true
          source_labels:
          - __meta_kubernetes_service_annotation_prometheus_io_probe
        - source_labels:
          - __address__
          target_label: __param_target
        - replacement: blackbox
          target_label: __address__
        - source_labels:
          - __param_target
          target_label: instance
        - action: labelmap
          regex: __meta_kubernetes_service_label_(.+)
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: kubernetes_namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: kubernetes_name
      
      - job_name: kubernetes-pods
        kubernetes_sd_configs:
        - role: pod
        relabel_configs:
        - action: keep
          regex: true
          source_labels:
          - __meta_kubernetes_pod_annotation_prometheus_io_scrape
        - action: replace
          regex: (.+)
          source_labels:
          - __meta_kubernetes_pod_annotation_prometheus_io_path
          target_label: __metrics_path__
        - action: replace
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
          source_labels:
          - __address__
          - __meta_kubernetes_pod_annotation_prometheus_io_port
          target_label: __address__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - action: replace
          source_labels:
          - __meta_kubernetes_namespace
          target_label: kubernetes_namespace
        - action: replace
          source_labels:
          - __meta_kubernetes_pod_name
          target_label: kubernetes_pod_name
        - action: replace
          source_labels:
          - __meta_kubernetes_pod_node_name
          target_label: kubernetes_pod_node_name
```
```yml:prometheus-service.yml
apiVersion: v1
kind: Service
metadata:
  name: prometheus-service
  namespace: monitoring
spec:
  selector:
    app: prometheus-server
  type: NodePort
  ports:
  - port: 9090
    targetPort: 9090
    nodePort: 30090
```
```bash
~ $ kubectl create -f prometheus-configmap.yml
~ $ kubectl create -f prometheus-deployment.yml
~ $ kubectl create -f prometheus-service.yml
```

## Grafanaのデプロイ
```yml:grafana-deployment.yml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: grafana-deployment
  namespace: monitoring
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
        - name: grafana
          image: grafana/grafana:latest
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3000
```
```yml:grafana-service.yml
apiVersion: v1
kind: Service
metadata:
  name: grafana-service
  namespace: monitoring
spec:
  selector:
    app: grafana
  type: NodePort
  ports:
    - port: 3000
      targetPort: 3000
      nodePort: 30100
```
```bash
~ $ kubectl create -f grafana-deployment.yml
~ $ kubectl create -f grafana-service.yml
```
私はClusterの監視も行いたく、下記記事を参考にさせていただきました。
[Prometheus+GrafanaでKubernetesクラスターを監視する ~Binaryファイルから起動+yamlファイルから構築](https://qiita.com/FY0323/items/72616d6e280ec7f2fdaf)

## 自作Exporterのデプロイ
AM2320から温湿度を吐いてくれる自作Exporter([am2320_exporter](https://github.com/yudaishimanaka/am2320_exporter))をデプロイしていきます。
AM2320センサが接続されているノードに1つずつ配置し、なおかつkubernetes-worker-[1~3]はスケジューリングから除外したいので
`nodeAffinity`によりkubernetes-worker-[4~6]だけにPodが作成されるようにしています。

```yml:tmp-and-hum-daemonset.yml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: am2320-exporter
  namespace: monitoring
  labels:
    name: am2320-exporter
spec:
  template:
    metadata:
      labels:
        app: am2320-exporter
      annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/port: '9430'
        prometheus.io/path: /metrics
    spec:
      containers:
      - name: am2320-exporter
        image: yudaishimanaka/am2320-exporter-armv7l:latest
        imagePullPolicy: IfNotPresent
        securityContext:
          privileged: true
        ports:
        - containerPort: 9430
      hostNetwork: true
      hostPID: true
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/hostname
                operator: In
                values:
                  - kubernetes-worker-4
                  - kubernetes-worker-5
                  - kubernetes-worker-6
```
```bash
~ $ kubectl create -f tmp-and-hum-daemonset.yml
```

## Grafanaダッシュボードの設定
ダッシュボードの設定を行う前にPrometheusからサービスディスカバリが正しく行えているか確認します。
`http://<master-ip>:30090/service-discovery`にアクセスして以下のように表示されれば問題なくサービスディスカバリが行えています。なにも表示されない場合はConfigMapの記述を見直してください。
※表示される内容はConfigMapに記述したprometheus.ymlの設定、クラスタにデプロイされているリソースの数によって異なります。
![Selection_043.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/122473/b786e103-5618-ae41-5c02-fd53939470cf.png)
Targetsではam2320_exporterが確認できています。
![Selection_044.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/122473/65e4b197-5e00-5601-600c-01491c4a0209.png)

Grafanaダッシュボードにログインします。
`http://<master-ip>:30100/login`
デフォルトのユーザとパスワードは`admin`,`admin`です。

サイドバーの`Configuration`->`Data Sources`->`Add data source`->`Prometheus`を選択し、`HTTP URL`に`http://<master-ip>:30090`と入力し`Save & Test`で保存します。
![Selection_047.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/122473/3ed4e348-92a9-9292-9d01-a4c7e38a7243.png)

## グラフの作成
グラフの作成ですが、am2320_exporterは`am2320_temperature_gauge`, `am2320_humidity_gauge`の2つのメトリクスしかサポートしていないので、それをよしなにやってくれれば良いと思います。
私はGrafanaのVariablesを使用し以下のように書きました。
![Selection_049.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/122473/78f5f407-2362-e2f1-e346-082cb96bd92e.png)
![Selection_050.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/122473/a4bf7cc0-0bc3-eff1-a289-37aa2e5b7653.png)
![Selection_051.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/122473/951eb662-8d08-0f78-40ac-79f6e803d642.png)

最終的なグラフ
![Selection_048.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/122473/d4593f7f-1423-9e75-c37c-6e951a24073a.png)

## 最後に
きっかけは[kubeedge/examplesのLチカ](https://github.com/kubeedge/examples/tree/master/led-raspberrypi)です。
同じようなことをしてみたいと思っている人の役に立てれば幸いです。