Prometheus OperatorでKubernetes Cluster外部をモニタリングする

# Prometheus OperatorでKubernetes Cluster外部をモニタリングする

## はじめに
API等をKubernetesで構築していて、データベースは外部のVMを使用しているユースケースがあると思う。
そのVMのモニタリングをPrometheusのExporterを使用して実現する。

## 前提条件
* KubernetesにPrometheus Operatorを導入済みである
    * 今回はGKEにHelmでPrometheus Operatorをインストールしている
* 監視対象のVMにデータベースを構築済みである
    * 今回はVM(GCE)にCassandra Clusterをインストールしている


## Cassandra Exporterを導入する

Cassandraの監視は[cassandra_exporter](https://github.com/criteo/cassandra_exporter)を使用する。
これは、[JMX exporter](https://github.com/prometheus/jmx_exporter)をforkしたプロジェクト。

### cassandra_exporterのユーザーを作成する

先ずはグループの作成

```shell
$ sudo groupadd -r cassandra_exporter
```

次にユーザーを作成
システムユーザーでログインできないようにしておく

```shell
$ sudo useradd -r -s /bin/false -g cassandra_exporter cassandra_exporter
```

ユーザー作成後確認

```shell
$ id cassandra_exporter
uid=997(cassandra_exporter) gid=994(cassandra_exporter) groups=994(cassandra_exporter)
```

### cassandra_exporter(本体)のダウンロード

[ここ](https://github.com/criteo/cassandra_exporter/releases)からダウンロードする


```shell
$ curl -L -O https://github.com/criteo/cassandra_exporter/releases/download/2.3.5/cassandra_exporter-2.3.5.jar

$ mv ./cassandra_exporter-2.3.5.jar /opt/cassandra_exporter.jar

$ chown cassandra_exporter:cassandra_exporter /opt/cassandra_exporter.jar

$ chmod 0755 /opt/cassandra_exporter.jar
```

### cassandra_exporter(設定ファイル)の作成

[ここ](https://github.com/criteo/cassandra_exporter/blob/master/config.yml)を参考にする

```shell
$ vi /etc/cassandra_exporter_config.yml

$ chown cassandra_exporter:cassandra_exporter /etc/cassandra_exporter_config.yml
```

```yaml:cassandra_exporter_config.yml
host: localhost:7199
ssl: False
user:
password:
listenAddress: 0.0.0.0
listenPort: 8080
# Regular expression to match environment variable names that will be added
# as labels to all data points. The name of the label will be either
# $1 from the regex below, or the entire environment variable name if no match groups are defined
#
# Example:
# additionalLabelsFromEnvvars: "^ADDL\_(.*)$"
additionalLabelsFromEnvvars:
blacklist:
   # To profile the duration of jmx call you can start the program with the following options
   # > java -Dorg.slf4j.simpleLogger.defaultLogLevel=trace -jar cassandra_exporter.jar config.yml --oneshot
   #
   # To get intuition of what is done by cassandra when something is called you can look in cassandra
   # https://github.com/apache/cassandra/tree/trunk/src/java/org/apache/cassandra/metrics
   # Please avoid to scrape frequently those calls that are iterating over all sstables

   # Unaccessible metrics (not enough privilege)
   - java:lang:memorypool:.*usagethreshold.*

   # Leaf attributes not interesting for us but that are presents in many path
   - .*:999thpercentile
   - .*:95thpercentile
   - .*:fifteenminuterate
   - .*:fiveminuterate
   - .*:durationunit
   - .*:rateunit
   - .*:stddev
   - .*:meanrate
   - .*:mean
   - .*:min

   # Path present in many metrics but uninterresting
   - .*:viewlockacquiretime:.*
   - .*:viewreadtime:.*
   - .*:cas[a-z]+latency:.*
   - .*:colupdatetimedeltahistogram:.*

   # Mostly for RPC, do not scrap them
   - org:apache:cassandra:db:.*

   # columnfamily is an alias for Table metrics
   # https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/src/java/org/apache/cassandra/metrics/TableMetrics.java#L162
   - org:apache:cassandra:metrics:columnfamily:.*

   # Should we export metrics for system keyspaces/tables ?
   - org:apache:cassandra:metrics:[^:]+:system[^:]*:.*

   # Logback doesn't have any useful metrics
   - ch:qos:logback:.*

   # Don't scrap us
   - com:criteo:nosql:cassandra:exporter:.*

maxScrapFrequencyInSec:
  50:
    - .*

  # Refresh those metrics only every hour as it is costly for cassandra to retrieve them
  3600:
    - .*:snapshotssize:.*
    - .*:estimated.*
    - .*:totaldiskspaceused:.*
```

### systemd serviceとして登録する

```shell:cassandra_exporter.sh
vi /opt/cassandra_exporter.sh

#!/bin/bash
java -jar /opt/cassandra_exporter.jar /etc/cassandra_exporter_config.yml


$ chown cassandra_exporter:cassandra_exporter /opt/cassandra_exporter.sh

$ chmod 0755 /opt/cassandra_exporter.sh
```


```shell
$ sudo  vi /etc/systemd/system/cassandra_exporter.service

[Unit]
Description=Prometheus Cassandra Exporter
After=network.target

[Service]
Type=simple
User=cassandra_exporter
Group=cassandra_exporter
ExecStart=/opt/cassandra_exporter.sh

SyslogIdentifier=prometheus_cassandra_exporter
Restart=always

[Install]
WantedBy=multi-user.target
```

daemon-reloadして認識させる

```shell
sudo systemctl daemon-reload
```

### cassandra_exporterを起動する

```shell
sudo systemctl start cassandra_exporter.service
```

### cassandra_exporterの確認

```shell
systemctl status cassandra_exporter
● cassandra_exporter.service - Prometheus Cassandra Exporter
   Loaded: loaded (/etc/systemd/system/cassandra_exporter.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-04-26 16:41:46 JST; 10s ago
 Main PID: 16849 (cassandra_expor)
   CGroup: /system.slice/cassandra_exporter.service
           ├─16849 /bin/bash /opt/cassandra_exporter.sh
           └─16850 java -jar /opt/cassandra_exporter.jar /etc/cassandra_exporter_config.yml

Apr 26 16:41:55 blcloud-cassandra-01 prometheus_cassandra_exporter[16849]: at java.security.AccessController.doPrivileged(Native Method)
Apr 26 16:41:55 blcloud-cassandra-01 prometheus_cassandra_exporter[16849]: at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
Apr 26 16:41:55 blcloud-cassandra-01 prometheus_cassandra_exporter[16849]: at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
Apr 26 16:41:55 blcloud-cassandra-01 prometheus_cassandra_exporter[16849]: at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
Apr 26 16:41:55 blcloud-cassandra-01 prometheus_cassandra_exporter[16849]: at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
Apr 26 16:41:55 blcloud-cassandra-01 prometheus_cassandra_exporter[16849]: at java.security.AccessController.doPrivileged(Native Method)
Apr 26 16:41:55 blcloud-cassandra-01 prometheus_cassandra_exporter[16849]: at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
Apr 26 16:41:55 blcloud-cassandra-01 prometheus_cassandra_exporter[16849]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Apr 26 16:41:55 blcloud-cassandra-01 prometheus_cassandra_exporter[16849]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Apr 26 16:41:55 blcloud-cassandra-01 prometheus_cassandra_exporter[16849]: at java.lang.Thread.run(Thread.java:748)
```

## prometheus OperatorにService Monitorを追加する

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/33101/4b470f9f-6b1b-01c7-40c3-ad7776edf56a.png)

上記のようにServiceMonitorは各Serviceを通してScrapeする。
そのため、今回作成したCassandraのExporterに対してKubernetesの `Endpoint` と `Service`を作成してKubernetes Clusterからアクセスする。

```yaml:cassandra-exporter-01.yaml
---
apiVersion: v1
kind: Endpoints
metadata:
    name: cassandra-metrics-01
    labels:
        release: prometheus-operator
    namespace: default
subsets:
    - addresses:
      - ip: Cassandra NodeのIPアドレス
      ports:
      - name: metrics
        port: 8080
        protocol: TCP

---
apiVersion: v1
kind: Service
metadata:
    name: cassandra-metrics-01
    namespace: default
    labels:
        release: prometheus-operator
        k8s-app: cassandra-metrics
spec:
    type: ExternalName
    externalName: Cassandra NodeのIPアドレス
    ports:
    - name: metrics
      port: 8080
      protocol: TCP
      targetPort: 8080
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
    name: cassandra-metrics-sm-01
    labels:
        release: prometheus-operator
        prometheus: kube-prometheus
    namespace: default
spec:
    selector:
        matchLabels:
            release: prometheus-operator
            k8s-app: cassandra-metrics-01
    endpoints:
    - port: metrics
      interval: 10s
      honorLabels: true
      path: /metrics
```

```shell
$ kubectl apply -f cassandra-exporter-01.yaml
endpoints/cassandra-metrics-01 created
service/cassandra-metrics-01 created
servicemonitor.monitoring.coreos.com/cassandra-metrics-sm-01 created
```

## 確認

今回、GrafanaのIngressは作ってなかったので、port-forwardで確認する

```shell
kubectl port-forward svc/prometheus-operator-grafana 8080:80
```

6400番のこの[dashboard](https://grafana.com/grafana/dashboards/6400)をインポートする

<img width="1396" alt="スクリーンショット 2020-04-26 17.27.10.png" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/33101/da9cf684-0ada-9e20-c0b2-face3685667c.png">

そして、Data Sourceを `--Grafana--` -> `Prometheus`に変更する
<img width="1151" alt="スクリーンショット 2020-04-26 18.02.44.png" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/33101/ac7fd332-c6c0-14c8-36a5-c593690dcef1.png">

すると、以下のようにメトリクスが表示されるようになる

![スクリーンショット 2020-04-26 18.05.57.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/33101/153b6d32-9f41-4e94-9901-e0bb0fc43ca5.png)