Prometheus サンプル設定

# 概要
 - k8sテスト環境構築<br>
<b>Prometheus サンプル設定</b>

# 構築目次
 - [全体目次] (https://qiita.com/prodigy413/items/89fcb8cfb71e80f541d8)

# 環境
 - Rancher: v2.4.8
 - kubernetes(Client): v1.19.1
 - kubernetes(Server): v1.18.8
 - kube-prometheus-stack Chart: v9.4.3
 - kube-prometheus-stack App: v0.38.1

# サンプル設定サマリ
1. Nginx metricsのexpose設定
2. NginxをPrometheusのtargetに追加
3. Nginxの`http request count`を表示するGrafana dashboard 作成
4. `http request count`を使ってアラート作成 → アラートを外部転送


# Nginx metrics 設定
 - 作業場所: ClientPC
 - ngx_http_stub_status_module Configuration Page<br>
https://nginx.org/en/docs/http/ngx_http_stub_status_module.html#stub_status
- Nginxの`nginx.conf`に以下を追加し、`ngx_http_stub_status_module`を有効にする

~~~
server {
    listen   8080;
    location /metrics {
        stub_status;
    }
}
~~~

 - nginx-prometheus-exporter Page<br>
https://github.com/nginxinc/nginx-prometheus-exporter
- PrometheusにNginx metricsを追加するため、`nginx-prometheus-exporter`をサイドカーとして追加
- マニフェスト作成

~~~yaml:test-nginx.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-configmap
  labels:
    app: nginx
data:
  nginx.conf: |2

    user  nginx;
    worker_processes  1;
    error_log  /var/log/nginx/error.log warn;
    pid        /var/run/nginx.pid;
    events {
        worker_connections  1024;
    }
    http {
        include       /etc/nginx/mime.types;
        default_type  application/octet-stream;
        log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for"';
        access_log  /var/log/nginx/access.log  main;

        server {
            listen   8080;
            location /metrics {
                stub_status;
            }
        }

        sendfile        on;
        keepalive_timeout  65;
        include /etc/nginx/conf.d/*.conf;
    }
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
  labels:
    app: nginx
spec:
  selector:
    app: nginx
  ports:
  - name: nginx-http
    protocol: TCP
    port: 80
    targetPort: 80
  - name: nginx-exporter
    protocol: TCP
    port: 9113
    targetPort: 9113
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deploy
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.19.2
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
        volumeMounts:
        - name: nginx-conf
          mountPath: /etc/nginx/nginx.conf
          subPath: nginx.conf
      - name: nginx-exporter
        image: nginx/nginx-prometheus-exporter:0.8.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9113
        args:
          - -nginx.scrape-uri=http://localhost:8080/metrics
      volumes:
      - name: nginx-conf
        configMap:
          name: nginx-configmap
          items:
          - key: nginx.conf
            path: nginx.conf
~~~

 - Nginx 起動

~~~
$ kubectl apply -f test-nginx.yaml
~~~

# Prometheus Target 追加
Prometheus Operatorのインストール時、追加されるCRD`ServiceMonitor`を使ってターゲット追加

 - serviceMonitorSelector 確認<br>
`serviceMonitorSelector.matchLabels`の値を`ServiceMonitor`に追加する<br>
この環境では`release: prometheus`を追加<br>
→ 設定しないとPrometheusはServiceMonitorを追加しない

~~~
$ kubectl get prometheus -n monitoring prometheus-kube-prometheus-prometheus -o yaml
..........
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector:
    matchLabels:
      release: prometheus
..........
~~~

 - マニフェスト作成

~~~yaml:nginx-servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: nginx-servicemonitor
  namespace: monitoring
  labels:
    app: nginx
    release: prometheus # serviceMonitorSelectorで確認したラベル
spec:
  endpoints:
  - port: nginx-exporter
  namespaceSelector:
    matchNames:
    - default
  selector:
    matchLabels:
      app: nginx # Serviceのラベル
~~~

 - ServiceMonitor 作成

~~~
$ kubectl apply -f nginx-servicemonitor.yaml

## 確認 ##
$ kubectl get servicemonitor -n monitoring
NAME                             AGE
..........
nginx-servicemonitor             64s
..........
~~~

 - Target 確認<br>
http://prometheus.test.local/targets<br>
![4-1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/561025/988e7118-f83c-9745-a15a-0e9a071d1a27.png)

Nginxのmetricsも使用可能
![5-1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/561025/8af859e8-ae0b-412a-0efa-5645d2f247f7.png)

# Grafana Dashboard 作成
 - Grafanaにログイン → [<b>+</b>] > [<b>Dashboard</b>]選択
![6-1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/561025/b431558b-1f95-a249-6d27-d1cc9d3acb8e.png)

 - [<b>Dashboard settings</b>]クリック
![13-1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/561025/020774e9-c7ac-7207-94bb-821a28596baa.png)

 - Dashboard名設定
[<b>General</b>] > [<b>Name</b>]設定
・Name: `Nginx HTTP Request Count`
![14-1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/561025/c246baac-b002-ae8c-1eca-003bb08768ca.png)

 - 変数設定
[<b>Variables</b>] > [<b>Add variable</b>]クリック
![15-1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/561025/d506ab56-484e-db17-cd26-68cad61da0f7.png)
以下変数を作成
※各変数設定後、[**Add**]クリックで保存 > [**New**]クリックで新規変数作成

| Name         | Data source  | Refresh            |  Query                                                                          |
| :----------- | :----------- | :----------------- | :------------------------------------------------------------------------------ |
| cluster      | Prometheus   | On Dashboard Load  | label_values(kube_pod_info, cluster)                                            |
| namespace    | Prometheus   | On Dashboard Load  | label_values(kube_pod_info{cluster="\$cluster"}, namespace)                     |

 - 確認
[<b>←</b>]クリック
![7-1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/561025/ebc68251-6c99-b1f5-6c6e-24ad86e44465.png)
変数を確認し、Nginx Podを作成したnamespaceに設定しておく
・cluster: <b>None</b>
・namespace: <b>default</b>
![1-1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/561025/63c52bcd-d02f-40bf-49e0-e47232b13156.png)


 - Panel 作成
[<b>Ad new panel</b>]クリック
![8-2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/561025/86bde2fd-6eab-9766-bde3-37e9705418af.png)
以下のように設定
・Panel title: <b>HTTP Request Count</b>
![9-3.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/561025/6b79efe7-9585-7418-7624-ca1bde14de4a.png)
・Metrics: <b>sum(increase(nginx_http_requests_total{namespace="\$namespace", cluster="\$cluster"}[1m])) by (pod)</b>
　→ 各PodのHTTP Request Countの合計/1分
・Legend: <b>{{pod}}</b>
![2-1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/561025/29ee1b5f-b4b8-f86c-d66d-f777bc93943e.png)
・Decimals: <b>0</b>
![9-2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/561025/7fddd493-b8c2-c91e-adb1-b3b47a41e7f1.png)
[<b>Save</b>]クリックして保存
・Dashboard name: <b>Nginx HTTP Request Count</b>
![9-4.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/561025/19df73c6-ce68-9ce8-6663-0e9434460807.png)
![18.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/561025/e9dd9c99-d9de-16db-803e-6c2c22ce423e.png)

 - 動作確認
- Nginx port forward 設定<br>
`$ kubectl port-forward svc/nginx-svc 8080:80`
- 他のTerminalでアクセス<br>
`$ while true ; do curl http://localhost:8080 ; sleep 5 ; done`<br><br>
- Dashboard 確認<br>
![3-1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/561025/9d015bf1-d9a0-5025-6ad8-6fa95e1163ca.png)

# Alert 設定

Prometheus Operatorのインストール時、追加されるCRD`PrometheusRule`を使ってターゲット追加

 - ruleSelector 確認<br>
`ruleSelector.matchLabels`の値を`PrometheusRule`に追加する<br>
この環境では`app: kube-prometheus-stack`、`release: prometheus`を追加<br>
→ 設定しないとPrometheusはPrometheusRuleを追加しない

~~~
$ kubectl get prometheus -n monitoring prometheus-kube-prometheus-prometheus -o yaml
..........
  ruleNamespaceSelector: {}
  ruleSelector:
    matchLabels:
      app: kube-prometheus-stack
      release: prometheus
..........
~~~

 - マニフェスト作成<br>
- 作成Rule: `HTTP Request Count`値が1分間`20`を超えたらアラート発生

~~~yaml:nginx-http-request-rule.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    app: kube-prometheus-stack # ruleSelectorで確認したラベル
    release: prometheus # ruleSelectorで確認したラベル
  name: http-request-rule.rules
  namespace: monitoring
spec:
  groups:
  - name: nginx-http-request.rules
    rules:
    - alert: NginxTooManyRequest
      annotations:
        message: HTTP Request to {{ $labels.namespace }}/{{ $labels.pod }} is over 20 / 1 minute.
      expr: sum(increase(nginx_http_requests_total{job="nginx-svc", namespace=~".*"}[1m])) by (namespace, pod) > 20
      for: 1m
      labels:
        severity: critical
~~~

 - PrometheusRule 作成

~~~
$ kubectl apply -f nginx-http-request-rule.yaml

## 確認 ##
$ kubectl get prometheusrule -n monitoring
NAME                                AGE
..........
http-request-rule.rules             27m
..........
~~~

- 確認
http://prometheus.test.local/alerts
![4-1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/561025/96c1ca3c-d3fb-86fb-c741-93b37236e583.png)

- 動作確認
 - Nginx port forward 設定<br>
`$ kubectl port-forward svc/nginx-svc 8080:80`
 - 他のTerminalでアクセス<br>
`$ while true ; do curl http://localhost:8080 ; sleep 1 ; done`<br><br>
 - Alert 確認<br>
http://prometheus.test.local/alerts<br>
![5-1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/561025/193dd652-916a-1d38-798e-9401a7ecaa66.png)

# Alert 外部転送
AlertをSlackに転送

 - Slack 設定 / Webhook URL 確認
[<b>Slackをブラウズする</b>] > [<b>App</b>]クリック
![2-1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/561025/e84aa9c4-06ba-affb-b2fd-2f189c363b92.png)
`incoming webhooks`検索 > [<b>追加</b>]クリック
![3-1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/561025/3f7bf942-9b7c-e5d5-2823-eba9f066308d.png)
[<b>Slackに追加</b>]クリック
![4-1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/561025/4e4e7bff-3002-a8e8-e4df-c2777d02b5a1.png)
チャンネル選択 > [<b>Incoming Webhook インテルレーションの追加</b>]クリック
![6-1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/561025/f6ca0df1-b999-19f4-de81-9a2f33019a35.png)
`Webhook URL`確認
![7-1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/561025/ca8932b8-3a7a-bf56-c606-42ffa041f75b.png)

 - 転送設定マニフェスト作成
- `slack_api_url`は上記で確認した`Webhook URL`
- `channel`はWebhook設定時、選択したチャンネル

~~~yaml:alertmanager.yaml
global:
  resolve_timeout: 1m
  slack_api_url: 'https://hooks.slack.com/services/xxxxx..............................'
route:
  receiver: 'slack_notifications'
  group_interval: 5m
  group_wait: 30s
  repeat_interval: 12h
  routes:
  - match:
      alertname: 'NginxTooManyRequest'
    receiver: 'nginx_request_count_error'
receivers:
- name: 'slack_notifications'
- name: 'nginx_request_count_error'
  slack_configs:
  - channel: '#it-test'
    send_resolved: true
    icon_url: https://avatars3.githubusercontent.com/u/3380462
~~~

 - 設定ファイルのsecret化<br>
上記マニフェストのbase64値を確認

~~~
$ kubectl create secret generic test --from-file=./alertmanager.yaml --dry-run=client -o yaml
apiVersion: v1
data:
  alertmanager.yaml: {base64値}
kind: Secret
metadata:
  creationTimestamp: null
  name: test
~~~

 - secret 更新用マニフェスト作成<br>
上記の`base64値`を使って更新用マニフェスト作成<br>

~~~yaml:alertmanager-slack.yaml
apiVersion: v1
kind: Secret
metadata:
  labels:
    release: prometheus
  name: alertmanager-prometheus-prometheus-oper-alertmanager
  namespace: monitoring
type: Opaque
data:
  alertmanager.yaml: {base64値}
~~~

 - secret 更新

~~~
## secret 確認 ##
$ kubectl get secret -n monitoring
NAME                                                          TYPE              DATA   AGE
..........
alertmanager-prometheus-kube-prometheus-alertmanager          Opaque            1      3d10h
..........

## 更新 ##
$ kubectl apply -f alertmanager-slack.yaml
~~~

 - 確認<br>
http://alertmanager.test.local/#/status
![6.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/561025/fbf868b2-e19a-dc49-5077-97ec244e2637.png)


 - 動作確認<br>
- Nginx port forward 設定<br>
`$ kubectl port-forward svc/nginx-svc 8080:80`<br>
- 他のTerminalでアクセス<br>
`$ while true ; do curl http://localhost:8080 ; sleep 1 ; done`<br><br>
- Slack 確認<br>
![7.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/561025/bd42ec35-3008-b487-5f93-710c916d697b.png)