Check! Node-RED で Prometheus のメトリクスを可視化する

こんにちは、 @dz_ こと大平かづみです。

[Node-RED Advent Calendar 2017](https://qiita.com/advent-calendar/2017/nodered) 11日目の投稿です。

Prologue - はじめに
====

Node-RED のダッシュボードがお手軽で便利なので、これはインフラのモニタリングにも使えるのではないかと試してみました！

題材は [Prometheus](https://prometheus.io/) のメトリクスを Node-RED で受け取り、可視化してみます。Prometheus は、[Grafana](https://grafana.com/ "Grafana - The open platform for analytics and monitoring") と組み合わせるのがスタンダードですが、物は試しです 	(ﾉ*’ω’)ﾉ

コードサンプル
====

今回書いたコードはこちらに公開しました。このコードをベースに話を進めます。

- [dzeyelid/nodered-prometheus-sample](https://github.com/dzeyelid/nodered-prometheus-sample)

準備
====

Docker で環境を用意する
----
Node-RED も Prometheus も  Docker イメージが提供されているので、それを利用します。

- [Node-RED : Running under Docker](https://nodered.org/docs/platforms/docker "Node-RED : Running under Docker")
- [Using Docker - Installation | Prometheus](https://prometheus.io/docs/prometheus/latest/installation/#using-docker)

これらを `docker-compose` で構成しました。

```yaml:docker-compose.yml
version: "3"
services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    volumes:
      - ./docker/prometheus/config/prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
  nodered:
    image: nodered/node-red-docker
    container_name: nodered
    volumes:
      - ./data:/data
      - ./nodes:/nodes
    ports:
      - "1880:1880"
    links:
      - prometheus
```

`prometheus.yml` は、Prometheus の設定ファイルで、この Getthing Started の内容を参考にしています。

- [Configuring Prometheus to monitor itsel - Getting started | Prometheus](https://prometheus.io/docs/prometheus/latest/getting_started/#configuring-prometheus-to-monitor-itself "Configuring Prometheus to monitor itsel - Getting started | Prometheus")

上記の準備ができたら、コンテナインスタンスを立ち上げます。

```bash
docker-compose up -d
```

これで、Node-RED エディタと、Prometheus の簡易GUIにアクセスできるようになりました。

| 項目 | URL |
| --- | --- |
| Node-RED エディタ | http://localhost:1880 |
| Prometheus の簡易GUI | http://localhost:9090 |

Node-RED ダッシュボードの準備
----

それから、Node-RED のダッシュボードが使えるように、「パレットの管理」> 「ノードを追加」から、 `node-red-dashuboard` を追加しておきます。

- [node-red-dashboard - Node-RED](https://flows.nodered.org/node/node-red-dashboard "node-red-dashboard - Node-RED")

Prometheus のメトリクスについて理解する
====

Promeheus は、Exporter と呼ばれる別のコンポーネントがメトリクスを収集してきています。そのフォーマットについてはこちらに記載があります。

- [Text format details - Exposition formats | Prometheus](https://prometheus.io/docs/instrumenting/exposition_formats/#text-format-details)

ざっくり理解するとこんな感じです。

```
metric_name{label_name1="label_value1", label_name2="label_value2", ... } value
```

ラベルが入れ子になったり、タイムスタンプがつくパターンもあるようですが、割愛します。

フローを作成する（簡易版）
====

まずは簡単なフローを組んでみました。任意のメトリクスを選択して可視化します。

1. `inject` ノードで定期的に実行する
2. `http request` ノードで Prometheus のメトリクスを取得する
3. `split` ノードで行ごとに分割、 `switch` ノードで取得したいパラメータでメトリクスを選択する
5. `split`、`switch` ノードを組み合わせて、メトリクスデータを取得する
6. ダッシュボードの `ui_chart` ノードに入力する

サンプルフロー
----
上記構成のサンプルコードです。Prometheus 自体が発行する `go_goroutines` というメトリクスをダッシュボードに表示しています。

```json
[{"id":"e8b15b7b.923128","type":"inject","z":"8f3a7677.4b1d08","name":"","topic":"","payload":"","payloadType":"date","repeat":"10","crontab":"","once":false,"x":132.00000762939453,"y":239.00003242492676,"wires":[["7b270a3d.128be4"]]},{"id":"7b270a3d.128be4","type":"http request","z":"8f3a7677.4b1d08","name":"","method":"GET","ret":"txt","url":"http://prometheus:9090/metrics","tls":"","x":318.00001525878906,"y":238.76392364501953,"wires":[["af09abb4.c607f8"]]},{"id":"af09abb4.c607f8","type":"split","z":"8f3a7677.4b1d08","name":"split by \\n","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":147.0000114440918,"y":313.33336544036865,"wires":[["579d6691.276068"]]},{"id":"40e71a0.454c7e8","type":"debug","z":"8f3a7677.4b1d08","name":"","active":true,"console":"false","complete":"true","x":333.00000381469727,"y":513.444465637207,"wires":[]},{"id":"579d6691.276068","type":"switch","z":"8f3a7677.4b1d08","name":"get go_goroutiens","property":"payload","propertyType":"msg","rules":[{"t":"regex","v":"^go_goroutines .*$","vt":"str","case":false}],"checkall":"true","outputs":1,"x":336.38890838623047,"y":313.2222099304199,"wires":[["b058c01d.d3354"]]},{"id":"b058c01d.d3354","type":"split","z":"8f3a7677.4b1d08","name":"split by space","splt":" ","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":166.3888931274414,"y":388.2222385406494,"wires":[["cc0b7c3f.10cc6"]]},{"id":"cc0b7c3f.10cc6","type":"switch","z":"8f3a7677.4b1d08","name":"get index[1]","property":"parts.index","propertyType":"msg","rules":[{"t":"eq","v":"1","vt":"num"}],"checkall":"true","outputs":1,"x":156.38890838623047,"y":457.2222385406494,"wires":[["ddbb5c4f.b44eb","40e71a0.454c7e8"]]},{"id":"ddbb5c4f.b44eb","type":"ui_chart","z":"8f3a7677.4b1d08","name":"","group":"215d19a8.9edbb6","order":0,"width":0,"height":0,"label":"chart","chartType":"line","legend":"false","xformat":"HH:mm:ss","interpolate":"linear","nodata":"","dot":false,"ymin":"","ymax":"","removeOlder":"1","removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"useOneColor":false,"colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"useOldStyle":false,"x":332.38891983032227,"y":456.7778215408325,"wires":[[],[]]},{"id":"215d19a8.9edbb6","type":"ui_group","z":"","name":"go_goroutines","tab":"356225f4.ea84ea","disp":true,"width":"10"},{"id":"356225f4.ea84ea","type":"ui_tab","z":"","name":"Prometheus","icon":"dashboard"}]
```

![image.png](https://qiita-image-store.s3.amazonaws.com/0/3812/50ab10bc-5f29-c6ab-9062-65af56f7c5a9.png)
![image.png](https://qiita-image-store.s3.amazonaws.com/0/3812/c9370676-2d6e-224e-5667-540ac95b648e.png)


ポイント
----
### `http request` ノードには Node-RED サーバーからアクセスできるURLを指定する

今回、`http request` ノードに指定するURLは、 `http://localhost:9090/metrics` ではなく、 `http://prometheus:9090/metrics` を指定します。

どういうことかというと、 `localhost:9090` は、実行マシンから みた
 Prometheus の URL なので、Node-RED からは見えません。そこで、前述の `docker-compose.yml` で、`nodered` コンテナから `prometheus` コンテナに対して `link` を貼って、 `prometheus:9090` でアクセスできるようにしました。

今回は Docker の場合ですが、例えばクラウドであれば、仮想ネットワークやセキュリティグループなどの設定で、ルートを確保する必要がありそうです。

### とはいえ、 `switch` ノードによるメトリクスの抽出がいまいち…

作ってみてなんですが、このフロー構成だと、複雑なメトリクスを扱うのが面倒ですね…。

よし！ノードを自作してみましょう！🙌

Prometheus のメトリクス解析ノードを作ってみる（改良版）
====

実はノードを自作したことなかったので、これを機にやってみます。ノードの自作については、こちらをご参照ください。

- [Node-RED日本ユーザ会  : Nodeの作成](https://nodered.jp/docs/creating-nodes/)

自作ノード `node-red-contrib-parser-prom-metrics` の仕様
----

自作ノードは、`http_request` ノードで取得した `/metrics` の結果を入力すると、下記のようなJSON構造に変換する仕様にしました。

```json
{
    "name": "<metric_name>",
    "value": <value>,
    "labels": {
        "label_name1": "label_value1",
        "label_name2": "label_value2",
        ...
    }
}

```

コードは、前述のコードサンプルの `nodes/parser-prom-metrics` をご参照ください。

サンプルフロー
----

さて、サンプルとして、`http_requests_total` の複数データを１つのグラフに表示してみました。ノードの構成もシンプルになりました！

```json
[{"id":"df1a33e9.993f5","type":"inject","z":"78d9cc24.3c55a4","name":"","topic":"","payload":"","payloadType":"date","repeat":"10","crontab":"","once":false,"x":154,"y":77.23610877990723,"wires":[["951f8fda.3b733"]]},{"id":"951f8fda.3b733","type":"http request","z":"78d9cc24.3c55a4","name":"","method":"GET","ret":"txt","url":"http://prometheus:9090/metrics","tls":"","x":340.00000762939453,"y":77,"wires":[["22dfb0e0.c8f01"]]},{"id":"22dfb0e0.c8f01","type":"parser-prom-metrics","z":"78d9cc24.3c55a4","name":"","x":151.388916015625,"y":153.23609733581543,"wires":[["209317b4.e62488"]]},{"id":"63747b53.8c22c4","type":"switch","z":"78d9cc24.3c55a4","name":"filter by http_requests_total","property":"payload.name","propertyType":"msg","rules":[{"t":"eq","v":"http_requests_total","vt":"str"}],"checkall":"true","outputs":1,"x":206.38889694213867,"y":235.01391696929932,"wires":[["fb01d069.8a2c3","9ffbd8dc.2f92d8"]]},{"id":"209317b4.e62488","type":"split","z":"78d9cc24.3c55a4","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":322.3888931274414,"y":152.01390266418457,"wires":[["63747b53.8c22c4"]]},{"id":"9f300d6.7394af","type":"ui_chart","z":"78d9cc24.3c55a4","name":"prometheus","group":"a815aaaf.67adb8","order":0,"width":0,"height":0,"label":"handler=pometheus only","chartType":"line","legend":"false","xformat":"HH:mm:ss","interpolate":"linear","nodata":"","dot":false,"ymin":"","ymax":"","removeOlder":1,"removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"useOneColor":false,"colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"useOldStyle":false,"x":548.3888893127441,"y":326.7917261123657,"wires":[[],[]]},{"id":"4702d48b.49a73c","type":"change","z":"78d9cc24.3c55a4","name":"","rules":[{"t":"move","p":"payload.labels.handler","pt":"msg","to":"topic","tot":"msg"},{"t":"move","p":"payload.value","pt":"msg","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":335.388916015625,"y":326.1250047683716,"wires":[["9f300d6.7394af","6cd20ad5.0a2794"]]},{"id":"fb01d069.8a2c3","type":"switch","z":"78d9cc24.3c55a4","name":"prometheus","property":"payload.labels.handler","propertyType":"msg","rules":[{"t":"eq","v":"prometheus","vt":"str"}],"checkall":"true","outputs":1,"x":151.3888931274414,"y":326.66667079925537,"wires":[["4702d48b.49a73c"]]},{"id":"61338a91.013f44","type":"ui_chart","z":"78d9cc24.3c55a4","name":"http_requests_total","group":"9e8c057f.89f1f8","order":0,"width":0,"height":0,"label":"other handler","chartType":"line","legend":"false","xformat":"HH:mm:ss","interpolate":"linear","nodata":"","dot":false,"ymin":"","ymax":"","removeOlder":1,"removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"useOneColor":false,"colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"useOldStyle":false,"x":570.9999465942383,"y":461.0000066757202,"wires":[[],[]]},{"id":"a70e5931.6cd578","type":"change","z":"78d9cc24.3c55a4","name":"","rules":[{"t":"move","p":"payload.labels.handler","pt":"msg","to":"topic","tot":"msg"},{"t":"move","p":"payload.value","pt":"msg","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":335.99999618530273,"y":461.3332862854004,"wires":[["61338a91.013f44","6cd20ad5.0a2794"]]},{"id":"6cd20ad5.0a2794","type":"debug","z":"78d9cc24.3c55a4","name":"","active":true,"console":"false","complete":"payload","x":549,"y":370.99997901916504,"wires":[]},{"id":"9ffbd8dc.2f92d8","type":"switch","z":"78d9cc24.3c55a4","name":"exclude prometheus","property":"payload.labels.handler","propertyType":"msg","rules":[{"t":"neq","v":"prometheus","vt":"str"}],"checkall":"true","outputs":1,"x":185.16666412353516,"y":396,"wires":[["a70e5931.6cd578"]]},{"id":"a815aaaf.67adb8","type":"ui_group","z":"","name":"http_requests_total (prometheus)","tab":"6832eea4.26d3","order":2,"disp":true,"width":"12"},{"id":"9e8c057f.89f1f8","type":"ui_group","z":"","name":"http_requests_total","tab":"6832eea4.26d3","order":1,"disp":true,"width":"12"},{"id":"6832eea4.26d3","type":"ui_tab","z":"","name":"Prometheus","icon":"dashboard"}]
```

![image.png](https://qiita-image-store.s3.amazonaws.com/0/3812/445f7cf1-0793-146f-5cd7-36d67964171d.png)
![image.png](https://qiita-image-store.s3.amazonaws.com/0/3812/b9a82ab5-6d78-2102-e359-aa4299bff9a4.png)


ポイント
----
### 自作ノードを有効にする

今回、既成のコンテナイメージを使っているので、自作ノードを有効にするには少しコツがいりました。それは、root 権限で `npm link` を貼らないとならなかったので、下記のように対応しました。

```bash
docker exec -it --user=root nodered /bin/bash

# nodered コンテナの中で npm link で相互にリンクさせます
cd /nodes/parser-prom-metrics && npm link
cd /usr/src/node-red && npm link node-red-contrib-parser-prom-metrics
exit

docker restart nodered
```

nodered コンテナが再起動し終わったら、自作ノードが使えるようになってるはずです。

### ダッシュボードの `ui-chart` で複数のグラフを表示させる

このような形式でデータを入力すると、複数のグラフを表示させることができます。詳細はドキュメントをご参照ください。

- [node-red-dashboard/Charts.md at master · node-red/node-red-dashboard](https://github.com/node-red/node-red-dashboard/blob/master/Charts.md)

```json
{
    "msg": {
        "topic": "<データのラベル>",
        "payload": <値>
    }
}
```


Epilogue - おわりに
====
既存のノードだけで作ってみたらあんまり格好良くなかったので、自作ノードに手を出してみました。簡易版よりスマートになって満足です！

ともあれ、こうして遊ぶのも勉強になっていいですね！🙌