Prometheus+blackboxをgrafanaで可視化

## 目的
Prometheusで監視対象のホストのメトリクスを収集したり、死活監視をする場合に、grafanaで可視化したいと思う。

## ディレクトリ構成
- docker-compose.yml
- blackbox_exporter
  - config.yml
- grafana
  - Dockerfile 
- prometheus
  -  prometheus.yml

## Docker系config
docker-compose.yml
```yml
version: '3'
services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - '9090:9090'


  blackbox_exporter:
    image: prom/blackbox-exporter:latest
    container_name: blackbox_exporter
    volumes:
      - ./blackbox_exporter/config.yml:/etc/blackbox_exporter/config.yml

  grafana:
#    image: grafana/grafana:latest
    build: ./grafana
    container_name: grafana
    hostname: grafana
    ports:
      - 3000:3000
    user: "$PID:$GID"
    volumes:
      - ./grafana:/var/lib/grafana
```

grafana/dockerfile
データソースにzabbixを使用する場合はこちらを導入する
```dockerfile
FROM grafana/grafana
RUN grafana-cli plugins install magnesium-wordcloud-panel
RUN grafana-cli plugins install alexanderzobnin-zabbix-app
```


## Prometheus系config
prometheus/prometheus.yml
```yml
global:
  scrape_interval: 15s # デフォルトの監視対象にpullする間隔
  external_labels: # 外部と通信するときのラベル
    monitor: 'codelab-monitor'

scrape_configs: # 監視対象ごとの設定
  - job_name: 'prometheus' # Prometheus自身の監視
    scrape_interval: 5s # デフォルトの間隔を上書き
    static_configs:
      - targets: ['prometheus:9090']


###### blackbox HTTP
  - job_name: 'blackbox_http'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - https://google.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox_exporter:9115


##### blackbox ICMP
  - job_name: 'blackbox_icmp'
    metrics_path: /probe
    params:
      module: [icmp]
    static_configs:
      - targets:
        - 8.8.8.8
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox_exporter:9115


##### blackbox TCP
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [tcp_connect]
    static_configs:
      - targets:
        - google.com:443
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox_exporter:9115
```

blackbox_exporter/config.yml
```yml
modules:
  http_2xx:
    prober: http
    http:

  http_post_2xx:
    prober: http
    http:
      method: POST

  tcp_connect:
    prober: tcp

  pop3s_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^+OK"
      tls: true
      tls_config:
        insecure_skip_verify: false

  ssh_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^SSH-2.0-"

  irc_banner:
    prober: tcp
    tcp:
      query_response:
      - send: "NICK prober"
      - send: "USER prober prober prober :prober"
      - expect: "PING :([^ ]+)"
        send: "PONG ${1}"
      - expect: "^:[^ ]+ 001"

  icmp:
    prober: icmp
```

## pluginの有効化
datasourceにzabbixを用いる場合は、grafanaのdashboardから有効化する必要がある
左側のパネルから「plugins」
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2738475/d065cc6b-87d7-c813-a23e-8fa4920e19a9.png)

下にスクロールするとzabbixがあるので、クリック
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2738475/0d5288d7-e7ae-2a6c-7de4-5f35b469f754.png)

Enableをクリック
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2738475/612582ba-8504-8935-0f6e-9ed97f4e7019.png)


## grafanaにpromqlを入力
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2738475/97b2992b-c9d5-c00b-de0f-fd9d934e40e3.png)
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2738475/0a215abf-e930-980f-333d-0a9e5f07efcf.png)

graphはgauge等にすると分かりやすい
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2738475/658e882b-7887-4fbc-8948-0c907743bf7a.png)
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2738475/16079bea-1df3-17a7-7c9e-5787d7d83c98.png)





## 動作確認
動いてる
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2738475/c2bf019d-039f-00a4-a2fc-ca233c6ca304.png)