blackbox_exporter でicmp,http監視、grafanaでslackにアラート通知

## icmp監視

```yml:docker-compose.yml
version: '3'
services:

  prometheus:
    image: prom/prometheus
    container_name: prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      #- ./prometheus-data:/prometheus
    ports:
      - 9090:9090

  blackbox_exporter:
    image: prom/blackbox-exporter
    volumes:
      - ./blackbox.yml:/etc/blackbox_exporter/config.yml:ro
    ports:
      - 9115:9115
```

* blackbox.yml に記述

```yml:blackbox.yml
modules:
  icmp:
    prober: icmp
```


* prometheus.yml

```yaml:prometheus.yml
global:
  scrape_interval:     15s
  evaluation_interval: 15s

scrape_configs:

  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [icmp]
    static_configs:
      - targets:
        - 192.168.100.55
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox_exporter:9115
```

## prometheusでグラフ表示

* `probe_success`で検索。
* 1がUPで、0がdown

![キャプチャ.PNG](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/25728/755b2052-97ab-32ee-085a-f0b2f5e69b57.png)

----

## http監視

* [prometheus - Probes returning zero - Stack Overflow](https://stackoverflow.com/questions/54773928/probes-returning-zero)

```yaml:blackbox.yml
modules:

  icmp:
    prober: icmp

  http_2xx:
    prober: http
    timeout: 15s
    http:
      valid_status_codes: []
      method: GET
      preferred_ip_protocol: "ip4"
```

```yaml:prometheus.yml
global:
  scrape_interval:     15s
  evaluation_interval: 15s

scrape_configs:

  - job_name: 'blackbox-icmp'
    metrics_path: /probe
    params:
      module: [icmp]
    static_configs:
      - targets:
        - 192.168.100.55
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox_exporter:9115

  - job_name: 'blackbox-http'
    scrape_timeout: 15s
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - https://qiita.com/
        labels:
          group: 'response'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox_exporter:9115
```

* `probe_success`

![無題.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/25728/69210fd0-677f-d43d-d795-3f38c45ac7fc.png)


----


## Grafana

### docker-compose

```yml:docker-compose.yml
version: '3'
services:

  prometheus:
    image: prom/prometheus
    container_name: prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      #- ./prometheus-data:/prometheus
    ports:
      - 9090:9090

  blackbox_exporter:
    image: prom/blackbox-exporter
    volumes:
      - ./blackbox.yml:/etc/blackbox_exporter/config.yml:ro
    ports:
      - 9115:9115

  grafana:
    image: grafana/grafana
    container_name: grafana
    volumes:
      - ./grafana-data:/var/lib/grafana
    ports:
      - 3000:3000
```


```sh
mkdir grafana-data
chmod 777 grafana-data
```

* http://localhost:3000
    * admin / admin

* `Data Sources`
* `Add data source`

![無題.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/25728/774f215d-45ac-d1b1-b9ea-67bbc911b74c.png)

* Prometheus - Select

![無題.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/25728/f7d8e89d-0a12-2062-1fe4-2d87b49e44f1.png)

* URL に `http://prometheus:9090/`

![無題.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/25728/5f3e9351-e1df-fe78-60da-7e38986a428f.png)
![無題.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/25728/b044c1d0-8965-2442-63ea-5256a080234b.png)

* `probe_success` を 指定するとグラフが描画される。

![無題.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/25728/03bbc800-f23b-0277-1dac-d97ae109a221.png)

* save

![無題.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/25728/e40956b0-c1d2-58f1-a8be-7d71d0592c85.png)


### アラート設定

![無題.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/25728/87f03e92-b5d2-9386-3e26-4ef68f0820d1.png)

![無題.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/25728/b9a741e9-5f40-9d10-e681-389c51fd4063.png)


* 1回でも疎通に失敗した場合はアラート通知の例

![無題.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/25728/096a2df8-95df-4ce8-8b89-bf0c7f0d3a9d.png)


### docker stats

* そんなに負荷は無いので使いながら様子を見てみる。

![無題.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/25728/16753ce4-5381-a6a7-19af-d773a9d44144.png)

### plugins

* https://grafana.com/grafana/plugins に多く用意されている。