IBM Cloud Monitoring with Sysdig と blackbox_exporter で死活監視する


# 概要
今回は [blackbox_exporter](https://github.com/prometheus/blackbox_exporter) を使用した死活監視の構成を試します。
Sysdig 自身には死活監視・外形監視がないので、 このような proxy server を設定し、インフラ監視を補完します。

また、Sysdig Agent `10.5.0` から Prometheus Native なスクレイピング設定が可能になっており、そちらを試します。

- [Sysdig Agent Release Notes - 10.5.0 September 23, 2020](https://docs.sysdig.com/en/sysdig-agent-release-notes.html)
- [Enable Prometheus Native Service Discovery](https://docs.sysdig.com/en/enable-prometheus-native-service-discovery.html)

>As of agent v10.5.0, Sysdig supports the native Prometheus service discovery and you can configure in prometheus.yaml in the same way you do for native Prometheus.
>The new version of promscrape is named promscrape.v2 .
>promscrape.v2 supports all types of scraping configuration, such as federation, blackbox-exporter, and so on.

![Kobito.TCK0H9.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/101361/2e5a2235-0b53-83b7-98bb-3a2307396b3c.png "Kobito.TCK0H9.png")




# `ibmcloud` CLI ログイン
CLI で設定を進めます。

```shell:

export REGION="jp-tok"
export RESOURCE_GROUP="khayama-rg"
ibmcloud login -a cloud.ibm.com -r $REGION -g $RESOURCE_GROUP
```

# Sysdig サービスインスタンス作成
監視結果の集約先として IBM Cloud Monitoring with Sysdig をプロビジョニングします。

```shell:

# ibmcloud catalog service sysdig-monitor 

export SYSDIG_NAME=khayama-sysdig
export SYSDIG_PLAN=graduated-tier # or lite

ibmcloud resource service-instance-create $SYSDIG_NAME sysdig-monitor $SYSDIG_PLAN $REGION
```

# Sysdig サービス認証情報の作成
プロビジョニングした Sysdig のインスタンス ID を取得します。

```shell:

export SYSDIG_ID=$(ibmcloud resource service-instance --output JSON $SYSDIG_NAME | jq -r '.[].id')
echo $SYSDIG_ID
```

Agent インストールに必要な認証情報を作成します。

```shell:

ibmcloud resource service-key-create "$SYSDIG_NAME"-service-key  --instance-id $SYSDIG_ID
ibmcloud resource service-keys 
```

Sysdig アクセスキーを確認します。

```shell:

export SYSDIG_ACCESS_KEY=$(ibmcloud resource service-key --output JSON "$SYSDIG_NAME"-service-key | jq -r '.[].credentials."Sysdig Access Key"')
echo $SYSDIG_ACCESS_KEY
```

# proxy server 環境準備
proxy server には Red Hat Enterprise Linux を使います。

```shell:

[root@khayama-proxy ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.2 (Ootpa)
```

# Sysdig Agent インストール
[Regions and endpoints](https://cloud.ibm.com/docs/Monitoring-with-Sysdig?topic=Monitoring-with-Sysdig-endpoints#endpoints_ingestion) からプライベートエンドポイントをセットし、
[Configuring a Sysdig agent](https://cloud.ibm.com/docs/Monitoring-with-Sysdig?topic=Monitoring-with-Sysdig-config_agent#config_agent_linux) に従って Proxy node に導入します。

```shell:

[root@khayama-proxy ~]# SYSDIG_ACCESS_KEY=xxxxxxxx-yyyy-zzzz-aaaa-bbbbbbcccccc
[root@khayama-proxy ~]# COLLECTOR_ENDPOINT=ingest.private.jp-tok.monitoring.cloud.ibm.com

[root@khayama-proxy ~]# curl -sL https://ibm.biz/install-sysdig-agent | sudo bash -s -- --access_key $SYSDIG_ACCESS_KEY --collector $COLLECTOR_ENDPOINT --collector_port 6443 --secure true --tags role:proxy,location:tok04 --additional_conf 'sysdig_capture_enabled: false'
```

```shell:result
* Detecting operating system
* Installing Sysdig public key
* Installing Sysdig repository
* Installing kernel headers
* Installing Sysdig Agent
* Setting access key
* Setting tags
* Setting collector endpoint
* Setting collector port
* Setting connection security
* Adding additional configuration to dragent.yaml
Restarting dragent (via systemctl):  [  OK  ]
```

Sysdig Agent のバージョンが `10.5.0` 以上であることを確認します。


```shell:

[root@khayama-proxy ~]# /opt/draios/bin/dragent --version
10.5.0
```

Agent の設定は以下のコマンドで確認できます。

```shell:

[root@khayama-proxy ~]# cat /opt/draios/etc/dragent.yaml
customerid: xxxxxxxx-yyyy-zzzz-aaaa-bbbbbbcccccc
tags: role:proxy,location:tok04
collector: ingest.private.jp-tok.monitoring.cloud.ibm.com
collector_port: 6443
ssl: true
sysdig_capture_enabled: false
```

# `blackbox_exporter` インストール

[prometheus/blackbox_exporter: Blackbox prober exporter](https://github.com/prometheus/blackbox_exporter) に従って proxy server に導入します。

GitHub からバイナリをダウンロードして展開します。

```shell:

wget -c https://github.com/prometheus/blackbox_exporter/releases/download/v0.17.0/blackbox_exporter-0.17.0.linux-amd64.tar.gz -O - | tar -xz
cd blackbox_exporter-0.17.0.linux-amd64
./blackbox_exporter -h
```

以下のようにバイナリと設定ファイルを配置します。

```shell:

mv blackbox_exporter /usr/local/bin
mkdir -p /etc/blackbox
mv blackbox.yml /etc/blackbox
```

`icmp` probe に追加の設定を入れます。

- [blackbox_exporter/CONFIGURATION.md at master · prometheus/blackbox_exporter](https://github.com/prometheus/blackbox_exporter/blob/master/CONFIGURATION.md)
- [blackbox_exporter/example.yml at master · prometheus/blackbox_exporter](https://github.com/prometheus/blackbox_exporter/blob/master/example.yml)

```shell:

cat <<EOF >> /etc/blackbox/blackbox.yml 
    timeout: 5s
    icmp:
      preferred_ip_protocol: "ip4"
EOF

blackbox_exporter --config.check --config.file="/etc/blackbox/blackbox.yml"
```

定常的に起動するようにサービスを構成します。

```shell:

cat <<EOF > /lib/systemd/system/blackbox.service
[Unit]
Description=Blackbox Exporter Service
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
ExecStart=/usr/local/bin/blackbox_exporter \
  --config.file=/etc/blackbox/blackbox.yml \
  --web.listen-address=":9115"

Restart=always

[Install]
WantedBy=multi-user.target
EOF
```

最後に権限を追加して `blackbox.service` を起動します。

```shell:
# to allow any user the ability to use ping
sysctl -w net.ipv4.ping_group_range='0 2147483647'

systemctl enable blackbox.service
systemctl start blackbox.service
systemctl status blackbox.service
```

# `blackbox_exporter` 稼働確認
`probe_success 1` なら　`Up` の意味になります。
詳細な情報を表示したい場合は、以下のように `debug=true` のパラメータを追加してください。

```
[root@khayama-proxy ~]# curl "http://localhost:9115/probe?module=icmp&target=10.193.37.176&debug=true"
Logs for the probe:
ts=2020-09-23T07:22:53.810006347Z caller=main.go:304 module=icmp target=10.193.37.176 level=info msg="Beginning probe" probe=icmp timeout_seconds=5
ts=2020-09-23T07:22:53.810227284Z caller=icmp.go:84 module=icmp target=10.193.37.176 level=info msg="Resolving target address" ip_protocol=ip4
ts=2020-09-23T07:22:53.810278023Z caller=icmp.go:84 module=icmp target=10.193.37.176 level=info msg="Resolved target address" ip=10.193.37.176
ts=2020-09-23T07:22:53.810324592Z caller=main.go:119 module=icmp target=10.193.37.176 level=info msg="Creating socket"
ts=2020-09-23T07:22:53.810535428Z caller=main.go:119 module=icmp target=10.193.37.176 level=info msg="Creating ICMP packet" seq=34885 id=25639
ts=2020-09-23T07:22:53.810577682Z caller=main.go:119 module=icmp target=10.193.37.176 level=info msg="Writing out packet"
ts=2020-09-23T07:22:53.810673503Z caller=main.go:119 module=icmp target=10.193.37.176 level=info msg="Waiting for reply packets"
ts=2020-09-23T07:22:53.813713568Z caller=main.go:119 module=icmp target=10.193.37.176 level=info msg="Found matching reply packet"
ts=2020-09-23T07:22:53.813791262Z caller=main.go:304 module=icmp target=10.193.37.176 level=info msg="Probe succeeded" duration_seconds=0.003687872



Metrics that would have been returned:
# HELP probe_dns_lookup_time_seconds Returns the time taken for probe dns lookup in seconds
# TYPE probe_dns_lookup_time_seconds gauge
probe_dns_lookup_time_seconds 4.7291e-05
# HELP probe_duration_seconds Returns how long the probe took to complete in seconds
# TYPE probe_duration_seconds gauge
probe_duration_seconds 0.003687872
# HELP probe_icmp_duration_seconds Duration of icmp request by phase
# TYPE probe_icmp_duration_seconds gauge
probe_icmp_duration_seconds{phase="resolve"} 4.7291e-05
probe_icmp_duration_seconds{phase="rtt"} 0.003104699
probe_icmp_duration_seconds{phase="setup"} 0.00025286
# HELP probe_ip_addr_hash Specifies the hash of IP address. It's useful to detect if the IP address changes.
# TYPE probe_ip_addr_hash gauge
probe_ip_addr_hash 3.984007035e+09
# HELP probe_ip_protocol Specifies whether probe ip protocol is IP4 or IP6
# TYPE probe_ip_protocol gauge
probe_ip_protocol 4
# HELP probe_success Displays whether or not the probe was a success
# TYPE probe_success gauge
probe_success 1



Module configuration:
prober: icmp
timeout: 5s
http:
    ip_protocol_fallback: true
tcp:
    ip_protocol_fallback: true
icmp:
    preferred_ip_protocol: ip4
    ip_protocol_fallback: true
dns:
    ip_protocol_fallback: true
```

# `blackbox_exporter` probe ターゲット設定

[Prometheus Configuration](https://github.com/prometheus/blackbox_exporter#prometheus-configuration) をベースに ICMP の死活監視ターゲットを設定します。
今回は `localhost` と `10.193.37.176` の複数を登録し、Sysdig でセグメント別にデータを見られるかを確認します。

```
cat <<EOF > /opt/draios/etc/prometheus.yaml
scrape_configs:
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [icmp]  # Look for a icmp response.
    static_configs:
      - targets:
        - localhost  # Target to probe with icmp
        - 10.193.37.176  # Target to probe with icmp
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115  # The blackbox exporter's real hostname:port.
EOF
```

# `blackbox_exporter` エンドポイントに対するスクレイピング設定
Sysdig Agent `10.5.0` から Prometheus Native なスクレイピング設定が可能になったので、その設定を追加して Agent を再起動します。


```
cat <<EOF >> /opt/draios/etc/dragent.yaml  
prometheus:
  enabled: true
  prom_service_discovery: true
EOF

service dragent restart
```

# データ確認
IBM Cloud Monitoring with Sysdig の「Dashboard > Add dashboard」画面から以下の設定でパネル追加できることを確認します。
同じ Metrics でも instance 別にデータを確認できたことがわかります。

- Metrics: `probe_success`
- Segmentation: `instance`
 
![Kobito.JgqHOV.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/101361/c29a6102-58bf-6b89-e9ae-a513201b255d.png "Kobito.JgqHOV.png")

# 通知設定

パネルが作成できたら、「Create Alert」からメール通知を設定します。
（事前にメール通知チャネルは追加しておきます。）
このように「Multiple Alerts」を作成すると、セグメント「instance」「job」ごとに一度にアラートが設定できます。
通知メール本文では「{{instance}}」「{{job}}」を変数として使用できます。

![Kobito.tuO6Or.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/101361/4b17ceb6-7884-f3c1-9b76-8b81ba8243b6.png "Kobito.tuO6Or.png")
![Kobito.jFHogt.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/101361/23cd5488-c3bd-5b79-bb6a-d68b06337619.png "Kobito.jFHogt.png")

# 通知確認
「10.193.37.176」をパワーオフして通知を確認します。
データは以下のように確認できました。

![Kobito.CCYrkK.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/101361/d7d7433c-7512-dd53-2c12-df097d46cf8f.png "Kobito.CCYrkK.png")

通知メールの受信も確認できました。

![貼り付けた画像_2020_09_28_14_10.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/101361/46af75ca-df6e-8e8c-782b-3204086c2945.png "貼り付けた画像_2020_09_28_14_10.png")

# さいごに
これで `blackbox_exporter` と Sysdig を使った死活監視を実装できました。
proxy server の冗長性を確保するには、複数立てるとよいでしょう。

# 参考リンク
- [Configuring Sysdig Agent - Sysdig Documentation](https://docs.sysdig.com/en/configuring-sysdig-agent.html#UUID-f9cb9589-c159-0f25-27e1-3953022e4ec2_UUID-a2bc7bd4-36b1-c4da-ac27-8f264d0bc744)
- [Working with Prometheus Metrics](https://docs.sysdig.com/en/working-with-prometheus-metrics.html)
- [How To Install and Configure Blackbox Exporter for Prometheus](https://devconnected.com/how-to-install-and-configure-blackbox-exporter-for-prometheus/)

# 参考：トラブルシューティング
`dragent.yaml` に設定を追加し、Agent を再起動することで詳細なログを確認できます。

```shell:

cat <<EOF >> /opt/draios/etc/dragent.yaml  
log:
  file_priority: debug
EOF

service dragent restart && tail -f /opt/draios/logs/draios.log
```

例えば、以下のようなログを確認できれば、今回の設定が正しく動いていることが確認できます。

```
2020-09-28 04:32:47.025, 21727.21762, Debug, promscrape:1010: have metrics for job 1
2020-09-28 04:32:47.025, 21727.21762, Debug, promscrape:1010: have metrics for job 2
```

# 参考：Python 3 で実行するには
Python3 をインストールせずに Sysdig Agent を実行すると、以下のエラーが確認できます。

```shell:

Error, sdchecks[0] /opt/draios/lib/python-deps2.7/OpenSSL/crypto.py:12: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in a future release.
```

Python3 をインストールします。

```shell:

dnf install python38 -y
python3 -V
alternatives --set python /usr/bin/python3
```

これだけでは以下のエラーが確認できます。

```
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] Traceback (most recent call last):
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] File "/opt/draios/bin/sdchecks", line 33, in <module>
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] from sdchecks import Application
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] File "/opt/draios/lib/python/sdchecks.py", line 28, in <module>
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] import config
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] File "/opt/draios/lib/python/config.py", line 22, in <module>
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] from util import get_os, yLoader
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] File "/opt/draios/lib/python/util.py", line 44, in <module>
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] from utils.platform import Platform
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] File "/opt/draios/lib/python/utils/platform.py", line 6, in <module>
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] from utils.dockerutil import get_client
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] File "/opt/draios/lib/python/utils/dockerutil.py", line 15, in <module>
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] from docker import Client
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] File "/opt/draios/lib/python-deps/docker/__init__.py", line 6, in <module>
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] from .client import Client, AutoVersionClient, from_env # flake8: noqa
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] File "/opt/draios/lib/python-deps/docker/client.py", line 5, in <module>
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] import requests
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] File "/opt/draios/lib/python-deps/requests/__init__.py", line 112, in <module>
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] from . import utils
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] File "/opt/draios/lib/python-deps/requests/utils.py", line 39, in <module>
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] DEFAULT_CA_BUNDLE_PATH = certs.where()
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] File "/opt/draios/lib/python-deps/certifi/core.py", line 37, in where
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] _CACERT_PATH = str(_CACERT_CTX.__enter__())
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] File "/usr/lib64/python3.8/contextlib.py", line 113, in __enter__
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] return next(self.gen)
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] File "/usr/lib64/python3.8/importlib/resources.py", line 201, in path
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] with open_binary(package, resource) as fp:
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] File "/usr/lib64/python3.8/importlib/resources.py", line 91, in open_binary
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] return reader.open_resource(resource)
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] File "<frozen importlib._bootstrap_external>", line 988, in open_resource
2020-09-28 03:36:57.979, 18226.18241, Error, sdchecks[0] FileNotFoundError: [Errno 2] No such file or directory: '/opt/draios/lib/python-deps/certifi/cacert.pem'
```

その場合は、続いて以下のコマンドを実施します。

```shell:

python3 -m pip install --upgrade pip
python3 -m pip --version
python3 -m pip install certifi
cp $(python -m certifi) /opt/draios/lib/python-deps/certifi/cacert.pem
```

最後にこのコマンドでエラーが出力されないことを確認します。

```shell:

service dragent restart && tail -f /opt/draios/logs/draios.log | grep Error
```

# 参考：メトリクス一覧

```
[root@khayama-proxy ~]# curl "http://localhost:9115/metrics"
# HELP blackbox_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which blackbox_exporter was built.
# TYPE blackbox_exporter_build_info gauge
blackbox_exporter_build_info{branch="HEAD",goversion="go1.14.4",revision="1bc768014cf6815f7e9d694e0292e77dd10f3235",version="0.17.0"} 1
# HELP blackbox_exporter_config_last_reload_success_timestamp_seconds Timestamp of the last successful configuration reload.
# TYPE blackbox_exporter_config_last_reload_success_timestamp_seconds gauge
blackbox_exporter_config_last_reload_success_timestamp_seconds 1.6008451361406033e+09
# HELP blackbox_exporter_config_last_reload_successful Blackbox exporter config loaded successfully.
# TYPE blackbox_exporter_config_last_reload_successful gauge
blackbox_exporter_config_last_reload_successful 1
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 10
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.14.4"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 834256
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 834256
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.444856e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 274
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 0
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 3.436808e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 834256
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 6.49216e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 1.794048e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 4258
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 6.49216e+07
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.6715648e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 0
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 4532
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 3472
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 28152
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 32768
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 4.473924e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 787456
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 393216
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 393216
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.2827136e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 7
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1024
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 9
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.8173952e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.60084513566e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 7.33978624e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes -1
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 0
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0
```