Nginx Vhost Traffic Status のデータを ELK に取り込む

# Nginx Vhost Traffic Status のデータを ELK に取り込む

## Nginx Vhost Traffic Status(以降 Nginx VTS) のデータを取り込む準備
### nginx-vts-exporter のインストール
Nginx VTS の JSONデータを Prometeus用に出力してくれる便利なやつ

```shell:git-clone
$ git clone https://github.com/hnlq715/nginx-vts-exporter.git
```

```shell:golangのinstall
$ wget https://dl.google.com/go/go1.13.7.linux-amd64.tar.gz
$ sudo tar go1.13.7.linux-amd64.tar.gz -C /usr/local/go
$ sudo cat <<EOF > /etc/profile.d/goenv.sh 
export GOROOT=/usr/local/go
export GOPATH=$HOME/go
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin
EOF
$ source /etc/profile.d/goenv.sh
```

```shell:nginx-vts-exporterのCompile&Install
$ cd nginx-vts-exporter
$ make
$ sudo cp nginx-vts-exporter /usr/bin
$ sudo cp nginx_vts_exporter.default /etc/sysconfig/nginx_vts_exporter
$ sudo cat <<EOF > /etc/sysconfig/nginx_vts_exporter
NGINX_VTS_EXPORTER_OPTS='-telemetry.address="localhost:9913" -nginx.scrape_uri=http://localhost/status/json'
EOF
$ sudo cat <<EOF > /usr/lib/systemd/system/nginx_vts_exporter.service
[Unit]
Description=Prometheus exporter thar exports Nginx VTS stats.
Documentation=https://github.com/hnlq715/nginx-vts-exporter
After=network.target

[Service]
EnvironmentFile=-/etc/sysconfig/nginx_vts_exporter
User=root
ExecStart=/usr/bin/nginx-vts-exporter $NGINX_VTS_EXPORTER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF
$ sudo systemctl enable nginx_vts_exporter
$ sudo systemctl start nginx_vts_exporter
```
### データを取得してみる
expoter の Lisner は localhost の port 9913 なので接続してみる。

```shell:port9913接続
$ curl http://localhost:9913
<html>
			<head><title>Nginx Exporter</title></head>
			<body>
			<h1>Nginx Exporter</h1>
			<p><a href="/metrics">Metrics</a></p>
			</body>
			</html>
```
おや？なんか変・・・・・

```shell:nginx-vts-exporterのoption
$ nginx-vts-exporter -h
Usage of nginx-vts-exporter:
  -insecure
    	Ignore server certificate if using https (default true)
  -metrics.namespace string
    	Prometheus metrics namespace. (default "nginx")
  -nginx.scrape_timeout int
    	The number of seconds to wait for an HTTP response from the nginx.scrape_uri (default 2)
  -nginx.scrape_uri string
    	URI to nginx stub status page (default "http://localhost/status")
  -telemetry.address string
    	Address on which to expose metrics. (default ":9913")
  -telemetry.endpoint string
    	Path under which to expose metrics. (default "/metrics")
  -version
    	Print version information.
```

ほう、endpoint が /metrics の模様なので再度試験

```shell:endpointに接続
$ curl http://localhost:9913/metrics
# HELP nginx_cache_bytes cache request/response bytes
# TYPE nginx_cache_bytes counter
nginx_cache_bytes{direction="in",zone="cache-zone"} 33366
nginx_cache_bytes{direction="out",zone="cache-zone"} 551016
# HELP nginx_cache_requests cache requests counter
# TYPE nginx_cache_requests counter
nginx_cache_requests{status="bypass",zone="cache-zone"} 65
nginx_cache_requests{status="expired",zone="cache-zone"} 0
nginx_cache_requests{status="hit",zone="cache-zone"} 4
nginx_cache_requests{status="miss",zone="cache-zone"} 7
nginx_cache_requests{status="revalidated",zone="cache-zone"} 0
nginx_cache_requests{status="scarce",zone="cache-zone"} 0
nginx_cache_requests{status="stale",zone="cache-zone"} 0
nginx_cache_requests{status="updating",zone="cache-zone"} 0
<snip>
```

cache hit などは Virtual Server毎に観たいので以下のように取得してみる。

```shell:取得する内容をgrepで絞る
$ /usr/bin/curl -s http://localhost:9913/metrics | grep -v ^# | grep -e "_server_requests" -e "_server_bytes"
nginx_server_bytes{direction="in",host="*"} 1.1809658e+07
nginx_server_bytes{direction="in",host="elk.nakacya.jp"} 146149
nginx_server_bytes{direction="in",host="localhost"} 1.1582905e+07
nginx_server_bytes{direction="in",host="test1.nakacya.jp"} 46807
nginx_server_bytes{direction="in",host="www.nakacya.jp"} 33797
nginx_server_bytes{direction="out",host="*"} 2.554692267e+09
nginx_server_bytes{direction="out",host="elk.nakacya.jp"} 332947
nginx_server_bytes{direction="out",host="localhost"} 2.553677271e+09
nginx_server_bytes{direction="out",host="test1.nakacya.jp"} 597605
nginx_server_bytes{direction="out",host="www.nakacya.jp"} 84444
nginx_server_requests{code="1xx",host="*"} 0
nginx_server_requests{code="1xx",host="elk.nakacya.jp"} 0
nginx_server_requests{code="1xx",host="localhost"} 0
nginx_server_requests{code="1xx",host="test1.nakacya.jp"} 0
nginx_server_requests{code="1xx",host="www.nakacya.jp"} 0
nginx_server_requests{code="2xx",host="*"} 33502
nginx_server_requests{code="2xx",host="elk.nakacya.jp"} 73
nginx_server_requests{code="2xx",host="localhost"} 33309
nginx_server_requests{code="2xx",host="test1.nakacya.jp"} 83
nginx_server_requests{code="2xx",host="www.nakacya.jp"} 37
nginx_server_requests{code="3xx",host="*"} 14
nginx_server_requests{code="3xx",host="elk.nakacya.jp"} 14
nginx_server_requests{code="3xx",host="localhost"} 0
nginx_server_requests{code="3xx",host="test1.nakacya.jp"} 0
nginx_server_requests{code="3xx",host="www.nakacya.jp"} 0
nginx_server_requests{code="4xx",host="*"} 219
nginx_server_requests{code="4xx",host="elk.nakacya.jp"} 19
nginx_server_requests{code="4xx",host="localhost"} 133
nginx_server_requests{code="4xx",host="test1.nakacya.jp"} 33
nginx_server_requests{code="4xx",host="www.nakacya.jp"} 34
nginx_server_requests{code="5xx",host="*"} 2
nginx_server_requests{code="5xx",host="elk.nakacya.jp"} 0
nginx_server_requests{code="5xx",host="localhost"} 0
nginx_server_requests{code="5xx",host="test1.nakacya.jp"} 0
nginx_server_requests{code="5xx",host="www.nakacya.jp"} 2
nginx_server_requests{code="total",host="*"} 33737
nginx_server_requests{code="total",host="elk.nakacya.jp"} 106
nginx_server_requests{code="total",host="localhost"} 33442
nginx_server_requests{code="total",host="test1.nakacya.jp"} 116
nginx_server_requests{code="total",host="www.nakacya.jp"} 73
```

### データ加工
direction のin/out が「指数表記」になっているのは困るので加工します。
#### ここの 1行perl が優秀
```shell:データ加工
#Server Metric GET
/usr/bin/curl -s http://localhost:9913/metrics | grep -v ^# | grep -e "_server_requests" -e "_server_bytes" > /var/log/nginx/vts_status.log

# an exponential -> real number
cat /var/log/nginx/vts_status.log | perl -anle 'printf "%s %d\n",$F[0],$F[1]' > /var/log/nginx/vts_status_number.log
mv -f /var/log/nginx/vts_status_number.log /var/log/nginx/vts_status.log
```
加工後

```shell:/var/log/nginx/vts_status.log
nginx_server_bytes{direction="in",host="*"} 11809961
nginx_server_bytes{direction="in",host="elk.nakacya.jp"} 146149
nginx_server_bytes{direction="in",host="localhost"} 11583208
nginx_server_bytes{direction="in",host="test1.nakacya.jp"} 46807
nginx_server_bytes{direction="in",host="www.nakacya.jp"} 33797
nginx_server_bytes{direction="out",host="*"} 2555083406
nginx_server_bytes{direction="out",host="elk.nakacya.jp"} 332947
nginx_server_bytes{direction="out",host="localhost"} 2554068410
nginx_server_bytes{direction="out",host="test1.nakacya.jp"} 597605
nginx_server_bytes{direction="out",host="www.nakacya.jp"} 84444
nginx_server_requests{code="1xx",host="*"} 0
nginx_server_requests{code="1xx",host="elk.nakacya.jp"} 0
nginx_server_requests{code="1xx",host="localhost"} 0
nginx_server_requests{code="1xx",host="test1.nakacya.jp"} 0
nginx_server_requests{code="1xx",host="www.nakacya.jp"} 0
nginx_server_requests{code="2xx",host="*"} 33505
nginx_server_requests{code="2xx",host="elk.nakacya.jp"} 73
nginx_server_requests{code="2xx",host="localhost"} 33312
nginx_server_requests{code="2xx",host="test1.nakacya.jp"} 83
nginx_server_requests{code="2xx",host="www.nakacya.jp"} 37
nginx_server_requests{code="3xx",host="*"} 14
nginx_server_requests{code="3xx",host="elk.nakacya.jp"} 14
nginx_server_requests{code="3xx",host="localhost"} 0
nginx_server_requests{code="3xx",host="test1.nakacya.jp"} 0
nginx_server_requests{code="3xx",host="www.nakacya.jp"} 0
nginx_server_requests{code="4xx",host="*"} 219
nginx_server_requests{code="4xx",host="elk.nakacya.jp"} 19
nginx_server_requests{code="4xx",host="localhost"} 133
nginx_server_requests{code="4xx",host="test1.nakacya.jp"} 33
nginx_server_requests{code="4xx",host="www.nakacya.jp"} 34
nginx_server_requests{code="5xx",host="*"} 2
nginx_server_requests{code="5xx",host="elk.nakacya.jp"} 0
nginx_server_requests{code="5xx",host="localhost"} 0
nginx_server_requests{code="5xx",host="test1.nakacya.jp"} 0
nginx_server_requests{code="5xx",host="www.nakacya.jp"} 2
nginx_server_requests{code="total",host="*"} 33740
nginx_server_requests{code="total",host="elk.nakacya.jp"} 106
nginx_server_requests{code="total",host="localhost"} 33445
nginx_server_requests{code="total",host="test1.nakacya.jp"} 116
nginx_server_requests{code="total",host="www.nakacya.jp"} 73
```
出た出た

```shell:sedで再加工&Host名でソート
$ cat /var/log/nginx/vts_status.log | sed -e s/\{/./g -e s/\}/" :"/g -e s/\"//g -e s/,/" "/g -e s/" host="/" "/g | sort -k2,2 > /tmp/temp.log
$ cat /tmp/temp.log
nginx_server_bytes.direction=in * : 11810163
nginx_server_bytes.direction=out * : 2555344161
nginx_server_requests.code=1xx * : 0
nginx_server_requests.code=2xx * : 33507
nginx_server_requests.code=3xx * : 14
nginx_server_requests.code=4xx * : 219
nginx_server_requests.code=5xx * : 2
nginx_server_requests.code=total * : 33742
nginx_server_bytes.direction=in elk.nakacya.jp : 146149
nginx_server_bytes.direction=out elk.nakacya.jp : 332947
nginx_server_requests.code=1xx elk.nakacya.jp : 0
nginx_server_requests.code=2xx elk.nakacya.jp : 73
nginx_server_requests.code=3xx elk.nakacya.jp : 14
nginx_server_requests.code=4xx elk.nakacya.jp : 19
nginx_server_requests.code=5xx elk.nakacya.jp : 0
nginx_server_requests.code=total elk.nakacya.jp : 106
nginx_server_bytes.direction=in localhost : 11583410
nginx_server_bytes.direction=out localhost : 2554329165
nginx_server_requests.code=1xx localhost : 0
nginx_server_requests.code=2xx localhost : 33314
nginx_server_requests.code=3xx localhost : 0
nginx_server_requests.code=4xx localhost : 133
nginx_server_requests.code=5xx localhost : 0
nginx_server_requests.code=total localhost : 33447
nginx_server_bytes.direction=in test1.nakacya.jp : 46807
nginx_server_bytes.direction=out test1.nakacya.jp : 597605
nginx_server_requests.code=1xx test1.nakacya.jp : 0
nginx_server_requests.code=2xx test1.nakacya.jp : 83
nginx_server_requests.code=3xx test1.nakacya.jp : 0
nginx_server_requests.code=4xx test1.nakacya.jp : 33
nginx_server_requests.code=5xx test1.nakacya.jp : 0
nginx_server_requests.code=total test1.nakacya.jp : 116
nginx_server_bytes.direction=in www.nakacya.jp : 33797
nginx_server_bytes.direction=out www.nakacya.jp : 84444
nginx_server_requests.code=1xx www.nakacya.jp : 0
nginx_server_requests.code=2xx www.nakacya.jp : 37
nginx_server_requests.code=3xx www.nakacya.jp : 0
nginx_server_requests.code=4xx www.nakacya.jp : 34
nginx_server_requests.code=5xx www.nakacya.jp : 2
nginx_server_requests.code=total www.nakacya.jp : 73
```

### filebeat用に微調整
改行が邪魔なのと日時がないと困るので
・日時追加
・改行コード削除

```
$ date "+%Y/%m/%dT%H:%M:%S" |sed -z 's/\n/ /g' && cat /tmp/temp.log | sed -z 's/\n/ /g' |sed -e "s/\$/\n/"
2020/02/05T04:04:33 nginx_server_bytes.direction=in * : 11810163 nginx_server_bytes.direction=out * : 2555344161 nginx_server_requests.code=1xx * : 0 nginx_server_requests.code=2xx * : 33507 nginx_server_requests.code=3xx * : 14 nginx_server_requests.code=4xx * : 219 nginx_server_requests.code=5xx * : 2 nginx_server_requests.code=total * : 33742
```

## 最後に
これを filebeat に食わせて ELK の Logstash で Grok に食わせればデータの完成
Grok はこんな感じ

```shell:Grok
^%{DATA:Date} nginx_server_bytes.direction=in %{DATA:Host} : %{DATA:In-Byte} nginx_server_bytes.direction=out .*: %{DATA:Out_Byte} nginx_server_cache.status=bypass .*: %{NUMBER:Cache_Bypass} nginx_server_cache.status=expired .*: %{NUMBER:Cache_Expired} nginx_server_cache.status=hit .*: %{NUMBER:Cache_Hit} nginx_server_cache.status=miss .*: %{NUMBER:Cache_Miss} nginx_server_cache.status=revalidated .*: %{NUMBER:Cache_Revalidated} nginx_server_cache.status=scarce .*: %{NUMBER:Cache_Scarce} nginx_server_cache.status=stale .*: %{NUMBER:Cache_Stale} nginx_server_cache.status=updating .*: %{NUMBER:Cache_Updating} nginx_server_requests.code=1xx .*: %{NUMBER:Requests_Code_1xx} nginx_server_requests.code=2xx .*: %{NUMBER:Requests_Code_2xx} nginx_server_requests.code=3xx .*: %{NUMBER:Requests_Code_3xx} nginx_server_requests.code=4xx .*: %{NUMBER:Requests_Code_4xx} nginx_server_requests.code=5xx .*: %{NUMBER:Requests_Code_5xx} nginx_server_requests.code=total .*: %{NUMBER:Requests_Code_Total}
```