Logstashを使ってApacheログを読み込んでみた（１）

# はじめに

- ``Apache``ログ解析のために``Elasticsearch``を使ってみることにした
- [Logstashのリファレンス](https://www.elastic.co/guide/en/logstash/current/index.html#)の手順に従って進めた様子を、ターミナルに返ってきた出力も（割と）細かく残した備忘録にしてみた

# まとめ

- ``macOS``環境で動かしてみた
- ``Homebrew``を使って簡単インストールできた
  - ``Java8``、``Elasticsearch``、``Logstash``、``Kibana``、``Filebeat``
- 残り容量は大事、少なくてエラーがでた
  - ``Elasticsearch``の設定で変えられるみたいだが、今回はそこまでやらなかった

# つづきの記事

- [Logstashを使ってApacheログを読み込んでみた（１）](#)
- [Logstashを使ってApacheログを読み込んでみた（２）](https://qiita.com/shotakaha/items/c1a3a728b0b0736dc51a)

---

# 動作環境

- MacBook Air（テスト機）とMac mini（本番機）で動かしてみた【``$ uname -a``】

```bash:MacBook-Air
$ uname -a
Darwin 17.7.0 Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64
```

```bash:Mac-mini
$ uname -a
Darwin 16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 21 20:07:39 PDT 2018; root:xnu-3789.73.14~1/RELEASE_X86_64 x86_64
```

# Java8をインストールした

- [Installing Logstash (6.2)](https://www.elastic.co/guide/en/logstash/6.2/installing-logstash.html)を参照する
- ``Java8``が必要、``Java9``はサポートしてない
- Mac miniには``Java8``がインストールされていなかったので、``Homebrew-Cask``を使ってインストールした【``$ brew cask install homebrew/cask-versions/java8``】
- Oracle のウェブサイトからダウンロードすることもできる
  - https://www.oracle.com/technetwork/java/javase/downloads/index.html
  - Java SE Development Kit 8u181にある``jdk-8u181-macosx-x64.dmg``をダウンロードする

```
$ brew cask install homebrew/cask-versions/java8
==> Tapping homebrew/cask-versions
...
==> Caveats
This Cask makes minor modifications to the JRE to prevent issues with
packaged applications, as discussed here:

  https://bugs.eclipse.org/bugs/show_bug.cgi?id=411361

If your Java application still asks for JRE installation, you might need
to reboot or logout/login.

Installing java8 means you have AGREED to the license at
  https://www.oracle.com/technetwork/java/javase/terms/license/index.html

==> Satisfying dependencies
==> Downloading http://download.oracle.com/otn-pub/java/jdk/8u181-b13/96a7b8442fe848ef90c96a2fad6ed6d1/jdk-8u181-macosx-x64.dmg
######################################################################## 100.0%
==> Verifying checksum for Cask java8
==> Installing Cask java8
==> Running installer for java8; your password may be necessary.
==> Package installers may write to any location; options such as --appdir are ignored.
Password:  ## <-- sudoのパスワード入力を求められる
installer: Package name is JDK 8 Update 181
installer: Installing at base path /
installer: The install was successful.
🍺  java8 was successfully installed!

$ java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
```


# ELKをインストールした

- ``Elasticsearch``、``Logstash``、``Kibana``（の頭文字を取って``ELK``）を``Homebrew``を使ってインストールした
  - 最新版は``6.3.x``
  - ``Homebrew``にあるのは``6.2.4``だったが、これを使うことにした
- ``Logstash``のチュートリアルを読むと``Filebeat``も必要だったのでインストールした


1. ``Elasticsearch 6.2.4``
1. ``Logstash 6.2.4``
1. ``Kibana 6.2.4``
1. ``Filebeat 6.2.4``

```bash
$ brew install elasticsearch
$ elasticsearch -V
Version: 6.2.4, Build: ccec39f/2018-04-12T20:37:28.497551Z, JVM: 1.8.0_181

$ brew install logstash
$ logstash -V
logstash 6.2.4

$ brew install kibana
$ kibana -V
6.2.4

$ brew install filebeat
$ filebeat version
filebeat version 6.2.4 (amd64), libbeat 6.2.4
```

## 公式ドキュメントはここ

Elasticsearchのトップページからだと、ドキュメントがぱっと探せなかったので、ここにメモしておく

- [ドキュメント一覧](https://www.elastic.co/guide/index.html)
- [Elasticsearch Reference](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html)
- [Logstash Reference](https://www.elastic.co/guide/en/logstash/current/index.html)
- [Kibana Reference](https://www.elastic.co/guide/en/kibana/current/index.html)
- [Beats Platform Reference](https://www.elastic.co/guide/en/beats/libbeat/current/index.html)
- [Filebeat Reference](https://www.elastic.co/guide/en/beats/filebeat/6.3/filebeat-getting-started.html)


# Apacheのログ

- ドキュメントに置いてあるサンプルログを使った
  - https://download.elastic.co/demos/logstash/gettingstarted/logstash-tutorial.log.gz
  - ``~/Downloads/tmp/``に置くことにした
  - 100行のログファイル

```bash:サンプルログの行数を調べた
$ cd ~/Downloads/tmp/
$ wc -l logstash-tutorial.log
    100 logstash-tutorial.log
```

> Apacheログは、あとで自分のものに置き換えて使う

# まずはLogstashを使ってみる

- [Stashing Your First Event (6.2)](https://www.elastic.co/guide/en/logstash/6.2/first-event.html)を参照した
- とりあえず、以下のコマンドを実行し、テストしてみる【``$ logstash -e 'input { stdin { } } output { stdout {} }'``】

```bash
$ logstash -e 'input { stdin { } } output { stdout {} }'
#### logstashオプションの説明 ####
## -e CONFIG_STRING : Use the given string as the configuration data.
## Same syntax as the config file. 
## If no input is specified, then the following is used as the default input:
##     "input { stdin { type => stdin } }"
## and if no output is specified, then the following is used as the default output:
##     "output { stdout { codec => rubydebug } }"
## If you wish to use both defaults, please use the empty string for the '-e' flag.
#### #### #### ####
## 20秒くらい待つ
Sending Logstashs logs to /usr/local/Cellar/logstash/6.2.4/libexec/logs which is now configured via log4j2.properties
[2018-08-20T16:28:31,715][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/local/Cellar/logstash/6.2.4/libexec/modules/fb_apache/configuration"}
[2018-08-20T16:28:31,741][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/local/Cellar/logstash/6.2.4/libexec/modules/netflow/configuration"}
[2018-08-20T16:28:31,869][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/local/Cellar/logstash/6.2.4/libexec/data/queue"}
[2018-08-20T16:28:31,873][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/local/Cellar/logstash/6.2.4/libexec/data/dead_letter_queue"}
[2018-08-20T16:28:32,053][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-08-20T16:28:32,108][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"93a06fb0-2047-4bb7-98db-4cb4fb1bf6c3", :path=>"/usr/local/Cellar/logstash/6.2.4/libexec/data/uuid"}
[2018-08-20T16:28:32,804][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.2.4"}
[2018-08-20T16:28:33,392][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-08-20T16:28:36,513][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
The stdin plugin is now waiting for input:
[2018-08-20T16:28:36,725][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x5814ebe0 run>"}
[2018-08-20T16:28:36,822][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["main"]}
## 何か文字列を入力する
はじめてのlogstash　　
{
          "host" => "thibetanus",
    "@timestamp" => 2018-08-20T07:29:15.826Z,
       "message" => "はじめてのlogstash",
      "@version" => "1"
}
## Control-Cで抜け出す
^C
[2018-08-20T16:30:01,454][WARN ][logstash.runner          ] SIGINT received. Shutting down.
[2018-08-20T16:30:01,797][INFO ][logstash.pipeline        ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x5814ebe0 run>"}
```

# いよいよログをパースしてみる

- [Parsing Logs with Logstash](https://www.elastic.co/guide/en/logstash/6.2/advanced-pipeline.html)の手順に沿って進めた
- 実際にログをパースする場合、``input``／``filter``／``output``に分割して考える
- 以下の流れを書いた設定ファイルを作成する

1. ``input`` : Apache web logs を ``Filebeat`` を使って読み込む
2. ``filter`` : web logs をパースする
3. ``output`` : パースしたデータを``Elasticsearch cluster``に書き出す

# Filebeatを設定する

- ``Filebeat``から``Logstash``にログを送信する設定をする
- 設定ファイル名は ``filebeat.yml`` なので探す【``$ mdfind filebeat.yml``】
- ``Logstash pipeline``を作成する前に、設定する必要がある【``$ emacs filebeat.yml``】
- ``Logstash``をインストールすると``Beats input``プラグインもインストールされる（らしい。実は別のことをいじっていて``Filebeat``をインストールしたので、よく分からない）

```bash
$ mdfind filebeat.yml
/usr/local/etc/filebeat/filebeat.yml
```

- コピーを取って編集する

```bash
$ cd /usr/local/etc/filebeat/
$ cp filebeat.yml filebeat.yml.original
$ emacs filebeat.yml
```

- 変更箇所と内容は以下の通り

```yml:/usr/local/etc/filebeat/filebeat.yml
filebeat.prospectors:
- type: log
  # Change to true to enable this prospector configuration.
  enabled: true    # false -> true に変更
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    - /Users/shotakaha/Downloads/tmp/logstash-tutorial.log

#-------------------------- Elasticsearch output ------------------------------
# -> コメントアウトした
#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

#----------------------------- Logstash output --------------------------------
# -> コメントを外した
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]
```

```bash:Filebeatを起動
$ cd /usr/local/etc/filebeat/
$ filebeat -e -c filebeat.yml -d "publish"
#### Filebeatのオプションの説明 ####
## -e, --e        : Log to stderr and disable syslog/file output
## -c, --c string : Configuration file, relative to path.config (default "filebeat.yml")
## -d, --d string : Enable certain debug selectors
2018-08-20T16:35:45.542+0900	INFO	instance/beat.go:468	Home path: [/usr/local/Cellar/filebeat/6.2.4] Config path: [/usr/local/etc/filebeat] Data path: [/usr/local/var/lib/filebeat] Logs path: [/usr/local/var/log/filebeat]
2018-08-20T16:35:45.543+0900	INFO	instance/beat.go:475	Beat UUID: 9a84ae5b-db57-4f01-a7a7-6abe9547c00c
2018-08-20T16:35:45.543+0900	INFO	instance/beat.go:213	Setup Beat: filebeat; Version: 6.2.4
2018-08-20T16:35:45.546+0900	INFO	pipeline/module.go:76	Beat name: thibetanus
2018-08-20T16:35:45.546+0900	ERROR	fileset/modules.go:95	Not loading modules. Module directory not found: /usr/local/Cellar/filebeat/6.2.4/module
2018-08-20T16:35:45.547+0900	INFO	[monitoring]	log/log.go:97	Starting metrics logging every 30s
2018-08-20T16:35:45.547+0900	INFO	instance/beat.go:301	filebeat start running.
2018-08-20T16:35:45.547+0900	INFO	registrar/registrar.go:73	No registry file found under: /usr/local/var/lib/filebeat/registry. Creating a new registry file.
2018-08-20T16:35:45.548+0900	INFO	registrar/registrar.go:110	Loading registrar data from /usr/local/var/lib/filebeat/registry
2018-08-20T16:35:45.548+0900	INFO	registrar/registrar.go:121	States Loaded from registrar: 0
2018-08-20T16:35:45.548+0900	WARN	beater/filebeat.go:261	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2018-08-20T16:35:45.548+0900	INFO	crawler/crawler.go:48	Loading Prospectors: 1
2018-08-20T16:35:45.550+0900	INFO	log/prospector.go:111	Configured paths: [/Users/shotakaha/Downloads/tmp/logstash-tutorial.log]
2018-08-20T16:35:45.551+0900	INFO	crawler/crawler.go:82	Loading and starting Prospectors completed. Enabled prospectors: 1
2018-08-20T16:35:45.551+0900	INFO	log/harvester.go:216	Harvester started for file: /Users/shotakaha/Downloads/tmp/logstash-tutorial.log
2018-08-20T16:35:45.551+0900	INFO	cfgfile/reload.go:127	Config reloader started
2018-08-20T16:35:45.551+0900	INFO	cfgfile/reload.go:219	Loading of config files completed.
2018-08-20T16:35:45.552+0900	DEBUG	[publish]	pipeline/processor.go:275	Publish event: {
  "@timestamp": "2018-08-20T07:35:45.551Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.2.4"
  },
  "source": "/Users/shotakaha/Downloads/tmp/logstash-tutorial.log",
  "offset": 325,
  "message": "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
  "prospector": {
    "type": "log"
  },
  "beat": {
    "hostname": "thibetanus",
    "version": "6.2.4",
    "name": "thibetanus"
  }
}
...（続）...
2018-08-20T16:35:47.563+0900	ERROR	pipeline/output.go:74	Failed to connect: dial tcp [::1]:5044: connect: connection refused
2018-08-20T16:35:49.565+0900	ERROR	pipeline/output.go:74	Failed to connect: dial tcp 127.0.0.1:5044: connect: connection refused
2018-08-20T16:35:53.571+0900	ERROR	pipeline/output.go:74	Failed to connect: dial tcp [::1]:5044: connect: connection refused
2018-08-20T16:36:01.574+0900	ERROR	pipeline/output.go:74	Failed to connect: dial tcp [::1]:5044: connect: connection refused
2018-08-20T16:36:15.554+0900	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":31,"time":31},"total":{"ticks":59,"time":59,"value":59},"user":{"ticks":28,"time":28}},"info":{"ephemeral_id":"c1b0fc97-b700-44ca-893b-1a95caa78444","uptime":{"ms":30036}},"memstats":{"gc_next":4194304,"memory_alloc":2367888,"memory_total":3854200,"rss":15699968}},"filebeat":{"events":{"active":100,"added":101,"done":1},"harvester":{"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"type":"logstash"},"pipeline":{"clients":1,"events":{"active":100,"filtered":1,"published":100,"retry":400,"total":101}}},"registrar":{"states":{"current":1,"update":1},"writes":2},"system":{"cpu":{"cores":4},"load":{"1":1.6953,"15":1.9009,"5":1.9126,"norm":{"1":0.4238,"15":0.4752,"5":0.4781}}}}}}
2018-08-20T16:36:17.576+0900	ERROR	pipeline/output.go:74	Failed to connect: dial tcp 127.0.0.1:5044: connect: connection refused
```

- この段階では、``Filebeat``が``5044``ポートに接続しようとしているが、``Logstash``が動いていないので、エラーっぽい表示がされていても問題ない
- ``Filebeat``はこのまま動かしたまま``Logstash``の設定を行う

# Logstashを設定する

- ``Logstash``の設定ファイルを作成する
- 任意のディレクトリでよいみたいなので、サンプルログと同じ場所（``~/Downloads/tmp/``）に ``first-pipeline.conf`` を作成した

```conf:~/Downloads/tmp/first-pipeline.confを編集した
# The # character at the beginning of a line indicates a comment.
# Use comments to describe your configuration.
input {
    beats {
        port => "5044"
    }
}
# The filter part of this file is commented out to indicate that
# it is optional.
#filter {
#}
output {
    stdout { codec => rubydebug }
}
```

## 設定ファイルの内容の確認

- ``--config.test_and_exit``オプションをつけて、設定ファイルの内容をチェックする

```bash
$ cd ~/Downloads/tmp/
$ logstash -f first-pipeline.conf --config.test_and_exit
####
## -f CONFIG_PATH : Load the logstash config from a specific file or directory.
## If a directory is given, all files in that directory will be concatenated
## in lexicographical order and then parsed as a single config file.
## You can also specify wildcards (globs) and any matched files will
## be loaded in the order described above.
##
## -t, --config.test_and_exit : Check configuration for valid syntax and then exit.
Sending Logstashs logs to /usr/local/Cellar/logstash/6.2.4/libexec/logs which is now configured via log4j2.properties
[2018-08-20T16:44:37,574][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/local/Cellar/logstash/6.2.4/libexec/modules/fb_apache/configuration"}
[2018-08-20T16:44:37,748][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/local/Cellar/logstash/6.2.4/libexec/modules/netflow/configuration"}
[2018-08-20T16:44:38,076][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2018-08-20T16:44:42,122][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
```

- ``Ignoring the 'pipelines.yml'``という警告（``WARN``）は無視してよい
- ``OK``なようなので、``--config.reload.automatic``オプションに変更して実行

## 設定ファイル読み込みモードで内容を追記していく

- ``--config.reload.automatic``オプションをつけておくと、設定ファイルを変更した際に``Logstash``を再起動する必要がなくなる


```bash
$ logstash -f first-pipeline.conf --config.reload.automatic
####
## -r, --config.reload.automatic : Monitor configuration changes and reload whenever it is changed.
Sending Logstashs logs to /usr/local/Cellar/logstash/6.2.4/libexec/logs which is now configured via log4j2.properties
[2018-08-20T16:46:15,522][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/local/Cellar/logstash/6.2.4/libexec/modules/fb_apache/configuration"}
[2018-08-20T16:46:15,543][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/local/Cellar/logstash/6.2.4/libexec/modules/netflow/configuration"}
[2018-08-20T16:46:15,913][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-08-20T16:46:16,620][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.2.4"}
[2018-08-20T16:46:17,081][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-08-20T16:46:20,468][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-08-20T16:46:21,223][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2018-08-20T16:46:21,340][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0xa65fcfe run>"}
[2018-08-20T16:46:21,453][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2018-08-20T16:46:21,653][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["main"]}

## ...（これ以降何も表示されなかった・・・）
```

- チュートリアルによると``pipline``がきちんと設定されていれば、読み込んだログがパースされて表示されるらしいが、何も表示されなかった
- ``/usr/local/var/lib/filebeat/registry``が空っぽだったので、``Filebeat``を停止してから``registry``を削除し、そしてまた``Filebeat``を再起動した

```bash
$ rm /usr/local/var/lib/filebeat/registry
$ cd /usr/local/etc/filebeat/
$ filebeat -e -c filebeat.yml -d "publish"
$ cd ~/Downloads/tmp/
$ logstash -f first-pipeline.conf --config.reload.automatic
Sending Logstashs logs to /usr/local/Cellar/logstash/6.2.4/libexec/logs which is now configured via log4j2.properties
[2018-08-20T16:49:00,055][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/local/Cellar/logstash/6.2.4/libexec/modules/fb_apache/configuration"}
[2018-08-20T16:49:00,325][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/local/Cellar/logstash/6.2.4/libexec/modules/netflow/configuration"}
[2018-08-20T16:49:00,651][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-08-20T16:49:01,635][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.2.4"}
[2018-08-20T16:49:02,218][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-08-20T16:49:06,834][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-08-20T16:49:07,622][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2018-08-20T16:49:07,757][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0xf3cf3c0 run>"}
[2018-08-20T16:49:08,004][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2018-08-20T16:49:08,127][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["main"]}
{
        "source" => "/Users/shotakaha/Downloads/tmp/logstash-tutorial.log",
    "prospector" => {
        "type" => "log"
    },
          "beat" => {
            "name" => "thibetanus",
         "version" => "6.2.4",
        "hostname" => "thibetanus"
    },
       "message" => "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-dashboard3.png HTTP/1.1\" 200 171717 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
        "offset" => 654,
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
          "host" => "thibetanus",
      "@version" => "1",
    "@timestamp" => 2018-08-20T07:48:14.328Z
}
{
        "source" => "/Users/shotakaha/Downloads/tmp/logstash-tutorial.log",
    "prospector" => {
        "type" => "log"
    },
          "beat" => {
            "name" => "thibetanus",
         "version" => "6.2.4",
        "hostname" => "thibetanus"
    },
       "message" => "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/sad-medic.png HTTP/1.1\" 200 430406 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
        "offset" => 1945,
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
          "host" => "thibetanus",
      "@version" => "1",
    "@timestamp" => 2018-08-20T07:48:14.328Z
}
...（続）...
```

- きちんと表示されたっぽい
- ``Filebeat``から送られてくるログを``Logstash``で読むことができた


# ``Grok``フィルターを作成する

- Apacheログをもっときれいに読み込めるようにフィルタを作成する
- ``Logstash``標準フィルタのひとつである``Grok Filter``プラグインを使う
- 実は ``%{COMBINEDAPACHELOG}`` という便利なフィルターパターンがある（←当たり前か）

```bash:Apacheのログ形式（コンバイン形式）
$ head -n 1 logstash-tutorial.log
83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
```


```conf:~/Downloads/tmp/first-pipeline.confのfilterを編集した
# The # character at the beginning of a line indicates a comment.
# Use comments to describe your configuration.
input {
    beats {
        port => "5044"
    }
}
# The filter part of this file is commented out to indicate that
# it is optional.
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
}

output {
    stdout { codec => rubydebug }
}
```

- 設定ファイルを保存すると``Logstash``が自動で再起動する

```bash:first-pipeline.ymlを編集したあとのLogstash出力
## Logstashが自動で再起動する
[2018-08-20T17:02:59,441][ERROR][logstash.shutdownwatcher ] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.
[2018-08-20T17:03:00,914][INFO ][logstash.pipeline        ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0xf3cf3c0 run>"}
[2018-08-20T17:03:01,219][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-08-20T17:03:01,607][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2018-08-20T17:03:01,624][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x55669250 sleep>"}
[2018-08-20T17:03:01,625][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2018-08-20T17:03:01,653][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["main"]}
## ここで表示が止まるので、Filebeatの再起動を行う
```

- ``Filebeat``のデータベース（``registry``）を削除してから、再実行する


```bash:Filebeatを再起動したときの出力
## Control-CでFilebeatを停止
^C
2018-08-20T17:06:44.921+0900	INFO	beater/filebeat.go:323	Stopping filebeat
2018-08-20T17:06:44.922+0900	INFO	crawler/crawler.go:109	Stopping Crawler
2018-08-20T17:06:44.922+0900	INFO	crawler/crawler.go:119	Stopping 1 prospectors
2018-08-20T17:06:44.922+0900	INFO	cfgfile/reload.go:222	Dynamic config reloader stopped
2018-08-20T17:06:44.922+0900	INFO	prospector/prospector.go:121	Prospector ticker stopped
2018-08-20T17:06:44.922+0900	INFO	prospector/prospector.go:138	Stopping Prospector: 3853306801411698003
2018-08-20T17:06:44.922+0900	INFO	crawler/crawler.go:135	Crawler stopped
2018-08-20T17:06:44.922+0900	INFO	registrar/registrar.go:239	Stopping Registrar
2018-08-20T17:06:44.922+0900	INFO	registrar/registrar.go:167	Ending Registrar
2018-08-20T17:06:44.923+0900	INFO	instance/beat.go:308	filebeat stopped.
2018-08-20T17:06:44.923+0900	INFO	[monitoring]	log/log.go:132	Total non-zero metrics	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":89,"time":89},"total":{"ticks":192,"time":192,"value":192},"user":{"ticks":103,"time":103}},"info":{"ephemeral_id":"71f49981-8388-488d-a685-5411777868d3","uptime":{"ms":1110582}},"memstats":{"gc_next":4194304,"memory_alloc":2011024,"memory_total":27248576,"rss":18726912}},"filebeat":{"events":{"added":102,"done":102},"harvester":{"closed":1,"open_files":0,"running":0,"started":1}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"events":{"acked":100,"batches":1,"total":100},"read":{"bytes":6},"type":"logstash","write":{"bytes":6607}},"pipeline":{"clients":0,"events":{"active":0,"filtered":2,"published":100,"retry":700,"total":102},"queue":{"acked":100}}},"registrar":{"states":{"current":1,"update":102},"writes":5},"system":{"cpu":{"cores":4},"load":{"1":2.3979,"15":2.0972,"5":2.1934,"norm":{"1":0.5995,"15":0.5243,"5":0.5483}}}}}}
2018-08-20T17:06:44.923+0900	INFO	[monitoring]	log/log.go:133	Uptime: 18m30.583259034s
2018-08-20T17:06:44.923+0900	INFO	[monitoring]	log/log.go:110	Stopping metrics logging.

$ rm /usr/local/var/lib/filebeat/registry

$ filebeat -e -c filebeat.yml -d "publish"
2018-08-20T17:08:06.702+0900	INFO	instance/beat.go:468	Home path: [/usr/local/Cellar/filebeat/6.2.4] Config path: [/usr/local/etc/filebeat] Data path: [/usr/local/var/lib/filebeat] Logs path: [/usr/local/var/log/filebeat]
2018-08-20T17:08:06.702+0900	INFO	instance/beat.go:475	Beat UUID: 9a84ae5b-db57-4f01-a7a7-6abe9547c00c
2018-08-20T17:08:06.702+0900	INFO	instance/beat.go:213	Setup Beat: filebeat; Version: 6.2.4
2018-08-20T17:08:06.703+0900	INFO	pipeline/module.go:76	Beat name: thibetanus
2018-08-20T17:08:06.703+0900	ERROR	fileset/modules.go:95	Not loading modules. Module directory not found: /usr/local/Cellar/filebeat/6.2.4/module
2018-08-20T17:08:06.703+0900	INFO	[monitoring]	log/log.go:97	Starting metrics logging every 30s
2018-08-20T17:08:06.704+0900	INFO	instance/beat.go:301	filebeat start running.
2018-08-20T17:08:06.704+0900	INFO	registrar/registrar.go:73	No registry file found under: /usr/local/var/lib/filebeat/registry. Creating a new registry file.
2018-08-20T17:08:06.704+0900	INFO	registrar/registrar.go:110	Loading registrar data from /usr/local/var/lib/filebeat/registry
2018-08-20T17:08:06.704+0900	INFO	registrar/registrar.go:121	States Loaded from registrar: 0
2018-08-20T17:08:06.704+0900	WARN	beater/filebeat.go:261	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2018-08-20T17:08:06.704+0900	INFO	crawler/crawler.go:48	Loading Prospectors: 1
2018-08-20T17:08:06.705+0900	INFO	log/prospector.go:111	Configured paths: [/Users/shotakaha/Downloads/tmp/logstash-tutorial.log]
2018-08-20T17:08:06.705+0900	INFO	crawler/crawler.go:82	Loading and starting Prospectors completed. Enabled prospectors: 1
2018-08-20T17:08:06.706+0900	INFO	cfgfile/reload.go:127	Config reloader started
2018-08-20T17:08:06.706+0900	INFO	cfgfile/reload.go:219	Loading of config files completed.
2018-08-20T17:08:06.707+0900	INFO	log/harvester.go:216	Harvester started for file: /Users/shotakaha/Downloads/tmp/logstash-tutorial.log
2018-08-20T17:08:06.707+0900	DEBUG	[publish]	pipeline/processor.go:275	Publish event: {
  "@timestamp": "2018-08-20T08:08:06.707Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.2.4"
  },
  "prospector": {
    "type": "log"
  },
  "beat": {
    "name": "thibetanus",
    "hostname": "thibetanus",
    "version": "6.2.4"
  },
  "source": "/Users/shotakaha/Downloads/tmp/logstash-tutorial.log",
  "offset": 325,
  "message": "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\""
}
...（続）...
```

```bash:Logstashの出力
## ($logstash -f first-pipeline.conf --config.reload.automatic)
{
         "source" => "/Users/shotakaha/Downloads/tmp/logstash-tutorial.log",
     "prospector" => {
        "type" => "log"
    },
           "verb" => "GET",
          "ident" => "-",
        "message" => "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
           "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
       "response" => "200",
           "host" => "thibetanus",
          "agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
       "@version" => "1",
        "request" => "/presentations/logstash-monitorama-2013/images/kibana-search.png",
           "auth" => "-",
           "beat" => {
            "name" => "thibetanus",
         "version" => "6.2.4",
        "hostname" => "thibetanus"
    },
       "clientip" => "83.149.9.216",
         "offset" => 325,
          "bytes" => "203023",
       "referrer" => "\"http://semicomplete.com/presentations/logstash-monitorama-2013/\"",
    "httpversion" => "1.1",
      "timestamp" => "04/Jan/2015:05:13:42 +0000",
     "@timestamp" => 2018-08-20T08:08:06.707Z
}
...（続）...
```

- ``Filebeat``と``Logstash``は別々のターミナルで起動しておいた
- さっきと比べて、いろいろと追加されているのを確認した
- 次はさらに``GeoIP``フィルターも追加してみる


# ``GeoIP``フィルターの追加

```conf:first-pipeline.confのfilterをさらに編集した
# The # character at the beginning of a line indicates a comment.
# Use comments to describe your configuration.
input {
    beats {
        port => "5044"
    }
}
# The filter part of this file is commented out to indicate that
# it is optional.
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
    geoip {
        source => "clientip"
    }
}
output {
    stdout { codec => rubydebug }
}
```

- 設定ファイル（``first-pipeline.conf``）を編集した
- ``Filebeat``を停止した
- データベース（``registry``）を削除した
- ``Filebeat``の再起動した
- （``Logstash``は``reload.automatic``していたので、再起動せずともきちんと動いた）


```bash:Filebeatを再起動したときの出力
$ rm /usr/local/var/lib/filebeat/registry
$ filebeat -e -c filebeat.yml -d "publish"
## ...（省略）...
2018-08-20T17:13:28.651+0900	DEBUG	[publish]	pipeline/processor.go:275	Publish event: {
  "@timestamp": "2018-08-20T08:13:28.651Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.2.4"
  },
  "source": "/Users/shotakaha/Downloads/tmp/logstash-tutorial.log",
  "offset": 325,
  "message": "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
  "prospector": {
    "type": "log"
  },
  "beat": {
    "name": "thibetanus",
    "hostname": "thibetanus",
    "version": "6.2.4"
  }
}
## ...（続）...
```

```bash:Logstashの出力
## (logstash -f first-pipeline.conf --config.reload.automatic)
## ...（省略）...
{
         "source" => "/Users/shotakaha/Downloads/tmp/logstash-tutorial.log",
     "prospector" => {
        "type" => "log"
    },
           "verb" => "GET",
          "ident" => "-",
        "message" => "83.149.9.216 - - [04/Jan/2015:05:13:44 +0000] \"GET /presentations/logstash-monitorama-2013/plugin/zoom-js/zoom.js HTTP/1.1\" 200 7697 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
           "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
       "response" => "200",
           "host" => "thibetanus",
          "agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
       "@version" => "1",
        "request" => "/presentations/logstash-monitorama-2013/plugin/zoom-js/zoom.js",
          "geoip" => {
           "region_name" => "Moscow",
             "longitude" => 37.6184,
         "country_code3" => "RU",
              "timezone" => "Europe/Moscow",
             "city_name" => "Moscow",
          "country_name" => "Russia",
           "postal_code" => "101194",
                    "ip" => "83.149.9.216",
              "latitude" => 55.7485,
           "region_code" => "MOW",
        "continent_code" => "EU",
         "country_code2" => "RU",
              "location" => {
            "lat" => 55.7485,
            "lon" => 37.6184
        }
    },
           "auth" => "-",
           "beat" => {
            "name" => "thibetanus",
         "version" => "6.2.4",
        "hostname" => "thibetanus"
    },
       "clientip" => "83.149.9.216",
         "offset" => 1304,
          "bytes" => "7697",
       "referrer" => "\"http://semicomplete.com/presentations/logstash-monitorama-2013/\"",
    "httpversion" => "1.1",
      "timestamp" => "04/Jan/2015:05:13:44 +0000",
     "@timestamp" => 2018-08-20T08:13:28.651Z
}
...（続）...
```

- ``GeoIP``の項目が、さらにいろいろ追加されていることを確認した


# Elasticsearchに書き出す

- ``output``を標準出力から``elasticsearch``に変更する

```conf:~/Downloads/tmp/first-pipeline.confのoutputを編集した
# The # character at the beginning of a line indicates a comment.
# Use comments to describe your configuration.
input {
    beats {
        port => "5044"
    }
}
# The filter part of this file is commented out to indicate that
# it is optional.
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
    geoip {
        source => "clientip"
    }
}
output {
    elasticsearch {
        hosts => [ "localhost:9200" ]
    }
}
```

```bash:Logstashの出力
## ($ logstash -f first-pipeline.conf --config.reload.automatic)
[2018-08-20T17:19:27,401][INFO ][logstash.pipelineaction.reload] Reloading pipeline {"pipeline.id"=>:main}
[2018-08-20T17:19:32,420][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>0, "stalling_thread_info"=>{"other"=>[{"thread_id"=>93, "name"=>"[main]<beats", "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.0.13-java/lib/logstash/inputs/beats.rb:198:in `run'"}], ["LogStash::Filters::GeoIP", {"source"=>"clientip", "id"=>"d22d425d24614b7079e7eb5dfcb5b07c15d270877a3ee1bc69819bd0a41ef389"}]=>[{"thread_id"=>89, "name"=>nil, "current_call"=>"[...]/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:90:in `read_batch'"}, {"thread_id"=>90, "name"=>nil, "current_call"=>"[...]/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:90:in `read_batch'"}, {"thread_id"=>91, "name"=>nil, "current_call"=>"[...]/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:90:in `read_batch'"}, {"thread_id"=>92, "name"=>nil, "current_call"=>"[...]/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:90:in `read_batch'"}]}}
[2018-08-20T17:19:32,421][ERROR][logstash.shutdownwatcher ] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.
[2018-08-20T17:19:33,847][INFO ][logstash.pipeline        ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x6a659349 run>"}
[2018-08-20T17:19:34,106][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-08-20T17:19:35,207][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2018-08-20T17:19:35,226][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-08-20T17:19:35,558][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2018-08-20T17:19:35,585][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-08-20T17:19:35,628][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Template file '' could not be found!", :class=>"ArgumentError", :backtrace=>["/usr/local/Cellar/logstash/6.2.4/libexec/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/template_manager.rb:31:in `read_template_file'", "/usr/local/Cellar/logstash/6.2.4/libexec/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/template_manager.rb:17:in `get_template'", "/usr/local/Cellar/logstash/6.2.4/libexec/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/template_manager.rb:7:in `install_template'", "/usr/local/Cellar/logstash/6.2.4/libexec/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:96:in `install_template'", "/usr/local/Cellar/logstash/6.2.4/libexec/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:26:in `register'", "/usr/local/Cellar/logstash/6.2.4/libexec/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:9:in `register'", "/usr/local/Cellar/logstash/6.2.4/libexec/logstash-core/lib/logstash/output_delegator.rb:42:in `register'", "/usr/local/Cellar/logstash/6.2.4/libexec/logstash-core/lib/logstash/pipeline.rb:342:in `register_plugin'", "/usr/local/Cellar/logstash/6.2.4/libexec/logstash-core/lib/logstash/pipeline.rb:353:in `block in register_plugins'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/local/Cellar/logstash/6.2.4/libexec/logstash-core/lib/logstash/pipeline.rb:353:in `register_plugins'", "/usr/local/Cellar/logstash/6.2.4/libexec/logstash-core/lib/logstash/pipeline.rb:730:in `maybe_setup_out_plugins'", "/usr/local/Cellar/logstash/6.2.4/libexec/logstash-core/lib/logstash/pipeline.rb:363:in `start_workers'", "/usr/local/Cellar/logstash/6.2.4/libexec/logstash-core/lib/logstash/pipeline.rb:290:in `run'", "/usr/local/Cellar/logstash/6.2.4/libexec/logstash-core/lib/logstash/pipeline.rb:250:in `block in start'"]}
[2018-08-20T17:19:35,653][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2018-08-20T17:19:35,700][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/usr/local/Cellar/logstash/6.2.4/libexec/vendor/bundle/jruby/2.3.0/gems/logstash-filter-geoip-5.0.3-java/vendor/GeoLite2-City.mmdb"}
[2018-08-20T17:19:35,732][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2018-08-20T17:19:35,751][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2018-08-20T17:19:35,759][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x62aa0dd4 run>"}
[2018-08-20T17:19:35,766][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["main"]}
[2018-08-20T17:19:40,601][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-08-20T17:19:40,617][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2018-08-20T17:19:45,628][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-08-20T17:19:45,633][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2018-08-20T17:19:50,641][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-08-20T17:19:50,644][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2018-08-20T17:19:55,652][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-08-20T17:19:55,656][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}

## ...（エラーが続く）...
```

- なんか警告（``WARN``）が出たので確認した
- ``Ignoring the 'pipeline.yml' ...``は無視する
- ``Running health check to see if an Elasticsearch connection is working``は無視してはいけなさそう
- ``Elasticsearch``を起動した


```bash:elasticsearchを起動
$ elasticsearch
Java HotSpot(TM) 64-Bit Server VM warning: Cannot open file logs/gc.log due to No such file or directory

[2018-08-20T17:22:46,943][INFO ][o.e.n.Node               ] [] initializing ...
[2018-08-20T17:22:47,031][INFO ][o.e.e.NodeEnvironment    ] [bOhJS94] using [1] data paths, mounts [[/ (/dev/disk2)]], net usable_space [1.3tb], net total_space [1.9tb], types [hfs]
[2018-08-20T17:22:47,031][INFO ][o.e.e.NodeEnvironment    ] [bOhJS94] heap size [990.7mb], compressed ordinary object pointers [true]
[2018-08-20T17:22:47,033][INFO ][o.e.n.Node               ] node name [bOhJS94] derived from node ID [bOhJS94JTEO6dkZgOfbX8g]; set [node.name] to override
[2018-08-20T17:22:47,033][INFO ][o.e.n.Node               ] version[6.2.4], pid[12778], build[ccec39f/2018-04-12T20:37:28.497551Z], OS[Mac OS X/10.12.6/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_181/25.181-b13]
[2018-08-20T17:22:47,033][INFO ][o.e.n.Node               ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/var/folders/k8/x_r4b8q52fg6_fv_7z32kj040000gn/T/elasticsearch.WbkW7qkz, -XX:+HeapDumpOnOutOfMemoryError, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/local/Cellar/elasticsearch/6.2.4/libexec, -Des.path.conf=/usr/local/etc/elasticsearch]
[2018-08-20T17:22:47,877][INFO ][o.e.p.PluginsService     ] [bOhJS94] loaded module [aggs-matrix-stats]
[2018-08-20T17:22:47,877][INFO ][o.e.p.PluginsService     ] [bOhJS94] loaded module [analysis-common]
[2018-08-20T17:22:47,877][INFO ][o.e.p.PluginsService     ] [bOhJS94] loaded module [ingest-common]
[2018-08-20T17:22:47,878][INFO ][o.e.p.PluginsService     ] [bOhJS94] loaded module [lang-expression]
[2018-08-20T17:22:47,878][INFO ][o.e.p.PluginsService     ] [bOhJS94] loaded module [lang-mustache]
[2018-08-20T17:22:47,878][INFO ][o.e.p.PluginsService     ] [bOhJS94] loaded module [lang-painless]
[2018-08-20T17:22:47,878][INFO ][o.e.p.PluginsService     ] [bOhJS94] loaded module [mapper-extras]
[2018-08-20T17:22:47,878][INFO ][o.e.p.PluginsService     ] [bOhJS94] loaded module [parent-join]
[2018-08-20T17:22:47,878][INFO ][o.e.p.PluginsService     ] [bOhJS94] loaded module [percolator]
[2018-08-20T17:22:47,878][INFO ][o.e.p.PluginsService     ] [bOhJS94] loaded module [rank-eval]
[2018-08-20T17:22:47,879][INFO ][o.e.p.PluginsService     ] [bOhJS94] loaded module [reindex]
[2018-08-20T17:22:47,879][INFO ][o.e.p.PluginsService     ] [bOhJS94] loaded module [repository-url]
[2018-08-20T17:22:47,879][INFO ][o.e.p.PluginsService     ] [bOhJS94] loaded module [transport-netty4]
[2018-08-20T17:22:47,879][INFO ][o.e.p.PluginsService     ] [bOhJS94] loaded module [tribe]
[2018-08-20T17:22:47,879][INFO ][o.e.p.PluginsService     ] [bOhJS94] no plugins loaded
[2018-08-20T17:22:51,601][INFO ][o.e.d.DiscoveryModule    ] [bOhJS94] using discovery type [zen]
[2018-08-20T17:22:52,210][INFO ][o.e.n.Node               ] initialized
[2018-08-20T17:22:52,211][INFO ][o.e.n.Node               ] [bOhJS94] starting ...
[2018-08-20T17:22:52,389][INFO ][o.e.t.TransportService   ] [bOhJS94] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-08-20T17:22:55,447][INFO ][o.e.c.s.MasterService    ] [bOhJS94] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {bOhJS94}{bOhJS94JTEO6dkZgOfbX8g}{ymM_Ll5vQ522PDzmHujc-g}{127.0.0.1}{127.0.0.1:9300}
[2018-08-20T17:22:55,451][INFO ][o.e.c.s.ClusterApplierService] [bOhJS94] new_master {bOhJS94}{bOhJS94JTEO6dkZgOfbX8g}{ymM_Ll5vQ522PDzmHujc-g}{127.0.0.1}{127.0.0.1:9300}, reason: apply cluster state (from master [master {bOhJS94}{bOhJS94JTEO6dkZgOfbX8g}{ymM_Ll5vQ522PDzmHujc-g}{127.0.0.1}{127.0.0.1:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-08-20T17:22:55,475][INFO ][o.e.h.n.Netty4HttpServerTransport] [bOhJS94] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-08-20T17:22:55,475][INFO ][o.e.n.Node               ] [bOhJS94] started
[2018-08-20T17:22:55,477][INFO ][o.e.g.GatewayService     ] [bOhJS94] recovered [0] indices into cluster_state

## MBAの場合
[2018-08-15T17:04:22,275][INFO ][o.e.g.GatewayService     ] [RYRm5T8] recovered [3] indices into cluster_state
[2018-08-15T17:04:23,160][INFO ][o.e.c.r.a.AllocationService] [RYRm5T8] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[logstash-2018.08.14][3], [.kibana][0]] ...]).
```

```bash:Logstashに追記された出力
[2018-08-20T17:22:57,233][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-08-20T17:22:57,420][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2018-08-20T17:22:57,500][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-08-20T17:22:57,509][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
```

- さっきの警告（``WARN``）がなくなった
- 別の警告（``WARN``）が出ているけれどとりあえず無視する
- ``Filebeat``を再起動する


```bash:Filebeatを再起動したときの出力
$ rm /usr/local/var/lib/filebeat/registry
$ filebeat -e -c filebeat.yml -d "publish"
2018-08-20T17:30:02.132+0900	INFO	instance/beat.go:468	Home path: [/usr/local/Cellar/filebeat/6.2.4] Config path: [/usr/local/etc/filebeat] Data path: [/usr/local/var/lib/filebeat] Logs path: [/usr/local/var/log/filebeat]
2018-08-20T17:30:02.133+0900	INFO	instance/beat.go:475	Beat UUID: 9a84ae5b-db57-4f01-a7a7-6abe9547c00c
2018-08-20T17:30:02.133+0900	INFO	instance/beat.go:213	Setup Beat: filebeat; Version: 6.2.4
2018-08-20T17:30:02.134+0900	INFO	pipeline/module.go:76	Beat name: tora.kek.jp
2018-08-20T17:30:02.135+0900	ERROR	fileset/modules.go:95	Not loading modules. Module directory not found: /usr/local/Cellar/filebeat/6.2.4/module
2018-08-20T17:30:02.135+0900	INFO	[monitoring]	log/log.go:97	Starting metrics logging every 30s
2018-08-20T17:30:02.136+0900	INFO	instance/beat.go:301	filebeat start running.
2018-08-20T17:30:02.136+0900	INFO	registrar/registrar.go:73	No registry file found under: /usr/local/var/lib/filebeat/registry. Creating a new registry file.
2018-08-20T17:30:02.138+0900	INFO	registrar/registrar.go:110	Loading registrar data from /usr/local/var/lib/filebeat/registry
2018-08-20T17:30:02.138+0900	INFO	registrar/registrar.go:121	States Loaded from registrar: 0
2018-08-20T17:30:02.138+0900	WARN	beater/filebeat.go:261	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2018-08-20T17:30:02.138+0900	INFO	crawler/crawler.go:48	Loading Prospectors: 1
2018-08-20T17:30:02.141+0900	INFO	log/prospector.go:111	Configured paths: [/Users/shotakaha/Downloads/tmp/logstash-tutorial.log]
2018-08-20T17:30:02.142+0900	INFO	crawler/crawler.go:82	Loading and starting Prospectors completed. Enabled prospectors: 1
2018-08-20T17:30:02.142+0900	INFO	cfgfile/reload.go:127	Config reloader started
2018-08-20T17:30:02.142+0900	INFO	cfgfile/reload.go:219	Loading of config files completed.
2018-08-20T17:30:02.142+0900	INFO	log/harvester.go:216	Harvester started for file: /Users/shotakaha/Downloads/tmp/logstash-tutorial.log
2018-08-20T17:30:02.143+0900	DEBUG	[publish]	pipeline/processor.go:275	Publish event: {
  "@timestamp": "2018-08-20T08:30:02.142Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.2.4"
  },
  "beat": {
    "hostname": "tora.kek.jp",
    "version": "6.2.4",
    "name": "tora.kek.jp"
  },
  "source": "/Users/shotakaha/Downloads/tmp/logstash-tutorial.log",
  "offset": 325,
  "message": "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
  "prospector": {
    "type": "log"
  }
}
...（続）...
```

```bash:Elasticsearchに追記された出力
[2018-08-20T17:30:03,707][INFO ][o.e.c.m.MetaDataCreateIndexService] [bOhJS94] [logstash-2018.08.20] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings []
[2018-08-20T17:30:04,107][INFO ][o.e.c.m.MetaDataMappingService] [bOhJS94] [logstash-2018.08.20/Cr1beaUFT1KDp3uC2YCrMw] create_mapping [doc]
[2018-08-20T17:30:04,127][INFO ][o.e.c.m.MetaDataMappingService] [bOhJS94] [logstash-2018.08.20/Cr1beaUFT1KDp3uC2YCrMw] update_mapping [doc]
[2018-08-20T17:30:04,281][INFO ][o.e.c.m.MetaDataMappingService] [bOhJS94] [logstash-2018.08.20/Cr1beaUFT1KDp3uC2YCrMw] update_mapping [doc]
```

- ちゃんと動いているかを確認する

```bash:日付を確認する
$ curl -XGET 'localhost:9200/_cat/indices?v'
health status index               uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   logstash-2018.08.20 Cr1beaUFT1KDp3uC2YCrMw   5   1        100            0    380.9kb        380.9kb
```

```bash:レスポンスを取ってくる
$ curl -XGET 'localhost:9200/logstash-2018.08.20/_search?pretty&q=response=200'
{
  "took" : 24,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 98,
    "max_score" : 2.5306559,
    "hits" : [
      {
        "_index" : "logstash-2018.08.20",
        "_type" : "doc",
        "_id" : "waF0VmUBuPIQbAmZjlw9",
        "_score" : 2.5306559,
        "_source" : {
          "source" : "/Users/shotakaha/Downloads/tmp/logstash-tutorial.log",
          "prospector" : {
            "type" : "log"
          },
          "verb" : "GET",
          "ident" : "-",
          "message" : "83.149.9.216 - - [04/Jan/2015:05:13:45 +0000] \"GET /presentations/logstash-monitorama-2013/images/frontend-response-codes.png HTTP/1.1\" 200 52878 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
          "tags" : [
            "beats_input_codec_plain_applied"
          ],
          "response" : "200",
          "host" : "tora.kek.jp",
          "agent" : "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
          "@version" : "1",
          "request" : "/presentations/logstash-monitorama-2013/images/frontend-response-codes.png",
          "geoip" : {
            "region_name" : "Moscow",
            "longitude" : 37.6184,
            "country_code3" : "RU",
            "timezone" : "Europe/Moscow",
            "city_name" : "Moscow",
            "country_name" : "Russia",
            "postal_code" : "101194",
            "ip" : "83.149.9.216",
            "latitude" : 55.7485,
            "region_code" : "MOW",
            "continent_code" : "EU",
            "country_code2" : "RU",
            "location" : {
              "lat" : 55.7485,
              "lon" : 37.6184
            }
          },
          "auth" : "-",
          "beat" : {
            "name" : "tora.kek.jp",
            "version" : "6.2.4",
            "hostname" : "tora.kek.jp"
          },
          "clientip" : "83.149.9.216",
          "offset" : 2932,
          "bytes" : "52878",
          "referrer" : "\"http://semicomplete.com/presentations/logstash-monitorama-2013/\"",
          "httpversion" : "1.1",
          "timestamp" : "04/Jan/2015:05:13:45 +0000",
          "@timestamp" : "2018-08-20T08:30:02.143Z"
        }
      },
      {
        "_index" : "logstash-2018.08.20",
        "_type" : "doc",
        "_id" : "rKF0VmUBuPIQbAmZjlwy",
        "_score" : 0.09646662,
        "_source" : {
          "source" : "/Users/shotakaha/Downloads/tmp/logstash-tutorial.log",
          "prospector" : {
            "type" : "log"
          },
          "verb" : "GET",
          "ident" : "-",
          "message" : "200.49.190.101 - - [04/Jan/2015:05:17:39 +0000] \"GET /reset.css HTTP/1.1\" 200 1015 \"-\" \"-\"",
          "tags" : [
            "beats_input_codec_plain_applied"
          ],
          "response" : "200",
          "host" : "tora.kek.jp",
          "agent" : "\"-\"",
          "@version" : "1",
          "request" : "/reset.css",
          "geoip" : {
            "region_name" : "Departamento de Huehuetenango",
            "longitude" : -91.3497,
            "country_code3" : "GT",
            "timezone" : "America/Guatemala",
            "country_name" : "Guatemala",
            "ip" : "200.49.190.101",
            "latitude" : 15.4731,
            "region_code" : "HU",
            "continent_code" : "NA",
            "country_code2" : "GT",
            "location" : {
              "lat" : 15.4731,
              "lon" : -91.3497
            }
          },
          "auth" : "-",
          "beat" : {
            "name" : "tora.kek.jp",
            "version" : "6.2.4",
            "hostname" : "tora.kek.jp"
          },
          "clientip" : "200.49.190.101",
          "offset" : 12114,
          "bytes" : "1015",
          "referrer" : "\"-\"",
          "httpversion" : "1.1",
          "timestamp" : "04/Jan/2015:05:17:39 +0000",
          "@timestamp" : "2018-08-20T08:30:02.145Z"
        }
      },
...（続）...
```

- やっていることがよくわからないけれど、なんだかちゃんと動いてそう
- ターミナルで``curl``コマンドを使ったが、ブラウザでも確認できる
  - http://localhost:9200/_cat/indices?v
  - http://localhost:9200/logstash-2018.08.20/_search?pretty&q=response=200
  - http://localhost:9200/logstash-2018.08.20/_search?pretty&q=geoip.city_name=Buffalo

# Kibanaで確認した

- ``Kibana``を起動する

```bash:Kibanaの出力
$ kibana
  log   [08:40:40.174] [info][status][plugin:kibana@6.2.4] Status changed from uninitialized to green - Ready
  log   [08:40:40.224] [info][status][plugin:elasticsearch@6.2.4] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [08:40:40.422] [info][status][plugin:timelion@6.2.4] Status changed from uninitialized to green - Ready
  log   [08:40:40.428] [info][status][plugin:console@6.2.4] Status changed from uninitialized to green - Ready
  log   [08:40:40.433] [info][status][plugin:metrics@6.2.4] Status changed from uninitialized to green - Ready
  log   [08:40:40.454] [info][listening] Server running at http://localhost:5601
  log   [08:40:40.507] [info][status][plugin:elasticsearch@6.2.4] Status changed from yellow to green - Ready
```

- ここに書いてある``localhost:5601``をブラウザで開く
- とりあえず``Discover``を開く
- 横軸``@timestamp``、縦軸``count``の棒グラフが表示された
  - 別の日に開いてみたら``No results found``と表示された
  - 期間が``Last 15 minuts``になっていたため、右上の時計マークのところをクリックして``This week``に変更した

> ``@timestamp``が、ログファイルを読み込んだ日時になっているためフィルタ時に変更する


# 残り容量の少ないPCでエラーがでた

- ``MacBook Air``のHD容量が5%を切っていたため、エラーに悩まされた・・・
- 解決法は見つかったが、試していない。放置状態


## エラーがでた

- ``Elasticsearch``を起動したターミナルと、``Logstash``を起動したターミナルの出力にもエラーが出ていた

```bash:Elasticsearchのエラー
[2018-08-15T17:04:52,065][WARN ][o.e.c.r.a.DiskThresholdMonitor] [RYRm5T8] flood stage disk watermark [95%] exceeded on [RYRm5T8DQI2Yz1xF7-PdiQ][RYRm5T8][/usr/local/var/lib/elasticsearch/nodes/0] free: 15.7gb[3.3%], all indices on this node will marked read-only
[2018-08-15T17:05:22,099][WARN ][o.e.c.r.a.DiskThresholdMonitor] [RYRm5T8] flood stage disk watermark [95%] exceeded on [RYRm5T8DQI2Yz1xF7-PdiQ][RYRm5T8][/usr/local/var/lib/elasticsearch/nodes/0] free: 15.7gb[3.3%], all indices on this node will marked read-only
[2018-08-15T17:05:52,131][WARN ][o.e.c.r.a.DiskThresholdMonitor] [RYRm5T8] flood stage disk watermark [95%] exceeded on [RYRm5T8DQI2Yz1xF7-PdiQ][RYRm5T8][/usr/local/var/lib/elasticsearch/nodes/0] free: 15.7gb[3.3%], all indices on this node will marked read-only
...（続）...
```

```bash:Logstashのエラー
[2018-08-15T17:09:51,873][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2018-08-15T17:09:51,873][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2018-08-15T17:09:51,881][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
...（続）...
```

## エラーについて調べてみた

- なんだかストレージ不足が原因みたい
  - https://discuss.elastic.co/t/forbidden-12-index-read-only-allow-delete-api/110282
  - https://discuss.elastic.co/t/forbidden-12-index-read-only-allow-delete-api/126067
  - https://stackoverflow.com/questions/48119189/elasticsearch-6-index-change-to-read-only-after-few-second
  - https://www.elastic.co/guide/en/elasticsearch/reference/6.2/disk-allocator.html