yumでLogstashをインストールして、困ったこと

# 概要

Logstash を``yum install``したときに、つまづいたことのメモ。

（自分が参照した）ネットや書籍では、Logstashインストール後の動作確認で、``sudo logstash -e ...``などとrootで起動させるような記述がなされているが、そうすると、その後の``systemctl start logstash``でエラーになるよ、という話。

# 検証環境

* CentOS 7.5 (VirtualBox)
* Logstash 6.8.1
* OpenJDK 1.8.0_161 

# yum install したときの環境

``sudo yum install logstash`` で、つくられる環境は、下記のとおり。

## OSユーザー、グループ

~~~
# id logstash
uid=986(logstash) gid=980(logstash) groups=980(logstash)
# grep logstash /etc/passwd
logstash:x:986:980:logstash:/usr/share/logstash:/sbin/nologin
# grep logstash /etc/group
logstash:x:980:
~~~

## /etc/systemd/system/logstash.service

このように、logstash ユーザーで起動するように書かれている。

~~~
[Unit]
Description=logstash

[Service]
Type=simple
User=logstash
Group=logstash
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target
~~~

## ディレクトリ・ファイル

|Directory/File name|Permission|User|Group|Memo|
|---|---|---|---|---|
|/usr/share/logstash/|drwxr-xr-x.|logstash|logstash|LS_HOME|
|/usr/share/logstash/bin/logstash|-rwxr-xr-x.|logstash|logstash|アプリケーション本体|
|/etc/logstash/|drwxr-xr-x.|root|root|path.settings|
|/etc/logstash/logstash.yml|-rw-r--r--.|root|root||
|/etc/logstash/pipelines.yml|-rw-r--r--.|root|root||
|/etc/logstash/logstash-sample.conf|-rw-r--r--.|root|root|設定ファイルのサンプル|
|/etc/logstash/conf.d/|drwxrwxr-x.|root|root|設定ファイルをおくディレクトリ|
|/var/log/logstash/|drwxrwxr-x.|logstash|root|path.logs|
|/var/lib/logstash/|drwxrwxr-x.|logstash|logstash|path.data|

どんな意図があるのか、設定まわりが root でつくられている。

## startup.options

参考として、/etc/logstash/startup.options の抜粋。

~~~
LS_HOME=/usr/share/logstash
LS_SETTINGS_DIR=/etc/logstash
LS_OPTS="--path.settings ${LS_SETTINGS_DIR}"
LS_USER=logstash
LS_GROUP=logstash
SERVICE_NAME="logstash"
SERVICE_DESCRIPTION="logstash"
~~~

# Error/Warningになるパターン

という環境で、自分が遭遇したケースは下記のとおり。

## 動作確認を root でおこなった場合

インストール直後の動作確認を root でおこなうと、/usr/share/logstash/data/ 内に root でディレクトリなどが作成される。

~~~
$ sudo /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2019-07-12 11:55:05.967 [main] writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[INFO ] 2019-07-12 11:55:05.997 [main] writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[WARN ] 2019-07-12 11:55:06.500 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-07-12 11:55:06.515 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.8.1"}
[INFO ] 2019-07-12 11:55:06.591 [LogStash::Runner] agent - No persistent UUID file found. Generating new UUID {:uuid=>"be79488d-c123-4b09-a080-72ce424307a2", :path=>"/usr/share/logstash/data/uuid"}
[INFO ] 2019-07-12 11:55:15.668 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
The stdin plugin is now waiting for input:
[INFO ] 2019-07-12 11:55:15.968 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x72418697 sleep>"}
[INFO ] 2019-07-12 11:55:16.029 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2019-07-12 11:55:16.320 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
Hello Logstash!
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
      "@version" => "1",
       "message" => "Hello Logstash!",
          "host" => "node160.migo.jp",
    "@timestamp" => 2019-07-12T02:55:37.212Z
}
[INFO ] 2019-07-12 11:55:40.254 [[main]-pipeline-manager] pipeline - Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x72418697 run>"}
[INFO ] 2019-07-12 11:55:41.014 [LogStash::Runner] runner - Logstash shut down.
~~~

~~~
$ ls -l /usr/share/logstash/data
合計 4
drwxrwxr-x.  4 logstash logstash  69  7月 12 13:40 .
drwxr-xr-x. 11 logstash logstash 241  7月 12 11:19 ..
-rw-r--r--.  1 root     root       0  7月 12 11:55 .lock
drwxr-xr-x.  2 root     root       6  7月 12 11:55 dead_letter_queue
drwxr-xr-x.  2 root     root       6  7月 12 11:55 queue
-rw-r--r--.  1 root     root      36  7月 12 11:55 uuid
~~~

そのため、その後 logstash ユーザーで実行するとエラーになる。

~~~
$ sudo -u logstash  /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[FATAL] 2019-07-12 13:23:43.303 [main] runner - An unexpected error occurred! {:error=>#<ArgumentError: Path "/usr/share/logstash/data/queue" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:447:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:229:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:140:in `block in validate_all'", "org/jruby/RubyHash.java:1419:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:139:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:278:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:237:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in `<main>'"]}
[ERROR] 2019-07-12 13:23:43.324 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
~~~

こうなったら、/usr/share/logstash/data の中身をすべて（.lock も忘れずに）削除して、logstash ユーザーで起動しなおせばよい。すると、...

~~~
$ sudo -u logstash /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'
...（省略）...
$ ls -la /usr/share/logstash/data
合計 4
drwxrwxr-x.  4 logstash logstash  69  7月 12 13:49 .
drwxr-xr-x. 11 logstash logstash 241  7月 12 11:19 ..
-rw-r--r--.  1 logstash logstash   0  7月 12 13:49 .lock
drwxr-xr-x.  2 logstash logstash   6  7月 12 13:49 dead_letter_queue
drwxr-xr-x.  2 logstash logstash   6  7月 12 13:49 queue
-rw-r--r--.  1 logstash logstash  36  7月 12 13:49 uuid
~~~

こうなります。

## 動作確認で --path.settings をつけなかった場合

起動オプションの --path.settings を指定しないと、こんな警告が表示されます。

~~~
$ sudo -u logstash  /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
...（省略）...
~~~

動作確認レベルでは別に構わないのですが、オプションをつけると警告は表示されなくなります。

~~~
$ sudo -u logstash /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }' --path.settings "/etc/logstash"
~~~

ただ、/etc/logstash/ 内の設定が使われるので、/var/log/logstash/ にログが出力されたり、queue/ などが /usr/share/logstash/data/ ではなく /var/lib/logstash/ に作成されたりします。

# まとめ

* インストール後の動作確認は、logstash ユーザーでおこない、--path.setting オプションも指定してあげる。

~~~
$ sudo -u logstash /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }' --path.settings "/etc/logstash"
~~~

# 付録

設定ファイルをつくって、試す場合。
これでエラーが無ければ、``systemctl start logstash``しても大丈夫。

~~~
$ pwd
/etc/logstash
$ ls -l
合計 36
drwxrwxr-x. 2 root root    6  6月 18 23:13 conf.d
-rw-r--r--. 1 root root 1829  6月 18 23:13 jvm.options
-rw-r--r--. 1 root root 4568  6月 18 23:13 log4j2.properties
-rw-r--r--. 1 root root  342  6月 18 23:13 logstash-sample.conf
-rw-r--r--. 1 root root 8236  7月 12 11:19 logstash.yml
-rw-r--r--. 1 root root  285  6月 18 23:13 pipelines.yml
-rw-------. 1 root root 1696  6月 18 23:13 startup.options
$ sudo cp logstash-sample.conf ./conf.d/logstash.conf
$ cd conf.d/
$ ls -l
合計 4
-rw-r--r--. 1 root root 342  7月 12 14:19 logstash.conf
$ sudo vi logstash.conf 
$ cat logstash.conf 
input {
  stdin { }
}
output {
  stdout { }
}
$ sudo -u logstash  /usr/share/logstash/bin/logstash --path.settings "/etc/logstash"
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2019-07-12T14:25:39,164][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.8.1"}
[2019-07-12T14:25:47,865][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
The stdin plugin is now waiting for input:
[2019-07-12T14:25:48,189][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x73860c9c sleep>"}
[2019-07-12T14:25:48,246][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-07-12T14:25:48,521][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
Hello Logstash!
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
       "message" => "Hello Logstash!",
      "@version" => "1",
          "host" => "node160.migo.jp",
    "@timestamp" => 2019-07-12T05:26:14.655Z
}
[2019-07-12T14:26:17,853][INFO ][logstash.pipeline        ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x73860c9c run>"}
[2019-07-12T14:26:18,358][INFO ][logstash.runner          ] Logstash shut down.
~~~