MacでLogstashをお試し利用する

# Logstashをインストールする

```
brew install logstash
```

### 標準入力からテスト文字列を入力して、標準出力へ出力する

```
sample.log
sample.conf
```

.logと.confファイルは以下の通り

```text:sample.log
this is test string!!
```

```conf:sample.conf
input { 
      stdin {} 
} 

filter {
  mutate {
    replace => {message => "%{message} こんにちは！"}
  }
}

output { 
      stdout {
             codec => rubydebug 
      }
}
```

実行する

```
$ cat sample.log | logstash -f sample.conf 
Sending Logstash logs to /usr/local/Cellar/logstash/7.6.2/libexec/logs which is now configured via log4j2.properties
[2020-05-10T01:15:18,807][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-05-10T01:15:18,960][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.6.2"}
[2020-05-10T01:15:21,148][INFO ][org.reflections.Reflections] Reflections took 51 ms to scan 1 urls, producing 20 keys and 40 values 
[2020-05-10T01:15:22,767][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.RubyArray) has been created for key: cluster_uuids. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.
[2020-05-10T01:15:22,784][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/Users/seigo/Desktop/python/elastic/sample/test1.conf"], :thread=>"#<Thread:0x2745a27 run>"}
[2020-05-10T01:15:24,334][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-05-10T01:15:24,481][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
/usr/local/Cellar/logstash/7.6.2/libexec/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
       "message" => "test string2!! こんにちは！",
      "@version" => "1",
    "@timestamp" => 2020-05-09T16:15:24.487Z,
          "host" => "xxxx.local"
}
{
       "message" => "test string3!! こんにちは！",
      "@version" => "1",
    "@timestamp" => 2020-05-09T16:15:24.489Z,
          "host" => "xxxx.local"
}
{
       "message" => "test string!! こんにちは！",
      "@version" => "1",
    "@timestamp" => 2020-05-09T16:15:24.460Z,
          "host" => "xxxx.local"
}
[2020-05-10T01:15:25,195][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2020-05-10T01:15:25,660][INFO ][logstash.runner          ] Logstash shut down.
```

### ファイル出力

sample.conf にfileプラグインを記載する。


```text:sample.log
this is test string!!
```

```conf:sample.conf
input {
    stdin {}
}

output {
    file {
     path => "./out.log"
     codec => line { format => "custom format: %{message}"}
    }
}
```

実行する

```
$ cat sample.log | logstash -f sample.conf
$ cat out.log 
custom format: this is test string!!
```

### 入力文字列をそのまま出力する場合

sample.conf にfileプラグインの記載を微修正

```text:sample.log
this is test string!!
```

```conf:sample.conf
input {
    stdin {}
}

output {
    file {
     path => "./out.log"
     codec => line { format => "%{message}"}
    }
}
```

実行する

```
$ cat sample.log | logstash -f sample.conf
$ cat out.log 
this is test string!!
```

### json形式で出力する場合

```text:sample.log
test string!!
test string2!!
test string3!!
```

```text:sample.conf
input {
      stdin {}
}

filter {
  mutate {
    replace => {message => "%{message} こんにちは！"}
  }
}

output {
      stdout {
             codec => rubydebug
      }
      file {
           path => "./out.log"
           codec => json_lines
    }
}
```

```
$ cat sample.log | logstash -f sample.conf
...
$ cat out.log
{"@version":"1","host":"seigonoMacBook-puro-3206.local","message":"test string3!! こんにちは！","@timestamp":"2020-05-09T16:27:07.541Z"}
{"@version":"1","host":"seigonoMacBook-puro-3206.local","message":"test string2!! こんにちは！","@timestamp":"2020-05-09T16:27:07.540Z"}
{"@version":"1","host":"seigonoMacBook-puro-3206.local","message":"test string!! こんにちは！","@timestamp":"2020-05-09T16:27:07.518Z"}
```

### 出力ファイル名に日付を加える場合

```text:sample.log
test string!!
test string2!!
test string3!!
```

```text:sample.conf
input {
      stdin {}
}

filter {
  mutate {
    replace => {message => "%{message} こんにちは！"}
  }
}

output {
      stdout {
             codec => rubydebug
      }
      file {
           path => "./out_%{+YYYY-MM-dd-hh-mm}.log"
           codec => json_lines
    }
}
```

```
$ cat sample.log | logstash -f sample.conf
...
$ cat out_2020-05-09-04-39.log
{"@version":"1","host":"seigonoMacBook-puro-3206.local","message":"test string3!! こんにちは！","@timestamp":"2020-05-09T16:27:07.541Z"}
{"@version":"1","host":"seigonoMacBook-puro-3206.local","message":"test string2!! こんにちは！","@timestamp":"2020-05-09T16:27:07.540Z"}
{"@version":"1","host":"seigonoMacBook-puro-3206.local","message":"test string!! こんにちは！","@timestamp":"2020-05-09T16:27:07.518Z"}
```



# 参考
[Logstash入門](https://qiita.com/nihei9/items/1ebc7934eb1e1ce66162)
[en/logstash/current/output-plugins](https://www.elastic.co/guide/en/logstash/current/output-plugins.html)
[logstash output to file and ignores codec](https://stackoverflow.com/questions/29782888/logstash-output-to-file-and-ignores-codec/39270081)
[Logstash Reference [7.6] » Output plugins » File output plugin](https://www.elastic.co/guide/en/logstash/current/advanced-pipeline.html)
[サーバ監視・性能測定ツールまとめ](https://qiita.com/tunepolo/items/25b7a33d473a6a37cde7)