Redmineのproduction.logを監視する

# 目的
Redmineのパフォーマンスダウン、エラーなどを監視する。


# 前提条件
## Redmineサーバ
* オンプレ
 * Redmine 3.4.6
 * nginx+unicornのリバースプロキシ構成のため、クライアントの解析は他のコンテンツも含めてnginxのログで行う。
* logstash-5.6.13
 
## Elasticsearch/Kibana
* AmazonES
 * Elasticsearch のバージョン6.3
 * アクセスポリシー：「特定のIPからのドメインへのアクセスを許可」として会社のゲートウェイIPを指定
 

# 監視対象
`/var/log/nginx/access.log`
`/usr/local/redmine/log/production.log`

production.logの解析対象は処理時間、ステータスコードを含む３パターンとした。

```
  Rendered issues/index.html.erb within layouts/base (54.0ms)
Completed 200 OK in 31ms (Views: 1.2ms | ActiveRecord: 8.1ms)
Completed 302 Found in 487ms (ActiveRecord: 465.0ms)
```

# Logstashの設定

```file:/etc/logstash/conf.d/input.conf
input {
  file {
    path => ["/var/log/nginx/access.log*"]
    start_position => beginning
    type => "nginx"
    exclude => ["*.gz"]
  }

  file {
    path => ["/usr/local/redmine/log/production.log"]
    start_position => beginning
    type => "redmine"
  }
}
```

```file:/etc/logstash/conf.d/filter.conf
filter {
  if [type] == "nginx" {
    grok {
      match => [ "message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:forwarder} %{NUMBER:responsetime}"]
      overwrite => [ "message" ]
    }
    date {
      match=>["timestamp","dd/MMM/yyyy:HH:mm:ss Z"]
      locale => us
      remove_field => [ "timestamp" ]
    }

    mutate {
      convert => ["response", "integer"]
      convert => ["bytes", "integer"]
      convert => ["responsetime", "float"]
      gsub => ["referrer","\"",""]
      gsub => ["agent","\"",""]
      gsub => ["forwarder","\"",""]
    }

    grok {
      match => { "request" => "^/%{WORD:first_path}/%{GREEDYDATA}$" }
      tag_on_failure => ["_request_parse_failure"]
    }

    if "_request_parse_failure" in [tags] {
      mutate {
        replace => {"first_path" => "/"}
      }
    } else {
      mutate {
        replace => {"first_path" => "/%{first_path}" }
      }
    }

    useragent {
      source => "agent"
    }
  }

  if [type] == "redmine" {
    grok {
      match => {
        "message" => [
          "  Rendered %{GREEDYDATA:access_path} .*\(%{NUMBER:rendering_time}ms\)",
          "Completed %{NUMBER:response}.*in %{NUMBER:total_time}ms \(Views: %{NUMBER:view_time}ms \| ActiveRecord: %{NUMBER:active_record_time}ms\)",
          "Completed %{NUMBER:response}.*in %{NUMBER:total_time}ms \(ActiveRecord: %{NUMBER:active_record_time}ms\)"
        ]
      }
    }

    mutate {
      convert => ["rendering_time", "float"]
      convert => ["total_time", "float"]
      convert => ["view_time", "float"]
      convert => ["active_record_time", "float"]
    }
  }
}
```

```file:/etc/logstash/conf.d/output.conf
output {

  if "_grokparsefailure" not in [tags] {

    if [type] == "nginx" {
      elasticsearch {
        hosts => ["search-XXXXXXXXXXXX.ap-northeast-1.es.amazonaws.com:443"]
        ssl => true
        index => "nginx-%{+YYYY.MM.dd}"
      }
    }

    if [type] == "redmine" {
      elasticsearch {
        hosts => ["search-XXXXXXXXXXXX.ap-northeast-1.es.amazonaws.com:443"]
        ssl => true
        index => "redmine-%{+YYYY.MM.dd}"
      }
    }
  }
}
```
既存の流用のため、AmazonESプラグインは使用しなかった。
ElasticsearchエンドポイントのFQDN＋ポート443、sslの使用を設定する。

# Kibanaの設定

![image.png](https://qiita-image-store.s3.amazonaws.com/0/63464/46e584a8-ea69-a77c-50a6-01143bc33b5d.png)

## Redmine_millisec
* Index:redmine-*
* Type:Line
* Y-Axis
 * Aggregation:Max
 * Field:total_time
* Y-Axis
 * Aggregation:Max
 * Field:active_record_time
* Y-Axis
 * Aggregation:Max
 * Field:rendering_time
* Y-Axis
 * Aggregation:Max
 * Field:view_time
* X-Axis
 * Aggregation:Date Histogram
 * Field: @&#65520;timestamp
 * Interval:Hourly
 
## Redmine_err_count
※あらかじめ、「response.keyword:/(4|5).*/」をSearchとして登録

* Index:redmine-*
* Type:Area
* Y-Axis
 * Aggregation:Count
* X-Axis
 * Aggregation:Date Histogram
 * Field: @&#65520;timestamp
 * Interval:Hourly
* Split Series
 * Sub Aggregation:Terms
 * Field:response.keyword
 * Order By:Term
 * Order:Ascending
 * Size:10