JavaアプリケーションのログをFluentd(Windows)へ送信する

# 概要

開発、検証目的としてWindows PCにインストールしたFluentdに、Javaアプリケーションが出力するログを送信する方法のまとめです。

**環境**

* Windows7 (64bit)
* Fluentd 0.12.16
 * Ruby 2.2.3
* Java 1.8.0_60
* Git 2.6

**参考**

* [fluent/fluentd - github] (https://github.com/fluent/fluentd)
* [Fluentd が Windows を正式サポートしたので動かした] (http://sqlazure.jp/r/tool/665/)
* [fluentd Windows版をWindowsサービスとして動かす] (http://qiita.com/okahashi117/items/9b55418e770f1f5ce4b7)


## Javaアプリケーションのログを送信する

### 1. [Fluent Logger for Java] (https://github.com/fluent/fluent-logger-java)を使用する方法

Fluent Loggerを使用してログを直接fluentdへ送信する方法です。

**Maven**

```xml:pom.xml
<dependency>
  <groupId>org.fluentd</groupId>
  <artifactId>fluent-logger</artifactId>
  <version>0.3.2</version>
</dependency>
```

**FluentLogger**

* tagPrefixに任意のタグを指定します。
* hostのデフォルトは`localhost`です。
* portのデフォルトは`24224`です。

```java:例
final static FluentLogger LOG = FluentLogger.getLogger("app");
```

hostとportに任意の値を指定することもできます。

```java:例
final static FluentLogger LOG = FluentLogger.getLogger("app", "localhost", 24224);
```

**LOG**

key/valueの形式でログを送信します。

```java:例
LOG.log("tag1", "key1", "value1");
```

```text:log
2015-10-20 19:41:19 +0900 app.tag1: {"key1":"value1"}
```

timeに任意の時刻を指定することができます。

```java:例
long epoch = System.currentTimeMillis()/1000L;
LOG.log("tag1", "key1", "value1", epoch + 10L);
```

Mapの値を送信します。

```java:例
Map<String, Object> data = new HashMap<>();
data.put("key2-A", "value2-A");
data.put("key2-B", "value2-B");
data.put("key2-C", "value2-C");
LOG.log("tag2", data);
```

fluentdの設定

```text:conf
<source>
  @type forward
</source>

<match app.*>
  @type stdout
</match>
```

上記のログを入力としたfluentdの出力結果

```text:fluentd_output
2015-10-20 19:41:19 +0900 app.tag2: {"key2-C":"value2-C","key2-B":"value2-B","key2-A":"value2-A"}
```

**close**

アプリケーション終了時にcloseAllメソッドを呼ぶ必要があります。

```java:例
FluentLogger.closeAll();
```


### 2. [logback] (http://logback.qos.ch/)のlogback-jsonを利用する方法

正確にはJavaアプリケーションから送信ではなくfluentdからログを読み取る方法になります。
logbackでjson形式のログを出力し、fluentdのin_tailプラグインでそのログを読み取ります。

**Maven**

```xml:pom.xml
<dependency>
  <groupId>ch.qos.logback</groupId>
  <artifactId>logback-classic</artifactId>
</dependency>
<dependency>
  <groupId>ch.qos.logback.contrib</groupId>
  <artifactId>logback-json-classic</artifactId>
  <version>0.1.2</version>
</dependency>
<dependency>
  <groupId>ch.qos.logback.contrib</groupId>
  <artifactId>logback-jackson</artifactId>
  <version>0.1.2</version>
</dependency>
<dependency>
  <groupId>com.fasterxml.jackson.core</groupId>
  <artifactId>jackson-core</artifactId>
</dependency>
<dependency>
  <groupId>com.fasterxml.jackson.core</groupId>
  <artifactId>jackson-databind</artifactId>
</dependency>
```

**logback.xml**

```xml:logback.xml
<appender name="FILE_FLUENTD" class="ch.qos.logback.core.rolling.RollingFileAppender">
  <file>d:/logs/fluentd.log</file>
  <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
    <fileNamePattern>d:/logs/fluentd.log.%d{yyyy-MM-dd}.zip</fileNamePattern>
    <maxHistory>7</maxHistory>
  </rollingPolicy>
  <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
    <layout class="ch.qos.logback.contrib.json.classic.JsonLayout">
      <jsonFormatter class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter">
        <prettyPrint>false</prettyPrint>
      </jsonFormatter>
      <timestampFormat>yyyy-MM-dd'T'HH:mm:ss.SSSXXX</timestampFormat>
      <timestampFormatTimezoneId>Asia/Tokyo</timestampFormatTimezoneId>
      <includeContextName>false</includeContextName>
      <appendLineSeparator>true</appendLineSeparator>
    </layout>
    <charset>UTF-8</charset>
  </encoder>
</appender>

<logger name="com.example" level="DEBUG">
  <appender-ref ref="FILE_FLUENTD" />
</logger>
```

ログ出力は通常の方法です。

```java:Logger
final static Logger logger = LoggerFactory.getLogger(FluentController.class);
```

```java:例
logger.debug("FluentController:[index] Passing through...");
```

logbackが出力するログ

```json:logback_log
{"timestamp":"2015-10-20T23:18:07.573+09:00","level":"DEBUG","thread":"http-nio-9000-exec-8","logger":"com.example.sbtip.web.FluentController","message":"FluentController:[index] Passing through..."}
```

fluentdの設定

```text:conf
<source>
  @type tail
  format json
  tag test
  path d:/logs/fluentd.log
  pos_file d:/logs/fluentd.log.pos
  rotate_wait 5
  read_from_head true
  refresh_interval 60
</source>

<match test>
  @type stdout
</match>
```

上記のログを入力としたfluentdの出力結果

```text:fluentd_output
2015-10-20 23:18:07 +0900 test: {"timestamp":"2015-10-20T23:18:07.573+09:00","level":"DEBUG","thread":"http-nio-9000-exec-8","logger":"com.example.sbtip.web.FluentController","message":"FluentController:[index] Passing through..."}
```



## Fluentdのインストール

Fluentdのインストールと実行にはRubyとGitが必要なため、まだ環境にインストールされていない場合は事前にインストールを行います。

### Ruby

ダウンロードページよりインストーラーをダウンロードしインストールします。

ダウンロードファイル: `rubyinstaller-2.2.3-x64.exe`

**version**

```bat:version
> ruby --version
ruby 2.2.3p173 (2015-08-18 revision 51636) [x64-mingw32]
```

### DevKit

ダウンロードページよりインストーラーをダウンロードしインストールします。
インストール先は`c:\devkit\`としました。

ダウンロードファイル: `DevKit-mingw64-64-4.7.2-20130224-1432-sfx.exe`

DevKitのインストールディレクトリへ移動します。

```bat
> cd c:\devkit
```

DevKitの初期化を行います。

```bat:init
> ruby dk.rb init
[INFO] found RubyInstaller v2.2.3 at C:/Ruby22-x64

Initialization complete! Please review and modify the auto-generated
'config.yml' file to ensure it contains the root directories to all
of the installed Rubies you want enhanced by the DevKit.
```

DevKitのインストールを行います。

```bat:install
> ruby dk.rb install
[INFO] Updating convenience notice gem override for 'C:/Ruby22-x64'
[INFO] Installing 'C:/Ruby22-x64/lib/ruby/site_ruby/devkit.rb'
```

### Git for Windows

ダウンロードページよりインストーラーをダウンロードしてインストールします。

ダウンロードファイル: `Git-2.6.2-64-bit.exe`

### Fluentd

Git for Windowsを立ち上げリポジトリをcloneします。

```bash:clone
$ git clone https://github.com/fluent/fluentd.git
Cloning into 'fluentd'...
remote: Counting objects: 12896, done.
remote: Total 12896 (delta 0), reused 0 (delta 0), pack-reused 12896
Receiving objects: 100% (12896/12896), 12.95 MiB | 424.00 KiB/s, done.
Resolving deltas: 100% (7282/7282), done.
Checking connectivity... done.
```

cloneしたローカルリポジトリへ移動します。

```bat
> cd fluentd
```

bundlerをインストールします。

```bat:bundler
> gem install bundler
Fetching: bundler-1.10.6.gem (100%)
Successfully installed bundler-1.10.6
Parsing documentation for bundler-1.10.6
Installing ri documentation for bundler-1.10.6
Done installing documentation for bundler after 7 seconds
1 gem installed
```

gemをアップデートします。

```bat:update
> gem update --system
Updating rubygems-update
Fetching: rubygems-update-2.4.8.gem (100%)
Successfully installed rubygems-update-2.4.8
Parsing documentation for rubygems-update-2.4.8
Installing ri documentation for rubygems-update-2.4.8
Installing darkfish documentation for rubygems-update-2.4.8
Done installing documentation for rubygems-update after 1 seconds
Parsing documentation for rubygems-update-2.4.8
Done installing documentation for rubygems-update after 0 seconds
Installing RubyGems 2.4.8
RubyGems 2.4.8 installed
Parsing documentation for rubygems-2.4.8
Installing ri documentation for rubygems-2.4.8

...省略...

RubyGems system software updated
```

依存パッケージをインストールします。

```bat:bundle
> bundle
Fetching gem metadata from https://rubygems.org/.........
Fetching version metadata from https://rubygems.org/..
Resolving dependencies...
Using rake 10.4.2
Installing cool.io 1.4.1
Installing ffi 1.9.10
Installing flexmock 1.3.3
Installing http_parser.rb 0.6.0 with native extensions
Installing json 1.8.3 with native extensions
Installing msgpack 0.5.12 with native extensions
Installing sigdump 0.2.3
Installing string-scrub 0.0.5 with native extensions
Installing thread_safe 0.3.5
Installing tzinfo 1.2.2
Installing tzinfo-data 1.2015.7
Installing win32-ipc 0.6.6
Installing win32-event 0.6.1
Installing win32-service 0.8.7
Installing win32-api 1.5.3
Installing windows-api 0.4.4
Installing windows-pr 1.2.4
Installing yajl-ruby 1.2.1 with native extensions
Using fluentd 0.12.16 from source at .
Using multi_json 1.11.2
Installing parallel 1.6.1
Installing parallel_tests 1.9.0
Installing power_assert 0.2.4
Installing rr 1.1.2
Installing simplecov-html 0.5.3
Installing simplecov 0.6.4
Installing test-unit 3.1.5
Installing test-unit-rr 1.0.3
Installing timecop 0.8.0
Installing bundler 1.10.6
Bundle complete! 9 Gemfile dependencies, 31 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.
```

fluentdをビルドします。

```bat:build
> bundle exec rake build
fluentd 0.12.16 built to pkg/fluentd-0.12.16.gem.
```

fluentdをインストールします。

```bat:install
> gem install pkg\fluentd-0.12.16.gem –no-ri –no-rdoc
Successfully installed fluentd-0.12.16
Parsing documentation for fluentd-0.12.16
Installing ri documentation for fluentd-0.12.16
Done installing documentation for fluentd after 6 seconds
1 gem installed
```

バージョンの確認

```bat:version
> ruby bin\fluentd --version
fluentd 0.12.16
```

### 動作確認

exampleで用意されているconfを使用して起動します。

```bat
> ruby bin\fluentd -c .\example\in_forward.conf -o fluent.log
```

ログファイルを参照して起動したことを確認します。

```text:fluent.log
2015-10-20 16:17:54 +0900 [info]: reading config file path=".\\example\\in_forward.conf"
2015-10-20 16:17:54 +0900 [info]: starting fluentd-0.12.16
2015-10-20 16:17:54 +0900 [info]: spawn command to main (windows) : C:/Ruby22-x64/bin/ruby.exe 'C:/Ruby22-x64/bin/fluentd' -c .\example\in_forward.conf -o fluent.log --no-supervisor
2015-10-20 16:17:55 +0900 [info]: reading config file path=".\\example\\in_forward.conf"
2015-10-20 16:17:55 +0900 [info]: starting fluentd-0.12.16 without supervision
2015-10-20 16:17:55 +0900 [info]: gem 'fluentd' version '0.12.16'
2015-10-20 16:17:55 +0900 [info]: adding match pattern="test" type="stdout"
2015-10-20 16:17:55 +0900 [info]: adding source type="forward"
2015-10-20 16:17:55 +0900 [info]: using configuration file: <ROOT>
  <source>
    type forward
  </source>
  <match test>
    type stdout
  </match>
</ROOT>
2015-10-20 16:17:55 +0900 [info]: listening fluent socket on 0.0.0.0:24224
```

fluent-catを使用してfluentdへログを送信します。

```bat
> echo {"id":100, "message": "fluentd - installing on Windows", "ts":"2015-10-20 00:00:00.000"} | ruby bin\fluent-cat test
```

先ほどのログファイルに出力されていることを確認します。

```text
2015-10-20 16:22:49 +0900 test: {"id":100,"message":"fluentd - installing on Windows","ts":"2015-10-20 00:00:00.000"}
```


## Fluentd v0.14.0.pre1のインストール (2015/11/04追記)

`v0.14.0.pre.1`をチェックアウトします。

```bash
$ git checkout -b v0.14.0.pre.1 refs/tags/v0.14.0.pre.1
Switched to a new branch 'v0.14.0.pre.1'
```

依存パッケージをインストールします。

```bat:bundle
> bundle
Fetching gem metadata from https://rubygems.org/.........
Fetching version metadata from https://rubygems.org/..
Resolving dependencies...
Using rake 10.4.2
Using cool.io 1.4.1
Using ffi 1.9.10
Using flexmock 1.3.3
Using http_parser.rb 0.6.0
Using json 1.8.3
Installing msgpack 0.7.0
Using sigdump 0.2.3
Installing strptime 0.1.4
Using thread_safe 0.3.5
Using tzinfo 1.2.2
Using tzinfo-data 1.2015.7
Using win32-ipc 0.6.6
Installing win32-event 0.6.3
Using win32-service 0.8.7
Using win32-api 1.5.3
Using windows-api 0.4.4
Using windows-pr 1.2.4
Using yajl-ruby 1.2.1
Using fluentd 0.14.0.pre.1 from source at .
Using multi_json 1.11.2
Using parallel 1.6.1
Using parallel_tests 1.9.0
Installing power_assert 0.2.5
Using rr 1.1.2
Using simplecov-html 0.5.3
Using simplecov 0.6.4
Using test-unit 3.1.5
Using test-unit-rr 1.0.3
Using timecop 0.8.0
Using bundler 1.10.6
Bundle complete! 9 Gemfile dependencies, 31 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.
```

fluentdをビルドします。

```bat
> bundle exec rake build
fluentd 0.14.0.pre.1 built to pkg/fluentd-0.14.0.pre.1.gem.
```

fluentdをインストールします。

```bat:install
> gem install pkg\fluentd-0.14.0.pre.1.gem -no-ri -no-rdoc
Successfully installed fluentd-0.14.0.pre.1
Parsing documentation for fluentd-0.14.0.pre.1
Installing ri documentation for fluentd-0.14.0.pre.1
Done installing documentation for fluentd after 6 seconds
1 gem installed
```

バージョンの確認

```bat:version
> ruby bin\fluentd --version
fluentd 0.14.0.pre.1
```



## メモ

### fluentdの設定について

#### sourceディレクティブ

**[forward Input Plugin](http://docs.fluentd.org/articles/in_forward)**

> The `in_forward` Input plugin listens to a TCP socket to receive the event stream.
 
```xml
<source>
  type forward
</source>
```

* type : `forward`, <strong>(required)</strong>
* port : The port to listen to. Default Value = <strong>24224</strong>
* bind : The bind address to listen to. Default Value = <strong>0.0.0.0</strong> (all addresses)


**[tail Input Plugin] (http://docs.fluentd.org/articles/in_tail)**

> The `in_tail` Input plugin allows Fluentd to read events from the tail of text files.

```xml
<source>
  type tail
  tag test
  path d:/logs/fluentd.log
  format json
  pos_file d:/logs/fluentd.log.pos
</source>
```

* type : `tail`, <strong>(required)</strong>
* tag : The tag of the event. <strong>(required)</strong>
* path : <strong>(required)</strong>
* format : <strong>(required)</strong>
* pos_file : (highly recommended)

***format***

* regexp
* apache2
* apache_error
* nginx
* syslog
* tsv
* csv
* ltsv (Labeled Tab-Separated Value)
* json
* none
* multiline


**[http Input Plugin] (http://docs.fluentd.org/articles/in_http)**

> The `in_http` Input plugin enables Fluentd to retrieve records from HTTP POST.

```xml
<source>
  type http
</source>
```

* type : `http`, <strong>(required)</strong>
* port : The port to listen to. Default Value = <strong>9880</strong>
* bind : The bind address to listen to. Default Value = <strong>0.0.0.0</strong>(all addresses)
* format : The format of the HTTP body. The default is <strong>default</strong>.

***format***

* default
* regexp
* json
* ltsv
* tsv
* csv
* none 

***例***

```bat
> curl -X POST -d "json={\"action\":\"login\",\"user\":2}" "http://localhost:9880/test"
```


#### matchディレクティブ

**[forward Output Plugin] (http://docs.fluentd.org/articles/out_forward)**

> The `out_forward` Buffered Output plugin forwards events to other fluentd nodes.

```xml
<match pattern>
  type forward

  <server>
    name other-server
    host 192.168.10.2
    port 24224
  </server>

  <secondary>
    type file
    path d:/logs/forward-failed.log
  </secondary>
</match>
```

* type : `forward`, <strong>(required)</strong>
* \<server\> : at least one is required.
 * name : The name of the server. This parameter is used in error messages.
 * host : The IP address or host name of the server. <strong>(required)</strong>
 * port : The port number of the host. The default is <strong>24224</strong>.
 * weight : The load balancing weight. The default weight is <strong>60</strong>.
* \<secondary\> : The backup destination that is used when all servers are unavailable. <strong>(optional)</strong>


**[stdout Output Plugin] (http://docs.fluentd.org/articles/out_stdout)**

> The `stdout` output plugin prints events to stdout (or logs if launched with daemon mode).

```xml
<match pattern>
  type stdout
</match>
```

* type : `stdout`, <strong>(required)</strong>


**[file Output Plugin] (http://docs.fluentd.org/articles/out_file)**

> The `out_file` TimeSliced Output plugin writes events to files.

```xml
<match pattern>
  type file
  path d:/logs/fluent-output.log
</match>
```

* type : `file`, <strong>(required)</strong>
* path : <strong>(required)</strong>
* format : The format of the file content. The default is <strong>out_file.</strong>

***format***

* out_file
* json
* hash
* ltsv
* single_value
* csv
* stdout

### fluentdのUsage

```text:Usage
> ruby bin\fluentd -h
Usage: fluentd [options]
    -s, --setup [DIR=/etc/fluent]    install sample configuration file to the directory
    -c, --config PATH                config file path (default: /etc/fluent/fluent.conf)
        --dry-run                    Check fluentd setup is correct or not
        --show-plugin-config=PLUGIN  Show PLUGIN configuration and exit(ex: input:dummy)
    -p, --plugin DIR                 add plugin directory
    -I PATH                          add library path
    -r NAME                          load library
    -d, --daemon PIDFILE             daemonize fluent process
        --no-supervisor              run without fluent supervisor
        --user USER                  change user
        --group GROUP                change group
    -o, --log PATH                   log file path
    -i CONFIG_STRING,                inline config which is appended to the config file on-fly
        --inline-config
        --emit-error-log-interval SECONDS
                                     suppress interval seconds of emit error logs
        --suppress-repeated-stacktrace [VALUE]
                                     suppress repeated stacktrace
        --without-source             invoke a fluentd without input plugins
        --use-v1-config              Use v1 configuration format (default)
        --use-v0-config              Use v0 configuration format
    -v, --verbose                    increase verbose level (-v: debug, -vv: trace)
    -q, --quiet                      decrease verbose level (-q: warn, -qq: error)
        --suppress-config-dump       suppress config dumping when fluentd starts
    -g, --gemfile GEMFILE            Gemfile path
    -G, --gem-path GEM_INSTALL_PATH  Gemfile install path (default: $(dirname $gemfile)/vendor/bundle)
    -x, --signame INTSIGNAME         an object name which is used for Windows Service signal (Windows only)
        --reg-winsvc MODE            install/uninstall as Windows Service. (i: install, u: uninstall) (Windows only)
        --reg-winsvc-fluentdopt OPTION
                                     specify fluentd option paramters for Windows Service. (Windows only)
```