LibertyからLogstash CollectorでELKへのログ出力を試す

WebSphere LibertyからLogstash Collectorフィーチャーを使って、ELK Stackへログを送る設定を試してみます。

以下のリンク先の手順に沿います。

[Using the Logstash collector version 1.1](https://www.ibm.com/support/knowledgecenter/ja/was_beta_liberty/com.ibm.websphere.wlp.nd.multiplatform.doc/ae/twlp_analytics_logstashv1_1.html)

# 鍵の準備

## 鍵と証明書の作成

LogstashとLogstash Connectorの接続をSSLとするため、Logstashが使用する鍵と証明書を作成します。

```
openssl req -x509 -newkey rsa:2048 -keyout logstash.key -out logstash.crt -days 365 -nodes
```

```shell-session
$ openssl req -x509 -newkey rsa:2048 -keyout logstash.key -out logstash.crt -days 365 -nodes
Generating a 2048 bit RSA private key
.........+++
...........+++
writing new private key to 'logstash.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) []:
State or Province Name (full name) []:
Locality Name (eg, city) []:
Organization Name (eg, company) []:
Organizational Unit Name (eg, section) []:
Common Name (eg, fully qualified host name) []:logstash
Email Address []:
$
```

## Liberty用のトラストストアの作成

作成したLogstash用の証明書から、この証明書を追加済みのトラストストアを作成しておきます。このトラストストアを後でLibertyで使います。

```
eytool -import -noprompt -alias logstash -file logstash.crt -keystore trust.jks -storepass Liberty
```

```shell-session
$ keytool -import -noprompt -alias logstash -file logstash.crt -keystore trust.jks -storepass Liberty
証明書がキーストアに追加されました
$ 
```

# Libertyの準備

## Liberty Betaのダウンロード

Logstash Collectorはバージョン1.0と1.1があり、送信するフィールド名が異なります。今回は1.1を使用してみますが、バージョン1.1はBetaのため、Liberty Betaを使用します。

フィールドの違いについてはKnowledge Centerの以下のトピックに記載があります。

[Logstash collector](https://www.ibm.com/support/knowledgecenter/en/was_beta_liberty/com.ibm.websphere.wlp.nd.multiplatform.doc/ae/cwlp_analytics_collectors.html)

以下サイトよりLiberty Betaのzipファイルをダウンロードし、適当なフォルダに展開します。

https://developer.ibm.com/wasdev/downloads/liberty-profile-beta/

## サーバーの作成

zipを展開したら、サーバーを作成します。サーバー名を指定しない場合はdefaultServerが作成されます。

```
sotoiwa@soto-no-air:~/opt/ibm/wlp/bin
$ ./server create
objc[5139]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/bin/java (0x10ba6d4c0) and /Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/jre/lib/libinstrument.dylib (0x10bb234e0). One of the two will be used. Which one is undefined.
サーバー defaultServer が作成されました。
sotoiwa@soto-no-air:~/opt/ibm/wlp/bin
$
```

## 証明書の作成

Libertyで使用する証明書を作成します。

```
sotoiwa@soto-no-air:~/opt/ibm/wlp/bin
$ ./securityUtility createSSLCertificate --server=defaultServer --password="Liberty" --subject=CN=myHostname,OU=defaultServer,O=ibm,C=us
鍵ストア /Users/sotoiwa/opt/ibm/wlp/usr/servers/defaultServer/resources/security/key.jks を作成中です

サーバー defaultServer の SSL 証明書を作成しました。 証明書は SubjectDN CN=myHostname,OU=defaultServer,O=ibm,C=us で作成されました。

SSL を使用可能にするには、server.xml に次の行を追加してください。

    <featureManager>
        <feature>ssl-1.0</feature>
    </featureManager>
    <keyStore id="defaultKeyStore" password="{xor}EzY9Oi0rJg==" />


sotoiwa@soto-no-air:~/opt/ibm/wlp/bin
$
```

## トラストストアの配置

今作成した鍵ストアとは別に、先ほど作成したLogstashの証明書が取り込み済みのトラストストアをLibertyサーバーに配置します。

```
sotoiwa@soto-no-air:~/opt/ibm/wlp/usr/servers/defaultServer/resources/security
$ ls -l
total 16
-rw-r--r--  1 sotoiwa  staff  2172  6 15 13:33 key.jks
-rw-r--r--  1 sotoiwa  staff   743  6 15 13:38 trust.jks
sotoiwa@soto-no-air:~/opt/ibm/wlp/usr/servers/defaultServer/resources/security
$
```

## server.xmlの修正

server.xmlを編集します。ここでの編集のポイントは以下です。

- Logstash Collectorフィーチャーを有効にし、接続先を構成します。
- HTTPアクセスロギングを有効にします。
- SSL構成を作成してキーストアとトラストストアを指定します。
- Logstash Colloctorの設定でタグを付けています。本質的な設定ではありませんが、タグがないと後でダッシュボードがエラーになります。

```xml
<?xml version="1.0" encoding="UTF-8"?>
<server description="Liberty beta">

    <featureManager>
        <feature>webProfile-8.0</feature>
        <feature>transportSecurity-1.0</feature>
        <feature>logstashCollector-1.1</feature>
    </featureManager>

    <httpEndpoint id="defaultHttpEndpoint"
                  httpPort="9080"
                  httpsPort="9443"
                  host="*"
                  accessLoggingRef="accessLogging" />

    <!-- Automatically expand WAR files and EAR files -->
    <applicationManager autoExpand="true" />

    <httpAccessLogging id="accessLogging" />

    <logstashCollector sslRef="mySSLConfig" hostName="localhost" port="5043" >
        <source>accesslog</source>
        <source>message</source>
        <source>ffdc</source>
        <source>trace</source>
        <source>garbageCollection</source>
        <tag>serverRackA5</tag>
        <tag>billingAppTeam</tag>
    </logstashCollector>

    <keyStore id="defaultKeyStore" password="Liberty" />
    <keyStore id="defaultTrustStore" location="trust.jks" password="Liberty" />
    <ssl id="mySSLConfig" trustStoreRef="defaultTrustStore" keyStoreRef="defaultKeyStore" />

</server>
```

## GCログ出力

IBM JDKの場合は、Logstash CollectorからGarbage Collectionの情報を送れます。残念ながらMacだとIBM JDKがないのでGarbage Collectionの情報は送れません。
`jvm.options`ファイルに以下を記載します。

```
-verbose:gc
```

## アプリケーションのデプロイ

何でもいいのですが、アプリケーションをデプロイします。ここでは以下からsample.warをダウンロードして、dropinsフォルダーに配置します。

https://tomcat.apache.org/tomcat-7.0-doc/appdev/sample/

## Libertyを起動

Libertyサーバーを起動します。コンソール出力を確認するため、ここでは`server start`の代わりに`server run`コマンドを使ってフォアグラウンドで起動します。

```
$ ./server run
objc[7950]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/bin/java (0x1060dc4c0) and /Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/jre/lib/libinstrument.dylib (0x1061924e0). One of the two will be used. Which one is undefined.
Java HotSpot(TM) 64-Bit Server VM バージョン 1.8.0_151-b12 (ja_JP) で、defaultServer (WebSphere Application Server 2018.6.0.0/wlp-1.0.21.20180531-2100) を起動しています
[AUDIT   ] CWWKE0001I: サーバー defaultServer が起動されました。
[AUDIT   ] CWWKE0104I: この製品はベータ版であり、実動には使用できません。 全ライセンス条項は以下で表示可能です: https://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/wasdev/license/early_access/ilar/2018.6.0.0/lafiles/ja.html
[GC (Allocation Failure)  33280K->10005K(125952K), 0.0327679 secs]
[GC (Allocation Failure)  43285K->15742K(159232K), 0.0260316 secs]
[GC (Allocation Failure)  82302K->24419K(159232K), 0.0419508 secs]
[GC (Metadata GC Threshold)  35740K->26052K(225792K), 0.0220855 secs]
[Full GC (Metadata GC Threshold)  26052K->19908K(213504K), 0.0908929 secs]
[AUDIT   ] CWWKZ0058I: アプリケーションの dropins をモニター中です。
[GC (Allocation Failure)  152993K->31758K(213504K), 0.0315995 secs]
[GC (Metadata GC Threshold)  87418K->36722K(307200K), 0.0435490 secs]
[Full GC (Metadata GC Threshold)  36722K->26108K(336896K), 0.1808532 secs]
[警告      ] TRAS4352W: Health Center エージェントが見つからなかったため、JVM モニタリング情報および診断情報は提供されません。
[AUDIT   ] CWWKF0012I: サーバーは次のフィーチャーをインストールしました。[beanValidation-2.0, servlet-4.0, ssl-1.0, jndi-1.0, cdi-2.0, transportSecurity-1.0, jdbc-4.2, appSecurity-3.0, appSecurity-2.0, jaxrs-2.1, webProfile-8.0, jpa-2.2, jsp-2.3, jsonb-1.0, ejbLite-3.2, managedBeans-1.0, jsf-2.3, logstashCollector-1.1, jsonp-1.1, jaxrsClient-2.1, el-3.0, jpaContainer-2.2, jaspic-1.1, distributedMap-1.0, websocket-1.1]。
[AUDIT   ] CWWKF0011I: サーバー defaultServer は、Smarter Planet に対応する準備ができました。
[警告      ] TRAS0216W: logstash コレクターが、指定されたホスト localhost およびポート番号 5043 上の logstash サーバーに接続できませんでした。
```

起動することが確認できたら、Ctrl+Cで停止します。

# ELKの起動

簡単に試すため、今回はELKをローカルで起動します。バージョンはKnowledge Centerにテスト済みと記載がある5.3.3を使用します。

## ダウンロード

以下からELKをダウンロードして展開します。
https://www.elastic.co/jp/downloads/logstash
https://www.elastic.co/jp/downloads/elasticsearch
https://www.elastic.co/jp/downloads/kibana

## Logstashへの接続確認

はじめに、Logstashだけを起動してLogstash Colloectorからデータが送られてきていることを確認します。

この手順の最初に作成した鍵と証明書を、Logstashを展開したディレクトリーのconfigディレクトリーに配置します。

以下のデバッグ用のLogstash構成ファイルを作成します。入力はSSLを使用して5043ポートで受け付けるようにし、SSLで使用する鍵と証明書のパスを指定しています。出力は標準出力にそのまま出力する設定です。

```ruby:config/logstash_debug.conf
input {
    beats {
        port => "5043"
        ssl => true
        ssl_certificate => "config/logstash.crt"
        ssl_key => "config/logstash.key"
    }
}
filter {
}
output {
    stdout {}
}
```

この設定ファイルを指定してLogstashを起動します。

```
sotoiwa@soto-no-air:~/opt/logstash/logstash-5.3.3
$ bin/logstash -f config/logstash_debug.conf
Sending Logstash's logs to /Users/sotoiwa/opt/logstash/logstash-5.3.3/logs which is now configured via log4j2.properties
[2018-06-19T18:18:14,300][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
log4j:WARN No appenders could be found for logger (io.netty.util.internal.logging.InternalLoggerFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[2018-06-19T18:18:14,973][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5043"}
[2018-06-19T18:18:15,062][INFO ][logstash.pipeline        ] Pipeline main started
[2018-06-19T18:18:15,150][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
```

Logstashが起動したあと、別のコンソールを開いてLibertyを起動するとLogstash Collectorからログが送られてくることが確認できます。

```
2018-06-19T09:18:35.118Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"CWWKE0001I: サーバー defaultServer が起動されました。","ibm_threadId":"00000001","ibm_datetime":"2018-06-19T18:18:20.294+0900","ibm_messageId":"CWWKE0001I","module":"com.ibm.ws.kernel.launch.internal.FrameworkManager","loglevel":"AUDIT","ibm_sequence":"1529399900294_0000000000001","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.123Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"CWWKE0104I: この製品はベータ版であり、実動には使用できません。 全ライセンス条項は以下で表示可能です: https:\/\/public.dhe.ibm.com\/ibmdl\/export\/pub\/software\/websphere\/wasdev\/license\/early_access\/ilar\/2018.6.0.0\/lafiles\/ja.html","ibm_threadId":"00000001","ibm_datetime":"2018-06-19T18:18:20.325+0900","ibm_messageId":"CWWKE0104I","module":"com.ibm.ws.kernel.launch.internal.FrameworkManager","loglevel":"AUDIT","ibm_sequence":"1529399900325_0000000000002","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.124Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"CWWKE0002I: カーネルは 2.527 秒 秒後に始動されました。","ibm_threadId":"00000001","ibm_datetime":"2018-06-19T18:18:22.481+0900","ibm_messageId":"CWWKE0002I","module":"com.ibm.ws.kernel.launch.internal.FrameworkManager","loglevel":"INFO","ibm_sequence":"1529399902481_0000000000003","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.125Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"CWWKF0007I: フィーチャー更新が開始しました。","ibm_threadId":"00000018","ibm_datetime":"2018-06-19T18:18:22.549+0900","ibm_messageId":"CWWKF0007I","module":"com.ibm.ws.kernel.feature.internal.FeatureManager","loglevel":"INFO","ibm_sequence":"1529399902549_0000000000004","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.126Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"CWWKS0007I: セキュリティー・サービスを開始しています...","ibm_threadId":"00000011","ibm_datetime":"2018-06-19T18:18:24.028+0900","ibm_messageId":"CWWKS0007I","module":"com.ibm.ws.security.ready.internal.SecurityReadyServiceImpl","loglevel":"INFO","ibm_sequence":"1529399904028_0000000000005","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.127Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"CWWKO0219I: TCP チャネル defaultHttpEndpoint が開始され、現在、ホスト *  (IPv6)、ポート 9080 の要求を listen しています。","ibm_threadId":"00000020","ibm_datetime":"2018-06-19T18:18:25.252+0900","ibm_messageId":"CWWKO0219I","module":"com.ibm.ws.tcpchannel.internal.TCPChannel","loglevel":"INFO","ibm_sequence":"1529399905252_0000000000006","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.128Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"CWWKO0219I: TCP チャネル defaultHttpEndpoint-ssl が開始され、現在、ホスト *  (IPv6)、ポート 9443 の要求を listen しています。","ibm_threadId":"00000020","ibm_datetime":"2018-06-19T18:18:25.283+0900","ibm_messageId":"CWWKO0219I","module":"com.ibm.ws.tcpchannel.internal.TCPChannel","loglevel":"INFO","ibm_sequence":"1529399905283_0000000000007","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.129Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"CWWKZ0058I: アプリケーションの dropins をモニター中です。","ibm_threadId":"00000011","ibm_datetime":"2018-06-19T18:18:25.293+0900","ibm_messageId":"CWWKZ0058I","module":"com.ibm.ws.app.manager.internal.monitor.DropinMonitor","loglevel":"AUDIT","ibm_sequence":"1529399905293_0000000000008","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.130Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"CWWKZ0064I: Jandex の使用はアプリケーション sample で有効になっています。","ibm_threadId":"00000013","ibm_datetime":"2018-06-19T18:18:25.321+0900","ibm_messageId":"CWWKZ0064I","module":"com.ibm.ws.app.manager.internal.ApplicationConfig","loglevel":"INFO","ibm_sequence":"1529399905321_0000000000009","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.131Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"CWWKS1123I: クラス名が NullCollectiveAuthenticationPlugin である集合認証プラグインがアクティブ化されました。 ","ibm_threadId":"00000011","ibm_datetime":"2018-06-19T18:18:25.939+0900","ibm_messageId":"CWWKS1123I","module":"com.ibm.ws.security.authentication.internal.jaas.JAASServiceImpl","loglevel":"INFO","ibm_sequence":"1529399905939_000000000000A","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.131Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"CWWKS4105I: LTPA 構成は、0.099 秒後に作動可能になります。","ibm_threadId":"0000001c","ibm_datetime":"2018-06-19T18:18:25.956+0900","ibm_messageId":"CWWKS4105I","module":"com.ibm.ws.security.token.ltpa.internal.LTPAKeyCreateTask","loglevel":"INFO","ibm_sequence":"1529399905956_000000000000B","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.132Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"CWWKS1655I: Java セキュリティー・プロパティー authconfigprovider.factory が設定されていないため、デフォルト Java Authentication SPI for Containers (JASPIC) AuthConfigFactory クラス com.ibm.ws.security.jaspi.ProviderRegistry が使用されます。 ","ibm_threadId":"00000011","ibm_datetime":"2018-06-19T18:18:26.340+0900","ibm_messageId":"CWWKS1655I","module":"com.ibm.ws.security.jaspi.AuthConfigFactoryWrapper","loglevel":"INFO","ibm_sequence":"1529399906340_000000000000C","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.133Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"DYNA1001I: WebSphere 動的キャッシュ・インスタンス baseCache が正常に初期化されました。","ibm_threadId":"00000011","ibm_datetime":"2018-06-19T18:18:28.198+0900","ibm_messageId":"DYNA1001I","module":"com.ibm.ws.cache.ServerCache","loglevel":"INFO","ibm_sequence":"1529399908198_000000000000D","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.134Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"DYNA1071I: キャッシュ・プロバイダー default が使用されています。","ibm_threadId":"00000011","ibm_datetime":"2018-06-19T18:18:28.208+0900","ibm_messageId":"DYNA1071I","module":"com.ibm.ws.cache.ServerCache","loglevel":"INFO","ibm_sequence":"1529399908208_000000000000E","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.135Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"DYNA1056I: 動的キャッシュ (オブジェクト・キャッシュ) が、正常に初期化されました。","ibm_threadId":"00000011","ibm_datetime":"2018-06-19T18:18:28.210+0900","ibm_messageId":"DYNA1056I","module":"com.ibm.ws.cache.CacheServiceImpl","loglevel":"INFO","ibm_sequence":"1529399908210_000000000000F","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.136Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"CWWKZ0018I: アプリケーション sample を開始中です。","ibm_threadId":"0000001c","ibm_datetime":"2018-06-19T18:18:28.663+0900","ibm_messageId":"CWWKZ0018I","module":"com.ibm.ws.app.manager.AppMessageHelper","loglevel":"INFO","ibm_sequence":"1529399908663_0000000000010","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.137Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"SESN8501I: セッション・マネージャーが、永続ストレージ・ロケーションを検出しませんでした。HttpSession オブジェクトはローカル・アプリケーション・サーバーのメモリーに保管されます。","ibm_threadId":"0000001c","ibm_datetime":"2018-06-19T18:18:29.159+0900","ibm_messageId":"SESN8501I","module":"com.ibm.ws.session.WASSessionCore","loglevel":"INFO","ibm_methodName":"initialize","ibm_className":"SessionMgrComponentImpl","ibm_sequence":"1529399909159_0000000000011","ext_thread":"Default Executor-thread-4","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.138Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"WELD-000900: 3.0.3 (Final)","ibm_threadId":"0000001c","ibm_datetime":"2018-06-19T18:18:29.249+0900","module":"org.jboss.weld.Version","loglevel":"INFO","ibm_methodName":"<clinit>","ibm_className":"org.jboss.weld.bootstrap.WeldStartup","ibm_sequence":"1529399909249_0000000000012","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.139Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"SRVE0169I: Web モジュールをロード中: Hello, World Application。","ibm_threadId":"0000001c","ibm_datetime":"2018-06-19T18:18:29.633+0900","ibm_messageId":"SRVE0169I","module":"com.ibm.ws.webcontainer.osgi.webapp.WebGroup","loglevel":"INFO","ibm_sequence":"1529399909633_0000000000013","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.139Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"SRVE0250I: Web モジュール Hello, World Application は default_host にバインドされています。","ibm_threadId":"0000001c","ibm_datetime":"2018-06-19T18:18:29.640+0900","ibm_messageId":"SRVE0250I","module":"com.ibm.ws.webcontainer","loglevel":"INFO","ibm_methodName":"addWebApplication","ibm_className":"com.ibm.ws.webcontainer.osgi.DynamicVirtualHost","ibm_sequence":"1529399909640_0000000000014","ext_thread":"Default Executor-thread-4","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.140Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"CWWKT0016I: Web アプリケーションが使用可能です (default_host): http:\/\/soto-no-air.dhcp.hakozaki.ibm.com:9080\/sample\/","ibm_threadId":"0000001c","ibm_datetime":"2018-06-19T18:18:29.642+0900","ibm_messageId":"CWWKT0016I","module":"com.ibm.ws.http.internal.VirtualHostImpl","loglevel":"AUDIT","ibm_sequence":"1529399909642_0000000000015","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.141Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"CWWKZ0001I: アプリケーション sample が 0.999 秒で開始しました。","ibm_threadId":"0000001c","ibm_datetime":"2018-06-19T18:18:29.662+0900","ibm_messageId":"CWWKZ0001I","module":"com.ibm.ws.app.manager.AppMessageHelper","loglevel":"AUDIT","ibm_sequence":"1529399909662_0000000000016","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.142Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"SESN0176I: アプリケーション・キー default_host\/sample に新しいセッション・コンテキストが作成されます。","ibm_threadId":"00000020","ibm_datetime":"2018-06-19T18:18:29.700+0900","ibm_messageId":"SESN0176I","module":"com.ibm.ws.session.WASSessionCore","loglevel":"INFO","ibm_methodName":"getSessionContext","ibm_className":"SessionContextRegistryImpl","ibm_sequence":"1529399909700_0000000000017","ext_thread":"Default Executor-thread-7","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.143Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"SESN0172I: セッション・マネージャーは、セッション ID の生成に Java のデフォルト SecureRandom 実装を使用しています。","ibm_threadId":"00000020","ibm_datetime":"2018-06-19T18:18:29.726+0900","ibm_messageId":"SESN0172I","module":"com.ibm.ws.util","loglevel":"INFO","ibm_methodName":"IDGeneratorImpl","ibm_className":"IDGeneratorImpl","ibm_sequence":"1529399909726_0000000000018","ext_thread":"Default Executor-thread-7","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.144Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"TRAS4352W: Health Center エージェントが見つからなかったため、JVM モニタリング情報および診断情報は提供されません。","ibm_threadId":"00000011","ibm_datetime":"2018-06-19T18:18:29.782+0900","ibm_messageId":"TRAS4352W","module":"com.ibm.ws.health.center.agent.HCAgentConnection","loglevel":"WARNING","ibm_sequence":"1529399909782_0000000000019","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.144Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"CWWKF0012I: サーバーは次のフィーチャーをインストールしました。[beanValidation-2.0, servlet-4.0, ssl-1.0, jndi-1.0, cdi-2.0, transportSecurity-1.0, jdbc-4.2, appSecurity-3.0, appSecurity-2.0, jaxrs-2.1, webProfile-8.0, jpa-2.2, jsp-2.3, jsonb-1.0, ejbLite-3.2, managedBeans-1.0, jsf-2.3, logstashCollector-1.1, jsonp-1.1, jaxrsClient-2.1, el-3.0, jpaContainer-2.2, jaspic-1.1, distributedMap-1.0, websocket-1.1]。","ibm_threadId":"00000018","ibm_datetime":"2018-06-19T18:18:29.806+0900","ibm_messageId":"CWWKF0012I","module":"com.ibm.ws.kernel.feature.internal.FeatureManager","loglevel":"AUDIT","ibm_sequence":"1529399909806_000000000001A","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.145Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"CWWKF0008I: フィーチャー更新が 7.326 秒で完了しました。","ibm_threadId":"00000018","ibm_datetime":"2018-06-19T18:18:29.807+0900","ibm_messageId":"CWWKF0008I","module":"com.ibm.ws.kernel.feature.internal.FeatureManager","loglevel":"INFO","ibm_sequence":"1529399909807_000000000001B","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.146Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"CWWKF0011I: サーバー defaultServer は、Smarter Planet に対応する準備ができました。","ibm_threadId":"00000018","ibm_datetime":"2018-06-19T18:18:29.808+0900","ibm_messageId":"CWWKF0011I","module":"com.ibm.ws.kernel.feature.internal.FeatureManager","loglevel":"AUDIT","ibm_sequence":"1529399909808_000000000001C","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.147Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"HV000001: Hibernate Validator 6.0.9.Final","ibm_threadId":"00000020","ibm_datetime":"2018-06-19T18:18:29.858+0900","module":"org.hibernate.validator.internal.util.Version","loglevel":"INFO","ibm_methodName":"<clinit>","ibm_className":"org.hibernate.validator.internal.util.Version","ibm_sequence":"1529399909858_000000000001D","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.147Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"SRVE9103I: Web サーバー・プラグイン構成ファイルは、このサーバー用に、\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/servers\/defaultServer\/logs\/state\/plugin-cfg.xml に自動的に生成されました。","ibm_threadId":"0000002a","ibm_datetime":"2018-06-19T18:18:30.054+0900","ibm_messageId":"SRVE9103I","module":"com.ibm.ws.webcontainer.osgi.mbeans.PluginGenerator","loglevel":"INFO","ibm_sequence":"1529399910054_000000000001E","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.148Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"DYNA1056I: 動的キャッシュ (オブジェクト・キャッシュ) が、正常に初期化されました。","ibm_threadId":"00000020","ibm_datetime":"2018-06-19T18:18:30.320+0900","ibm_messageId":"DYNA1056I","module":"com.ibm.ws.cache.CacheServiceImpl","loglevel":"INFO","ibm_sequence":"1529399910320_000000000001F","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:35.149Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"Using org.apache.myfaces.ee.MyFacesContainerInitializer","ibm_threadId":"00000020","ibm_datetime":"2018-06-19T18:18:30.347+0900","module":"org.apache.myfaces.ee.MyFacesContainerInitializer","loglevel":"INFO","ibm_sequence":"1529399910347_0000000000020","ext_thread":"Default Executor-thread-7","ibm_tags":["serverRackA5","billingAppTeam"]}
2018-06-19T09:18:39.732Z %{host} {"type":"liberty_message","host":"soto-no-air.dhcp.hakozaki.ibm.com","ibm_userDir":"\/Users\/sotoiwa\/opt\/ibm\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"TRAS0218I: logstash コレクターが、指定されたホスト localhost およびポート番号 5043 上の logstash サーバーに接続されました。","ibm_threadId":"00000031","ibm_datetime":"2018-06-19T18:18:35.049+0900","ibm_messageId":"TRAS0218I","module":"com.ibm.ws.logstash.collector.internal.LogstashClient","loglevel":"INFO","ibm_sequence":"1529399915049_0000000000021","ibm_tags":["serverRackA5","billingAppTeam"]}
```

各行は

```
時刻 ホスト名 json
```

となっており、json部分を整形すると、次のような内容です。

```json
{
	"type": "liberty_message",
	"host": "soto-no-air.dhcp.hakozaki.ibm.com",
	"ibm_userDir": "/Users/sotoiwa/opt/ibm/wlp/usr/",
	"ibm_serverName": "defaultServer",
	"message": "CWWKF0011I: サーバー defaultServer は、Smarter Planet に対応する準備ができました。",
	"ibm_threadId": "00000018",
	"ibm_datetime": "2018-06-19T18:18:29.808+0900",
	"ibm_messageId": "CWWKF0011I",
	"module": "com.ibm.ws.kernel.feature.internal.FeatureManager",
	"loglevel": "AUDIT",
	"ibm_sequence": "1529399909808_000000000001C",
	"ibm_tags": [
		"serverRackA5",
		"billingAppTeam"
	]
}
```

接続できたらCtrl-CでLibertyとLogstashをそれぞれ停止します。

## Liberty用のLogstash設定ファイルの準備

https://github.com/WASdev/sample.logstash.collector
から以下のファイルを取得してconfigディレクトリーに配置します。

- [liberty_logstash.conf](https://github.com/WASdev/sample.logstash.collector/blob/master/version-1.1/Kibana5/Beta/20170516/liberty_logstash.conf)
- [liberty_logstash_template_ibm.json](https://github.com/WASdev/sample.logstash.collector/blob/master/version-1.1/Kibana5/Beta/20170516/liberty_logstash_template_ibm.json)

ダウンロードした`liberty_logstash.conf`を修正します。
また、鍵ファイル、証明書、索引テンプレート・ファイルのパスを実際の配置場所に合わせて修正します。

```
input {
    beats {
・・・
        ssl_certificate => "config/logstash.crt" 
        ssl_key => "config/logstash.key" 
    }
```

てElasticsearchにログを送るようにします。Kubernetes環境であれば、`elasticsearch`というホスト名でElasticsearchのサービスにアクセスするケースが多そうですが、今回はローカルでElasticsearchを起動しているので、ホスト名を`localhost`に書き換えます。

```
output {
    elasticsearch {
        hosts => "localhost:9200"
・・・
    }
}
```

[liberty_logstash.conf](https://github.com/WASdev/sample.logstash.collector/blob/master/version-1.1/Kibana5/Beta/20170516/liberty_logstash.conf)には次のようなフィルターも設定されています。

```ruby
filter {
    if [type] =~ "liberty_*" {
        if [type] == "liberty_accesslog" {
            json {
                source => "message"
                remove_field => [ "message" ]
            }
        }
        else if [type] == "liberty_trace" {
            json {
                source => "message"
            }
        }
        else if [type] == "liberty_gc" {
            json {
                source => "message"
                remove_field => [ "message" ]
            }
        }
        else if [type] == "liberty_message" {
            json {
                source => "message"
            }
        }
        else if [type] == "liberty_ffdc" {
            json {
                source => "message"
                remove_field => [ "message" ]
            }
        }
    }
}
```

Logstashの出力設定を`stdout { codec => rubydebug }`や`stdout { codec => json }`に変更することで、LogstashからElasticsearchに送られるはずのjsonを確認することができます。jsonだと整形まではしてくれないので、rubydebugのほうが見やすいです。

```ruby
output {
    #stdout {}
    stdout { codec => json }
}
```

filterがない状態で、出力をjsonとして出力して内容を確認すると以下のようになっていました。`message`フィールドにjson部分が入っていていて、それ以外に`@timestamp`や`@version`や`type`といったフィールドが設定されています。

```json
{
	"@timestamp": "2018-06-19T07:25:41.269Z",
	"@version": "1",
	"message": "{\"type\":\"liberty_message\",\"host\":\"soto-no-air.hogehoge.com\",\"ibm_userDir\":\"\\/Users\\/sotoiwa\\/opt\\/ibm\\/wlp\\/usr\\/\",\"ibm_serverName\":\"defaultServer\",\"message\":\"CWWKF0011I: サーバー defaultServer は、Smarter Planet に対応する準備ができました。\",\"ibm_threadId\":\"00000018\",\"ibm_datetime\":\"2018-06-19T16:25:35.921+0900\",\"ibm_messageId\":\"CWWKF0011I\",\"module\":\"com.ibm.ws.kernel.feature.internal.FeatureManager\",\"loglevel\":\"AUDIT\",\"ibm_sequence\":\"1529393135921_000000000001C\",\"ibm_tags\":[\"serverRackA5\",\"billingAppTeam\"]}",
	"type": "liberty_message",
	"tags": [
		"beats_input_codec_plain_applied"
	]
}
```

[liberty_logstash.conf](https://github.com/WASdev/sample.logstash.collector/blob/master/version-1.1/Kibana5/Beta/20170516/liberty_logstash.conf)では、[jsonフィルタープラグイン](https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html)を使い、`message`フィールドがjsonであることを指定しているので、次のように、jsonの内容が取り出されてフィールドに設定されます。

```json
{
	"module": "com.ibm.ws.kernel.feature.internal.FeatureManager",
	"ibm_sequence": "1529393526665_000000000001C",
	"ibm_tags": [
		"serverRackA5",
		"billingAppTeam"
	],
	"message": "CWWKF0011I: サーバー defaultServer は、Smarter Planet に対応する準備ができました。",
	"type": "liberty_message",
	"tags": [
		"beats_input_codec_plain_applied"
	],
	"ibm_userDir": "/Users/sotoiwa/opt/ibm/wlp/usr/",
	"ibm_serverName": "defaultServer",
	"@timestamp": "2018-06-19T07:32:12.134Z",
	"loglevel": "AUDIT",
	"@version": "1",
	"host": "soto-no-air.hogehoge.com",
	"ibm_threadId": "00000018",
	"ibm_datetime": "2018-06-19T16:32:06.665+0900",
	"ibm_messageId": "CWWKF0011I"
}
```

なお、[liberty_logstash.conf](https://github.com/WASdev/sample.logstash.collector/blob/master/version-1.1/Kibana5/Beta/20170516/liberty_logstash.conf)では、`type`で条件分岐がされています。`type`が`liberty_message`または`liberty_trace`の場合、`message`フィールドのjsonのなかに入れ子で`message`のフィールドがあるので、フィルター後はjsonのなかの`message`フィールドが`message`フィールドに設定されます。

`type`が`liberty_gc`、`libberty_ffdc`、`liberty_accesslog`の場合は、jsonのなかに`message`フィールドがないので、json全体が`message`フィールドに入ったままとなりますが、この`message`フィールドは不要なので削除しています。

## Elasticsearchを起動

Elasticsearchを起動します。

```shell-session
sotoiwa@soto-no-air:~/opt/elasticsearch/elasticsearch-5.3.3
$ bin/elasticsearch
[2018-06-19T14:00:58,167][INFO ][o.e.n.Node               ] [] initializing ...
[2018-06-19T14:00:58,362][INFO ][o.e.e.NodeEnvironment    ] [GcxldH9] using [1] data paths, mounts [[/ (/dev/disk1s1)]], net usable_space [51.4gb], net total_space [233.5gb], spins? [unknown], types [apfs]
[2018-06-19T14:00:58,363][INFO ][o.e.e.NodeEnvironment    ] [GcxldH9] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-06-19T14:00:58,594][INFO ][o.e.n.Node               ] node name [GcxldH9] derived from node ID [GcxldH9xQkuG5qEtogRhgw]; set [node.name] to override
[2018-06-19T14:00:58,595][INFO ][o.e.n.Node               ] version[5.3.3], pid[9909], build[65e26e6/2017-05-22T12:03:12.318Z], OS[Mac OS X/10.13.4/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_151/25.151-b12]
[2018-06-19T14:01:00,929][INFO ][o.e.p.PluginsService     ] [GcxldH9] loaded module [aggs-matrix-stats]
[2018-06-19T14:01:00,931][INFO ][o.e.p.PluginsService     ] [GcxldH9] loaded module [ingest-common]
[2018-06-19T14:01:00,933][INFO ][o.e.p.PluginsService     ] [GcxldH9] loaded module [lang-expression]
[2018-06-19T14:01:00,934][INFO ][o.e.p.PluginsService     ] [GcxldH9] loaded module [lang-groovy]
[2018-06-19T14:01:00,934][INFO ][o.e.p.PluginsService     ] [GcxldH9] loaded module [lang-mustache]
[2018-06-19T14:01:00,934][INFO ][o.e.p.PluginsService     ] [GcxldH9] loaded module [lang-painless]
[2018-06-19T14:01:00,935][INFO ][o.e.p.PluginsService     ] [GcxldH9] loaded module [percolator]
[2018-06-19T14:01:00,935][INFO ][o.e.p.PluginsService     ] [GcxldH9] loaded module [reindex]
[2018-06-19T14:01:00,936][INFO ][o.e.p.PluginsService     ] [GcxldH9] loaded module [transport-netty3]
[2018-06-19T14:01:00,936][INFO ][o.e.p.PluginsService     ] [GcxldH9] loaded module [transport-netty4]
[2018-06-19T14:01:00,938][INFO ][o.e.p.PluginsService     ] [GcxldH9] no plugins loaded
[2018-06-19T14:01:06,566][INFO ][o.e.n.Node               ] initialized
[2018-06-19T14:01:06,567][INFO ][o.e.n.Node               ] [GcxldH9] starting ...
[2018-06-19T14:01:07,115][INFO ][o.e.t.TransportService   ] [GcxldH9] publish_address {127.0.0.1:9300}, bound_addresses {[fe80::1]:9300}, {[::1]:9300}, {127.0.0.1:9300}
[2018-06-19T14:01:10,371][INFO ][o.e.c.s.ClusterService   ] [GcxldH9] new_master {GcxldH9}{GcxldH9xQkuG5qEtogRhgw}{FXdz9K49R2mbOoWmvBWDfA}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-06-19T14:01:10,412][INFO ][o.e.h.n.Netty4HttpServerTransport] [GcxldH9] publish_address {127.0.0.1:9200}, bound_addresses {[fe80::1]:9200}, {[::1]:9200}, {127.0.0.1:9200}
[2018-06-19T14:01:10,419][INFO ][o.e.n.Node               ] [GcxldH9] started
[2018-06-19T14:01:11,322][INFO ][o.e.g.GatewayService     ] [GcxldH9] recovered [5] indices into cluster_state
[2018-06-19T14:01:13,081][INFO ][o.e.c.r.a.AllocationService] [GcxldH9] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[metricbeat-2018.05.21][3]] ...]).
```

起動したら、[cat API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cat.html)を使ってインデックスの一覧とテンプレートの一覧を取得してみます。

```
$ curl -XGET 'localhost:9200/_cat/indices?v&pretty'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
$
$ curl -XGET 'localhost:9200/_cat/templates?v&pretty'
name template order version
$
```

まだインデックスもテンプレートもありません。

`liberty_logstash.conf`を指定してLogstashを起動します。

```shell-session
sotoiwa@soto-no-air:~/opt/logstash/logstash-5.3.3
$ bin/logstash -f config/liberty_logstash.conf
Sending Logstash's logs to /Users/sotoiwa/opt/logstash/logstash-5.3.3/logs which is now configured via log4j2.properties
[2018-06-19T14:17:14,480][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2018-06-19T14:17:14,493][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
log4j:WARN No appenders could be found for logger (org.apache.http.client.protocol.RequestAuthCache).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[2018-06-19T14:17:14,729][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<URI::HTTP:0x6acdef31 URL:http://localhost:9200/>}
[2018-06-19T14:17:14,739][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"config/liberty_logstash_template_ibm.json"}
[2018-06-19T14:17:14,865][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"liberty_message"=>{"properties"=>{"host"=>{"type"=>"keyword", "index"=>"true"}, "ibm_userDir"=>{"type"=>"keyword", "index"=>"true"}, "ibm_serverName"=>{"type"=>"keyword", "index"=>"true"}, "ibm_datetime"=>{"type"=>"date", "format"=>"yyyy-MM-dd'T'HH:mm:ss.SSSZ"}, "ibm_sequence"=>{"type"=>"keyword", "index"=>"true"}, "ibm_className"=>{"type"=>"keyword", "index"=>"true"}, "module"=>{"type"=>"keyword", "index"=>"true"}, "message"=>{"type"=>"text", "index"=>"true"}, "ibm_messageId"=>{"type"=>"keyword", "index"=>"true"}, "ibm_methodName"=>{"type"=>"keyword", "index"=>"true"}, "loglevel"=>{"type"=>"keyword", "index"=>"true"}, "ibm_threadId"=>{"type"=>"keyword", "index"=>"true"}, "ibm_tags"=>{"type"=>"keyword", "index"=>"true"}}}, "liberty_ffdc"=>{"properties"=>{"host"=>{"type"=>"keyword", "index"=>"true"}, "ibm_userDir"=>{"type"=>"keyword", "index"=>"true"}, "ibm_serverName"=>{"type"=>"keyword", "index"=>"true"}, "ibm_datetime"=>{"type"=>"date", "format"=>"yyyy-MM-dd'T'HH:mm:ss.SSSZ"}, "ibm_sequence"=>{"type"=>"keyword", "index"=>"true"}, "ibm_className"=>{"type"=>"keyword", "index"=>"true"}, "ibm_exceptionName"=>{"type"=>"keyword", "index"=>"true"}, "ibm_objectDetails"=>{"type"=>"keyword", "index"=>"true"}, "ibm_probeID"=>{"type"=>"keyword", "index"=>"true"}, "ibm_stackTrace"=>{"type"=>"keyword", "index"=>"true"}, "ibm_tags"=>{"type"=>"keyword", "index"=>"true"}}}, "liberty_trace"=>{"properties"=>{"host"=>{"type"=>"keyword", "index"=>"true"}, "ibm_userDir"=>{"type"=>"keyword", "index"=>"true"}, "ibm_serverName"=>{"type"=>"keyword", "index"=>"true"}, "ibm_datetime"=>{"type"=>"date", "format"=>"yyyy-MM-dd'T'HH:mm:ss.SSSZ"}, "ibm_sequence"=>{"type"=>"keyword", "index"=>"true"}, "ibm_className"=>{"type"=>"keyword", "index"=>"true"}, "module"=>{"type"=>"keyword", "index"=>"true"}, "message"=>{"type"=>"text", "index"=>"true"}, "ibm_methodName"=>{"type"=>"keyword", "index"=>"true"}, "loglevel"=>{"type"=>"keyword", "index"=>"true"}, "ibm_threadId"=>{"type"=>"keyword", "index"=>"true"}, "ibm_tags"=>{"type"=>"keyword", "index"=>"true"}}}, "liberty_gc"=>{"properties"=>{"host"=>{"type"=>"keyword", "index"=>"true"}, "ibm_userDir"=>{"type"=>"keyword", "index"=>"true"}, "ibm_serverName"=>{"type"=>"keyword", "index"=>"true"}, "ibm_datetime"=>{"type"=>"date", "format"=>"yyyy-MM-dd'T'HH:mm:ss.SSSZ"}, "ibm_sequence"=>{"type"=>"keyword", "index"=>"true"}, "ibm_heap"=>{"type"=>"long", "index"=>"true"}, "ibm_usedHeap"=>{"type"=>"long", "index"=>"true"}, "ibm_maxHeap"=>{"type"=>"long", "index"=>"true"}, "ibm_duration"=>{"type"=>"long", "index"=>"true"}, "ibm_gcType"=>{"type"=>"keyword", "index"=>"true"}, "ibm_reason"=>{"type"=>"keyword", "index"=>"true"}, "ibm_tags"=>{"type"=>"keyword", "index"=>"true"}}}, "liberty_accesslog"=>{"properties"=>{"host"=>{"type"=>"keyword", "index"=>"true"}, "ibm_userDir"=>{"type"=>"keyword", "index"=>"true"}, "ibm_serverName"=>{"type"=>"keyword", "index"=>"true"}, "ibm_datetime"=>{"type"=>"date", "format"=>"yyyy-MM-dd'T'HH:mm:ss.SSSZ"}, "ibm_sequence"=>{"type"=>"keyword", "index"=>"true"}, "ibm_requestHost"=>{"type"=>"keyword", "index"=>"true"}, "ibm_requestPort"=>{"type"=>"keyword", "index"=>"true"}, "ibm_remoteHost"=>{"type"=>"keyword", "index"=>"true"}, "ibm_requestMethod"=>{"type"=>"keyword", "index"=>"true"}, "ibm_requestProtocol"=>{"type"=>"keyword", "index"=>"true"}, "ibm_bytesReceived"=>{"type"=>"long", "index"=>"true"}, "ibm_elapsedTime"=>{"type"=>"long", "index"=>"true"}, "ibm_queryString"=>{"type"=>"text", "index"=>"true"}, "ibm_responseCode"=>{"type"=>"integer", "index"=>"true"}, "ibm_uriPath"=>{"type"=>"keyword", "index"=>"true"}, "ibm_userAgent"=>{"type"=>"keyword", "index"=>"true"}, "ibm_tags"=>{"type"=>"keyword", "index"=>"true"}}}}}}
[2018-06-19T14:17:14,897][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/logstash
[2018-06-19T14:17:15,045][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#<URI::Generic:0x5cd552c7 URL://localhost:9200>]}
[2018-06-19T14:17:15,048][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2018-06-19T14:17:16,435][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5043"}
[2018-06-19T14:17:16,610][INFO ][logstash.pipeline        ] Pipeline main started
[2018-06-19T14:17:16,739][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
```

起動時にテンプレートがPOSTされ作成されたことが確認できます。

```
$ curl -XGET 'localhost:9200/_cat/indices?v&pretty'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
$
$ curl -XGET 'localhost:9200/_cat/templates?v&pretty'
name     template   order version
logstash logstash-* 0
$
```

Libertyを起動します。

```
$ curl -XGET 'localhost:9200/_cat/indices?v&pretty'
health status index               uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   logstash-2018.06.19 aF0gR54USJ6C0AqBrYpzVw   5   1         19            0       82kb           82kb
$
```

インデックスが作成されたことが確認できます。日毎に作成されます。

アクセスログを出したいので、アプリケーションにアクセスしておきます。

http://localhost:9080/sample/

## Kibanaの起動

Kibanaを起動します。

```shell-session
sotoiwa@soto-no-air:~/opt/kibana/kibana-5.3.3-darwin-x86_64
$ bin/kibana
  log   [05:46:07.758] [info][status][plugin:kibana@5.3.3] Status changed from uninitialized to green - Ready
  log   [05:46:07.945] [info][status][plugin:elasticsearch@5.3.3] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [05:46:08.028] [info][status][plugin:console@5.3.3] Status changed from uninitialized to green - Ready
  log   [05:46:08.569] [info][status][plugin:timelion@5.3.3] Status changed from uninitialized to green - Ready
  log   [05:46:08.575] [info][listening] Server running at http://localhost:5601
  log   [05:46:08.576] [info][status][ui settings] Status changed from uninitialized to yellow - Elasticsearch plugin is yellow
  log   [05:46:13.920] [info][status][plugin:elasticsearch@5.3.3] Status changed from yellow to yellow - No existing Kibana index found
  log   [05:46:14.699] [info][status][plugin:elasticsearch@5.3.3] Status changed from yellow to green - Kibana index ready
  log   [05:46:14.702] [info][status][ui settings] Status changed from yellow to green - Ready
```

Kibanaのコンソールにアクセスします。

http://localhost:5601

Time-field nameに`ibm_datetime`を指定して、Createをクリックします。

![image.png](https://qiita-image-store.s3.amazonaws.com/0/86449/25b19b3b-2825-0331-778e-0820c1cee143.png)

Discoverタブに移動すると、ログが送られてきていることが確認できます。

![image.png](https://qiita-image-store.s3.amazonaws.com/0/86449/1ffc77ef-8a3a-e89d-58d1-0de41c69939f.png)

リポジトリーからLiberty用のダッシュボードのサンプルをダウンロードします。

https://github.com/WASdev/sample.logstash.collector/blob/master/version-1.1/Kibana5/Beta/20170516/liberty-kibana5.json

KibanaコンソールのManagement > Saved Object > Importからダッシュボードをインポートします。

![image.png](https://qiita-image-store.s3.amazonaws.com/0/86449/e758fe2c-bb91-a938-055a-0e361830ad37.png)

インポートした内容がいくつかエラーになってしまいました。これはGarbageCollectionなど、検索で使用するいくつかのフォールドについて、実際のログが送られてきていないためです。

Liberty Problem DashboardとTraffic DashboardについてはDashboardタブからダッシュボードの表示が確認できると思います。

![image.png](https://qiita-image-store.s3.amazonaws.com/0/86449/7cdf1366-11c8-acbe-34c7-dd8df3a5d4ad.png)

![image.png](https://qiita-image-store.s3.amazonaws.com/0/86449/1c64d1b1-f04e-d1d2-ab2b-6baf692328aa.png)

表示ができない場合、フィールドが検索可能になっていないことが考えられます。
例えば、Management > Index Patternsを確認すると、以下の例では`ibm_classNameフィールド`がsearchableになっていないので、このフィールドを使うVisualizationやDashboardがエラーになります。

![image.png](https://qiita-image-store.s3.amazonaws.com/0/86449/8c792790-064b-27ef-5242-74f7f09bf030.png)

このフィールドを含むログを出力させた後、Index Patternを再作成することで、フィールドがsearchableとなり、エラーが解消するはずです。

![image.png](https://qiita-image-store.s3.amazonaws.com/0/86449/dcfefb96-1c86-5001-383d-31d77407f117.png)

# 続き

[Libertyのログをjson形式で出力してFilebeatでELKに送る](https://qiita.com/sotoiwa/items/13e08d2426289e5c5c67)

# 参考リンク

[Logstash のログ・コレクター](https://www.ibm.com/support/knowledgecenter/ja/SSEQTP_liberty/com.ibm.websphere.wlp.doc/ae/cwlp_analytics_collectors.html)
[Logstash collector](https://www.ibm.com/support/knowledgecenter/en/was_beta_liberty/com.ibm.websphere.wlp.nd.multiplatform.doc/ae/cwlp_analytics_collectors.html)

[Logstash コレクター V1.0 の使用](https://www.ibm.com/support/knowledgecenter/ja/SSEQTP_liberty/com.ibm.websphere.wlp.doc/ae/twlp_analytics_logstash.html)
[Using the Logstash collector version 1.1](https://www.ibm.com/support/knowledgecenter/ja/was_beta_liberty/com.ibm.websphere.wlp.nd.multiplatform.doc/ae/twlp_analytics_logstashv1_1.html)

[Logstash Collector 1.0](https://www.ibm.com/support/knowledgecenter/en/SSEQTP_liberty/com.ibm.websphere.liberty.autogen.base.doc/ae/rwlp_feature_logstashCollector-1.0.html)
[Logstash Collector 1.1](https://www.ibm.com/support/knowledgecenter/ja/was_beta_liberty/com.ibm.websphere.liberty.autogen.beta.doc/ae/rwlp_feature_logstashCollector-1.1.html)