Elastalert kibana plugin で gmail にアラート送付

Elasticsearch に入れたデータを元にアラートを送付する仕組みの一つに elastalert があります。
https://elastalert.readthedocs.io/en/latest/index.html

Elasticsearch 公式ではないですが、広く使われているようなので試しに使ってみます。
また、今回は kibana plugin 経由で実行してます。

マシンスペック等
OS: VirtualBox 上 Ubuntu 18.04
メモリ: 8196 MB
CPUコア数: 2
Elasticsearch と kibana の version: 7.6.2


### テスト環境セットアップ
ローカル環境で実行するため、手軽に Docker を使って環境をセットアップしていきます。

#### ディレクトリ構造
```
├── docker-compose.yml
├── elastalert
│   ├── bin
│   │   ├── elastalert-start.sh
│   │   └── elastic_search_status.sh
│   ├── config
│   │   ├── config.json
│   │   ├── elastalert-test.yaml
│   │   └── elastalert.yaml
│   ├── Dockerfile
│   ├── pass
│   │   └── smtp_auth_user.yaml
│   ├── rules
│   └── rule_templates
├── elasticsearch
│   ├── config
│   │   └── elasticsearch.yml
│   └── Dockerfile
├── kibana
│   ├── config
│   │   └── kibana.xml
│   ├── Dockerfile
│   └── plugin
│       └── elastalert-kibana-plugin-1.1.0-7.6.2.zip
```

#### elasticsearch
```bash:Dockerfile
FROM docker.elastic.co/elasticsearch/elasticsearch:7.6.2
```

```bash:config/elasticsearch.yml
cluster.name: "docker-cluster"
node.name: "node1"
node.master: true
node.data: true
network.host: 0.0.0.0
network.publish_host: _local_
discovery.seed_hosts: ["172.40.0.2"]
cluster.initial_master_nodes: ["node1"]
```

#### kibana
```bash:Dockerfile
FROM docker.elastic.co/kibana/kibana:7.6.2
```

kibana.xml 内で elastalert-kibana-plugin を使用する旨を宣言します。

```bash:config/kibana.xml
server.name: kibana
server.host: "0"
elasticsearch.hosts: [ "http://doc-elastic101:9200" ]
xpack.monitoring.ui.container.elasticsearch.enabled: true
elasticsearch.requestTimeout: 60000

# elastalert-kibana-plugin
elastalert-kibana-plugin.serverHost: elastalert
elastalert-kibana-plugin.serverPort: 3030
```

plugin/elastalert-kibana-plugin-1.1.0-7.6.2.zip については前バージョンを元に作成します。

```plugin/elastalert-kibana-plugin-1.1.0-7.6.2.zip
# Download necessary files
cd /tmp
curl -L -O https://github.com/bitsensor/elastalert-kibana-plugin/releases/download/1.1.0/elastalert-kibana-plugin-1.1.0-7.5.0.zip
curl -L -O https://raw.githubusercontent.com/mmguero-dev/Malcolm/development/kibana/elastalert-kibana-plugin/server/routes/elastalert.js

# update elasticsearch package to 7.6.2
mv elastalert.js elastalert-server-routes.js
mv elastalert-kibana-plugin-1.1.0-7.5.0.zip elastalert-kibana-plugin-1.1.0-7.6.2.zip
unzip elastalert-kibana-plugin-1.1.0-7.6.2.zip kibana/elastalert-kibana-plugin/package.json
sed -i "s/7\.5\.0/7\.6\.2/g" kibana/elastalert-kibana-plugin/package.json
mkdir -p kibana/elastalert-kibana-plugin/server/routes/
mv /tmp/elastalert-server-routes.js kibana/elastalert-kibana-plugin/server/routes/elastalert.js
zip elastalert-kibana-plugin-1.1.0-7.6.2.zip kibana/elastalert-kibana-plugin/package.json kibana/elastalert-kibana-plugin/server/routes/elastalert.js

# delete remaining directory
rm -rf kibana

# copy the created package to your workspace
cp /tmp/elastalert-kibana-plugin-1.1.0-7.6.2.zip {your_workspace}/kibana/plugin
```

#### elastalert
```bash:Dockerfile
FROM bitsensor/elastalert:3.0.0-beta.1

USER root

RUN apk update && \
    apk add bash curl && \
    rm -rf /var/cache/apk/*

ADD elastalert/bin/elastalert-start.sh /usr/local/bin/
ADD elastalert/bin/elastic_search_status.sh /usr/local/bin/

RUN chmod +x /usr/local/bin/elastalert-start.sh 

USER node

ENTRYPOINT ["/usr/local/bin/elastalert-start.sh"]
```

```bash:bin/elastalert-start.sh
#!/bin/bash

set -e

echo "Giving Elasticsearch at $ELASTICSEARCH_URL time to start..."

elastic_search_status.sh

echo "Starting ElastAlert!"
npm start
```

```bash:elastic_search_status.sh
#!/bin/bash

set -e

if [ $# -gt 0 ]; then
  ES_URL="$1"
elif [[ -n $ELASTICSEARCH_URL ]]; then
  ES_URL="$ELASTICSEARCH_URL"
elif [[ -n $ES_HOST ]] && [[ -n $ES_PORT ]]; then
  ES_URL="http://$ES_HOST:$ES_PORT"
else
  ES_URL="http://doc-elastic101:9200"
fi

until [[ "$(curl -fsSL "$ES_URL/_cat/health?h=status" | sed -r 's/^[[:space:]]+|[[:space:]]+$//g')" =~ ^(yellow|green)$ ]]; do
  # printf '+' >&2
  sleep 1
done

echo "Elasticsearch is up and healthy at "$ES_URL"" >&2
```

```bash:config/config.json
{
  "appName": "elastalert-server",
  "port": 3030,
  "wsport": 3333,
  "elastalertPath": "/opt/elastalert",
  "verbose": true,
  "es_debug": false,
  "debug": false,
  "rulesPath": {
    "relative": true,
    "path": "/rules"
  },
  "templatesPath": {
    "relative": true,
    "path": "/rule_templates"
  },
  "es_host": "elasticsearch",
  "es_port": 9200,
  "writeback_index": "elastalert_status"
}
```

```bash:config/elastalert-test.yaml
# NOTE: This config is used when testing a rule

# The elasticsearch hostname for metadata writeback
# Note that every rule can have its own elasticsearch host
es_host: doc-elastic101

# The elasticsearch port
es_port: 9200

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: rules

# How often ElastAlert will query elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  seconds: 5

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 1

# Optional URL prefix for elasticsearch
#es_url_prefix: elasticsearch

# Connect with TLS to elasticsearch
#use_ssl: True

# Verify TLS certificates
#verify_certs: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for elasticsearch
#es_username: someusername
#es_password: somepassword

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2
```

```bash:config/elastalert.yaml
# The elasticsearch hostname for metadata writeback
# Note that every rule can have its own elasticsearch host
es_host: doc-elastic101

# The elasticsearch port
es_port: 9200

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: rules

# How often ElastAlert will query elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  seconds: 5

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 1

# Optional URL prefix for elasticsearch
#es_url_prefix: elasticsearch

# Connect with TLS to elasticsearch
#use_ssl: True

# Verify TLS certificates
#verify_certs: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for elasticsearch
#es_username: someusername
#es_password: somepassword

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2
```

後で出てきますが、今回は gmail にメールを送付したいため、認証用のユーザとパスワードが書いたファイルを用意しておきます。

```bash:pass/smtp_auth_user.yaml
user: "xxxx@gmail.com"
password: "xxxx"
```

```bash:docker-compose.yml
version: '3.2'
services:
    elasticsearch:
        build:
            context: elasticsearch/
        hostname: doc-elastic101
        container_name: elastic1
        ports:
            - "9200:9200/tcp"
            - "9300:9300/tcp"
        networks:
            elk_nw:
                ipv4_address: 172.60.0.2
        volumes:
          - type: bind
            source: ./elasticsearch/config/elasticsearch.yml
            target: /usr/share/elasticsearch/config/elasticsearch.yml
            read_only: true
          - type: volume
            source: elasticsearch-data
            target: /usr/share/elasticsearch/data
        extra_hosts:
            - "doc-kibana101:172.60.0.4"

    kibana:
        build:
            context: kibana/
        hostname: doc-kibana101
        container_name: kibana1
        command: sh -c './bin/kibana-plugin list | grep elastalert-kibana-plugin@1.1.0; result=`echo $$?`; if [ $$result = 1 ]; then  ./bin/kibana-plugin install file:///usr/share/kibana/work/elastalert-kibana-plugin-1.1.0-7.6.2.zip && exec /usr/local/bin/kibana-docker; else exec /usr/local/bin/kibana-docker; fi'
        ports:
            - "5601:5601/tcp"
        networks:
            elk_nw:
                ipv4_address: 172.60.0.4
        volumes:
            - type: bind
              source: ./kibana/config/kibana.xml
              target: /usr/share/kibana/config/kibana.yml
              read_only: true
            - type: bind
              source: ./kibana/plugin
              target: /usr/share/kibana/work
              read_only: true
        extra_hosts:
            - "doc-elastic101:172.60.0.2"
        depends_on:
            - elasticsearch

    elastalert:
        container_name: elastalert
        build:
            context: .
            dockerfile: elastalert/Dockerfile
        image: elastalert:0.2.1
        ports:
            - 3030:3030
            - 3333:3333
        depends_on:
            - elasticsearch
            - kibana
        networks:
            elk_nw:
                ipv4_address: 172.60.0.5
        volumes:
            - ./elastalert/config/elastalert.yaml:/opt/elastalert/config.yaml
            - ./elastalert/config/elastalert-test.yaml:/opt/elastalert/config-test.yaml
            - ./elastalert/config/config.json:/opt/elastalert-server/config/config.json
            - ./elastalert/rules:/opt/elastalert/rules
            - ./elastalert/rule_templates:/opt/elastalert/rule_templates
            - ./elastalert/pass:/opt/elastalert/pass
        extra_hosts:
            - "doc-elastic101:172.60.0.2"
            - "doc-kibana101:172.60.0.4"

volumes:
    elasticsearch-data:
        driver: local

networks:
    elk_nw:
        driver: bridge
        ipam:
            driver: default
            config:
                - subnet: 172.60.0.0/16
```

### 実行
以下のコマンドで
1. 元になる DockerImage のダウンロード
2. DockerImage の生成
3. 各コンテナの起動
まで行ってくれます。
※自分の環境下ではダウンロードにもやや時間がかかりましたが、コンテナの起動後、全てのサービスが起動状態になるまで15分ほど要しました。

```bash
$ docker-compose up --build -d
```

とりあえず、以下のように3つのコンテナが起動している状態になるまで待ちます。

```bash
$ docker ps
CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS                                            NAMES
9b38f592fd57        elastalert:0.2.1          "/usr/local/bin/elas…"   x days ago        Up 3 minutes        0.0.0.0:3030->3030/tcp, 0.0.0.0:3333->3333/tcp   elastalert
696929e31b45        dockerelk_kibana          "/usr/local/bin/dumb…"   x days ago          Up 3 minutes        0.0.0.0:5601->5601/tcp                           kibana1
a06d24afc2d3        dockerelk_elasticsearch   "/usr/local/bin/dock…"   x days ago          Up 3 minutes        0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   elastic1
```

#### kibana UI
http://localhost:5601 にアクセスします。
正しくプラグインがインストールされていれば、次のようなアイコンが生成されているはずです。
![elastalertkibana0.PNG](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63951/9d970c39-6b95-ab16-c8cc-6189893f1450.png)

#### ルールの作成
Create Rule ボタンからルールを作成していきます。
構文については、以下を参照。
https://elastalert.readthedocs.io/en/latest/ruletypes.html

例)
Rule name: test

```bash
es_host: doc-elastic101
es_port: 9200
name: First rule
type: frequency
index: alert*
is_enabled: true

num_events: 1
timeframe:
    hours: 10
filter: []

#subject to email
alert_subject: "Search Spike"

alert:
- "email"
email:
    - "xxxx@gmail.com"
smtp_host: "smtp.gmail.com"
smtp_port: 465 
smtp_ssl: true
from_addr: "xxxx@gmail.com"
smtp_auth_file: "/opt/elastalert/pass/smtp_auth_user.yaml"
```

- filter: [] とすることで、(アラートを送信する条件を設けず) elasticsearch にメッセージが届いただけでアラートを飛ばすようにしています。
- 送信元と送信先のメールアドレスは同じでOK

### テスト
#### gmail 安全性の低いアプリのアクセスを許可する
以下にアクセスして、安全性の低いアプリの許可: 有効 にしておきます。
https://myaccount.google.com/lesssecureapps

#### テストデータの作成: Index patter の生成用
kibana UI から以下のクエリを投げます。
@timestamp は適当な直近の時刻をしていします。

```
PUT /alert/_doc/1
{
  "title": "Test Alart1",
  "name": {
    "first": "test",
    "last": "taro"
  },
  "@timestamp": "2020-MM-ddThh:mm:ss+0900"
}
```

#### Index pattern の生成
kibana UI から Index pattern を生成します。
Index pattern名は "alert*" とします。
![elastalertkibana3.PNG](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63951/a1d14720-e736-7df3-56ba-4d120e0d94cd.png)

![elastalertkibana4.PNG](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63951/a271ed37-22cf-5025-b960-2b277b5c30ac.png)

#### テストデータの作成: アラート送信用
kibana UI から以下のクエリを投げます。
@timestamp は現在時刻から直後の時刻(1分後など)を指定します

```
PUT /alert/_doc/2
{
  "title": "Test Alart2",
  "name": {
    "first": "test",
    "last": "taro"
  },
  "@timestamp": "2020-MM-ddThh:mm:ss+0900"
}
```

指定したメールアドレスに以下のようなメッセージが到着していれば成功です。

```
First rule

At least 1 events occurred between 2020-MM-dd hh:mm UTC and 2020-MM-dd hh:mm UTC

@timestamp: 2020-MM-dd hh:mm:ss+09:00
_id: 2
_index: alert
_type: _doc
name: {
    "first": "test",
    "last": "taro"
}
num_hits: 1
num_matches: 1
title: Test Alart2
```

### ハマった箇所
- 安全性の低いアプリの許可: 無効 だとメール送信できない
セキュアな設定があると思われますが、未確認です

- ドキュメント内に @timestamp のフィールドが含まれていないと動かない
ProcessController:  ERROR:root:Error running query: RequestError(400, u'search_phase_execution_exception', u'No mapping found for [@timestamp] in order to sort on')
ドキュメントの到着時刻を elastalert が判断するために必要

- elastalert config で "debug": true だとアラート送信しない
デバッグモードだと検知してもアラート送信しませんでした（公式ドキュメントでは未確認）

- ElastAlert Kibana Plugin の UI から Edit rule を選択すると編集モードになる
編集モード内に Test ボタンがあるが、実行してもアラートは送信されない
以下のようにルールに match するメッセージが何件あったか表示される

```bash
Successfully loaded First rule

Would have written the following documents to writeback index (default is elastalert_status):

silence - {'rule_name': 'First rule', '@timestamp': ...}

elastalert_status - {'hits': 2, 'matches': 2, '@timestamp': ...}
```

- docker logs elastalert -f でログが表示できるが、"verbose": true にしておかないと詳細なログが表示されず問題に気づけない。

- AttributeError: 'Namespace' object has no attribute 'verbose'
Docker の version: 3.0.0-beta.0 では動作しなかった
-> 3.0.0-beta.1 に変更することで解消した

### 次の課題
- フィルター条件を具体的に指定してみる
- メールのコンテンツをカスタマイズしてみる

### 参考
#### Trouble shooting
[Elastalert not alerting via emai](https://github.com/Yelp/elastalert/issues/667)
[How to resolve 'Error connecting to SMTP host' in elastalert?](http://quabr.com:8182/52269118/how-to-resolve-error-connecting-to-smtp-host-in-elastalert)
[My elastalert is running normally, but I have not received any email alerts](https://github.com/Yelp/elastalert/issues/2000)
[Elastalert Test Rules doesn't send any alert](https://github.com/Yelp/elastalert/issues/1315)
[elastalert not sending out email](https://stackoverflow.com/questions/40142912/elastalert-not-sending-out-email)
[Email sending error in elastalert. SMTPSenderRefused: (530, '5.5.1 Authentication Required)](https://stackoverflow.com/questions/47474069/email-sending-error-in-elastalert-smtpsenderrefused-530-5-5-1-authenticatio)
[Matches found but 0 alerts sent](https://github.com/Yelp/elastalert/issues/256)
[unable to connect smtp server with correct username and password](https://github.com/Yelp/elastalert/issues/854)
[how to fix “send-mail: Authorization failed 534 5.7.14](https://serverfault.com/questions/635139/how-to-fix-send-mail-authorization-failed-534-5-7-14)
[Elasticsearchのデータを基にアラート通知する方法の調査](https://qiita.com/rururu_kenken/items/6207ab68378b70f408bb)
[bitsensor/elastalert](https://github.com/bitsensor/elastalert)
[elastalertでアプリの異常を自動検知して通知したい](https://qiita.com/AknYk416/items/e9e0e2132f6996735ec9)
[Praeco (ElastAlert GUI)でElasticsearchログアラートする](https://designetwork.daichi703n.com/entry/2019/03/24/Praeco-ElastAlert-GUI)
[bitsensor/elastalert-kibana-plugin](https://github.com/bitsensor/elastalert-kibana-plugin)
[Error executing elastalert-test-rule during args/conf readout](https://github.com/Yelp/elastalert/issues/2274)

#### Tips
[ElastAlert for first time and Configuration](https://medium.com/everythingatonce/elastalert-for-first-time-and-configuration-part-2-fdf49635fe46)
[Integrating ElastAlert Email Alerting with Elasticsearch](https://qbox.io/blog/integrating-elastalert-email-alerting-with-elasticsearch)
[ElastAlert Documentation](https://readthedocs.org/projects/elastalert/downloads/pdf/latest/)
[ElastAlert —Configurations & Frequency Rule Type to Email](https://medium.com/everythingatonce/elastalert-configurations-frequency-rule-type-to-email-day-2-fd9aa944322)
[Rule Types and Configuration Options](https://elastalert.readthedocs.io/en/latest/ruletypes.html)