docker-composeでElasticsearch/Kibana/ElastAlertを同時に起動して初回起動時にElastAlertのIndexが作成されるようにする

# 対象バージョン

・Elasticsearch 7.6.2
・Kibana 7.6.2
・servercentral/elastalert:latest
　→ ElastAlert Server & ElastAlert 0.2.1

# エラーメッセージ出力例

以下のようなエラーメッセージが大量に出るのと、ElastAlertのIndexがElasticsearchに作られないので、ElastAlertの再起動が必要

```sh
16:42:28.793Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
16:42:59.055Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
16:43:06.877Z ERROR elastalert-server:
    ProcessController:  WARNING:elasticsearch:GET http://elasticsearch:9200/elastalert_status/_search?size=1000 [status:404 request:0.009s]
    
16:43:06.877Z ERROR elastalert-server:
    ProcessController:  ERROR:root:Error finding recent pending alerts: NotFoundError(404, 'index_not_found_exception', 'no such index [elastalert_status]', elastalert_status, index_or_alias) {'query': {'bool': {'must': {'query_string': {'query': '!_exists_:aggregate_id AND alert_sent:false'}}, 'filter': {'range': {'alert_time': {'from': '2020-04-25T16:43:06.867791Z', 'to': '2020-04-27T16:43:06.867819Z'}}}}}, 'sort': {'alert_time': {'order': 'asc'}}}
    Traceback (most recent call last):
      File "/opt/elastalert/elastalert/elastalert.py", line 1625, in find_recent_pending_alerts
        res = self.writeback_es.search(index=self.writeback_index, body=query, size=1000)
      File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
        return func(*args, params=params, **kwargs)
      File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/client/__init__.py", line 819, in search
        "GET", _make_path(index, "_search"), params=params, body=body
      File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/transport.py", line 350, in perform_request
        timeout=timeout,
      File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/connection/http_requests.py", line 156, in perform_request
        self._raise_error(response.status_code, raw_data)
      File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/connection/base.py", line 181, in _raise_error
        status_code, error_message, additional_info
    elasticsearch.exceptions.NotFoundError: NotFoundError(404, 'index_not_found_exception', 'no such index [elastalert_status]', elastalert_status, index_or_alias)
    
16:43:29.504Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
16:43:59.813Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
16:44:06.871Z ERROR elastalert-server:
    ProcessController:  WARNING:elasticsearch:GET http://elasticsearch:9200/elastalert_status/_search?size=1000 [status:404 request:0.007s]
    
16:44:06.874Z ERROR elastalert-server:
    ProcessController:  ERROR:root:Error finding recent pending alerts: NotFoundError(404, 'index_not_found_exception', 'no such index [elastalert_status]', elastalert_status, index_or_alias) {'query': {'bool': {'must': {'query_string': {'query': '!_exists_:aggregate_id AND alert_sent:false'}}, 'filter': {'range': {'alert_time': {'from': '2020-04-25T16:44:06.864223Z', 'to': '2020-04-27T16:44:06.864250Z'}}}}}, 'sort': {'alert_time': {'order': 'asc'}}}
    Traceback (most recent call last):
      File "/opt/elastalert/elastalert/elastalert.py", line 1625, in find_recent_pending_alerts
        res = self.writeback_es.search(index=self.writeback_index, body=query, size=1000)
      File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
        return func(*args, params=params, **kwargs)
      File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/client/__init__.py", line 819, in search
        "GET", _make_path(index, "_search"), params=params, body=body
      File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/transport.py", line 350, in perform_request
        timeout=timeout,
      File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/connection/http_requests.py", line 156, in perform_request
        self._raise_error(response.status_code, raw_data)
      File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/connection/base.py", line 181, in _raise_error
        status_code, error_message, additional_info
    elasticsearch.exceptions.NotFoundError: NotFoundError(404, 'index_not_found_exception', 'no such index [elastalert_status]', elastalert_status, index_or_alias)
    
16:44:30.100Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
```

再起動すると作られるが。。。
最初から作られて欲しいなぁと思う

```sh
> @bitsensor/elastalert@0.0.14 start /opt/elastalert-server
> sh ./scripts/start.sh

16:46:38.792Z  INFO elastalert-server: Config:  No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json.
16:46:38.794Z  INFO elastalert-server: Config:  Proceeding to look for normal config file.
16:46:38.795Z  INFO elastalert-server: Config:  A config file was found in /opt/elastalert-server/config/config.json. Using that config.
16:46:38.804Z  INFO elastalert-server: Router:  Listening for GET request on /.
16:46:38.804Z  INFO elastalert-server: Router:  Listening for GET request on /status.
16:46:38.805Z  INFO elastalert-server: Router:  Listening for GET request on /status/errors.
16:46:38.805Z  INFO elastalert-server: Router:  Listening for GET request on /rules.
16:46:38.807Z  INFO elastalert-server: Router:  Listening for GET request on /rules/:id*.
16:46:38.807Z  INFO elastalert-server: Router:  Listening for POST request on /rules/:id*.
16:46:38.807Z  INFO elastalert-server: Router:  Listening for DELETE request on /rules/:id*.
16:46:38.807Z  INFO elastalert-server: Router:  Listening for GET request on /templates.
16:46:38.807Z  INFO elastalert-server: Router:  Listening for GET request on /templates/:id*.
16:46:38.808Z  INFO elastalert-server: Router:  Listening for POST request on /templates/:id*.
16:46:38.808Z  INFO elastalert-server: Router:  Listening for DELETE request on /templates/:id*.
16:46:38.808Z  INFO elastalert-server: Router:  Listening for PUT request on /folders/:type/:path*.
16:46:38.808Z  INFO elastalert-server: Router:  Listening for DELETE request on /folders/:type/:path*.
16:46:38.808Z  INFO elastalert-server: Router:  Listening for POST request on /test.
16:46:38.808Z  INFO elastalert-server: Router:  Listening for POST request on /silence/:path*.
16:46:38.808Z  INFO elastalert-server: Router:  Listening for GET request on /config.
16:46:38.809Z  INFO elastalert-server: Router:  Listening for POST request on /config.
16:46:38.809Z  INFO elastalert-server: Router:  Listening for POST request on /download.
16:46:38.809Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert.
16:46:38.809Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert_status.
16:46:38.809Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/silence.
16:46:38.809Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert_error.
16:46:38.809Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/past_elastalert.
16:46:38.809Z  INFO elastalert-server: Router:  Listening for GET request on /indices.
16:46:38.810Z  INFO elastalert-server: Router:  Listening for GET request on /mapping/:index.
16:46:38.810Z  INFO elastalert-server: Router:  Listening for POST request on /search/:index.
16:46:38.810Z  INFO elastalert-server: Router:  Listening for GET request on /config.
16:46:38.814Z  INFO elastalert-server: ProcessController:  Starting ElastAlert
16:46:38.814Z  INFO elastalert-server: ProcessController:  Creating index
16:46:43.074Z  INFO elastalert-server:
    ProcessController:  Elastic Version: 7.6.2
    Reading Elastic 6 index mappings:
    Reading index mapping 'es_mappings/6/silence.json'
    Reading index mapping 'es_mappings/6/elastalert_status.json'
    Reading index mapping 'es_mappings/6/elastalert.json'
    Reading index mapping 'es_mappings/6/past_elastalert.json'
    Reading index mapping 'es_mappings/6/elastalert_error.json'
    New index elastalert_status created
    Done!
    
16:46:43.075Z  INFO elastalert-server: ProcessController:  Index create exited with code 0
16:46:43.076Z  INFO elastalert-server: ProcessController:  Starting elastalert with arguments [none]
16:46:43.087Z  INFO elastalert-server: ProcessController:  Started Elastalert (PID: 36)
16:46:43.091Z  INFO elastalert-server: Server:  Server listening on port 3030
16:46:43.092Z  INFO elastalert-server: Server:  Websocket listening on port 3333
16:46:43.095Z  INFO elastalert-server: Server:  Server started
16:47:06.693Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
16:47:37.013Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
16:48:07.292Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
```

![キャプチャ.PNG](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/208807/7e86c765-2560-5f02-f472-5ff517379653.png)



# 対応

Elasticsearchが接続できる状態になってからElastAlertを起動するようにする

```
|--docker-compose.yml
|--Dockerfiles
|  |--Dockerfile-elastalert
|
|--es
|  |--config
|  |  |--elasticsearch.yml
|  |--data
|
|--kibana
|  |--config
|  |  |--kibana.yml
|
|--elastalert
|  |--bin
|  |  |--elastalert-start.sh
|  |  |--elastic_search_status.sh
|  |--config
|  |  |--api.config.json
|  |  |--elastalert.yaml
|  |--rule_templates
|  |--rules
```

```docker-compose.yml
version: "3.7"
services:
  elasticsearch:
    container_name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:7.6.2
    ports:
      - 9200:9200
      - 9300:9300
    environment:
      - ES_JAVA_OPTS=-Xms256m -Xmx256m
      - discovery.type=single-node
    restart: always
    volumes:
      - ./es/data:/usr/share/elasticsearch/data
      - ./es/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:9200 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 180s

  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:7.6.2
    ports:
      - 5601:5601
    depends_on:
      - elasticsearch
    restart: always
    volumes:
      - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:5601/api/status || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s

  elastalert:
    container_name: elastalert
    build:
      context: .
      dockerfile: Dockerfiles/Dockerfile-elastalert
    image: elastalert:0.2.1
    ports:
      - 3030:3030
      - 3333:3333
    depends_on:
      - elasticsearch
      - kibana
    restart: always
    volumes:
      - ./elastalert/config/elastalert.yaml:/opt/elastalert/config.yaml
      - ./elastalert/config/api.config.json:/opt/elastalert-server/config/config.json
      - ./elastalert/rules:/opt/elastalert/rules
      - ./elastalert/rule_templates:/opt/elastalert/rule_templates
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:3030 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s
```

```es/config/elasticsearch.yml
cluster.name: "docker-cluster"
network.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1
```

```kibana/config/kibana.yml
server.name: kibana
server.host: "0"
elasticsearch.hosts: http://elasticsearch:9200
xpack.monitoring.ui.container.elasticsearch.enabled: true
```

```:Dockerfiles/Dockerfile-elastalert
FROM servercentral/elastalert:latest

USER root

RUN apk update && \
    apk add bash curl && \
    rm -rf /var/cache/apk/*

ADD elastalert/bin/elastalert-start.sh /usr/local/bin/
ADD elastalert/bin/elastic_search_status.sh /usr/local/bin/

RUN chmod +x /usr/local/bin/elastalert-start.sh && chmod +x /usr/local/bin/elastic_search_status.sh

USER node

ENTRYPOINT ["/usr/local/bin/elastalert-start.sh"]
```

```elastalert/bin/elastic_search_status.sh
#!/bin/bash

set -e

if [ $# -gt 0 ]; then
  ES_URL="$1"
elif [[ -n $ELASTICSEARCH_URL ]]; then
  ES_URL="$ELASTICSEARCH_URL"
elif [[ -n $ES_HOST ]] && [[ -n $ES_PORT ]]; then
  ES_URL="http://$ES_HOST:$ES_PORT"
else
  ES_URL="http://elasticsearch:9200"
fi

until [[ "$(curl -fsSL "$ES_URL/_cat/health?h=status" | sed -r 's/^[[:space:]]+|[[:space:]]+$//g')" =~ ^(yellow|green)$ ]]; do
  # printf '+' >&2
  sleep 1
done

echo "Elasticsearch is up and healthy at "$ES_URL"" >&2
```

```elastalert/bin/elastalert-start.sh
#!/bin/bash

set -e

echo "Giving Elasticsearch at $ELASTICSEARCH_URL time to start..."

elastic_search_status.sh

echo "Starting ElastAlert!"
npm start
es/config/elasticsearch.yml
```

```elastalert/config/api.config.json
{
  "appName": "elastalert-server",
  "port": 3030,
  "wsport": 3333,
  "elastalertPath": "/opt/elastalert",
  "verbose": false,
  "es_debug": false,
  "debug": false,
  "rulesPath": {
    "relative": true,
    "path": "/rules"
  },
  "templatesPath": {
    "relative": true,
    "path": "/rule_templates"
  },
  "es_host": "elasticsearch",
  "es_port": 9200,
  "es_username": "",
  "es_password": "",
  "es_ssl": false,
  "writeback_index": "elastalert_status"
}
```

```elastalert/config/elastalert.yml
# The elasticsearch hostname for metadata writeback
# Note that every rule can have its own elasticsearch host
es_host: elasticsearch

# The elasticsearch port
es_port: 9200

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: rules

# How often ElastAlert will query elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  seconds: 60

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 1

# Optional URL prefix for elasticsearch
#es_url_prefix: elasticsearch

# Connect with TLS to elasticsearch
#use_ssl: True

# Verify TLS certificates
#verify_certs: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for elasticsearch
#es_username: someusername
#es_password: somepassword

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

skip_invalid: True
```

```sh
$ chmod 777 es/data
$ chmod 777 elastalert/rules
$ chmod 777 elastalert/rule_templates
$ docker-compose up -d
```

```sh
$ docker logs -f elastalert

Giving Elasticsearch at  time to start...
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
Elasticsearch is up and healthy at http://elasticsearch:9200
Starting ElastAlert!

> @bitsensor/elastalert@0.0.14 start /opt/elastalert-server
> sh ./scripts/start.sh

16:53:28.919Z  INFO elastalert-server: Config:  No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json.
16:53:28.927Z  INFO elastalert-server: Config:  Proceeding to look for normal config file.
16:53:28.931Z  INFO elastalert-server: Config:  A config file was found in /opt/elastalert-server/config/config.json. Using that config.
16:53:28.942Z  INFO elastalert-server: Router:  Listening for GET request on /.
16:53:28.943Z  INFO elastalert-server: Router:  Listening for GET request on /status.
16:53:28.944Z  INFO elastalert-server: Router:  Listening for GET request on /status/errors.
16:53:28.944Z  INFO elastalert-server: Router:  Listening for GET request on /rules.
16:53:28.947Z  INFO elastalert-server: Router:  Listening for GET request on /rules/:id*.
16:53:28.948Z  INFO elastalert-server: Router:  Listening for POST request on /rules/:id*.
16:53:28.949Z  INFO elastalert-server: Router:  Listening for DELETE request on /rules/:id*.
16:53:28.949Z  INFO elastalert-server: Router:  Listening for GET request on /templates.
16:53:28.950Z  INFO elastalert-server: Router:  Listening for GET request on /templates/:id*.
16:53:28.951Z  INFO elastalert-server: Router:  Listening for POST request on /templates/:id*.
16:53:28.952Z  INFO elastalert-server: Router:  Listening for DELETE request on /templates/:id*.
16:53:28.952Z  INFO elastalert-server: Router:  Listening for PUT request on /folders/:type/:path*.
16:53:28.953Z  INFO elastalert-server: Router:  Listening for DELETE request on /folders/:type/:path*.
16:53:28.954Z  INFO elastalert-server: Router:  Listening for POST request on /test.
16:53:28.958Z  INFO elastalert-server: Router:  Listening for POST request on /silence/:path*.
16:53:28.959Z  INFO elastalert-server: Router:  Listening for GET request on /config.
16:53:28.960Z  INFO elastalert-server: Router:  Listening for POST request on /config.
16:53:28.961Z  INFO elastalert-server: Router:  Listening for POST request on /download.
16:53:28.962Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert.
16:53:28.963Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert_status.
16:53:28.964Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/silence.
16:53:28.964Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert_error.
16:53:28.965Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/past_elastalert.
16:53:28.966Z  INFO elastalert-server: Router:  Listening for GET request on /indices.
16:53:28.967Z  INFO elastalert-server: Router:  Listening for GET request on /mapping/:index.
16:53:28.967Z  INFO elastalert-server: Router:  Listening for POST request on /search/:index.
16:53:28.968Z  INFO elastalert-server: Router:  Listening for GET request on /config.
16:53:28.973Z  INFO elastalert-server: ProcessController:  Starting ElastAlert
16:53:28.974Z  INFO elastalert-server: ProcessController:  Creating index
16:53:35.304Z  INFO elastalert-server:
    ProcessController:  Elastic Version: 7.6.2
    Reading Elastic 6 index mappings:
    Reading index mapping 'es_mappings/6/silence.json'
    Reading index mapping 'es_mappings/6/elastalert_status.json'
    Reading index mapping 'es_mappings/6/elastalert.json'
    Reading index mapping 'es_mappings/6/past_elastalert.json'
    Reading index mapping 'es_mappings/6/elastalert_error.json'
    New index elastalert_status created
    Done!
    
16:53:35.304Z  INFO elastalert-server: ProcessController:  Index create exited with code 0
16:53:35.307Z  INFO elastalert-server: ProcessController:  Starting elastalert with arguments [none]
16:53:35.332Z  INFO elastalert-server: ProcessController:  Started Elastalert (PID: 241)
16:53:35.352Z  INFO elastalert-server: Server:  Server listening on port 3030
16:53:35.353Z  INFO elastalert-server: Server:  Websocket listening on port 3333
16:53:35.358Z  INFO elastalert-server: Server:  Server started
16:53:45.843Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
16:54:16.146Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
```
![キャプチャ.PNG](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/208807/fb812f5d-2c9d-86b3-2cd8-e48d7494b9e9.png)