DockerでELK基盤構築

# 内容
* 下記環境を作成した際（ELK基盤の箇所）の備忘録です。
![elk-01.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2523652/29e8f2ce-d4ab-8301-699f-e678f349dfe7.png)

* FESSを使用して運用で使用しているOSSツールを全文検索する。
* ELKでSyslogサーバのログ分析を行う。
* portainerでDockerの管理を行う。

# 環境
* AlmaLinux release 8.5
* Docker version 20.10.14

# 構築手順
## ELKコンテナ作成
* ダウンロード
```bash
cd ~
git clone  https://github.com/deviantony/docker-elk.git
```

* 起動
```bash
docker compose -f /root/docker-elk/docker-compose.yml up -d
```

* パスワード取得
```bash
docker compose exec elasticsearch bin/elasticsearch-reset-password --batch --user elastic
docker compose exec elasticsearch bin/elasticsearch-reset-password --batch --user logstash_internal
docker compose exec elasticsearch bin/elasticsearch-reset-password --batch --user kibana_system
```

* envファイル修正

```bash:/root/docker-elk/.env
ELASTIC_PASSWORD='**********'
LOGSTASH_INTERNAL_PASSWORD='**********'
KIBANA_SYSTEM_PASSWORD='**********'
```

* コンテナrebuild

```bash
docker compose up -d logstash kibana
```

* kibanaにアクセス確認
http:\//xx.xx.xx.xx:5601/
user: elastic
password: **********


* 下記を追加
既にFESS用にESが動いているためポート番号変更
/mntの箇所は後でSyslogサーバの/var/logをNFSマウントする


```bash:/root/docker-elk/compose-elk.yml
elasticsearch:
  ports:
    - "9201:9200"
    - "9301:9300"

logstash:
  volumes:
    - /mnt:/mnt:ro
```

* 有償機能を無効化

```bash:/root/docker-elk/elasticsearch/config/elasticsearch.yml
xpack.license.self_generated.type: basic
```

## systemd設定変更 

```bash:/usr/lib/systemd/system/docker-compose-elk.service
ExecStart=/usr/bin/docker compose -f /root/docker-elk/docker-compose.yml up -d
```
```bash
systemctl daemon-reload
systemctl enable docker-compose-elk.service
systemctl restart docker-compose-elk.service
```

## Syslogサーバ設定

```bash
yum -y install nfs-utils
systemctl enable nfs-server
systemctl start nfs-server
```

```bash:/etc/exports
/var/log 192.168.108.0/24(rw,no_root_squash)
```

```bash
systemctl restart nfs-server
```

## Dockerホストサーバ設定

```bash
yum -y install nfs-utils
systemctl enable nfs-server
systemctl start nfs-server
mount -t nfs 192.168.108.148:/var/log /mnt
```

```bash:/etc/fstab
192.168.108.148:/var/log /mnt               nfs     defaults        0 0
```

## portainer追加

* docker systemdの設定変更
```bash:/usr/lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd -H unix:// -H tcp://0.0.0.0:12345
```

```bash
systemctl daemon-reload
systemctl restart docker.service
netstat -lntp | grep docker
```

* イメージ取得とテスト起動

```bash
docker pull portainer/portainer
docker run -d -p 9000:9000 --name test portainer/portainer
```

* Composeから起動設定

```bash:/root/portainer/compose/portainer.yaml
services:

  portainer01:
    image: portainer/portainer
    container_name: portainer01
    ports:
      - "9000:9000"
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - data:/data

volumes:
  data:
    driver: local
```

```bash:/usr/lib/systemd/system/docker-compose.service
[Unit]
After=docker.service
Description=Docker-Compose
[Service]
ExecStart=/usr/bin/docker compose -f /root/portainer/compose/portainer.yaml up -d
Type=simple
[Install]
WantedBy=multi-user.target
```

```bash
systemctl daemon-reload
systemctl restart docker-compose
```

# Syslog取込設定
## 各種設定ファイル修正

```bash:/root/docker-elk/docker-compose.yml
下記追加
  logstash:
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline:ro,Z
      - ./logstash/patterns:/usr/share/logstash/patterns:ro,Z
      - /mnt:/mnt:ro
```

```bash:/root/docker-elk/logstash/Dockerfile
下記追加
RUN rm -f /usr/share/logstash/pipeline/logstash.conf
ADD pipeline/ /usr/share/logstash/pipeline/
ADD config/ /usr/share/logstash/config/

```

```bash:/root/docker-elk/logstash/config/pipelines.yml
- pipeline.id: server-syslog
  pipeline.batch.size: 125
  path.config: "/usr/share/logstash/pipeline/server-syslog.conf"
  pipeline.workers: 1
```

```bash:/root/docker-elk/logstash/pipeline/server-syslog.conf
input {
     file {
        path => "/mnt/Server/********.log"
        start_position => "beginning"
     }
}
filter {
    grok {
      patterns_dir => ["/usr/share/logstash/patterns"]
      match => { "message" => "%{SYSLOG_SERVER_PATTERN}" }
    }
}
output {
     elasticsearch {
         hosts => ["elasticsearch:9200"]
         index => "server-syslog-%{+YYYY-MM-dd}"
         user => "logstash_internal"
         password => "${LOGSTASH_INTERNAL_PASSWORD}"
     }
}

```

```bash:/root/docker-elk/logstash/patterns/server-pattern
SYSLOG_PROG [^:]+
SYSLOG_MSG (?<=: ).+

SYSLOG_SERVER_PATTERN %{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{SYSLOG_PROG:syslog_program}: %{SYSLOG_MSG:syslog_messages}
```

## Logstash_internalユーザにロール追加

* 書き込み権限がなくエラーとなってしまったため追加
* テストのためsuperuser追加

![logstash2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2523652/126d261a-37c4-2dd4-b0aa-5b07246ec4ea.png)

## コンテナ起動・ログ確認

```bash
docker compose -f /root/docker-elk/docker-compose.yml up -d
docker logs docker-elk-logstash-1 -f
```

## kibana設定

* 「Stack Management」⇒「Data Views」⇒「Create data view」でdata view作成
* 「Analytics」⇒「Discover」からデータ確認


![logstash.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2523652/c282c719-f2ed-17ec-b54d-613d14a97688.png)