(調査中)FluentdをDockerで動かしてみる

AWS Workspaces
Amazon Linux2
Fluentd 1.7.4
MariaDB 10.4.8
Redis 5.0.5

# 調査中
・MariaDBのgeneral_logに関するプラグインがあるか調査
　→なければ複数行のログに関することを調査
・nginxの追加(アクセスログ、エラーログ)
・Redisのスローログに関するプラグインがあるか調査
・ElasticsearchとKibanaのコンテナ追加
　→Kibanaでログを参照

```
|--docker-compose.yml
|--fluentd
|  |--dockerfiles
|  |  |--Dockerfile
|  |--etc
|  |  |--fluent.conf
|--mariadb
|  |--etc
|  |  |--mymariadb.cnf
|  |--log
```

```docker-compose.yml
version: "3.7"

services:
  fluentd:
    container_name: fluentd
    build: ./fluentd/dockerfiles
    ports:
      - 24224:24224
      - 24224:24224/udp
    environment:
      - FLUENTD_CONF=fluent.conf
    volumes:
      - ./fluentd/etc:/fluentd/etc
      - ./mariadb/log:/var/log/mysql
    user: root # MariaDBのログ参照時にエラーになったので追加

  mariadb:
    container_name: mariadb
    image: mariadb:10.4.8
    ports:
      - 3306:3306
    environment:
      - MYSQL_ROOT_PASSWORD=mariadb
    volumes:
      - ./mariadb/etc:/etc/mysql/conf.d
      - ./mariadb/log:/var/log/mysql

  redis:
    container_name: redis
    image: redis:5.0.5
    ports:
      - 6379:6379
```

```:./fluentd/dockerfile/Dockerfile
FROM fluent/fluentd:v1.7.4-debian-1.0

# Use root account to use apt
USER root

# below RUN includes plugin as examples elasticsearch is not required
# you may customize including plugins as you wish
RUN buildDeps="sudo make gcc g++ libc-dev" \
 && apt-get update \
 && apt-get install -y --no-install-recommends $buildDeps \
 && sudo gem install fluent-plugin-mysqlslowquery \
 && sudo gem install fluent-plugin-redis-slowlog \
 && sudo gem sources --clear-all \
 && SUDO_FORCE_REMOVE=yes \
    apt-get purge -y --auto-remove \
                  -o APT::AutoRemove::RecommendsImportant=false \
                  $buildDeps \
 && rm -rf /var/lib/apt/lists/* \
 && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

USER fluent
```

```./fluentd/etc/fluent.conf
<source>
  @type mysql_slow_query
  path /var/log/mysql/slow.log
  pos_file /tmp/mysql/slow.pos
  tag mysqld.slow_query
   <parse>
     @type none
   </parse>
</source>
<source>
  @type tail
  format none
  path /var/log/mysql/general.log
  pos_file /tmp/mysql/general.pos
  tag mysqld.general
</source>
<source>
  @type tail
  format none
  path /var/log/mysql/error.log
  pos_file /tmp/mysql/error.pos
  tag mysqld.error
</source>

<match **.**>
    @type stdout
</match>
```

```./mariadb/etc/mymariadb.cnf
[mysqld]
general_log
general_log_file=/var/log/mysql/general.log
slow_query_log
slow_query_log_file=/var/log/mysql/slow.log
long_query_time=5 # 5秒以上処理に時間がかかったクエリを記録
log-queries-not-using-indexes # インデックスが使用されていないクエリをログに出力
log-error=/var/log/mysql/error.log
```

# 準備・実行

```
$ mkdir -p mariadb/log
$ chmod 777 mariadb/log

$ docker-compose up -d
```

# 標準出力の出力例

MariaDBのslow_query_logの一部

```
2019-11-10 15:04:26.000000000 +0000 mysqld.slow_query: {"sql":"mysqld, Version: 10.4.8-MariaDB-1:10.4.8+maria~bionic-log (mariadb.org binary distribution). started with: Tcp port: 0  Unix socket: /var/run/mysqld/mysqld.sock Time\t\t    Id Command\tArgument","db":"mysql"}
2019-11-10 15:04:26.000000000 +0000 mysqld.slow_query: {"user":"root[root]","host":"localhost","host_ip":"","thread_id":9,"schema":"mysql","qc_hit":"No","query_time":0.065842,"lock_time":0.0,"rows_sent":0,"rows_examined":1,"rows_affected":0,"bytes_sent":119,"db":"mysql","sql":"use mysql; SET timestamp=1573398266; IF (select count(*) from information_schema.global_variables where variable_name='wsrep_on') = 1 THEN ALTER TABLE time_zone ENGINE=InnoDB; ALTER TABLE time_zone_name ENGINE=InnoDB; ALTER TABLE time_zone_transition ENGINE=InnoDB; ALTER TABLE time_zone_transition_type ENGINE=InnoDB; END IF;"}
```

MariaDBのgeneral_logの一部

```
2019-11-10 15:04:39.843549472 +0000 mysqld.general: {"message":"\t\t     9 Query\tINSERT INTO time_zone_transition_type (Time_zone_id, Transition_type_id, Offset, Is_DST, Abbreviation) VALUES"}
2019-11-10 15:04:39.843555589 +0000 mysqld.general: {"message":" (@time_zone_id, 0, 3600, 1, 'WEST')"}
2019-11-10 15:04:39.843558069 +0000 mysqld.general: {"message":",(@time_zone_id, 1, 0, 0, 'WET')"}
2019-11-10 15:04:39.843896720 +0000 mysqld.general: {"message":"\t\t     9 Query\tINSERT INTO time_zone (Use_leap_seconds) VALUES ('Y')"}
2019-11-10 15:04:39.844847218 +0000 mysqld.general: {"message":"\t\t     9 Query\tSET @time_zone_id= LAST_INSERT_ID()"}
2019-11-10 15:04:39.844852250 +0000 mysqld.general: {"message":"\t\t     9 Query\tINSERT INTO time_zone_name (Name, Time_zone_id) VALUES ('right/Zulu', @time_zone_id)"}
2019-11-10 15:04:39.846381634 +0000 mysqld.general: {"message":"\t\t     9 Query\tINSERT INTO time_zone_transition_type (Time_zone_id, Transition_type_id, Offset, Is_DST, Abbreviation) VALUES"}
2019-11-10 15:04:39.846385918 +0000 mysqld.general: {"message":" (@time_zone_id, 0, 0, 0, 'UTC')"}
2019-11-10 15:04:39.848711320 +0000 mysqld.general: {"message":"\t\t     9 Query\tALTER TABLE time_zone_transition ORDER BY Time_zone_id, Transition_time"}
2019-11-10 15:04:40.163073544 +0000 mysqld.general: {"message":"191110 15:04:40\t     9 Query\tALTER TABLE time_zone_transition_type ORDER BY Time_zone_id, Transition_type_id"}
2019-11-10 15:04:40.235542296 +0000 mysqld.general: {"message":"\t\t     9 Query\tIF (select count(*) from information_schema.global_variables where"}
2019-11-10 15:04:40.235549524 +0000 mysqld.general: {"message":"variable_name='wsrep_on') = 1 THEN"}
2019-11-10 15:04:40.235551677 +0000 mysqld.general: {"message":"ALTER TABLE time_zone ENGINE=Aria;"}
2019-11-10 15:04:40.235553780 +0000 mysqld.general: {"message":"ALTER TABLE time_zone_name ENGINE=Aria;"}
2019-11-10 15:04:40.235556402 +0000 mysqld.general: {"message":"ALTER TABLE time_zone_transition ENGINE=Aria;"}
2019-11-10 15:04:40.235558628 +0000 mysqld.general: {"message":"ALTER TABLE time_zone_transition_type ENGINE=Aria;"}
2019-11-10 15:04:40.235560786 +0000 mysqld.general: {"message":"END IF"}
2019-11-10 15:04:40.783979234 +0000 mysqld.general: {"message":"\t\t     9 Quit\t"}
2019-11-10 15:04:40.790090092 +0000 mysqld.general: {"message":"\t\t    10 Connect\troot@localhost as anonymous on "}
2019-11-10 15:04:40.791960613 +0000 mysqld.general: {"message":"\t\t    10 Query\tSET @@SESSION.SQL_LOG_BIN=0"}
2019-11-10 15:04:40.792238993 +0000 mysqld.general: {"message":"\t\t    10 Query\tDELETE FROM mysql.user WHERE user NOT IN ('mysql.sys', 'mysqlxsys', 'root') OR host NOT IN ('localhost')"}
2019-11-10 15:04:40.796164198 +0000 mysqld.general: {"message":"\t\t    10 Query\tSET PASSWORD FOR 'root'@'localhost'=PASSWORD('mariadb')"}
2019-11-10 15:04:40.796582865 +0000 mysqld.general: {"message":"\t\t    10 Query\tGRANT ALL ON *.* TO 'root'@'localhost' WITH GRANT OPTION"}
2019-11-10 15:04:40.797369557 +0000 mysqld.general: {"message":"\t\t    10 Query\tCREATE USER 'root'@'%' IDENTIFIED BY 'mariadb'"}
2019-11-10 15:04:40.798257568 +0000 mysqld.general: {"message":"\t\t    10 Query\tGRANT ALL ON *.* TO 'root'@'%' WITH GRANT OPTION"}
2019-11-10 15:04:40.798900930 +0000 mysqld.general: {"message":"\t\t    10 Query\tDROP DATABASE IF EXISTS test"}
2019-11-10 15:04:40.800663164 +0000 mysqld.general: {"message":"\t\t    10 Query\tFLUSH PRIVILEGES"}
2019-11-10 15:04:40.800960012 +0000 mysqld.general: {"message":"\t\t    10 Quit\t"}
2019-11-10 15:04:42.162846659 +0000 mysqld.general: {"message":"mysqld, Version: 10.4.8-MariaDB-1:10.4.8+maria~bionic-log (mariadb.org binary distribution). started with:"}
2019-11-10 15:04:42.162852043 +0000 mysqld.general: {"message":"Tcp port: 3306  Unix socket: /var/run/mysqld/mysqld.sock"}
2019-11-10 15:04:42.162855166 +0000 mysqld.general: {"message":"Time\t\t    Id Command\tArgument"}
```

MariaDBのerror_logの一部

```
2019-11-10 15:04:42.161608960 +0000 mysqld.error: {"message":"2019-11-10 15:04:42 0 [Note] Plugin 'FEEDBACK' is disabled."}
2019-11-10 15:04:42.163697207 +0000 mysqld.error: {"message":"2019-11-10 15:04:42 0 [Note] Server socket created on IP: '::'."}
2019-11-10 15:04:42.168289913 +0000 mysqld.error: {"message":"2019-11-10 15:04:42 0 [Note] Reading of all Master_info entries succeeded"}
2019-11-10 15:04:42.168296473 +0000 mysqld.error: {"message":"2019-11-10 15:04:42 0 [Note] Added new Master_info '' to hash table"}
2019-11-10 15:04:42.168299929 +0000 mysqld.error: {"message":"2019-11-10 15:04:42 0 [Note] mysqld: ready for connections."}
2019-11-10 15:04:42.168302198 +0000 mysqld.error: {"message":"Version: '10.4.8-MariaDB-1:10.4.8+maria~bionic-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution"}
2019-11-10 15:04:42.176705708 +0000 mysqld.error: {"message":"2019-11-10 15:04:42 0 [Note] InnoDB: Buffer pool(s) load completed at 191110 15:04:42"}
```