MariaDB Galera Clusterの構築


Galera Clusterを使えば簡単にマルチマスタのDBを構築できたのでその時の構築メモです。

LB配下にインスタンスを複数ぶら下げて各インスタンスにAPサーバとDBサーバを同居させて、各インスタンスで更新・参照をしなければならなかったので、試してみました。

PostgreSQLでマルチマスタの構成を組んでみたかったけど、難しそうだったので簡単そうなMariaDBで組んでみました。

* OS: CentOS7, 最小3台で構築
* MariaDB: 10.3.8

とても参考になった記事。  
というかほぼコピペになってしまった。  
http://www.orangetakam.com/blog/archives/1403


## OS準備

### Vagrantfileの設定

```ruby
# *- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|

  config.vm.define "db1" do |node|
    node.vm.provider "virtualbox" do |vm|
      vm.name = "db1"
      vm.customize ["modifyvm", :id, "--memory", "2048"]
    end
    node.vm.box = "centos/7"
    node.vm.hostname = "db1"
    node.vm.network "private_network", ip: "192.168.56.10"
  end

  config.vm.define "db2" do |node|
    node.vm.provider "virtualbox" do |vm|
      vm.name = "db2"
      vm.customize ["modifyvm", :id, "--memory", "2048"]
    end
    node.vm.box = "centos/7"
    node.vm.hostname = "db2"
    node.vm.network "private_network", ip: "192.168.56.20"
  end

  config.vm.define "db3" do |node|
    node.vm.provider "virtualbox" do |vm|
      vm.name = "db3"
      vm.customize ["modifyvm", :id, "--memory", "2048"]
    end
    node.vm.box = "centos/7"
    node.vm.hostname = "db3"
    node.vm.network "private_network", ip: "192.168.56.30"
  end

end
```

### OS起動


```
vagrant up db1 db2 db3
```

### SELINUXの無効化

`/etc/sysconfig/selinux`を以下のように編集して再起動する。

```
SELINUX=disabled
SELINUXTYPE=targeted
```

## MariaDBのインストール

以下の作業を`db1`-`db3`まですべて実施する。

### バンドルされているMariaDBを削除する

```
sudo yum info mariadb-libs.x86_64
sudo yum -y remove mariadb-libs.x86_64
```

### リポジトリの設定

```
curl -sS https://downloads.mariadb.com/MariaDB/mariadb_repo_setup | sudo bash
```

### MariaDB最新版インストール

```
sudo yum -y install MariaDB-server MariaDB-client
```

### MariaDBの初期セットアップ

```
sudo systemctl disable mariadb
sudo systemctl start mariadb
```

```
sudo mysql_secure_installation

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current
password for the root user.  If you've just installed MariaDB, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none):
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.

Set root password? [Y/n] Y
New password: Pass@0077
Re-enter new password: Pass@0077
Password updated successfully!
Reloading privilege tables..
 ... Success!


By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them.  This is intended only for testing, and to make the installation
go a bit smoother.  You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] Y
 ... Success!

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] Y
 ... Success!

By default, MariaDB comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] Y
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] Y
 ... Success!

Cleaning up...

All done!  If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!
```

### MariaDBの停止

あとでGalera Clusterで起動するので、停止しておく。

```
sudo systemctl stop mariadb
```


## Galera Clusterの初期設定(`/etc/my.cnf.d/server.cnf`)

### db1の初期設定

```
sudo vi /etc/my.cnf.d/server.cnf

sudo mkdir -p /var/log/mariadb
sudo chown -R mysql:mysql /var/log/mariadb
```

```
[server]

[mysqld]
character-set-server=utf8

log-output=FILE

# Error log
log-error="/var/log/mariadb/mysqld.log"

# Query log
general-log=0
general_log_file="/var/log/mariadb/sql.log"

# Slow Query log
slow-query-log=1
slow_query_log_file="/var/log/mariadb/slow.log"
log_queries_not_using_indexes
log_slow_admin_statements
long_query_time=10

[galera]
wsrep_node_address=192.168.56.10
wsrep_node_name=db1

wsrep_cluster_address='gcomm://192.168.56.10,192.168.56.20,192.168.56.30'

wsrep_cluster_name=DevelopCluster

wsrep_provider=/usr/lib64/galera/libgalera_smm.so

binlog_format=row

default_storage_engine=InnoDB

innodb_autoinc_lock_mode=2

wsrep_on=ON

wsrep_sst_method=rsync

wsrep_slave_threads=2

innodb_locks_unsafe_for_binlog=1

[embedded]

[mariadb]

[mariadb-10.3]
```

### db2の初期設定


```
sudo vi /etc/my.cnf.d/server.cnf

sudo mkdir /var/log/mariadb
sudo chown -R mysql:mysql /var/log/mariadb
```

```
[server]

[mysqld]
character-set-server=utf8

log-output=FILE

# Error log
log-error="/var/log/mariadb/mysqld.log"

# Query log
general-log=0
general_log_file="/var/log/mariadb/sql.log"

# Slow Query log
slow-query-log=1
slow_query_log_file="/var/log/mariadb/slow.log"
log_queries_not_using_indexes
log_slow_admin_statements
long_query_time=10

[galera]
wsrep_node_address=192.168.56.20
wsrep_node_name=db2

wsrep_cluster_address='gcomm://192.168.56.10,192.168.56.20,192.168.56.30'

wsrep_cluster_name=DevelopCluster

wsrep_provider=/usr/lib64/galera/libgalera_smm.so

binlog_format=row

default_storage_engine=InnoDB

innodb_autoinc_lock_mode=2

wsrep_on=ON

wsrep_sst_method=rsync

wsrep_slave_threads=2

innodb_locks_unsafe_for_binlog=1

[embedded]

[mariadb]

[mariadb-10.3]
```


### db3の初期設定

```
sudo vi /etc/my.cnf.d/server.cnf

sudo mkdir -p /var/log/mariadb
sudo chown -R mysql:mysql /var/log/mariadb
```

```
[server]

[mysqld]
character-set-server=utf8

log-output=FILE

# Error log
log-error="/var/log/mariadb/mysqld.log"

# Query log
general-log=0
general_log_file="/var/log/mariadb/sql.log"


# Slow Query log
slow-query-log=1
slow_query_log_file="/var/log/mariadb/slow.log"
log_queries_not_using_indexes
log_slow_admin_statements
long_query_time=10

[galera]
wsrep_node_address=192.168.56.30
wsrep_node_name=db3

wsrep_cluster_address='gcomm://192.168.56.10,192.168.56.20,192.168.56.30'

wsrep_cluster_name=DevelopCluster

wsrep_provider=/usr/lib64/galera/libgalera_smm.so

binlog_format=row

default_storage_engine=InnoDB

innodb_autoinc_lock_mode=2

wsrep_on=ON

wsrep_sst_method=rsync

wsrep_slave_threads=2

innodb_locks_unsafe_for_binlog=1

[embedded]

[mariadb]

[mariadb-10.3]
```

## Galera Clusterの起動

### db1のMariaDB Galera Cluster起動確認

db1の起動確認です。
このとき、db2とdb3は停止状態です。

※ Galera Clusterでは、初期設定ファイルの「`wsrep_cluster_address`」に記述されたノードのどれかが起動していないと、起動できません。
1台目（ファーストノード）を起動する際には、他のノードは停止状態です。

ファーストノードは`galera_new_cluster`コマンドで起動します。

```
sudo galera_new_cluster
```

`show status like 'wsrep_%';`を実行して以下の事項を確認してください。

**[確認事項]**

* wsrep_cluster_status: Primary
* wsrep_incoming_addresses: 設定したアドレス
* wsrep_local_state_comment: Synced

```
[vagrant@db1 my.cnf.d]$ mysql -u root -p
Enter password:

MariaDB [(none)]> show status like 'wsrep_%';
+------------------------------+--------------------------------------+
| Variable_name                | Value                                |
+------------------------------+--------------------------------------+
| wsrep_apply_oooe             | 0.000000                             |
| wsrep_apply_oool             | 0.000000                             |
| wsrep_apply_window           | 0.000000                             |
| wsrep_causal_reads           | 0                                    |
| wsrep_cert_deps_distance     | 0.000000                             |
| wsrep_cert_index_size        | 0                                    |
| wsrep_cert_interval          | 0.000000                             |
| wsrep_cluster_conf_id        | 1                                    |
| wsrep_cluster_size           | 1                                    |
| wsrep_cluster_state_uuid     | 5ea4eb1f-8bff-11e8-934d-7aa2eaa06ac1 |
| wsrep_cluster_status         | Primary                              |
| wsrep_commit_oooe            | 0.000000                             |
| wsrep_commit_oool            | 0.000000                             |
| wsrep_commit_window          | 0.000000                             |
| wsrep_connected              | ON                                   |
| wsrep_desync_count           | 0                                    |
| wsrep_evs_delayed            |                                      |
| wsrep_evs_evict_list         |                                      |
| wsrep_evs_repl_latency       | 0/0/0/0/0                            |
| wsrep_evs_state              | OPERATIONAL                          |
| wsrep_flow_control_paused    | 0.000000                             |
| wsrep_flow_control_paused_ns | 0                                    |
| wsrep_flow_control_recv      | 0                                    |
| wsrep_flow_control_sent      | 0                                    |
| wsrep_gcomm_uuid             | 5ea35535-8bff-11e8-90ef-6b5cfed70296 |
| wsrep_incoming_addresses     | 192.168.56.10:3306                   |
| wsrep_last_committed         | 0                                    |
| wsrep_local_bf_aborts        | 0                                    |
| wsrep_local_cached_downto    | 18446744073709551615                 |
| wsrep_local_cert_failures    | 0                                    |
| wsrep_local_commits          | 0                                    |
| wsrep_local_index            | 0                                    |
| wsrep_local_recv_queue       | 0                                    |
| wsrep_local_recv_queue_avg   | 0.500000                             |
| wsrep_local_recv_queue_max   | 2                                    |
| wsrep_local_recv_queue_min   | 0                                    |
| wsrep_local_replays          | 0                                    |
| wsrep_local_send_queue       | 0                                    |
| wsrep_local_send_queue_avg   | 0.000000                             |
| wsrep_local_send_queue_max   | 1                                    |
| wsrep_local_send_queue_min   | 0                                    |
| wsrep_local_state            | 4                                    |
| wsrep_local_state_comment    | Synced                               |
| wsrep_local_state_uuid       | 5ea4eb1f-8bff-11e8-934d-7aa2eaa06ac1 |
| wsrep_protocol_version       | 8                                    |
| wsrep_provider_name          | Galera                               |
| wsrep_provider_vendor        | Codership Oy <info@codership.com>    |
| wsrep_provider_version       | 25.3.23(r3789)                       |
| wsrep_ready                  | ON                                   |
| wsrep_received               | 2                                    |
| wsrep_received_bytes         | 140                                  |
| wsrep_repl_data_bytes        | 0                                    |
| wsrep_repl_keys              | 0                                    |
| wsrep_repl_keys_bytes        | 0                                    |
| wsrep_repl_other_bytes       | 0                                    |
| wsrep_replicated             | 0                                    |
| wsrep_replicated_bytes       | 0                                    |
| wsrep_thread_count           | 3                                    |
+------------------------------+--------------------------------------+
58 rows in set (0.001 sec)

MariaDB [(none)]> exit
```


### db2のMariaDB Galera Cluster起動確認

db2の起動確認になります。  
このときは、db1は起動状態で、db3は停止状態です。

```
sudo systemctl start mariadb
```

クラスタに参加できていることを確認してください。

```
$ mysql -u root -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 11
Server version: 10.3.8-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> show status like 'wsrep_%';
+------------------------------+---------------------------------------+
| Variable_name                | Value                                 |
+------------------------------+---------------------------------------+
| wsrep_apply_oooe             | 0.000000                              |
| wsrep_apply_oool             | 0.000000                              |
| wsrep_apply_window           | 0.000000                              |
| wsrep_causal_reads           | 0                                     |
| wsrep_cert_deps_distance     | 0.000000                              |
| wsrep_cert_index_size        | 0                                     |
| wsrep_cert_interval          | 0.000000                              |
| wsrep_cluster_conf_id        | 2                                     |
| wsrep_cluster_size           | 2                                     |
| wsrep_cluster_state_uuid     | 5ea4eb1f-8bff-11e8-934d-7aa2eaa06ac1  |
| wsrep_cluster_status         | Primary                               |
| wsrep_commit_oooe            | 0.000000                              |
| wsrep_commit_oool            | 0.000000                              |
| wsrep_commit_window          | 0.000000                              |
| wsrep_connected              | ON                                    |
| wsrep_desync_count           | 0                                     |
| wsrep_evs_delayed            |                                       |
| wsrep_evs_evict_list         |                                       |
| wsrep_evs_repl_latency       | 0/0/0/0/0                             |
| wsrep_evs_state              | OPERATIONAL                           |
| wsrep_flow_control_paused    | 0.000000                              |
| wsrep_flow_control_paused_ns | 0                                     |
| wsrep_flow_control_recv      | 0                                     |
| wsrep_flow_control_sent      | 0                                     |
| wsrep_gcomm_uuid             | 58f28b22-8c03-11e8-b913-db00a1269d15  |
| wsrep_incoming_addresses     | 192.168.56.10:3306,192.168.56.20:3306 |
| wsrep_last_committed         | 0                                     |
| wsrep_local_bf_aborts        | 0                                     |
| wsrep_local_cached_downto    | 18446744073709551615                  |
| wsrep_local_cert_failures    | 0                                     |
| wsrep_local_commits          | 0                                     |
| wsrep_local_index            | 1                                     |
| wsrep_local_recv_queue       | 0                                     |
| wsrep_local_recv_queue_avg   | 0.000000                              |
| wsrep_local_recv_queue_max   | 1                                     |
| wsrep_local_recv_queue_min   | 0                                     |
| wsrep_local_replays          | 0                                     |
| wsrep_local_send_queue       | 0                                     |
| wsrep_local_send_queue_avg   | 0.000000                              |
| wsrep_local_send_queue_max   | 1                                     |
| wsrep_local_send_queue_min   | 0                                     |
| wsrep_local_state            | 4                                     |
| wsrep_local_state_comment    | Synced                                |
| wsrep_local_state_uuid       | 5ea4eb1f-8bff-11e8-934d-7aa2eaa06ac1  |
| wsrep_protocol_version       | 8                                     |
| wsrep_provider_name          | Galera                                |
| wsrep_provider_vendor        | Codership Oy <info@codership.com>     |
| wsrep_provider_version       | 25.3.23(r3789)                        |
| wsrep_ready                  | ON                                    |
| wsrep_received               | 3                                     |
| wsrep_received_bytes         | 216                                   |
| wsrep_repl_data_bytes        | 0                                     |
| wsrep_repl_keys              | 0                                     |
| wsrep_repl_keys_bytes        | 0                                     |
| wsrep_repl_other_bytes       | 0                                     |
| wsrep_replicated             | 0                                     |
| wsrep_replicated_bytes       | 0                                     |
| wsrep_thread_count           | 3                                     |
+------------------------------+---------------------------------------+
58 rows in set (0.001 sec)

MariaDB [(none)]> exit
```


### db3のMariaDB Galera Cluster起動確認

db3の起動確認になります。

```
sudo systemctl start  mariadb
```


```
mysql -u root -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 11
Server version: 10.3.8-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> show status like 'wsrep_%';
+------------------------------+----------------------------------------------------------+
| Variable_name                | Value                                                    |
+------------------------------+----------------------------------------------------------+
| wsrep_apply_oooe             | 0.000000                                                 |
| wsrep_apply_oool             | 0.000000                                                 |
| wsrep_apply_window           | 0.000000                                                 |
| wsrep_causal_reads           | 0                                                        |
| wsrep_cert_deps_distance     | 0.000000                                                 |
| wsrep_cert_index_size        | 0                                                        |
| wsrep_cert_interval          | 0.000000                                                 |
| wsrep_cluster_conf_id        | 3                                                        |
| wsrep_cluster_size           | 3                                                        |
| wsrep_cluster_state_uuid     | 5ea4eb1f-8bff-11e8-934d-7aa2eaa06ac1                     |
| wsrep_cluster_status         | Primary                                                  |
| wsrep_commit_oooe            | 0.000000                                                 |
| wsrep_commit_oool            | 0.000000                                                 |
| wsrep_commit_window          | 0.000000                                                 |
| wsrep_connected              | ON                                                       |
| wsrep_desync_count           | 0                                                        |
| wsrep_evs_delayed            |                                                          |
| wsrep_evs_evict_list         |                                                          |
| wsrep_evs_repl_latency       | 0.000878738/0.00319743/0.00973027/0.003294/5             |
| wsrep_evs_state              | OPERATIONAL                                              |
| wsrep_flow_control_paused    | 0.000000                                                 |
| wsrep_flow_control_paused_ns | 0                                                        |
| wsrep_flow_control_recv      | 0                                                        |
| wsrep_flow_control_sent      | 0                                                        |
| wsrep_gcomm_uuid             | f93e0362-8c03-11e8-b220-a3600473017a                     |
| wsrep_incoming_addresses     | 192.168.56.10:3306,192.168.56.20:3306,192.168.56.30:3306 |
| wsrep_last_committed         | 0                                                        |
| wsrep_local_bf_aborts        | 0                                                        |
| wsrep_local_cached_downto    | 18446744073709551615                                     |
| wsrep_local_cert_failures    | 0                                                        |
| wsrep_local_commits          | 0                                                        |
| wsrep_local_index            | 2                                                        |
| wsrep_local_recv_queue       | 0                                                        |
| wsrep_local_recv_queue_avg   | 0.000000                                                 |
| wsrep_local_recv_queue_max   | 1                                                        |
| wsrep_local_recv_queue_min   | 0                                                        |
| wsrep_local_replays          | 0                                                        |
| wsrep_local_send_queue       | 0                                                        |
| wsrep_local_send_queue_avg   | 0.000000                                                 |
| wsrep_local_send_queue_max   | 1                                                        |
| wsrep_local_send_queue_min   | 0                                                        |
| wsrep_local_state            | 4                                                        |
| wsrep_local_state_comment    | Synced                                                   |
| wsrep_local_state_uuid       | 5ea4eb1f-8bff-11e8-934d-7aa2eaa06ac1                     |
| wsrep_protocol_version       | 8                                                        |
| wsrep_provider_name          | Galera                                                   |
| wsrep_provider_vendor        | Codership Oy <info@codership.com>                        |
| wsrep_provider_version       | 25.3.23(r3789)                                           |
| wsrep_ready                  | ON                                                       |
| wsrep_received               | 3                                                        |
| wsrep_received_bytes         | 284                                                      |
| wsrep_repl_data_bytes        | 0                                                        |
| wsrep_repl_keys              | 0                                                        |
| wsrep_repl_keys_bytes        | 0                                                        |
| wsrep_repl_other_bytes       | 0                                                        |
| wsrep_replicated             | 0                                                        |
| wsrep_replicated_bytes       | 0                                                        |
| wsrep_thread_count           | 3                                                        |
+------------------------------+----------------------------------------------------------+
58 rows in set (0.001 sec)

MariaDB [(none)]> exit
```

## データ同期の確認

* db1でデータベース作成

```
[vagrant@db1 ~]$ mysql -u root -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 11
Server version: 10.3.8-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> create database dev_nagakuray;
Query OK, 1 row affected (0.002 sec)

MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| dev_nagakuray      |
| information_schema |
| mysql              |
| performance_schema |
+--------------------+
4 rows in set (0.001 sec)

MariaDB [(none)]> use dev_nagakuray
Database changed

MariaDB [dev_nagakuray]> create table test (id int, name varchar(100));
Query OK, 0 rows affected (0.008 sec)

MariaDB [dev_nagakuray]> insert into test (id, name) values (1, 'hogehoge');
Query OK, 1 row affected (0.003 sec)

MariaDB [dev_nagakuray]>
```

* db2で確認

```
[vagrant@db2 ~]$ mysql -u root -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 11
Server version: 10.3.8-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| dev_nagakuray      |
| information_schema |
| mysql              |
| performance_schema |
+--------------------+
4 rows in set (0.001 sec)

MariaDB [(none)]> use dev_nagakuray
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [dev_nagakuray]> select * from test;
+------+----------+
| id   | name     |
+------+----------+
|    1 | hogehoge |
+------+----------+
1 row in set (0.000 sec)

MariaDB [dev_nagakuray]> insert into test (id, name) values (2, 'fugafuga');
Query OK, 1 row affected (0.005 sec)
```

* db3で確認

```
[vagrant@db3 ~]$ mysql -u root -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 12
Server version: 10.3.8-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| dev_nagakuray      |
| information_schema |
| mysql              |
| performance_schema |
+--------------------+
4 rows in set (0.000 sec)

MariaDB [(none)]> use dev_nagakuray
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [dev_nagakuray]> select * from test;
+------+----------+
| id   | name     |
+------+----------+
|    1 | hogehoge |
|    2 | fugafuga |
+------+----------+
2 rows in set (0.000 sec)

MariaDB [dev_nagakuray]> insert into test (id, name) values (3, 'hanako');
Query OK, 1 row affected (0.003 sec)
```

## 注意事項: MariaDB Galera Clusterの停止→起動の順序について

* 起動状態時のdb1

```
sudo cat /var/lib/mysql/grastate.dat

# GALERA saved state
version: 2.1
uuid:    5ea4eb1f-8bff-11e8-934d-7aa2eaa06ac1
seqno:   -1
safe_to_bootstrap: 0

```

* 起動状態時のdb2

```
sudo cat /var/lib/mysql/grastate.dat

# GALERA saved state
version: 2.1
uuid:    5ea4eb1f-8bff-11e8-934d-7aa2eaa06ac1
seqno:   -1
safe_to_bootstrap: 0
```

* 起動状態時のdb3

```
sudo cat /var/lib/mysql/grastate.dat

# GALERA saved state
version: 2.1
uuid:    5ea4eb1f-8bff-11e8-934d-7aa2eaa06ac1
seqno:   -1
safe_to_bootstrap: 0
```

上記の通り、該当ファイルの「safe_to_bootstrap」の値は「0」となっています。

では、1号機 → 2号機 → 3号機 の順番に停止します。

```
[vagrant@db1 ~]$ sudo systemctl stop mariadb
```

```
[vagrant@db2 ~]$ sudo systemctl stop mariadb
```

```
[vagrant@db3 ~]$ sudo systemctl stop mariadb
```

この時の`/var/lib/mysql/grastate.dat`を確認します。

* 停止状態時のdb1

```
[vagrant@db1 ~]$ sudo cat /var/lib/mysql/grastate.dat
# GALERA saved state
version: 2.1
uuid:    5ea4eb1f-8bff-11e8-934d-7aa2eaa06ac1
seqno:   0
safe_to_bootstrap: 0
```

* 停止状態時のdb2

```
[vagrant@db2 ~]$ sudo cat /var/lib/mysql/grastate.dat
# GALERA saved state
version: 2.1
uuid:    5ea4eb1f-8bff-11e8-934d-7aa2eaa06ac1
seqno:   0
safe_to_bootstrap: 0
```

* 停止状態時のdb3

```
[vagrant@db3 ~]$ sudo cat /var/lib/mysql/grastate.dat
# GALERA saved state
version: 2.1
uuid:    5ea4eb1f-8bff-11e8-934d-7aa2eaa06ac1
seqno:   0
safe_to_bootstrap: 1
```

**この時、1号機から起動すると、以下の通りエラーになります**。

```
 [vagrant@db1 ~]$ sudo galera_new_cluster
Job for mariadb.service failed because the control process exited with error code. See "systemctl status mariadb.service" and "journalctl -xe" for details.
```

`safe_to_bootstrap: 1`となっている**3号機から起動する必要があります**

```
[vagrant@db3 ~]$ sudo galera_new_cluster
```

```
[vagrant@db1 ~]$ sudo systemctl start mariadb
```

```
[vagrant@db2 ~]$ sudo systemctl start mariadb
```