PostgreSQL + PGroongaセットアップテスト

# PostgreSQL + PGroongaセットアップテスト
## 目的
本番導入の前にセットアップ手順や動作の検証
master -> slaveの物理レプリケーションに於いて、pgroongaの動作テストを行う

## 環境

| - | - |
| :--: | :-- |
| HostOS | Windows10 1709 (64bit) |
| GuestHW | VMware Workstation15 |
| GuestOS | CentOS Linux release 7.5.1804 (Core) |
| GeustMEM | 4096MB |
| GuestCPU(core) | 1(2) |

| hostname | roll | ipaddr(static) |
| :-- | :-- | :--: |
| pgrntest1 | master | 192.168.50.130 |
| pgrntest2 | slave | 192.168.50.131 | 

| SW | version|
| :--: | --: |
| msgpack | 1.4.1 |
| zstandard | 1.3.5 |
| groonga | 8.0.7 |
| postgresql | 10.5 |
| pgroonga | 2.1.4 |
*(2018.10.3 時点の最新)*

各種ソフトウェアは再現性を考慮して
まずオフィシャル手順とパッケージになるべく沿って構築して確認を行う


## OSセットアップ
スレーブはマスターノードをVMwareでcloneするのでミドルウェアのセットアップ後に構築する
したがってまずマスターノードのみ構築を行なう。

* インストールメディア: http://isoredirect.centos.org/centos/7/isos/x86_64/CentOS-7-x86_64-DVD-1804.iso
* インストールは`最小限のインストール`で行う
* 一般ユーザ`pgrn3`セットアップ

```bash
#rootでログインして
useradd -c "user" -G wheel pgrn3
passwd pgrn3
ユーザー pgrn3 のパスワードを変更。
新しいパスワード:
新しいパスワードを再入力してください:
passwd: すべての認証トークンが正しく更新できました。
#以降は全てこのユーザでログインして作業する
```

* FW無効化(今回はテストなので問題の切り分けを楽にする為に)

```terminal
$ sudo systemctl disable firewalld
$ sudo systemctl stop firewalld
$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
```

* selinux無効化(今回はテストなので問題の切り分けを楽にする為に)

```terminal
$ getenforce
Enforcing
$ sudo vi -es /etc/selinux/config
7c
#SELINUX=enforcing
SELINUX=permissive
.
:wq!
$ sudo setenforce permissive
$ getenforce
Permissive
```

* hosts設定

```terminal
$ sudo vi -es /etc/hosts
2a

192.168.50.130  pgrntest1
192.168.50.131  pgrntest2
.
:wq!
```

* カーネルパラメータ設定

```terminal
$ sudo vi -es /etc/sysctl.d/mysite.conf
0c
vm.overcommit_memory = 1
.
:wq!

```

* OSアップデート&再起動

```terminal
$ sudo yum update -y
$ sudo shutdown -r now
$ cat /etc/centos-release
CentOS Linux release 7.5.1804 (Core)
$ uname -a
Linux pgrntest1 3.10.0-862.14.4.el7.x86_64 #1 SMP Wed Sep 26 15:12:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
```

* 開発ツールセットアップ(あとでいろいろやってみる為に)

```terminal
$  sudo yum groupinstall -y "Development"
```


## パッケージセットアップ
以下のドキュメントを参考にミドルウェアのインストールを行う
https://pgroonga.github.io/install/centos.html#install-on-7

* レポジトリ設定インストール
 
```terminal
# postgresql10,groongaのレポジトリ設定をインストールする
$ sudo -H yum install -y http://yum.postgresql.org/10/redhat/rhel-$(rpm -qf --queryformat="%{VERSION}" /etc/redhat-release)-$(rpm -qf --queryformat="%{ARCH}" /etc/redhat-release)/pgdg-centos10-10-2.noarch.rpm
[sudo] pgrn3 のパスワード:
読み込んだプラグイン:fastestmirror
pgdg-centos10-10-2.noarch.rpm                                                          | 4.6 kB  00:00:00
/var/tmp/yum-root-EafJO1/pgdg-centos10-10-2.noarch.rpm を調べています: pgdg-centos10-10-2.noarch
/var/tmp/yum-root-EafJO1/pgdg-centos10-10-2.noarch.rpm をインストール済みとして設定しています
依存性の解決をしています
--> トランザクションの確認を実行しています。
---> パッケージ pgdg-centos10.noarch 0:10-2 を インストール
--> 依存性解決を終了しました。

依存性を解決しました

==============================================================================================================
 Package                   アーキテクチャー   バージョン         リポジトリー                            容量
==============================================================================================================
インストール中:
 pgdg-centos10             noarch             10-2               /pgdg-centos10-10-2.noarch             2.7 k

トランザクションの要約
==============================================================================================================
インストール  1 パッケージ

合計容量: 2.7 k
インストール容量: 2.7 k
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  インストール中          : pgdg-centos10-10-2.noarch                                                     1/1
  検証中                  : pgdg-centos10-10-2.noarch                                                     1/1

インストール:
  pgdg-centos10.noarch 0:10-2

完了しました!

$ sudo -H yum install -y https://packages.groonga.org/centos/groonga-release-latest.noarch.rpm
読み込んだプラグイン:fastestmirror
groonga-release-latest.noarch.rpm                                                      |  17 kB  00:00:00
/var/tmp/yum-root-EafJO1/groonga-release-latest.noarch.rpm を調べています: groonga-release-1.4.0-1.noarch
/var/tmp/yum-root-EafJO1/groonga-release-latest.noarch.rpm をインストール済みとして設定しています
依存性の解決をしています
--> トランザクションの確認を実行しています。
---> パッケージ groonga-release.noarch 0:1.4.0-1 を インストール
--> 依存性の処理をしています: epel-release のパッケージ: groonga-release-1.4.0-1.noarch
Loading mirror speeds from cached hostfile
 * base: ftp.iij.ad.jp
 * extras: ftp.iij.ad.jp
 * updates: ftp.iij.ad.jp
pgdg10                                                                                 | 4.1 kB  00:00:00
(1/2): pgdg10/7/x86_64/group_gz                                                        |  245 B  00:00:00
(2/2): pgdg10/7/x86_64/primary_db                                                      | 173 kB  00:00:01
--> トランザクションの確認を実行しています。
---> パッケージ epel-release.noarch 0:7-11 を インストール
--> 依存性解決を終了しました。

依存性を解決しました

==============================================================================================================
 Package                   アーキテクチャー バージョン         リポジトリー                              容量
==============================================================================================================
インストール中:
 groonga-release           noarch           1.4.0-1            /groonga-release-latest.noarch            10 k
依存性関連でのインストールをします:
 epel-release              noarch           7-11               extras                                    15 k

トランザクションの要約
==============================================================================================================
インストール  1 パッケージ (+1 個の依存関係のパッケージ)

合計容量: 25 k
総ダウンロード容量: 15 k
インストール容量: 35 k
Downloading packages:
epel-release-7-11.noarch.rpm                                                           |  15 kB  00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  インストール中          : epel-release-7-11.noarch                                                      1/2
  インストール中          : groonga-release-1.4.0-1.noarch                                                2/2
  検証中                  : groonga-release-1.4.0-1.noarch                                                1/2
  検証中                  : epel-release-7-11.noarch                                                      2/2

インストール:
  groonga-release.noarch 0:1.4.0-1

依存性関連をインストールしました:
  epel-release.noarch 0:7-11

完了しました!
```

* postgresql, pgroongaと依存パッケージをインストールする

```terminal
$ sudo -H yum install -y postgresql10-pgroonga
読み込んだプラグイン:fastestmirror
Loading mirror speeds from cached hostfile
epel/x86_64/metalink                                                                   | 7.3 kB  00:00:00
 * base: ftp.iij.ad.jp
 * epel: ftp.iij.ad.jp
 * extras: ftp.iij.ad.jp
 * updates: ftp.iij.ad.jp
epel                                                                                   | 3.2 kB  00:00:00
groonga                                                                                | 2.9 kB  00:00:00
(1/4): epel/x86_64/group_gz                                                            |  88 kB  00:00:00
(2/4): epel/x86_64/updateinfo                                                          | 948 kB  00:00:00
(3/4): epel/x86_64/primary                                                             | 3.6 MB  00:00:00
(4/4): groonga/7/x86_64/primary_db                                                     | 500 kB  00:00:00
epel                                                                                              12701/12701
依存性の解決をしています
--> トランザクションの確認を実行しています。
---> パッケージ postgresql10-pgroonga.x86_64 0:2.1.4-1.el7 を インストール
--> 依存性の処理をしています: groonga-libs >= 8.0.6 のパッケージ: postgresql10-pgroonga-2.1.4-1.el7.x86_64
--> 依存性の処理をしています: msgpack のパッケージ: postgresql10-pgroonga-2.1.4-1.el7.x86_64
--> 依存性の処理をしています: postgresql10-server のパッケージ: postgresql10-pgroonga-2.1.4-1.el7.x86_64
--> 依存性の処理をしています: libmsgpackc.so.2()(64bit) のパッケージ: postgresql10-pgroonga-2.1.4-1.el7.x86_64
--> 依存性の処理をしています: libgroonga.so.0()(64bit) のパッケージ: postgresql10-pgroonga-2.1.4-1.el7.x86_64
--> トランザクションの確認を実行しています。
---> パッケージ groonga-libs.x86_64 0:8.0.7-1.el7 を インストール
--> 依存性の処理をしています: libzstd のパッケージ: groonga-libs-8.0.7-1.el7.x86_64
--> 依存性の処理をしています: libzstd.so.1()(64bit) のパッケージ: groonga-libs-8.0.7-1.el7.x86_64
---> パッケージ msgpack.x86_64 0:1.4.1-1.el7 を インストール
---> パッケージ postgresql10-server.x86_64 0:10.5-1PGDG.rhel7 を インストール
--> 依存性の処理をしています: postgresql10-libs(x86-64) = 10.5-1PGDG.rhel7 のパッケージ: postgresql10-server-10.5-1PGDG.rhel7.x86_64
--> 依存性の処理をしています: postgresql10(x86-64) = 10.5-1PGDG.rhel7 のパッケージ: postgresql10-server-10.5-1PGDG.rhel7.x86_64
--> 依存性の処理をしています: libpq.so.5()(64bit) のパッケージ: postgresql10-server-10.5-1PGDG.rhel7.x86_64
--> トランザクションの確認を実行しています。
---> パッケージ libzstd.x86_64 0:1.3.5-1.el7 を インストール
---> パッケージ postgresql10.x86_64 0:10.5-1PGDG.rhel7 を インストール
---> パッケージ postgresql10-libs.x86_64 0:10.5-1PGDG.rhel7 を インストール
--> 依存性解決を終了しました。

依存性を解決しました

==============================================================================================================
 Package                           アーキテクチャー   バージョン                    リポジトリー         容量
==============================================================================================================
インストール中:
 postgresql10-pgroonga             x86_64             2.1.4-1.el7                   groonga             113 k
依存性関連でのインストールをします:
 groonga-libs                      x86_64             8.0.7-1.el7                   groonga             1.9 M
 libzstd                           x86_64             1.3.5-1.el7                   epel                239 k
 msgpack                           x86_64             1.4.1-1.el7                   epel                 23 k
 postgresql10                      x86_64             10.5-1PGDG.rhel7              pgdg10              1.6 M
 postgresql10-libs                 x86_64             10.5-1PGDG.rhel7              pgdg10              354 k
 postgresql10-server               x86_64             10.5-1PGDG.rhel7              pgdg10              4.5 M

トランザクションの要約
==============================================================================================================
インストール  1 パッケージ (+6 個の依存関係のパッケージ)

総ダウンロード容量: 8.7 M
インストール容量: 40 M
Downloading packages:
警告: /var/cache/yum/x86_64/7/epel/packages/libzstd-1.3.5-1.el7.x86_64.rpm: ヘッダー V3 RSA/SHA256 Signature、鍵 ID 352c64e5: NOKEY
libzstd-1.3.5-1.el7.x86_64.rpm の公開鍵がインストールされていません
(1/7): libzstd-1.3.5-1.el7.x86_64.rpm                                                  | 239 kB  00:00:00
(2/7): msgpack-1.4.1-1.el7.x86_64.rpm                                                  |  23 kB  00:00:00
警告: /var/cache/yum/x86_64/7/groonga/packages/postgresql10-pgroonga-2.1.4-1.el7.x86_64.rpm: ヘッダー V4 DSA/SHA1 Signature、鍵 ID 45499429: NOKEY
postgresql10-pgroonga-2.1.4-1.el7.x86_64.rpm の公開鍵がインストールされていません
(3/7): postgresql10-pgroonga-2.1.4-1.el7.x86_64.rpm                                    | 113 kB  00:00:00
(4/7): groonga-libs-8.0.7-1.el7.x86_64.rpm                                             | 1.9 MB  00:00:00
(5/7): postgresql10-libs-10.5-1PGDG.rhel7.x86_64.rpm                                   | 354 kB  00:00:01
(6/7): postgresql10-10.5-1PGDG.rhel7.x86_64.rpm                                        | 1.6 MB  00:00:02
(7/7): postgresql10-server-10.5-1PGDG.rhel7.x86_64.rpm                                 | 4.5 MB  00:00:01
--------------------------------------------------------------------------------------------------------------
合計                                                                          2.9 MB/s | 8.7 MB  00:00:02
file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 から鍵を取得中です。
Importing GPG key 0x352C64E5:
 Userid     : "Fedora EPEL (7) <epel@fedoraproject.org>"
 Fingerprint: 91e9 7d7c 4a5e 96f1 7f3e 888f 6a2f aea2 352c 64e5
 Package    : epel-release-7-11.noarch (@extras)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
file:///etc/pki/rpm-gpg/RPM-GPG-KEY-groonga から鍵を取得中です。
Importing GPG key 0x45499429:
 Userid     : "groonga Key (groonga Official Signing Key) <packages@groonga.org>"
 Fingerprint: c97e 4649 a205 1d0c ea1a 73f9 72a7 496b 4549 9429
 Package    : groonga-release-1.4.0-1.noarch (installed)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-groonga
file:///etc/pki/rpm-gpg/RPM-GPG-KEY-groonga-RSA4096 から鍵を取得中です。
Importing GPG key 0x34839225:
 Userid     : "Groonga Key (Groonga Official Signing Key) <packages@groonga.org>"
 Fingerprint: 2701 f317 cfcc cb97 5cad e9c2 624c f774 3483 9225
 Package    : groonga-release-1.4.0-1.noarch (installed)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-groonga-RSA4096
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  インストール中          : msgpack-1.4.1-1.el7.x86_64                                                    1/7
  インストール中          : postgresql10-libs-10.5-1PGDG.rhel7.x86_64                                     2/7
  インストール中          : postgresql10-10.5-1PGDG.rhel7.x86_64                                          3/7
  インストール中          : postgresql10-server-10.5-1PGDG.rhel7.x86_64                                   4/7
  インストール中          : libzstd-1.3.5-1.el7.x86_64                                                    5/7
  インストール中          : groonga-libs-8.0.7-1.el7.x86_64                                               6/7
  インストール中          : postgresql10-pgroonga-2.1.4-1.el7.x86_64                                      7/7
  検証中                  : postgresql10-libs-10.5-1PGDG.rhel7.x86_64                                     1/7
  検証中                  : libzstd-1.3.5-1.el7.x86_64                                                    2/7
  検証中                  : postgresql10-server-10.5-1PGDG.rhel7.x86_64                                   3/7
  検証中                  : groonga-libs-8.0.7-1.el7.x86_64                                               4/7
  検証中                  : postgresql10-pgroonga-2.1.4-1.el7.x86_64                                      5/7
  検証中                  : postgresql10-10.5-1PGDG.rhel7.x86_64                                          6/7
  検証中                  : msgpack-1.4.1-1.el7.x86_64                                                    7/7

インストール:
  postgresql10-pgroonga.x86_64 0:2.1.4-1.el7

依存性関連をインストールしました:
  groonga-libs.x86_64 0:8.0.7-1.el7                    libzstd.x86_64 0:1.3.5-1.el7
  msgpack.x86_64 0:1.4.1-1.el7                         postgresql10.x86_64 0:10.5-1PGDG.rhel7
  postgresql10-libs.x86_64 0:10.5-1PGDG.rhel7          postgresql10-server.x86_64 0:10.5-1PGDG.rhel7

完了しました!
```

```terminal
# 以下パッケージのインストールを確認
$ rpm -qa|egrep '(msgpack|zstd|groonga|postgresql)'
groonga-libs-8.0.7-1.el7.x86_64
postgresql10-10.5-1PGDG.rhel7.x86_64
libzstd-1.3.5-1.el7.x86_64
postgresql10-pgroonga-2.1.4-1.el7.x86_64
postgresql10-libs-10.5-1PGDG.rhel7.x86_64
postgresql10-server-10.5-1PGDG.rhel7.x86_64
groonga-release-1.4.0-1.noarch
msgpack-1.4.1-1.el7.x86_64

# pgroongaエクステンション確認
$ ldd /usr/pgsql-10/lib/pgroonga.so
        linux-vdso.so.1 =>  (0x00007ffe3a3c4000)
        libmsgpackc.so.2 => /lib64/libmsgpackc.so.2 (0x00007f073f032000)
        libgroonga.so.0 => /lib64/libgroonga.so.0 (0x00007f073e4f8000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f073e12b000)
        liblz4.so.1 => /lib64/liblz4.so.1 (0x00007f073df16000)
        libzstd.so.1 => /lib64/libzstd.so.1 (0x00007f073dc7f000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f073da69000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f073d865000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f073d649000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f073d342000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f073d040000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f073ce2a000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f073f477000)

```
問題無さそう

## PostgreSQLセットアップ
同じく以下のドキュメントを参考にPostgreSQLのセットアップを行う
https://pgroonga.github.io/install/centos.html#install-on-7

* データベース初期化

```terminal
# 環境確認
$ ll /etc/localtime
lrwxrwxrwx. 1 root root 32 10月  3  2018 /etc/localtime -> ../usr/share/zoneinfo/Asia/Tokyo
$ date
2018年 10月  3日 水曜日 14:13:22 JST
$ sudo -i -u postgres echo $LANG
ja_JP.UTF-8


# DB初期化(locale=Cで初期化)
$ sudo -H PGSETUP_INITDB_OPTIONS='--no-locale' /usr/pgsql-10/bin/postgresql-10-setup initdb
Initializing database ... OK

# 初期化メッセージ確認
$ sudo -u postgres cat /var/lib/pgsql/10/initdb.log
データベースシステム内のファイルの所有者は"postgres"となります。
このユーザがサーバプロセスも所有する必要があります。

データベースクラスタはロケール"C"で初期化されます。
そのためデフォルトのデータベース符号化方式はSQL_ASCIIに設定されました。
デフォルトのテキスト検索設定はenglishに設定されました。

データベージのチェックサムは無効です。

既存のディレクトリ/var/lib/pgsql/10/dataの権限を修正します ... 完了
サブディレクトリを作成します ... 完了
max_connectionsのデフォルト値を選択します ... 100
shared_buffersのデフォルト値を選択します ... 128MB
動的共有メモリの実装を選択します ... posix
設定ファイルを作成します ... 完了
ブートストラップスクリプトを実行します ... 完了
ブートストラップ後の初期化を行っています ... 完了
データをディスクに同期します...完了

成功しました。以下のようにしてデータベースサーバを起動できます。

    /usr/pgsql-10/bin/pg_ctl -D /var/lib/pgsql/10/data/ -l <ログファイル> start
```

* 起動設定&起動

```terminal
$ sudo -H systemctl enable postgresql-10
Created symlink from /etc/systemd/system/multi-user.target.wants/postgresql-10.service to /usr/lib/systemd/system/postgresql-10.service.
$ sudo -H systemctl start postgresql-10
$ ps -ef|grep postg
postgres  23183      1  0 14:26 ?        00:00:00 /usr/pgsql-10/bin/postmaster -D /var/lib/pgsql/10/data/
postgres  23185  23183  0 14:26 ?        00:00:00 postgres: logger process
postgres  23187  23183  0 14:26 ?        00:00:00 postgres: checkpointer process
postgres  23188  23183  0 14:26 ?        00:00:00 postgres: writer process
postgres  23189  23183  0 14:26 ?        00:00:00 postgres: wal writer process
postgres  23190  23183  0 14:26 ?        00:00:00 postgres: autovacuum launcher process
postgres  23191  23183  0 14:26 ?        00:00:00 postgres: stats collector process
postgres  23192  23183  0 14:26 ?        00:00:00 postgres: bgworker: logical replication launcher

$ sudo -i -u postgres psql
psql (10.5)
"help" でヘルプを表示します。

postgres=# \l
                                        データベース一覧
   名前    |  所有者  | エンコーディング | 照合順序 | Ctype(変換演算子) |     アクセス権限
-----------+----------+------------------+----------+-------------------+-----------------------
 postgres  | postgres | SQL_ASCII        | C        | C                 |
 template0 | postgres | SQL_ASCII        | C        | C                 | =c/postgres          +
           |          |                  |          |                   | postgres=CTc/postgres
 template1 | postgres | SQL_ASCII        | C        | C                 | =c/postgres          +
           |          |                  |          |                   | postgres=CTc/postgres
(3 行)

```

* データベース作成&extension登録

```terminal
$ sudo -i -u postgres -H psql --command "CREATE DATABASE pgroonga_test WITH ENCODING='UTF-8' template=template0"
CREATE DATABASE
$ sudo -i -u postgres -H psql -d pgroonga_test --command 'CREATE EXTENSION pgroonga'
CREATE EXTENSION
$ sudo -i -u postgres -H psql -d pgroonga_test --command '\l'
                                          データベース一覧
     名前      |  所有者  | エンコーディング | 照合順序 | Ctype(変換演算子) |     アクセス権限
---------------+----------+------------------+----------+-------------------+-----------------------
 pgroonga_test | postgres | UTF8             | C        | C                 |
 postgres      | postgres | SQL_ASCII        | C        | C                 |
 template0     | postgres | SQL_ASCII        | C        | C                 | =c/postgres          +
               |          |                  |          |                   | postgres=CTc/postgres
 template1     | postgres | SQL_ASCII        | C        | C                 | =c/postgres          +
               |          |                  |          |                   | postgres=CTc/postgres
(4 行)

$ sudo -i -u postgres -H psql -d pgroonga_test --command '\dx'
                                             インストール済みの拡張一覧
   名前   | バージョン |  スキーマ  |                                      説明

----------+------------+------------+-------------------------------------------------------------------------
-------
 pgroonga | 2.1.4      | public     | Super fast and all languages supported full text search index based on G
roonga
 plpgsql  | 1.0        | pg_catalog | PL/pgSQL procedural language
(2 行)
```

## PGroongaテスト

* 上で作成したDBに接続する

```terminal
$ sudo -i -u postgres -H psql -d pgroonga_test
[sudo] pgrn3 のパスワード:
psql (10.5)
"help" でヘルプを表示します。

pgroonga_test=#
```

* テストデータを作る

```sql
--テーブル&index作成&サンプルデータ投入
CREATE TABLE memos (
  id integer,
  content text
);
CREATE INDEX pgroonga_content_index ON memos USING pgroonga (content);
INSERT INTO memos VALUES (1, 'PostgreSQL is a relational database management system.');
INSERT INTO memos VALUES (2, 'Groonga is a fast full text search engine that supports all languages.');
INSERT INTO memos VALUES (3, 'PGroonga is a PostgreSQL extension that uses Groonga as index.');
INSERT INTO memos VALUES (4, 'There is groonga command.');
INSERT INTO memos VALUES (10, 'あいうえお');
INSERT INTO memos VALUES (11, 'かきくけこ');
INSERT INTO memos VALUES (12, 'さしすせそ');
INSERT INTO memos VALUES (13, 'たちつてと');
INSERT INTO memos VALUES (14, 'なにぬねの');
INSERT INTO memos VALUES (15, 'はひふへほ');
INSERT INTO memos VALUES (16, 'まみむめも');
INSERT INTO memos VALUES (17, 'やゆよ');
INSERT INTO memos VALUES (18, 'らりるれろ');
INSERT INTO memos VALUES (19, 'わをん');
```

```terminal
#--- 結果
pgroonga_test=# \d+ memos
                                  テーブル "public.memos"
   列    |   型    | 照合順序 | Null 値を許容 | デフォルト | ストレージ | 統計の対象 | 説明
---------+---------+----------+---------------+------------+------------+------------+------
 id      | integer |          |               |            | plain      |            |
 content | text    |          |               |            | extended   |            |
インデックス:
    "pgroonga_content_index" pgroonga (content)

pgroonga_test=#
pgroonga_test=# select * from memos;
 id |                                content
----+------------------------------------------------------------------------
  1 | PostgreSQL is a relational database management system.
  2 | Groonga is a fast full text search engine that supports all languages.
  3 | PGroonga is a PostgreSQL extension that uses Groonga as index.
  4 | There is groonga command.
 10 | あいうえお
 11 | かきくけこ
 12 | さしすせそ
 13 | たちつてと
 14 | なにぬねの
 15 | はひふへほ
 16 | まみむめも
 17 | やゆよ
 18 | らりるれろ
 19 | わをん
(14 行)

```

* 検証

```terminal
pgroonga_test=# SET enable_seqscan = off;
SET
pgroonga_test=# SELECT * FROM memos WHERE content &@~ 'PGroonga OR PostgreSQL';
 id |                            content
----+----------------------------------------------------------------
  3 | PGroonga is a PostgreSQL extension that uses Groonga as index.
  1 | PostgreSQL is a relational database management system.
(2 行)

pgroonga_test=# explain SELECT * FROM memos WHERE content &@~ 'PGroonga OR PostgreSQL';
                                      QUERY PLAN
--------------------------------------------------------------------------------------
 Index Scan using pgroonga_content_index on memos  (cost=0.00..43.63 rows=1 width=36)
   Index Cond: (content &@~ 'PGroonga OR PostgreSQL'::text)
(2 行)
pgroonga_test=# SELECT * FROM memos WHERE content &@~ 'きく';
 id |  content
----+------------
 11 | かきくけこ
(1 行)

pgroonga_test=# explain SELECT * FROM memos WHERE content &@~ 'きく';
                                      QUERY PLAN
--------------------------------------------------------------------------------------
 Index Scan using pgroonga_content_index on memos  (cost=0.00..40.91 rows=1 width=36)
   Index Cond: (content &@~ 'きく'::text)
(2 行)

pgroonga_test=# INSERT INTO memos VALUES (20, 'あめんぼあかいなあいうえお');
INSERT 0 1
pgroonga_test=# SELECT * FROM MEMOS;
 id |                                content
----+------------------------------------------------------------------------
  1 | PostgreSQL is a relational database management system.
  2 | Groonga is a fast full text search engine that supports all languages.
  3 | PGroonga is a PostgreSQL extension that uses Groonga as index.
  4 | There is groonga command.
 10 | あいうえお
 11 | かきくけこ
 12 | さしすせそ
 13 | たちつてと
 14 | なにぬねの
 15 | はひふへほ
 16 | まみむめも
 17 | やゆよ
 18 | らりるれろ
 19 | わをん
 20 | あめんぼあかいなあいうえお
(15 行)

pgroonga_test=# explain SELECT * FROM MEMOS;
                        QUERY PLAN
----------------------------------------------------------
 Seq Scan on memos  (cost=0.00..22.70 rows=1270 width=36)
(1 行)

pgroonga_test=# SELECT * FROM memos WHERE content &@~ 'あいう';
 id |          content
----+----------------------------
 10 | あいうえお
 20 | あめんぼあかいなあいうえお
(2 行)

pgroonga_test=# explain SELECT * FROM memos WHERE content &@~ 'あいう';
                                      QUERY PLAN
--------------------------------------------------------------------------------------
 Index Scan using pgroonga_content_index on memos  (cost=0.00..42.54 rows=1 width=36)
   Index Cond: (content &@~ 'あいう'::text)
(2 行)

```
問題無さそう


## スレーブノード構築
VMwareで構築した上記マスターノードをクローンし、IPアドレス/ホスト名を変更する。

```terminal
$ sudo vi -es /etc/sysconfig/network-scripts/ifcfg-ens33
16c
IPADDR="192.168.50.131"
.
:wq!
$ sudo vi -es /etc/hostname
1c
pgrntest2
.
:wq!
# 再起動
$ sudo shutdown -r now
```

変更&疎通確認

* マスターノード(そのまま): pgrntest1

```terminal
[pgrn3@pgrntest1 ~]$ ip a show dev ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:9f:88:5f brd ff:ff:ff:ff:ff:ff
    inet 192.168.50.130/24 brd 192.168.50.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::cdbb:125d:48d3:c858/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
[pgrn3@pgrntest1 ~]$
[pgrn3@pgrntest1 ~]$ ping -c 2 pgrntest2
PING pgrntest2 (192.168.50.131) 56(84) bytes of data.
64 bytes from pgrntest2 (192.168.50.131): icmp_seq=1 ttl=64 time=0.243 ms
64 bytes from pgrntest2 (192.168.50.131): icmp_seq=2 ttl=64 time=0.274 ms

--- pgrntest2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.243/0.258/0.274/0.022 ms
```

* スレーブノード: pgrntest2

```terminal
[pgrn3@pgrntest2 ~]$ ip a show dev ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:ef:f3:5b brd ff:ff:ff:ff:ff:ff
    inet 192.168.50.131/24 brd 192.168.50.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::5aa1:1de3:d820:c978/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
    inet6 fe80::cdbb:125d:48d3:c858/64 scope link tentative noprefixroute dadfailed
       valid_lft forever preferred_lft forever
[pgrn3@pgrntest2 ~]$
[pgrn3@pgrntest2 ~]$ ping -c 2 pgrntest1
PING pgrntest1 (192.168.50.130) 56(84) bytes of data.
64 bytes from pgrntest1 (192.168.50.130): icmp_seq=1 ttl=64 time=0.173 ms
64 bytes from pgrntest1 (192.168.50.130): icmp_seq=2 ttl=64 time=0.245 ms

--- pgrntest1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.173/0.209/0.245/0.036 ms
```

## レプリケーション設定
以下の手順を参考にするが、
https://pgroonga.github.io/ja/reference/replication.html
手順3迄は上記でここまでで完了しているので割愛する。

* 1. 割愛 [通常] マスターとスレーブでPostgreSQLをインストールする
* 2. 割愛 [固有] マスターとスレーブでPGroongaをインストールする
* 3. 割愛 [通常] マスターでPostgreSQLのデータベース初期化する
* 4. [通常] マスターで`postgresql.conf`と`pg_hba.conf`にストリーミングレプリケーション用の設定を追加する


```terminal
# /var/lib/pgsql/10/data/postgresql.conf へ以下の様に変更を加える
$ diff -urN /var/lib/pgsql/10/data/postgresql.conf.org /var/lib/pgsql/10/data/postgresql.conf
--- /var/lib/pgsql/10/data/postgresql.conf.org  2018-10-03 14:22:33.602051241 +0900
+++ /var/lib/pgsql/10/data/postgresql.conf      2018-10-03 16:06:19.017563829 +0900
@@ -56,7 +56,7 @@

 # - Connection Settings -

-#listen_addresses = 'localhost'                # what IP address(es) to listen on;
+listen_addresses = '*'         # what IP address(es) to listen on;
                                        # comma-separated list of addresses;
                                        # defaults to 'localhost'; use '*' for all
                                        # (change requires restart)
@@ -177,7 +177,7 @@

 # - Settings -

-#wal_level = replica                   # minimal, replica, or logical
+wal_level = replica                    # minimal, replica, or logical
                                        # (change requires restart)
 #fsync = on                            # flush data to disk for crash safety
                                        # (turning this off can cause
@@ -232,7 +232,7 @@

 # Set these on the master and on any standby that will send replication data.

-#max_wal_senders = 10          # max number of walsender processes
+max_wal_senders = 10           # max number of walsender processes
                                # (change requires restart)
 #wal_keep_segments = 0         # in logfile segments, 16MB each; 0 disables
 #wal_sender_timeout = 60s      # in milliseconds; 0 disables
```

```terminal
# /var/lib/pgsql/10/data/pg_hba.conf へ以下の様に変更を加える
# テストなので全許可にしてみる
$ diff -urN /var/lib/pgsql/10/data/pg_hba.conf.org /var/lib/pgsql/10/data/pg_hba.conf
--- /var/lib/pgsql/10/data/pg_hba.conf.org      2018-10-03 14:22:33.602051241 +0900
+++ /var/lib/pgsql/10/data/pg_hba.conf  2018-10-03 16:12:08.148306883 +0900
@@ -87,3 +87,8 @@
 local   replication     all                                     peer
 host    replication     all             127.0.0.1/32            ident
 host    replication     all             ::1/128                 ident
+
+# test setting
+host    all             all             0/0            trust
+host    replication     all             0/0            trust
+

```

```terminal
# 反映と確認を行う
[pgrn3@pgrntest1 ~]$ sudo systemctl restart postgresql-10
[sudo] pgrn3 のパスワード:
[pgrn3@pgrntest1 ~]$ sudo -i -u postgres -H psql -d pgroonga_test
psql (10.5)
"help" でヘルプを表示します。

pgroonga_test=# select name,setting from pg_settings where name ~ '(listen_address|wal_level|max_wal_senders)';
       name       | setting
------------------+---------
 listen_addresses | *
 max_wal_senders  | 10
 wal_level        | replica
(3 行)

```

* 5. [固有] マスターでpostgresql.confにPGroonga関連の設定を追加する

```terminal
# /var/lib/pgsql/10/data/postgresql.conf へ以下の様に追記する
$ diff -urN /var/lib/pgsql/10/data/postgresql.conf.bak  /var/lib/pgsql/10/data/postgresql.conf
--- /var/lib/pgsql/10/data/postgresql.conf.bak  2018-10-03 16:06:19.017563829 +0900
+++ /var/lib/pgsql/10/data/postgresql.conf      2018-10-03 16:25:15.772807622 +0900
@@ -656,3 +656,7 @@
 #------------------------------------------------------------------------------

 # Add settings for extensions here
+
+# pgroonga settings
+pgroonga.enable_wal = on
+pgroonga.log_level = debug

```

```terminal
# 反映と確認を行う
[pgrn3@pgrntest1 ~]$ sudo systemctl restart postgresql-10
[sudo] pgrn3 のパスワード:
[pgrn3@pgrntest1 ~]$ sudo -i -u postgres -H psql -d pgroonga_test
psql (10.5)
"help" でヘルプを表示します。
pgroonga_test=# show pgroonga.enable_wal;
 pgroonga.enable_wal
---------------------
 on
(1 行)

pgroonga_test=# show pgroonga.log_level;
 pgroonga.log_level
--------------------
 debug
(1 行)
```


* 6. [通常] マスターでデータを投入する

マスターでユーザを作成する
 
```terminal
[pgrn3@pgrntest1 ~]$ sudo -i -u postgres -H createuser ${USER}
```
マスターでDBを作成する
 
```terminal
[pgrn3@pgrntest1 ~]$ sudo -i -u postgres -H createdb --locale C --encoding UTF-8 --owner ${USER} --template template0 blog
```

作成したblogデータベースへ接続する

```terminal
[pgrn3@pgrntest1 ~]$ psql blog
psql (10.5)
"help" でヘルプを表示します。

blog=>
```

entriesテーブル作成&サンプルデータを追加

```sql
CREATE TABLE entries (
  title text,
  body text
);
INSERT INTO entries VALUES ('PGroonga', 'PGroonga is a PostgreSQL extension for fast full text search that supports all languages. It will help us.');
INSERT INTO entries VALUES ('Groonga', 'Groonga is a full text search engine used by PGroonga. We did not know about it.');
INSERT INTO entries VALUES ('PGroonga and replication', 'PGroonga 1.1.6 supports WAL based streaming replication. We should try it!');
```

結果確認

```sql
blog=> SELECT * FROM entries;
          title           |                                                    body

--------------------------+----------------------------------------------------------------------------------
-------------------------
 PGroonga                 | PGroonga is a PostgreSQL extension for fast full text search that supports all la
guages. It will help us.
 Groonga                  | Groonga is a full text search engine used by PGroonga. We did not know about it.
 PGroonga and replication | PGroonga 1.1.6 supports WAL based streaming replication. We should try it!
(3 行)
```

* 7. [固有] マスターでPGroongaのインデックスを作成する

このデータベースにPGroongaをインストールする。(DBスーパーユーザで実行)

```terminal
[pgrn3@pgrntest1 ~]$ sudo -i -u postgres -H psql blog --command "CREATE EXTENSION pgroonga;"
CREATE EXTENSION
[pgrn3@pgrntest1 ~]$ sudo -i -u postgres -H psql blog --command "GRANT USAGE ON SCHEMA pgroonga TO ${USER};"
GRANT
```

一般ユーザで接続確認

```terminal
[pgrn3@pgrntest1 ~]$ psql blog
psql (10.5)
"help" でヘルプを表示します。

blog=>
```

PGroongaインデックスの作成

```terminal
blog=> CREATE INDEX entries_full_text_search ON entries USING pgroonga (title, body);
CREATE INDEX
blog=> \d+ entries
                               テーブル "public.entries"
  列   |  型  | 照合順序 | Null 値を許容 | デフォルト | ストレージ | 統計の対象 | 説明
-------+------+----------+---------------+------------+------------+------------+------
 title | text |          |               |            | extended   |            |
 body  | text |          |               |            | extended   |            |
インデックス:
    "entries_full_text_search" pgroonga (title, body)

blog=>

```

作成したindexの確認

```terminal
blog=> SET enable_seqscan TO off;
SET
blog=> SELECT title FROM entries WHERE title %% 'replication';
          title
--------------------------
 PGroonga and replication
(1 行)

blog=> explain SELECT title FROM entries WHERE title %% 'replication';
                                          QUERY PLAN
----------------------------------------------------------------------------------------------
 Index Only Scan using entries_full_text_search on entries  (cost=0.00..4.01 rows=1 width=32)
   Index Cond: (title %% 'replication'::text)
(2 行)

blog=>

```


* 8. [固有] マスターでPGroonga関連のデータをフラッシュする

```terminal
[pgrn3@pgrntest1 ~]$ sudo -i -u postgres -H psql blog
psql (10.5)
"help" でヘルプを表示します。

blog=# SELECT pgroonga_command('io_flush') AS command;
                     command
-------------------------------------------------
 [[0,1538553024.67985,0.03089427947998047],true]
(1 行)

blog=#

```

* 9. [通常] スレーブでpg_basebackupを実行する

スレーブでpg_basebackupを実行し、データベースクラスタのコピーを行なう

```terminal
# DBを一度停止し、データベースクラスタを削除する
[pgrn3@pgrntest2 ~]$ sudo systemctl stop postgresql-10
[sudo] pgrn3 のパスワード:
[pgrn3@pgrntest2 ~]$
[pgrn3@pgrntest2 ~]$ sudo su -c 'rm -rf /var/lib/pgsql/10/data/*'
```

```
[pgrn3@pgrntest2 ~]$ sudo -i -u postgres -H pg_basebackup --host pgrntest1 --pgdata /var/lib/pgsql/10/data -X fetch --progress --username postgres --password --write-recovery-conf
[sudo] pgrn3 のパスワード:
330115/330115 kB (100%), 1/1 テーブル空間
[pgrn3@pgrntest2 ~]$
```

* 10. [通常] スレーブでpostgresql.confにストリーミングレプリケーション用の設定を追加する

```terminal
# /var/lib/pgsql/10/data/postgresql.conf を以下の様に修正する
$ diff -urN /var/lib/pgsql/10/data/postgresql.conf.slaveorg  /var/lib/pgsql/10/data/postgresql.conf
--- /var/lib/pgsql/10/data/postgresql.conf.slaveorg     2018-10-03 17:20:49.397952715 +0900
+++ /var/lib/pgsql/10/data/postgresql.conf      2018-10-03 17:23:41.649789358 +0900
@@ -256,7 +256,7 @@

 # These settings are ignored on a master server.

-#hot_standby = on                      # "off" disallows queries during recovery
+hot_standby = on                       # "off" disallows queries during recovery
                                        # (change requires restart)
 #max_standby_archive_delay = 30s       # max delay before canceling queries
                                        # when reading WAL from archive;
```

ログを削除しておく

```terminal
$ sudo rm -f /var/lib/pgsql/10/data/pgroonga.log
```

追記:スレーブ側でもpostgresql.confに以下の設定が入っている事

```
# pgroonga settings
pgroonga.enable_wal = on
pgroonga.log_level = debug
```

* 11. [通常] スレーブでPostgreSQLを起動する

```terminal
[pgrn3@pgrntest2 ~]$ sudo systemctl start postgresql-10
[pgrn3@pgrntest2 ~]$
[pgrn3@pgrntest2 ~]$ ps -ef|grep postg
postgres  10958      1  0 17:26 ?        00:00:00 /usr/pgsql-10/bin/postmaster -D /var/lib/pgsql/10/data/
postgres  10960  10958  0 17:26 ?        00:00:00 postgres: logger process
postgres  10961  10958  0 17:26 ?        00:00:00 postgres: startup process   recovering 000000010000000000000003
postgres  10962  10958  0 17:26 ?        00:00:00 postgres: checkpointer process
postgres  10963  10958  0 17:26 ?        00:00:00 postgres: writer process
postgres  10964  10958  0 17:26 ?        00:00:00 postgres: stats collector process
postgres  10965  10958  0 17:26 ?        00:00:00 postgres: wal receiver process   streaming 0/3000140
pgrn3     10967  10492  0 17:26 pts/0    00:00:00 grep --color=auto postg

```

ログ確認

```terminal
[pgrn3@pgrntest2 ~]$ sudo tail -6 /var/lib/pgsql/10/data/log/postgresql-Wed.log
2018-10-03 17:26:27.647 JST [10961] LOG:  database system was interrupted; last known up at 2018-10-03 17:20:46 JST
2018-10-03 17:26:27.660 JST [10961] LOG:  entering standby mode
2018-10-03 17:26:27.661 JST [10961] LOG:  redo starts at 0/2000028
2018-10-03 17:26:27.662 JST [10961] LOG:  consistent recovery state reached at 0/2000130
2018-10-03 17:26:27.663 JST [10958] LOG:  database system is ready to accept read only connections
2018-10-03 17:26:27.674 JST [10965] LOG:  started streaming WAL from primary at 0/3000000 on timeline 1
[pgrn3@pgrntest2 ~]$
````

* スレーブ動作確認

```terminal
[pgrn3@pgrntest2 ~]$ psql blog
psql (10.5)
"help" でヘルプを表示します。

blog=>
blog=> SET enable_seqscan TO off;
SET
blog=> SELECT title FROM entries WHERE title %% 'replication';
          title
--------------------------
 PGroonga and replication
(1 行)

blog=> explain SELECT title FROM entries WHERE title %% 'replication';
                                          QUERY PLAN
----------------------------------------------------------------------------------------------
 Index Only Scan using entries_full_text_search on entries  (cost=0.00..4.01 rows=1 width=32)
   Index Cond: (title %% 'replication'::text)
(2 行)

blog=>

```
 ここまでは問題無し

マスターにデータを追加してスレーブに反映されるかテストしてみる
マスターノード:pgrmtest1

```terminal
[pgrn3@pgrntest1 ~]$ psql blog
psql (10.5)
"help" でヘルプを表示します。

blog=> INSERT INTO entries VALUES ('PostgreSQL 9.6 and replication', 'PostgreSQL supports generic WAL since 9.6. It is required for replication for PGroonga.');
INSERT 0 1
blog=>
blog=> SELECT * FROM entries;
             title              |                                                    body

--------------------------------+----------------------------------------------------------------------------
--------------------------------
 PGroonga                       | PGroonga is a PostgreSQL extension for fast full text search that supports
all languages. It will help us.
 Groonga                        | Groonga is a full text search engine used by PGroonga. We did not know abou
t it.
 PGroonga and replication       | PGroonga 1.1.6 supports WAL based streaming replication. We should try it!
 PostgreSQL 9.6 and replication | PostgreSQL supports generic WAL since 9.6. It is required for replication f
or PGroonga.
(4 行)

blog=>
blog=> explain SELECT * FROM entries;
                       QUERY PLAN
--------------------------------------------------------
 Seq Scan on entries  (cost=0.00..1.03 rows=3 width=64)
(1 行)

blog=>
blog=> SET enable_seqscan TO off;
SET
blog=> SELECT title FROM entries WHERE title %% 'replication';
             title
--------------------------------
 PGroonga and replication
 PostgreSQL 9.6 and replication
(2 行)

blog=> explain SELECT title FROM entries WHERE title %% 'replication';
                                          QUERY PLAN
----------------------------------------------------------------------------------------------
 Index Only Scan using entries_full_text_search on entries  (cost=0.00..4.02 rows=1 width=32)
   Index Cond: (title %% 'replication'::text)
(2 行)
```
マスターノードは問題なし



スレーブノード:pgrntest2

```terminal
blog=> SET enable_seqscan TO off;
SET
blog=> SELECT title FROM entries WHERE title %% 'replication';
ERROR:  cannot acquire lock mode ShareUpdateExclusiveLock on database objects while recovery is in progress
HINT:  Only RowExclusiveLock or less can be acquired on database objects during recovery.
blog=>
blog=> SELECT * FROM entries;
ERROR:  cannot acquire lock mode ShareUpdateExclusiveLock on database objects while recovery is in progress
HINT:  Only RowExclusiveLock or less can be acquired on database objects during recovery.
blog=>
blog=> explain SELECT * FROM entries;
ERROR:  cannot acquire lock mode ShareUpdateExclusiveLock on database objects while recovery is in progress
HINT:  Only RowExclusiveLock or less can be acquired on database objects during recovery.
blog=>

```

エラーになってしまう。
スレーブ側pgroonga.log(全)

```terminal

[pgrn3@pgrntest2 ~]$ sudo cat /var/lib/pgsql/10/data/pgroonga.log
[sudo] pgrn3 のパスワード:
2018-10-03 19:20:12.826337|n|14449: grn_init: <8.0.7>
2018-10-03 19:20:12.826915|n|14449: pgroonga: initialize: <2.1.4>
2018-10-03 19:20:12.845614|d|14449: pgroonga: [initialize][scan-opaque][start] 0
2018-10-03 19:20:12.845648|d|14449: pgroonga: [initialize][scan-opaque][end] 1: <0x125b9c0>
2018-10-03 19:20:12.845652|d|14449: pgroonga: [scan][begin] <0x125b9c0>
2018-10-03 19:20:12.846072|i|14449: [object][search][index][key][exact] <Lexicon20423_0.index>
2018-10-03 19:20:12.846091|i|14449: grn_ii_sel > (replication)
2018-10-03 19:20:12.846122|i|14449: n=1 (replication)
2018-10-03 19:20:12.846417|i|14449: exact: 1
2018-10-03 19:20:12.846428|i|14449: hits=1
2018-10-03 19:20:12.846654|d|14449: pgroonga: [scan][end] <0x125b9c0>
2018-10-03 19:20:12.846665|d|14449: pgroonga: [release][scan-opaques][before-locks] 1: skip
2018-10-03 19:20:12.846669|d|14449: pgroonga: [release][scan-opaques][locks] 1: skip
2018-10-03 19:20:12.846672|d|14449: pgroonga: [release][scan-opaques][after-locks][start] 1
2018-10-03 19:20:12.846679|d|14449: pgroonga: [release][scan-opaques][top-level][before-locks] 1: skip
2018-10-03 19:20:12.846685|d|14449: pgroonga: [release][scan-opaques][top-level][locks] 1: skip
2018-10-03 19:20:12.846687|d|14449: pgroonga: [release][scan-opaques][top-level][after-locks][start] 1
2018-10-03 19:20:12.846690|d|14449: pgroonga: [finalize][scan-opaque][start] 1: <0x125b9c0>
2018-10-03 19:20:12.846860|d|14449: pgroonga: [finalize][scan-opaque][end] 0: <0x125b9c0>
2018-10-03 19:20:12.846866|d|14449: pgroonga: [release][scan-opaques][top-level][after-locks][end] 0
2018-10-03 19:20:21.685340|d|14449: pgroonga: [release][scan-opaques][before-locks] 0: skip
2018-10-03 19:20:21.685364|d|14449: pgroonga: [release][scan-opaques][locks] 0: skip
2018-10-03 19:20:21.685367|d|14449: pgroonga: [release][scan-opaques][after-locks][start] 0
2018-10-03 19:20:21.685374|d|14449: pgroonga: [release][scan-opaques][top-level][before-locks] 0: skip
2018-10-03 19:20:21.685379|d|14449: pgroonga: [release][scan-opaques][top-level][locks] 0: skip
2018-10-03 19:20:21.685382|d|14449: pgroonga: [release][scan-opaques][top-level][after-locks][start] 0
2018-10-03 19:20:21.685384|d|14449: pgroonga: [release][scan-opaques][top-level][after-locks][end] 0
2018-10-03 19:22:18.103721|d|14449: pgroonga: [release][scan-opaques][before-locks] 0: skip
2018-10-03 19:22:18.103768|d|14449: pgroonga: [release][scan-opaques][locks] 0: skip
2018-10-03 19:22:18.103772|d|14449: pgroonga: [release][scan-opaques][after-locks][start] 0
2018-10-03 19:22:18.103779|d|14449: pgroonga: [release][scan-opaques][top-level][before-locks] 0: skip
2018-10-03 19:22:18.103839|d|14449: pgroonga: [release][scan-opaques][top-level][locks] 0: skip
2018-10-03 19:22:18.103846|d|14449: pgroonga: [release][scan-opaques][top-level][after-locks][start] 0
2018-10-03 19:22:18.103849|d|14449: pgroonga: [release][scan-opaques][top-level][after-locks][end] 0
2018-10-03 19:22:29.571677|d|14449: pgroonga: [release][scan-opaques][top-level][before-locks] 0: skip
2018-10-03 19:22:29.571702|d|14449: pgroonga: [release][scan-opaques][top-level][locks] 0: skip
2018-10-03 19:22:29.571706|d|14449: pgroonga: [release][scan-opaques][top-level][after-locks][start] 0
2018-10-03 19:22:29.571709|d|14449: pgroonga: [release][scan-opaques][top-level][after-locks][end] 0
2018-10-03 19:22:37.803344|d|14449: pgroonga: [release][scan-opaques][top-level][before-locks] 0: skip
2018-10-03 19:22:37.803375|d|14449: pgroonga: [release][scan-opaques][top-level][locks] 0: skip
2018-10-03 19:22:37.803379|d|14449: pgroonga: [release][scan-opaques][top-level][after-locks][start] 0
2018-10-03 19:22:37.803397|d|14449: pgroonga: [release][scan-opaques][top-level][after-locks][end] 0
2018-10-03 19:22:45.790493|d|14449: pgroonga: [release][scan-opaques][top-level][before-locks] 0: skip
2018-10-03 19:22:45.790514|d|14449: pgroonga: [release][scan-opaques][top-level][before-locks] 0: skip
2018-10-03 19:22:45.790519|d|14449: pgroonga: [release][scan-opaques][top-level][locks] 0: skip
2018-10-03 19:22:45.790523|d|14449: pgroonga: [release][scan-opaques][top-level][locks] 0: skip
2018-10-03 19:22:45.790535|d|14449: pgroonga: [release][scan-opaques][top-level][after-locks][start] 0
2018-10-03 19:22:45.790538|d|14449: pgroonga: [release][scan-opaques][top-level][after-locks][end] 0
2018-10-03 19:22:45.790540|d|14449: pgroonga: [release][scan-opaques][top-level][after-locks][start] 0
2018-10-03 19:22:45.790543|d|14449: pgroonga: [release][scan-opaques][top-level][after-locks][end] 0
2018-10-03 19:22:55.939738|d|14449: pgroonga: [exit][db][opened]
2018-10-03 19:22:55.939765|d|14449: pgroonga: [exit][finalize][auto-close]
2018-10-03 19:22:55.940087|d|14449: pgroonga: [exit][finalize][query-extract-keywords]
2018-10-03 19:22:55.940352|d|14449: pgroonga: [exit][finalize][match-positions-byte]
2018-10-03 19:22:55.940686|d|14449: pgroonga: [exit][finalize][match-positions-character]
2018-10-03 19:22:55.940843|d|14449: pgroonga: [exit][finalize][highlight-html]
2018-10-03 19:22:55.940851|d|14449: pgroonga: [exit][finalize][keywords]
2018-10-03 19:22:55.940853|d|14449: pgroonga: [exit][finalize][jsonb]
2018-10-03 19:22:55.941891|d|14449: pgroonga: [exit][finalize][sequential-search-data]
2018-10-03 19:22:55.942894|d|14449: pgroonga: [exit][finalize][prefix-rk-sequential-search-data]
2018-10-03 19:22:55.943150|d|14449: pgroonga: [exit][finalize][options]
2018-10-03 19:22:55.943166|d|14449: pgroonga: [exit][db][close]
2018-10-03 19:22:55.945402|d|14449: pgroonga: [exit][finalize][buffers]
2018-10-03 19:22:55.945432|d|14449: pgroonga: [exit][finalize][context]
2018-10-03 19:22:55.947321|d|14449: pgroonga: [exit][finalize]
2018-10-03 19:22:55.949344|n|14449: grn_fin (0)

```

マスター側pgroonga.log (全)

```text
[pgrn3@pgrntest1 ~]$ sudo cat /var/lib/pgsql/10/data/pgroonga.log
[sudo] pgrn3 のパスワード:
2018-10-03 19:11:42.630492|n|11085: grn_init: <8.0.7>
2018-10-03 19:11:42.630880|n|11085: pgroonga: initialize: <2.1.4>
2018-10-03 19:11:42.636256|n|11085: spec:2:update:Object:32(type):8
2018-10-03 19:11:42.636960|n|11085: spec:3:update:Bool:32(type):1
2018-10-03 19:11:42.636980|n|11085: spec:4:update:Int8:32(type):1
2018-10-03 19:11:42.636984|n|11085: spec:5:update:UInt8:32(type):1
2018-10-03 19:11:42.636989|n|11085: spec:6:update:Int16:32(type):2
2018-10-03 19:11:42.636993|n|11085: spec:7:update:UInt16:32(type):2
2018-10-03 19:11:42.636997|n|11085: spec:8:update:Int32:32(type):4
2018-10-03 19:11:42.637001|n|11085: spec:9:update:UInt32:32(type):4
2018-10-03 19:11:42.637004|n|11085: spec:10:update:Int64:32(type):8
2018-10-03 19:11:42.637008|n|11085: spec:11:update:UInt64:32(type):8
2018-10-03 19:11:42.637013|n|11085: spec:12:update:Float:32(type):8
2018-10-03 19:11:42.637018|n|11085: spec:13:update:Time:32(type):8
2018-10-03 19:11:42.637022|n|11085: spec:14:update:ShortText:32(type):4096
2018-10-03 19:11:42.637025|n|11085: spec:15:update:Text:32(type):65536
2018-10-03 19:11:42.637029|n|11085: spec:16:update:LongText:32(type):2147483648
2018-10-03 19:11:42.637033|n|11085: spec:17:update:TokyoGeoPoint:32(type):8
2018-10-03 19:11:42.637037|n|11085: spec:18:update:WGS84GeoPoint:32(type):8
2018-10-03 19:11:42.656854|n|11085: DDL:257:table_create Aliases
2018-10-03 19:11:42.657365|n|11085: spec:257:update:Aliases:48(table:hash_key):0
2018-10-03 19:11:42.657401|n|11085: DDL:258:column_create Aliases real_name
2018-10-03 19:11:42.657783|n|11085: spec:258:update:Aliases.real_name:65(column:var_size):14(ShortText)
2018-10-03 19:11:42.661617|n|11085: DDL:259:table_create IndexStatuses
2018-10-03 19:11:42.662156|n|11085: spec:259:update:IndexStatuses:48(table:hash_key):0
2018-10-03 19:11:42.662185|n|11085: DDL:260:column_create IndexStatuses max_record_size
2018-10-03 19:11:42.662476|n|11085: spec:260:update:IndexStatuses.max_record_size:64(column:fix_size):9(UInt32)
2018-10-03 19:11:42.662494|n|11085: DDL:261:column_create IndexStatuses wal_applied_position
2018-10-03 19:11:42.662780|n|11085: spec:261:update:IndexStatuses.wal_applied_position:64(column:fix_size):11(UInt64)
2018-10-03 19:11:42.677489|d|11085: pgroonga: [release][scan-opaques][before-locks] 0: skip
2018-10-03 19:11:42.677518|d|11085: pgroonga: [release][scan-opaques][locks] 0: skip
2018-10-03 19:11:42.677521|d|11085: pgroonga: [release][scan-opaques][after-locks][start] 0
2018-10-03 19:11:42.700151|d|11085: pgroonga: [release][scan-opaques][before-locks] 0: skip
2018-10-03 19:11:42.700175|d|11085: pgroonga: [release][scan-opaques][locks] 0: skip
2018-10-03 19:11:42.700178|d|11085: pgroonga: [release][scan-opaques][after-locks][start] 0
2018-10-03 19:11:42.701372|d|11085: pgroonga: [release][scan-opaques][top-level][before-locks] 0: skip
2018-10-03 19:11:42.701491|d|11085: pgroonga: [release][scan-opaques][top-level][locks] 0: skip
2018-10-03 19:11:42.701496|d|11085: pgroonga: [release][scan-opaques][top-level][after-locks][start] 0
2018-10-03 19:11:42.701498|d|11085: pgroonga: [release][scan-opaques][top-level][after-locks][end] 0
2018-10-03 19:11:42.701742|d|11085: pgroonga: [exit][db][opened]
2018-10-03 19:11:42.701753|d|11085: pgroonga: [exit][finalize][auto-close]
2018-10-03 19:11:42.701774|d|11085: pgroonga: [exit][finalize][query-extract-keywords]
2018-10-03 19:11:42.702014|d|11085: pgroonga: [exit][finalize][match-positions-byte]
2018-10-03 19:11:42.702036|d|11085: pgroonga: [exit][finalize][match-positions-character]
2018-10-03 19:11:42.702053|d|11085: pgroonga: [exit][finalize][highlight-html]
2018-10-03 19:11:42.702059|d|11085: pgroonga: [exit][finalize][keywords]
2018-10-03 19:11:42.702062|d|11085: pgroonga: [exit][finalize][jsonb]
2018-10-03 19:11:42.702554|d|11085: pgroonga: [exit][finalize][sequential-search-data]
2018-10-03 19:11:42.702970|d|11085: pgroonga: [exit][finalize][prefix-rk-sequential-search-data]
2018-10-03 19:11:42.703003|d|11085: pgroonga: [exit][finalize][options]
2018-10-03 19:11:42.703008|d|11085: pgroonga: [exit][db][close]
2018-10-03 19:11:42.703884|d|11085: pgroonga: [exit][finalize][buffers]
2018-10-03 19:11:42.703910|d|11085: pgroonga: [exit][finalize][context]
2018-10-03 19:11:42.705253|d|11085: pgroonga: [exit][finalize]
2018-10-03 19:11:42.706229|n|11085: grn_fin (0)
2018-10-03 19:12:36.186031|n|11124: grn_init: <8.0.7>
2018-10-03 19:12:36.186290|n|11124: pgroonga: initialize: <2.1.4>
2018-10-03 19:12:36.205748|n|11124: DDL:263:table_create BuildingSources20423
2018-10-03 19:12:36.206497|n|11124: spec:263:update:BuildingSources20423:48(table:hash_key):0
2018-10-03 19:12:36.210717|n|11124: DDL:264:table_create Lexicon20423_0
2018-10-03 19:12:36.212450|n|11124: spec:264:update:Lexicon20423_0:49(table:pat_key):0
2018-10-03 19:12:36.213500|n|11124: DDL:265:column_create BuildingSources20423 title
2018-10-03 19:12:36.214141|n|11124: spec:265:update:BuildingSources20423.title:65(column:var_size):16(LongText)
2018-10-03 19:12:36.214197|n|11124: DDL:266:column_create Lexicon20423_0 index
2018-10-03 19:12:36.215416|n|11124: spec:266:update:Lexicon20423_0.index:72(column:index):263(BuildingSources20423)
2018-10-03 19:12:36.215488|n|11124: DDL:267:table_create Lexicon20423_1
2018-10-03 19:12:36.217083|n|11124: spec:267:update:Lexicon20423_1:49(table:pat_key):0
2018-10-03 19:12:36.217212|n|11124: DDL:268:column_create BuildingSources20423 body
2018-10-03 19:12:36.217826|n|11124: spec:268:update:BuildingSources20423.body:65(column:var_size):16(LongText)
2018-10-03 19:12:36.217870|n|11124: DDL:269:column_create Lexicon20423_1 index
2018-10-03 19:12:36.218852|n|11124: spec:269:update:Lexicon20423_1.index:72(column:index):263(BuildingSources20423)
2018-10-03 19:12:36.220997|d|11124: pgroonga: [insert] <entries_full_text_search>(20423): <1>: <(0,0),1>(1)
2018-10-03 19:12:36.224605|d|11124: pgroonga: [insert] <entries_full_text_search>(20423): <2>: <(0,0),2>(2)
2018-10-03 19:12:36.224654|d|11124: pgroonga: [insert] <entries_full_text_search>(20423): <3>: <(0,0),3>(3)
2018-10-03 19:12:36.225594|n|11124: DDL:266:set_source Lexicon20423_0.index BuildingSources20423.title
2018-10-03 19:12:36.225615|n|11124: spec:265:update:BuildingSources20423.title:65(column:var_size):16(LongText)
2018-10-03 19:12:36.229787|d|11124: n_terms = 1, chunk_offset = 7, chunk_size = 1048576, total = 1024KB
2018-10-03 19:12:36.229929|i|11124: [ii][builder][fin] removed path: <base/19751/pgrn.000010AZhZ9FK>
2018-10-03 19:12:36.230122|n|11124: spec:266:update:Lexicon20423_0.index:72(column:index):263(BuildingSources20423)
2018-10-03 19:12:36.230196|n|11124: DDL:269:set_source Lexicon20423_1.index BuildingSources20423.body
2018-10-03 19:12:36.230203|n|11124: spec:268:update:BuildingSources20423.body:65(column:var_size):16(LongText)
2018-10-03 19:12:36.233296|d|11124: n_terms = 11, chunk_offset = 89, chunk_size = 1048576, total = 1024KB
2018-10-03 19:12:36.233448|i|11124: [ii][builder][fin] removed path: <base/19751/pgrn.000010DDL5Wi6>
2018-10-03 19:12:36.233499|n|11124: spec:269:update:Lexicon20423_1.index:72(column:index):263(BuildingSources20423)
2018-10-03 19:12:36.237665|d|11124: pgroonga: [release][scan-opaques][before-locks] 0: skip
2018-10-03 19:12:36.237690|d|11124: pgroonga: [release][scan-opaques][locks] 0: skip
2018-10-03 19:12:36.237693|d|11124: pgroonga: [release][scan-opaques][after-locks][start] 0
2018-10-03 19:12:36.238004|d|11124: pgroonga: [release][scan-opaques][top-level][before-locks] 0: skip
2018-10-03 19:12:36.238026|d|11124: pgroonga: [release][scan-opaques][top-level][locks] 0: skip
2018-10-03 19:12:36.238030|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][start] 0
2018-10-03 19:12:36.238032|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][end] 0
2018-10-03 19:12:40.463459|d|11124: pgroonga: [release][scan-opaques][before-locks] 0: skip
2018-10-03 19:12:40.463486|d|11124: pgroonga: [release][scan-opaques][locks] 0: skip
2018-10-03 19:12:40.463489|d|11124: pgroonga: [release][scan-opaques][after-locks][start] 0
2018-10-03 19:12:40.463495|d|11124: pgroonga: [release][scan-opaques][top-level][before-locks] 0: skip
2018-10-03 19:12:40.463501|d|11124: pgroonga: [release][scan-opaques][top-level][locks] 0: skip
2018-10-03 19:12:40.463510|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][start] 0
2018-10-03 19:12:40.463513|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][end] 0
2018-10-03 19:12:40.464564|d|11124: pgroonga: [release][scan-opaques][before-locks] 0: skip
2018-10-03 19:12:40.464582|d|11124: pgroonga: [release][scan-opaques][locks] 0: skip
2018-10-03 19:12:40.464586|d|11124: pgroonga: [release][scan-opaques][after-locks][start] 0
2018-10-03 19:12:40.464591|d|11124: pgroonga: [release][scan-opaques][top-level][before-locks] 0: skip
2018-10-03 19:12:40.464595|d|11124: pgroonga: [release][scan-opaques][top-level][locks] 0: skip
2018-10-03 19:12:40.464598|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][start] 0
2018-10-03 19:12:40.464601|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][end] 0
2018-10-03 19:12:40.466310|d|11124: pgroonga: [release][scan-opaques][before-locks] 0: skip
2018-10-03 19:12:40.466329|d|11124: pgroonga: [release][scan-opaques][locks] 0: skip
2018-10-03 19:12:40.466332|d|11124: pgroonga: [release][scan-opaques][after-locks][start] 0
2018-10-03 19:12:40.466338|d|11124: pgroonga: [release][scan-opaques][top-level][before-locks] 0: skip
2018-10-03 19:12:40.466343|d|11124: pgroonga: [release][scan-opaques][top-level][locks] 0: skip
2018-10-03 19:12:40.466346|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][start] 0
2018-10-03 19:12:40.466348|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][end] 0
2018-10-03 19:12:40.466877|d|11124: pgroonga: [release][scan-opaques][before-locks] 0: skip
2018-10-03 19:12:40.466891|d|11124: pgroonga: [release][scan-opaques][locks] 0: skip
2018-10-03 19:12:40.466895|d|11124: pgroonga: [release][scan-opaques][after-locks][start] 0
2018-10-03 19:12:40.466899|d|11124: pgroonga: [release][scan-opaques][top-level][before-locks] 0: skip
2018-10-03 19:12:40.466904|d|11124: pgroonga: [release][scan-opaques][top-level][locks] 0: skip
2018-10-03 19:12:40.466907|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][start] 0
2018-10-03 19:12:40.466909|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][end] 0
2018-10-03 19:12:40.467978|d|11124: pgroonga: [release][scan-opaques][before-locks] 0: skip
2018-10-03 19:12:40.467998|d|11124: pgroonga: [release][scan-opaques][locks] 0: skip
2018-10-03 19:12:40.468001|d|11124: pgroonga: [release][scan-opaques][after-locks][start] 0
2018-10-03 19:12:40.468006|d|11124: pgroonga: [release][scan-opaques][top-level][before-locks] 0: skip
2018-10-03 19:12:40.468012|d|11124: pgroonga: [release][scan-opaques][top-level][locks] 0: skip
2018-10-03 19:12:40.468015|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][start] 0
2018-10-03 19:12:40.468017|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][end] 0
2018-10-03 19:12:40.469180|d|11124: pgroonga: [release][scan-opaques][before-locks] 0: skip
2018-10-03 19:12:40.469198|d|11124: pgroonga: [release][scan-opaques][locks] 0: skip
2018-10-03 19:12:40.469201|d|11124: pgroonga: [release][scan-opaques][after-locks][start] 0
2018-10-03 19:12:40.469207|d|11124: pgroonga: [release][scan-opaques][top-level][before-locks] 0: skip
2018-10-03 19:12:40.469213|d|11124: pgroonga: [release][scan-opaques][top-level][locks] 0: skip
2018-10-03 19:12:40.469216|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][start] 0
2018-10-03 19:12:40.469218|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][end] 0
2018-10-03 19:12:40.470166|d|11124: pgroonga: [release][scan-opaques][before-locks] 0: skip
2018-10-03 19:12:40.470185|d|11124: pgroonga: [release][scan-opaques][locks] 0: skip
2018-10-03 19:12:40.470189|d|11124: pgroonga: [release][scan-opaques][after-locks][start] 0
2018-10-03 19:12:40.470194|d|11124: pgroonga: [release][scan-opaques][top-level][before-locks] 0: skip
2018-10-03 19:12:40.470199|d|11124: pgroonga: [release][scan-opaques][top-level][locks] 0: skip
2018-10-03 19:12:40.470208|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][start] 0
2018-10-03 19:12:40.470211|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][end] 0
2018-10-03 19:12:40.470741|d|11124: pgroonga: [release][scan-opaques][before-locks] 0: skip
2018-10-03 19:12:40.470766|d|11124: pgroonga: [release][scan-opaques][locks] 0: skip
2018-10-03 19:12:40.470769|d|11124: pgroonga: [release][scan-opaques][after-locks][start] 0
2018-10-03 19:12:40.470774|d|11124: pgroonga: [release][scan-opaques][top-level][before-locks] 0: skip
2018-10-03 19:12:40.470779|d|11124: pgroonga: [release][scan-opaques][top-level][locks] 0: skip
2018-10-03 19:12:40.470781|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][start] 0
2018-10-03 19:12:40.470784|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][end] 0
2018-10-03 19:12:40.471208|d|11124: pgroonga: [release][scan-opaques][before-locks] 0: skip
2018-10-03 19:12:40.471220|d|11124: pgroonga: [release][scan-opaques][locks] 0: skip
2018-10-03 19:12:40.471223|d|11124: pgroonga: [release][scan-opaques][after-locks][start] 0
2018-10-03 19:12:40.471227|d|11124: pgroonga: [release][scan-opaques][top-level][before-locks] 0: skip
2018-10-03 19:12:40.471232|d|11124: pgroonga: [release][scan-opaques][top-level][locks] 0: skip
2018-10-03 19:12:40.471235|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][start] 0
2018-10-03 19:12:40.471237|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][end] 0
2018-10-03 19:12:40.471637|d|11124: pgroonga: [release][scan-opaques][before-locks] 0: skip
2018-10-03 19:12:40.471648|d|11124: pgroonga: [release][scan-opaques][locks] 0: skip
2018-10-03 19:12:40.471651|d|11124: pgroonga: [release][scan-opaques][after-locks][start] 0
2018-10-03 19:12:40.471656|d|11124: pgroonga: [release][scan-opaques][top-level][before-locks] 0: skip
2018-10-03 19:12:40.471661|d|11124: pgroonga: [release][scan-opaques][top-level][locks] 0: skip
2018-10-03 19:12:40.471663|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][start] 0
2018-10-03 19:12:40.471666|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][end] 0
2018-10-03 19:12:48.922784|d|11124: pgroonga: [release][scan-opaques][before-locks] 0: skip
2018-10-03 19:12:48.922815|d|11124: pgroonga: [release][scan-opaques][locks] 0: skip
2018-10-03 19:12:48.922818|d|11124: pgroonga: [release][scan-opaques][after-locks][start] 0
2018-10-03 19:12:48.922824|d|11124: pgroonga: [release][scan-opaques][top-level][before-locks] 0: skip
2018-10-03 19:12:48.922830|d|11124: pgroonga: [release][scan-opaques][top-level][locks] 0: skip
2018-10-03 19:12:48.922833|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][start] 0
2018-10-03 19:12:48.922835|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][end] 0
2018-10-03 19:12:53.843306|d|11124: pgroonga: [initialize][scan-opaque][start] 0
2018-10-03 19:12:53.843340|d|11124: pgroonga: [initialize][scan-opaque][end] 1: <0x2866e10>
2018-10-03 19:12:53.843344|d|11124: pgroonga: [scan][begin] <0x2866e10>
2018-10-03 19:12:53.843442|i|11124: [object][search][index][key][exact] <Lexicon20423_0.index>
2018-10-03 19:12:53.843533|i|11124: grn_ii_sel > (replication)
2018-10-03 19:12:53.843633|i|11124: n=1 (replication)
2018-10-03 19:12:53.845278|i|11124: exact: 1
2018-10-03 19:12:53.845338|i|11124: hits=1
2018-10-03 19:12:53.845493|d|11124: pgroonga: [scan][end] <0x2866e10>
2018-10-03 19:12:53.845504|d|11124: pgroonga: [release][scan-opaques][before-locks] 1: skip
2018-10-03 19:12:53.845509|d|11124: pgroonga: [release][scan-opaques][locks] 1: skip
2018-10-03 19:12:53.845512|d|11124: pgroonga: [release][scan-opaques][after-locks][start] 1
2018-10-03 19:12:53.845520|d|11124: pgroonga: [release][scan-opaques][top-level][before-locks] 1: skip
2018-10-03 19:12:53.845526|d|11124: pgroonga: [release][scan-opaques][top-level][locks] 1: skip
2018-10-03 19:12:53.845529|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][start] 1
2018-10-03 19:12:53.845531|d|11124: pgroonga: [finalize][scan-opaque][start] 1: <0x2866e10>
2018-10-03 19:12:53.845916|d|11124: pgroonga: [finalize][scan-opaque][end] 0: <0x2866e10>
2018-10-03 19:12:53.845936|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][end] 0
2018-10-03 19:12:59.780819|d|11124: pgroonga: [release][scan-opaques][before-locks] 0: skip
2018-10-03 19:12:59.780850|d|11124: pgroonga: [release][scan-opaques][locks] 0: skip
2018-10-03 19:12:59.780853|d|11124: pgroonga: [release][scan-opaques][after-locks][start] 0
2018-10-03 19:12:59.780860|d|11124: pgroonga: [release][scan-opaques][top-level][before-locks] 0: skip
2018-10-03 19:12:59.780866|d|11124: pgroonga: [release][scan-opaques][top-level][locks] 0: skip
2018-10-03 19:12:59.780869|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][start] 0
2018-10-03 19:12:59.780871|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][end] 0
2018-10-03 19:13:23.702407|d|11124: pgroonga: [release][scan-opaques][before-locks] 0: skip
2018-10-03 19:13:23.702427|d|11124: pgroonga: [release][scan-opaques][locks] 0: skip
2018-10-03 19:13:23.702430|d|11124: pgroonga: [release][scan-opaques][after-locks][start] 0
2018-10-03 19:13:23.702438|d|11124: pgroonga: [release][scan-opaques][top-level][before-locks] 0: skip
2018-10-03 19:13:23.702444|d|11124: pgroonga: [release][scan-opaques][top-level][locks] 0: skip
2018-10-03 19:13:23.702447|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][start] 0
2018-10-03 19:13:23.702449|d|11124: pgroonga: [release][scan-opaques][top-level][after-locks][end] 0
2018-10-03 19:13:24.707677|d|11124: pgroonga: [exit][db][opened]
2018-10-03 19:13:24.707704|d|11124: pgroonga: [exit][finalize][auto-close]
2018-10-03 19:13:24.708007|d|11124: pgroonga: [exit][finalize][query-extract-keywords]
2018-10-03 19:13:24.708296|d|11124: pgroonga: [exit][finalize][match-positions-byte]
2018-10-03 19:13:24.708319|d|11124: pgroonga: [exit][finalize][match-positions-character]
2018-10-03 19:13:24.708334|d|11124: pgroonga: [exit][finalize][highlight-html]
2018-10-03 19:13:24.708340|d|11124: pgroonga: [exit][finalize][keywords]
2018-10-03 19:13:24.708343|d|11124: pgroonga: [exit][finalize][jsonb]
2018-10-03 19:13:24.708958|d|11124: pgroonga: [exit][finalize][sequential-search-data]
2018-10-03 19:13:24.709405|d|11124: pgroonga: [exit][finalize][prefix-rk-sequential-search-data]
2018-10-03 19:13:24.709432|d|11124: pgroonga: [exit][finalize][options]
2018-10-03 19:13:24.709437|d|11124: pgroonga: [exit][db][close]
2018-10-03 19:13:24.712561|d|11124: pgroonga: [exit][finalize][buffers]
2018-10-03 19:13:24.712598|d|11124: pgroonga: [exit][finalize][context]
2018-10-03 19:13:24.714024|d|11124: pgroonga: [exit][finalize]
2018-10-03 19:13:24.714965|n|11124: grn_fin (0)
2018-10-03 19:21:24.977406|n|11160: grn_init: <8.0.7>
2018-10-03 19:21:24.977674|n|11160: pgroonga: initialize: <2.1.4>
2018-10-03 19:21:24.996674|d|11160: pgroonga: [insert] <entries_full_text_search>(20423): <4>: <(0,0),4>(4)
2018-10-03 19:21:24.998047|d|11160: pgroonga: [release][scan-opaques][before-locks] 0: skip
2018-10-03 19:21:24.998064|d|11160: pgroonga: [release][scan-opaques][locks] 0: skip
2018-10-03 19:21:24.998067|d|11160: pgroonga: [release][scan-opaques][after-locks][start] 0
2018-10-03 19:21:24.998300|d|11160: pgroonga: [release][scan-opaques][top-level][before-locks] 0: skip
2018-10-03 19:21:24.998314|d|11160: pgroonga: [release][scan-opaques][top-level][locks] 0: skip
2018-10-03 19:21:24.998317|d|11160: pgroonga: [release][scan-opaques][top-level][after-locks][start] 0
2018-10-03 19:21:24.998319|d|11160: pgroonga: [release][scan-opaques][top-level][after-locks][end] 0
2018-10-03 19:21:32.998580|d|11160: pgroonga: [release][scan-opaques][before-locks] 0: skip
2018-10-03 19:21:32.998604|d|11160: pgroonga: [release][scan-opaques][locks] 0: skip
2018-10-03 19:21:32.998608|d|11160: pgroonga: [release][scan-opaques][after-locks][start] 0
2018-10-03 19:21:32.998614|d|11160: pgroonga: [release][scan-opaques][top-level][before-locks] 0: skip
2018-10-03 19:21:32.998620|d|11160: pgroonga: [release][scan-opaques][top-level][locks] 0: skip
2018-10-03 19:21:32.998629|d|11160: pgroonga: [release][scan-opaques][top-level][after-locks][start] 0
2018-10-03 19:21:32.998632|d|11160: pgroonga: [release][scan-opaques][top-level][after-locks][end] 0
2018-10-03 19:21:45.050190|d|11160: pgroonga: [release][scan-opaques][before-locks] 0: skip
2018-10-03 19:21:45.050222|d|11160: pgroonga: [release][scan-opaques][locks] 0: skip
2018-10-03 19:21:45.050226|d|11160: pgroonga: [release][scan-opaques][after-locks][start] 0
2018-10-03 19:21:45.050233|d|11160: pgroonga: [release][scan-opaques][top-level][before-locks] 0: skip
2018-10-03 19:21:45.050239|d|11160: pgroonga: [release][scan-opaques][top-level][locks] 0: skip
2018-10-03 19:21:45.050241|d|11160: pgroonga: [release][scan-opaques][top-level][after-locks][start] 0
2018-10-03 19:21:45.050244|d|11160: pgroonga: [release][scan-opaques][top-level][after-locks][end] 0
2018-10-03 19:21:51.497832|d|11160: pgroonga: [release][scan-opaques][before-locks] 0: skip
2018-10-03 19:21:51.497866|d|11160: pgroonga: [release][scan-opaques][locks] 0: skip
2018-10-03 19:21:51.497870|d|11160: pgroonga: [release][scan-opaques][after-locks][start] 0
2018-10-03 19:21:51.497887|d|11160: pgroonga: [release][scan-opaques][top-level][before-locks] 0: skip
2018-10-03 19:21:51.497893|d|11160: pgroonga: [release][scan-opaques][top-level][locks] 0: skip
2018-10-03 19:21:51.497896|d|11160: pgroonga: [release][scan-opaques][top-level][after-locks][start] 0
2018-10-03 19:21:51.497898|d|11160: pgroonga: [release][scan-opaques][top-level][after-locks][end] 0
2018-10-03 19:21:58.257531|d|11160: pgroonga: [initialize][scan-opaque][start] 0
2018-10-03 19:21:58.257563|d|11160: pgroonga: [initialize][scan-opaque][end] 1: <0x2816bc0>
2018-10-03 19:21:58.257651|d|11160: pgroonga: [scan][begin] <0x2816bc0>
2018-10-03 19:21:58.258113|i|11160: [object][search][index][key][exact] <Lexicon20423_0.index>
2018-10-03 19:21:58.258124|i|11160: grn_ii_sel > (replication)
2018-10-03 19:21:58.258148|i|11160: n=1 (replication)
2018-10-03 19:21:58.258380|i|11160: exact: 2
2018-10-03 19:21:58.258386|i|11160: hits=2
2018-10-03 19:21:58.258486|d|11160: pgroonga: [scan][end] <0x2816bc0>
2018-10-03 19:21:58.258495|d|11160: pgroonga: [release][scan-opaques][before-locks] 1: skip
2018-10-03 19:21:58.258499|d|11160: pgroonga: [release][scan-opaques][locks] 1: skip
2018-10-03 19:21:58.258501|d|11160: pgroonga: [release][scan-opaques][after-locks][start] 1
2018-10-03 19:21:58.258509|d|11160: pgroonga: [release][scan-opaques][top-level][before-locks] 1: skip
2018-10-03 19:21:58.258515|d|11160: pgroonga: [release][scan-opaques][top-level][locks] 1: skip
2018-10-03 19:21:58.258517|d|11160: pgroonga: [release][scan-opaques][top-level][after-locks][start] 1
2018-10-03 19:21:58.258520|d|11160: pgroonga: [finalize][scan-opaque][start] 1: <0x2816bc0>
2018-10-03 19:21:58.258771|d|11160: pgroonga: [finalize][scan-opaque][end] 0: <0x2816bc0>
2018-10-03 19:21:58.258782|d|11160: pgroonga: [release][scan-opaques][top-level][after-locks][end] 0
2018-10-03 19:22:07.339922|d|11160: pgroonga: [release][scan-opaques][before-locks] 0: skip
2018-10-03 19:22:07.339948|d|11160: pgroonga: [release][scan-opaques][locks] 0: skip
2018-10-03 19:22:07.339952|d|11160: pgroonga: [release][scan-opaques][after-locks][start] 0
2018-10-03 19:22:07.339959|d|11160: pgroonga: [release][scan-opaques][top-level][before-locks] 0: skip
2018-10-03 19:22:07.339964|d|11160: pgroonga: [release][scan-opaques][top-level][locks] 0: skip
2018-10-03 19:22:07.339967|d|11160: pgroonga: [release][scan-opaques][top-level][after-locks][start] 0
2018-10-03 19:22:07.339969|d|11160: pgroonga: [release][scan-opaques][top-level][after-locks][end] 0
2018-10-03 19:22:58.309821|d|11160: pgroonga: [exit][db][opened]
2018-10-03 19:22:58.309848|d|11160: pgroonga: [exit][finalize][auto-close]
2018-10-03 19:22:58.310026|d|11160: pgroonga: [exit][finalize][query-extract-keywords]
2018-10-03 19:22:58.310258|d|11160: pgroonga: [exit][finalize][match-positions-byte]
2018-10-03 19:22:58.310488|d|11160: pgroonga: [exit][finalize][match-positions-character]
2018-10-03 19:22:58.310530|d|11160: pgroonga: [exit][finalize][highlight-html]
2018-10-03 19:22:58.310538|d|11160: pgroonga: [exit][finalize][keywords]
2018-10-03 19:22:58.310540|d|11160: pgroonga: [exit][finalize][jsonb]
2018-10-03 19:22:58.311465|d|11160: pgroonga: [exit][finalize][sequential-search-data]
2018-10-03 19:22:58.312167|d|11160: pgroonga: [exit][finalize][prefix-rk-sequential-search-data]
2018-10-03 19:22:58.312288|d|11160: pgroonga: [exit][finalize][options]
2018-10-03 19:22:58.312294|d|11160: pgroonga: [exit][db][close]
2018-10-03 19:22:58.315818|d|11160: pgroonga: [exit][finalize][buffers]
2018-10-03 19:22:58.315852|d|11160: pgroonga: [exit][finalize][context]
2018-10-03 19:22:58.317608|d|11160: pgroonga: [exit][finalize]
2018-10-03 19:22:58.319153|n|11160: grn_fin (0)
```

何度も見直し、やり直してみたりしたが原因がよく分からず。。。
スレーブ側ではShareUpdateExclusiveLockは使えないよ、RowExclusiveLock以下のものを使ってと言っているエラーの気がするが、
かと言ってスレーブpgroonga.enable_wal = onを外してみると、スレーブ側のpgroongaインデックスが更新されず。
RowExclusiveLockに書き換えてよいものかも良く分からず。

※追記[2018-11-27]
2018-10-14にリリースされた2.1.6にて、ロックモードの修正が入り、
上記手順でのレプリケーション動作については問題無くなりました。
https://pgroonga.github.io/news/