Postgres 12.4 replication with Docker
By: Bhushan Patil
Tags: Docker, Postgresql, Replication, Postgres

Postgresql Replication with Docker

PostgreSQL with built-in replication is a great choice for a reliable and scalable database engine. The setup is going to be a master instance that handles both read and write operations. The other is a slave instance that will only serve read requests and that is because writing data is a whole different story than reading them.

Let’s create a master database.

Dockerfile

FROM postgres:12.4-alpine

COPY ./setup-master.sh /docker-entrypoint-initdb.d/setup-master.sh

RUN chmod 0666 /docker-entrypoint-initdb.d/setup-master.sh

And there is a setup-master.sh file which needs to be copied to the image which makes the Postgres ready for being a master in the replication process.

setup-master.sh

#!/bin/bash

echo “host replication all 0.0.0.0/0 trust” >> “$PGDATA/pg_hba.conf”

set -e

psql -v ON_ERROR_STOP=1 — username “$POSTGRES_USER” — dbname “$POSTGRES_DB” <<-EOSQL

CREATE USER $PG_REP_USER REPLICATION LOGIN CONNECTION LIMIT 100 ENCRYPTED PASSWORD ‘$PG_REP_PASSWORD’;

CREATE USER postgres;

EOSQL cat >> ${PGDATA}/postgresql.conf <<EOF

wal_level = replica

archive_mode = on

archive_command = ‘cd .’

max_wal_senders = 8

wal_keep_segments = 8

hot_standby = on

EOF

Now let’s create a Dockerfile for the slave database.

Dockerfile



ENV GOSU_VERSION 1.12

RUN set -eux; \

\

apk add --no-cache --virtual .gosu-deps \

ca-certificates \

dpkg \

gnupg \

; \

\

dpkgArch="$(dpkg --print-architecture | awk -F- '{ print $NF }')"; \

wget -O /usr/local/bin/gosu "

wget -O /usr/local/bin/gosu.asc "

\

# verify the signature

export GNUPGHOME="$(mktemp -d)"; \

gpg --keyserver hkp://ipv4.pool.sks-keyservers.net --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4; \

gpg --verify /usr/local/bin/gosu.asc /usr/local/bin/gosu; \

command -v gpgconf && gpgconf --kill all || :; \

rm -rf "$GNUPGHOME" /usr/local/bin/gosu.asc; \

\

# clean up fetch dependencies

apk del --no-network .gosu-deps; \

\

chmod +x /usr/local/bin/gosu; \

# verify that the binary works

gosu --version; \

gosu nobody true FROM postgres:12.4-alpineENV GOSU_VERSION 1.12RUN set -eux; \apk add --no-cache --virtual .gosu-deps \ca-certificates \dpkg \gnupg \; \dpkgArch="$(dpkg --print-architecture | awk -F- '{ print $NF }')"; \wget -O /usr/local/bin/gosu " https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$dpkgArch "; \wget -O /usr/local/bin/gosu.asc " https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$dpkgArch.asc "; \# verify the signatureexport GNUPGHOME="$(mktemp -d)"; \gpg --keyserver hkp://ipv4.pool.sks-keyservers.net --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4; \gpg --verify /usr/local/bin/gosu.asc /usr/local/bin/gosu; \command -v gpgconf && gpgconf --kill all || :; \rm -rf "$GNUPGHOME" /usr/local/bin/gosu.asc; \# clean up fetch dependenciesapk del --no-network .gosu-deps; \chmod +x /usr/local/bin/gosu; \# verify that the binary worksgosu --version; \gosu nobody true RUN apk add --update iputils

COPY ./docker-entrypoint.sh /docker-entrypoint.sh

RUN chmod +x /docker-entrypoint.sh

ENTRYPOINT ["/docker-entrypoint.sh"]

CMD ["gosu", "postgres","postgres"]

We need the gosu binary to execute the postgres as a root

We also need the iputils package to be able to ping the master.

The next step is to prepare the slave images to actually be slaves. We use a docker-entrypoint.sh file which will be the first thing that docker executes upon creating the container.

#!/bin/bash

if [ ! -s “$PGDATA/PG_VERSION” ]; then

echo “*:*:*:$PG_REP_USER:$PG_REP_PASSWORD” > ~/.pgpass

chmod 0600 ~/.pgpass until ping -c 1 -W 1 ${PG_MASTER_HOST}

do

echo “Waiting for master to ping…”

sleep 1s

done

until pg_basebackup -h ${PG_MASTER_HOST} -D ${PGDATA} -p 5432 -U ${PG_REP_USER} -R -vP -W

do

echo “Waiting for master to connect…”

sleep 1s

done

fi cat >> ${PGDATA}/postgresql.conf <<EOF

wal_level = replica

archive_mode = on

archive_command = ‘cd .’

max_wal_senders = 8

wal_keep_segments = 8

hot_standby = on

EOF set -e

chown postgres. ${PGDATA} -R

chmod 700 ${PGDATA} -R sed -i ‘s/wal_level = hot_standby/wal_level = replica/g’ ${PGDATA}/postgresql.conf

exec “$@”

On the first line, we check that this instance has already been set up or not by checking the PG_VERSION file in the PG_DATA the path so as not to do it on every startup of the container.

file in the the path so as not to do it on every startup of the container. We copy the replication user and password in the .pgpass so Postgres can access it.

so Postgres can access it. We start pinging the Master to make sure that it’s already up and running.

It’s almost done now! :)

Now, need to create a docker-compose.yml file to spin up our databases and build our containers.

version: "3"

services: master_db:

build: ./postgres_master/

container_name: master.db

restart: unless-stopped

ports:

- 5451:5432

volumes:

- pgdata_master:/var/lib/postgresql/data

env_file:

- .env

environment:

- PGDATA=/var/lib/postgresql/data

- POSTGRESQL_REPLICATION_MODE=${MASTER_REPLICATION_MODE}

- PG_REP_USER=${DB_REPLICATION_USER}

- PG_REP_PASSWORD=${DB_REPLICATION_PASSWORD}

- POSTGRES_DB=${DATABASE}

- POSTGRES_PASSWORD=${DB_PASSWORD}

- POSTGRES_USER=${DB_USER} slave_db:

build: ./postgres_slave/

container_name: slave.db

restart: unless-stopped

volumes:

- pgdata_slave:/var/lib/postgresql/data

ports:

- 7571:5432

depends_on:

- master_db

env_file:

- .env

environment:

- PGDATA=/var/lib/postgresql/data

- POSTGRESQL_REPLICATION_MODE=${SLAVE_REPLICATION_MODE}

- PG_REP_USER=${DB_REPLICATION_USER}

- PG_REP_PASSWORD=${DB_REPLICATION_PASSWORD}

- PG_MASTER_HOST=${MASTER_HOST}

- PG_MASTER_PORT=${MASTER_PORT}

Now let’s create a .env file to store the variables Or you can just directly put the values in docker-compose.yml

.env

DATABASE=bhushan12

DB_PASSWORD=docker

DB_USER=docker MASTER_REPLICATION_MODE=master

SLAVE_REPLICATION_MODE=slave

DB_REPLICATION_USER=repl_user

DB_REPLICATION_PASSWORD=repl_password

MASTER_HOST=master_db

MASTER_PORT=5432

Just keep in mind that we have put the Dockerfile and setup-master.sh file the postgres_master directory and the slave’s Dockerfile and docker-entrypoint.sh inside the postgres_slave directory.

Let’s run this setup: