**Fault Tolerance with Gold**

One of the design goals for Gold is fault tolerance. Gold implements
fault tolerance in several ways:

-   Robust communication infrastructure

-   Transactions

-   Hot backup server

**[Robust Communication Infrastructure]{.underline}**

The Gold server and client communications have been designed and tested
for robustness. Goldd is a forking server which allows it to handle
multiple requests simultaneously, each in a separate process. This
approach prevents one hanging or problematic connection from preventing
other requests from getting serviced. Additionally, messages between the
client and server are handled with a timeout select in non-blocking read
and write loops in order to prevent hangs due to dropped connections.
The use of strong security for authentication and encryption can prevent
third parties from hijacking or eavesdropping on the communications.

**[Transactions]{.underline}**

Gold uses a database backend for data persistence. Each request performs
its work within a transaction, which is committed or rolled-back as a
unit depending on the success or failure of the request. This approach
ensures that all requests are atomic, even if the client, server or
database is interrupted at any point in the request.

**[Hot Backup Server]{.underline}**

Gold can be configured to employ a hot backup server. Such a
configuration would also require the use of a synchronous multi-master
replicated database such as pgcluster. If a gold client detects that the
primary gold server or its database is unavailable, it will
automatically forward the request to the backup gold server if one is
configured.

**[Example configuring gold with a hot backup server using
pgcluster:]{.underline}**

For this example we will assume that we have two head nodes called
*primary* and *backup*.

Unpack and install Gold on both the primary and backup nodes. Follow the
installation instructions for [Preparation]{.underline} and [Install
Prerequisites]{.underline} with the exception that pgcluster will need
to be installed in the place of postgresql (pgcluster is a repackaging
of postgresql with patches for replication).

o PGCluster 1.3 or higher \[OPTIONAL\]:

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\--

PGCluster is a synchronous multi-master replication system based on
postgres. PGCluster 1.3c is based on PostgreSQL 8.0.1.

PGCluster is available at:
\<http://hiroshima.sraw.co.jp/people/mitani/jpug/pgcluster/en/index.html\>

\# Follow the INSTALL_PGCLUSTER instructions for installing pgcluster.

\# The following are some of the highlights:

\[root\] useradd postgres

\[root\] cd /usr/local/src

\[root\] wget
[[http://hiroshima.sraw.co.jp/people/mitani/jpug/pgcluster/src/1_3/pgcluster-1.3.0c.tar.gz]{.underline}](http://hiroshima.sraw.co.jp/people/mitani/jpug/pgcluster/src/1_3/pgcluster-1.3.0c.tar.gz)

\[root\] tar -zxvf /tmp/pgcluster-1.3.0c.tar.gz

\[root\] chown -R postgres.postgres pgcluster-1.3.0c

\[postgres\] cd /usr/local/src/pgcluster-1.3.0c

\[postgres\] ./configure \--enable-thread-safety

\[postgres\] make

\[root\] make install

\[root\] chown -R postgres /usr/local/pgsql

\[postgres\] /usr/local/pgsql/bin/initdb -D /usr/local/pgsql/data

\[postgres\] vi /usr/local/pgsql/data/pg_hba.conf

host all all \<ip address for primary\>/32 trust

host all all \<ip address for backup\>/32 trust

\# This example configure pgcluster with a pgcluster and a replication
server on the

\# the primary node, and a pgcluster on the backup node. This was found
through testing

\# to be the most stable configuration. No load balance server is used.

\[postgres@primary,backup\] vi /usr/local/pgsql/data/cluster.conf:

\<Replicate_Server_Info\>

\<Host_Name\> primary.emsl.pnl.gov \</Host_Name\>

\<Port\> 8001 \</Port\>

\<Recovery_Port\> 8101 \</Recovery_Port\>

\<LifeCheck_Port\> 8201 \</LifeCheck_Port\> \</Replicate_Server_Info\>

\<Rsync_Option\> ssh -x \</Rsync_Option\>

\<When_Stand_Alone\> read_write \</When_Stand_Alone\>

\# A single replication server is configured on the primary node only

\[postgres@primary\] vi /usr/local/pgsql/etc/pgreplicate.conf:

\<Cluster_Server_Info\>

\<Host_Name\> sst.emsl.pnl.gov \</Host_Name\>

\<Port\> 5432 \</Port\>

\<Recovery_Port\> 7101 \</Recovery_Port\>

\<LifeCheck_Port\> 7201 \</LifeCheck_Port\>

\</Cluster_Server_Info\>

\# Rsync and openssh have to be installed allowing non-interactive ssh
connections

\# Install openssh

\[root\] cd /usr/local/src

\[root\] wget
http://mirror.mcs.anl.gov/openssh/portable/openssh-3.9p1.tar.gz

\[root\] tar -zxvf openssh-3.9p1.tar.gz

\[root\] cd openssh-3.9p1

\[root\] LIBS=-lcrypt ./configure \--prefix=/usr \--sysconfdir=/etc/ssh

\[root\] make

\[root\] make install

\[root\] vi /etc/ssh/ssh_config

\# ForwardX11 no

\[root\] vi /etc/ssh/sshd_config

RSAAuthentication yes

PubkeyAuthentication yes

AuthorizedKeysFile .ssh/authorized_keys

\[root\] /etc/init.d/ssh restart

\[postgres\] cd \~/.ssh

\[postgres\] ssh-keygen -t rsa \# No pass phrase

\[postgres\] cp id_rsa.pub authorized_keys

\[postgres\] cat \<id_rsa.pub from other host\> \>\> authorized_keys

\# Test ssh to/from both hosts to ensure non-prompting operation and
clear known_hosts

\# Test rsync

\# **The following is how you startup the replicated database on the two
servers**

\# The gold server should be in a quiesced state (or not started yet)

\# Start the pgcluster and the replication server on the primary

\[postgres@primary\] /usr/local/pgsql/bin/pg_ctl -D
/usr/local/pgsql/data -o \"-i\" start

\[postgres@primary\] /usr/local/pgsql/bin/pgreplicate -D
/usr/local/pgsql/etc -l

\# Start the pgcluster on the backup

\[postgres@backup\] /usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data
-o \"-i\" start

\# **The following is how you stop the replicated database on the two
servers**

\# The gold server should be in a quiesced state (or already shut down)

\# Stop the replication server on the primary, then the pgcluster

\[postgres@primary\] /usr/local/pgsql/bin/pgreplicate -D
/usr/local/pgsql/etc stop

\[postgres@primary\] /usr/local/pgsql/bin/pg_ctl -D
/usr/local/pgsql/data stop

\# Stop the pgcluster on the backup

\[postgres@backup\] /usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data
stop

\# **The following is how you recover the replicated database on the
primary**

\# The gold server should be in a quiesced state (or shut down on
primary and backup)

\# If the pgcluster on the backup is not already running then start it

\# \[postgres@backup\] /usr/local/pgsql/bin/pg_ctl -D
/usr/local/pgsql/data -o \"-i\" start

\# Start the replication server on the primary

\[postgres@primary\] /usr/local/pgsql/bin/pgreplicate -D
/usr/local/pgsql/etc -l

\# Start the pgcluster on the primary in recovery mode

\[postgres@primary\] /usr/local/pgsql/bin/pg_ctl -D
/usr/local/pgsql/data -o \"-i -R\" start

\# **The following is how you recover the replicated database on the
backup**

\# The gold server should be in a quiesced state (or shut down on
primary and backup)

\# If the pgcluster and replication server on the backup are not already
running then start them

\# \[postgres@primary\] /usr/local/pgsql/bin/pg_ctl -D
/usr/local/pgsql/data -o \"-i\" start

\# \[postgres@primary\] /usr/local/pgsql/bin/pgreplicate -D
/usr/local/pgsql/etc -l

\# Start the pgcluster on the backup in recovery mode

\[postgres@backup\] /usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data
-o \"-i -R\" start

*\....Now back to the Gold installation\....*

Continue following the Gold installation instructions for
[Configuration]{.underline}, [Compilation]{.underline}, [Perl Module
Dependencies]{.underline}, and [Installation]{.underline}. In [General
Setup]{.underline}, when you edit the Gold client config file (i.e.
/usr/local/gold/etc/gold.conf), uncomment the server.backup property and
specify the name of the backup host.

server.backup = backup

The [Database Setup]{.underline} has partly been done during the
pgcluster setup above. The databases should already have been
initialized on both the primary and the backup. The
/usr/local/pgsql/data/pg_hba.conf should already have been modified to
define the hosts for both the primary and backup servers.

Start up the replicated databases.

\[postgres@primary\] /usr/local/pgsql/bin/pg_ctl -D
/usr/local/pgsql/data -o \"-i\" start

\[postgres@primary\] /usr/local/pgsql/bin/pgreplicate -D
/usr/local/pgsql/etc -l

\[postgres@backup\] /usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data
-o \"-i\" start

Create the gold user *on the primary only*.

\[postgres@primary\] /usr/local/pgsql/bin/createuser gold

Shall the new user be allowed to create databases? y

Shall the new user be allowed to create more new users? n

Create the gold database *as the gold user on the primary only*.

\[gold@primary\] /usr/local/pgsql/bin/createdb gold

Perform the [Bootstrap]{.underline} step *on the primary only*.

\[gold@primary\] /usr/local/pgsql/bin/psql gold \< bank.sql

Startup the gold server daemons on both the primary and the backup.

\[gold@primary,backup\] /usr/local/gold/sbin/goldd

Proceed as normal, with the [Initialization]{.underline} and
[Customization]{.underline} steps.

**Note**: Only the non-interactive clients will automatically switch to
use the hot backup. The goldsh client and the web-based gui will not do
so. Also, 3^rd^ party applications that communicate with the Gold server
may not automatically fail-over such as the Maui Scheduler and the Moab
Cluster Manager.