Package: pghoard
PGHoardpghoardis a PostgreSQL backup daemon and restore tooling that stores backup data in cloud object stores.Features:Automatic periodic basebackupsAutomatic transaction log (WAL/xlog) backups (using eitherpg_receivexlog,archive_commandor experimental PG native replication protocol support withwalreceiver)Optional Standalone Hot Backup supportCloud object storage support (AWS S3, Google Cloud, OpenStack Swift, Azure, Ceph)Backup restoration directly from object storage, compressed and encryptedPoint-in-time-recovery (PITR)Initialize a new standby from object storage backups, automatically configured as
a replicating hot-standbyFault-resilience and monitoring:Persists over temporary object storage connectivity issues by retrying transfersVerifies WAL file headers before upload (backup) and after download (restore),
so that e.g. files recycled by PostgreSQL are ignoredAutomatic history cleanup (backups and related WAL files older than N days)“Archive sync” tool for detecting holes in WAL backup streams and fixing them“Archive cleanup” tool for deleting obsolete WAL files from the archiveKeeps statistics updated in a file on disk (for monitoring tools)Creates alert files on disk on problems (for monitoring tools)Performance:Parallel compression and encryptionWAL pre-fetching on restoreOverviewPostgreSQL Point In Time Replication (PITR) consists of a having a database
basebackup and changes after that point go into WAL log files that can be
replayed to get to the desired replication point.PGHoard supports multiple operating models.  The basic mode where you have a
separate backup machine,pghoardcan simply connect withpg_receivexlogto receive WAL files from the database as they’re
written.  Another model is to usepghoard_postgres_commandas a
PostgreSQLarchive_command. There is also experimental support for PGHoard to
use PostgreSQL’s native replication protocol with the experimentalwalreceivermode.With both modes of operations PGHoard creates periodic basebackups usingpg_basebackupthat is run against the database in question.The PostgreSQL write-ahead log (WAL) and basebackups are compressed with
Snappy (default), Zstandard (configurable, level 3 by default) or LZMA (configurable,
level 0 by default) in order to ensure good compression speed and relatively small backup size.
For performance critical applications it is recommended to test compression
algorithms to find the most suitable trade-off for the particular use-case.
E.g. Snappy is fast but yields larger compressed files, Zstandard (zstd) on the other hand
offers a very wide range of compression/speed trade-off.Optionally, PGHoard can encrypt backed up data at rest. Each individual
file is encrypted and authenticated with file specific keys. The file
specific keys are included in the backup in turn encrypted with a master
RSA private/public key pair.PGHoard supports backing up and restoring from either a local filesystem or
from various object stores (AWS S3, Azure, Ceph, Google Cloud and OpenStack
Swift.)In case you just have a single database machine, it is heavily recommended
to utilize one of the object storage services to allow backup recovery even
if the host running PGHoard is incapacitated.RequirementsPGHoard can backup and restore PostgreSQL versions 9.3 and above.  The
daemon is implemented in Python and works with CPython version 3.3 or newer.
The following Python modules are required:psycopg2to look up transaction log metadatarequestsfor the internal client-server architectureOptional requirements include:azurefor Microsoft Azure object storagebotocorefor AWS S3 (or Ceph-S3) object storagegoogle-api-clientfor Google Cloud object storagecryptographyfor backup encryption and decryption (version 0.8 or newer required)snappyfor Snappy compression and decompressionzstandardfor Zstandard (zstd) compression and decompressionsystemdfor systemd integrationswiftclientfor OpenStack Swift object storageparamikofor sftp object storageDeveloping and testing PGHoard also requires the following utilities:flake8,pylintandpytest.PGHoard has been developed and tested on modern Linux x86-64 systems, but
should work on other platforms that provide the required modules.VagrantThe Vagrantfile can be used to setup a vagrant development environment, consisting of two
vagrant virtual machines.Postgresql 9.4, python 3.5 and 3.6:vagrant up
vagrant ssh postgres9
cd /vagrant
source ~/venv3/bin/activate
make test
source ~/venv3.6/bin/activate
make testPostgresql 10 and python 3.7:vagrant ssh postgres10
cd /vagrant
make testNote: make deb does not work from vagrant at the moment, hopefully this will be resolved soonBuildingTo build an installation package for your distribution, go to the root
directory of a PGHoard Git checkout and run:Debian:make debThis will produce a.debpackage into the parent directory of the Git
checkout.Fedora:make rpmThis will produce a.rpmpackage usually intorpm/RPMS/noarch/.Python/Other:python setup.py bdist_eggThis will produce an egg file into a dist directory within the same folder.InstallationTo install it run as root:Debian:dpkg -i ../pghoard*.debFedora:dnf install rpm/RPMS/noarch/*On Linux systems it is recommended to simply runpghoardundersystemd:systemctl enable pghoard.serviceand eventually after the setup section, you can just run:systemctl start pghoard.servicePython/Other:easy_install dist/pghoard-1.7.0-py3.6.eggOn systems withoutsystemdit is recommended that you runpghoardunderSupervisoror other similar process control system.SetupAfter this you need to create a suitable JSON configuration file for your
installation.Make sure PostgreSQL is configured to allow WAL archival and retrieval.postgresql.confshould havewal_levelset toarchiveor
higher andmax_wal_sendersset to at least1(archive_commandmode)
or at least2(pg_receivexlogandwalreceivermodes), for example:wal_level = archive
max_wal_senders = 4Note that changingwal_levelormax_wal_senderssettings requires
restarting PostgreSQL.Create a suitable PostgreSQL user account forpghoard:CREATE USER pghoard PASSWORD 'putyourpasswordhere' REPLICATION;Edit the localpg_hba.confto allow access for the newly created
account to thereplicationdatabase from the master and standby
nodes. For example:# TYPE  DATABASE     USER     ADDRESS       METHOD
host    replication  pghoard  127.0.0.1/32  md5After editing, please reload the configuration with either:SELECT pg_reload_conf();or by sending directly aSIGHUPto the PostgreSQLpostmasterprocess.Fill in the created user account and master/standby addresses into the
configuration filepghoard.jsonto the sectionbackup_sites.Fill in the possible object storage user credentials into the
configuration filepghoard.jsonunder sectionobject_storagein case you wishpghoardto back up into the cloud.Now copy the samepghoard.jsonconfiguration to the standby
node if there are any.Other possible configuration settings are covered in more detail under theConfiguration keyssection of this README.If all has been set up correctly up to this point,pghoardshould now be
ready to be started.Backing up your databasePostgreSQL backups consist of full database backups,basebackups, plus
write ahead logs and related metadata,WAL.  BothbasebackupsandWALare required to create and restore a consistent database (does not apply
for standalone hot backups).To enable backups with PGHoard thepghoarddaemon must be running
locally.  The daemon will periodically take full basebackups of the database
files to the object store.  Additionally, PGHoard and PostgreSQL must be set
up correctly to archive the WAL.  There are two ways to do this:The default option is to use PostgreSQL’s own WAL-archive mechanism withpghoardby running thepghoarddaemon locally and entering the
following configuration keys inpostgresql.conf:archive_mode = on
archive_command = pghoard_postgres_command --mode archive --site default --xlog %fThis instructs PostgreSQL to call thepghoard_postgres_commandwhenever
a new WAL segment is ready.  The command instructs PGHoard to store the
segment in its object store.The other option is to set up PGHoard to read the WAL stream directly from
PostgreSQL.  To do thisarchive_modemust be disabled inpostgresql.confandpghoard.jsonmust setactive_backup_modetopg_receivexlogin the relevant site, for example:{
    "backup_sites": {
        "default": {
            "active_backup_mode": "pg_receivexlog",
            ...
         },
     },
     ...
 }Note that as explained in theSetupsection,postgresql.confsettingwal_levelmust always be set toarchive,hot_standbyorlogicalandmax_wal_sendersmust allow 2 connections from PGHoard,
i.e. it should be set to 2 plus the number of streaming replicas, if any.Whilepghoardis running it may be useful to read the JSON state filepghoard_state.jsonthat exists wherejson_state_file_pathpoints.
The JSON state file is human readable and is meant to describe the current
state ofpghoard‘s backup activities.Standalone Hot Backup SupportPghoard has the option to enable standalone hot backups.To do thisarchive_modemust be disabled inpostgresql.confandpghoard.jsonmust setactive_backup_modetostandalone_hot_backupin the relevant site, for example:{
    "backup_sites": {
        "default": {
            "active_backup_mode": "standalone_hot_backup",
            ...
         },
     },
     ...
 }For more information refer to the postgresql documentationhttps://www.postgresql.org/docs/9.5/continuous-archiving.html#BACKUP-STANDALONERestoring databasesYou can list your database basebackups by running:pghoard_restore list-basebackups --config /var/lib/pghoard/pghoard.json

Basebackup                       Size  Start time            Metadata
-------------------------------  ----  --------------------  ------------
default/basebackup/2016-04-12_0  8 MB  2016-04-12T07:31:27Z  {'original-file-size': '48060928',
                                                              'start-wal-segment': '000000010000000000000012',
                                                              'compression-algorithm': 'snappy'}If we’d want to restore to the latest point in time we could fetch the
required basebackup by running:pghoard_restore get-basebackup --config /var/lib/pghoard/pghoard.json \
    --target-dir /var/lib/pgsql/9.5/data --restore-to-master

Basebackup complete.
You can start PostgreSQL by running pg_ctl -D foo start
On systemd based systems you can run systemctl start postgresql
On SYSV Init based systems you can run /etc/init.d/postgresql startNote that thetarget-dirneeds to be either an empty or non-existent
directory in which case PGHoard will automatically create it.After this we’d proceed to start both the PGHoard server process and the
PostgreSQL server normally by running (on systemd based systems, assuming
PostgreSQL 9.5 is used):systemctl start pghoard
systemctl start postgresql-9.5Which will make PostgreSQL start recovery process to the latest point
in time. PGHoard must be running before you start up the
PostgreSQL server. To see other possible restoration options please run:pghoard_restore --helpCommandsOnce correctly installed, there are six commands available:pghoardis the main daemon process that should be run under a service
manager, such assystemdorsupervisord.  It handles the backup of
the configured sites.pghoard_restoreis a command line tool that can be used to restore a
previous database backup from eitherpghoarditself or from one of the
supported object stores.pghoard_restorecan also configurerecovery.confto usepghoard_postgres_commandas the WALrestore_commandinrecovery.conf.pghoard_archive_cleanupcan be used to clean up any orphan WAL files
from the object store.  After the configured number of basebackups has been
exceeded (configuration keybasebackup_count),pghoarddeletes the
oldest basebackup and all WAL associated with it.  Transient object storage
failures and other interruptions can cause the WAL deletion process to leave
orphan WAL files behind, they can be deleted with this tool.pghoard_archive_synccan be used to see if any local files should
be archived but haven’t been or if any of the archived files have unexpected
content and need to be archived again. The other usecase it has is to determine
if there are any gaps in the required files in the WAL archive
from the current WAL file on to to the latest basebackup’s first WAL file.pghoard_create_keyscan be used to generate and output encryption keys
in thepghoardconfiguration format.pghoard_postgres_commandis a command line tool that can be used as
PostgreSQL’sarchive_commandorrecovery_command.  It communicates withpghoard‘s locally running webserver to let it know there’s a new file that
needs to be compressed, encrypted and stored in an object store (in archive
mode) or it’s inverse (in restore mode.)Configuration keysactive(defaulttrue)Can be set on a perbackup_sitelevel tofalseto disable the taking
of new backups and to stop the deletion of old ones.active_backup_mode(defaultpg_receivexlog)Can be eitherpg_receivexlogorarchive_command. If set topg_receivexlog,pghoardwill start up apg_receivexlogprocess to be
run against the database server.  Ifarchive_commandis set, we rely on the
user setting the correctarchive_commandinpostgresql.conf. You can also set this to the experimentalwalreceivermode
whereby pghoard will start communicating directly with PostgreSQL
through the replication protocol. (Note requires an unreleased version
of psycopg2 library)alert_file_dir(defaultbackup_locationif set elseos.getcwd())Directory in which alert files for replication warning and failover are
created.backup_location(no default)Place wherepghoardwill create its internal data structures for local state
data and the actual backups.  (if no object storage is used)backup_sites(default{})This object contains names and configurations for the different PostgreSQL
clusters (here calledsites) from which to take backups.  The
configuration keys for sites are listed below.compressionWAL/basebackup compression parametersalgorithmdefault"snappy"if available, otherwise"lzma"or"zstd"leveldefault"0"compression level for"lzma"or"zstd"compressionthread_count(default max(cpu_count,5)) number of parallel compression threadshash_algorithm(default"sha1")The hash algorithm used for calculating checksums for WAL or other files. Must
be one of the algorithms supported by Python’s hashlib.http_address(default"127.0.0.1")Address to bind the PGHoard HTTP server to.  Set to an empty string to
listen to all available addresses.http_port(default16000)HTTP webserver port. Used for the archive command and for fetching of
basebackups/WAL’s when restoring if not using an object store.json_state_file_path(default"/var/lib/pghoard/pghoard_state.json")Location of a JSON state file which describes the state of thepghoardprocess.log_level(default"INFO")Determines log level ofpghoard.maintenance_mode_file(default"/var/lib/pghoard/maintenance_mode_file")If a file exists in this location, no new backup actions will be started.restore_prefetch(defaulttransfer.thread_count)Number of files to prefetch when performing archive recovery.  The default
is the number of Transfer Agent threads to try to utilize them all.statsd(default: disabled)Enables metrics sending to a statsd daemon that supports Telegraf
or DataDog syntax with tags.The value is a JSON object:{
    "host": "<statsd address>",
    "port": <statsd port>,
    "format": "<statsd message format>",
    "tags": {
        "<tag>": "<value>"
    }
}format(default:"telegraf")Determines statsd message format. Following formats are supported:telegrafTelegraf specdatadogDataDog specThetagssetting can be used to enter optional tag values for the metrics.pushgateway(default: disabled)Enables metrics sending to a Prometheus Pushgateway with tags.The value is a JSON object:{
    "endpoint": "<pushgateway address>",
    "tags": {
        "<tag>": "<value>"
    }
}Thetagssetting can be used to enter optional tag values for the metrics.prometheus(default: disabled)Expose metrics through a Prometheus endpoint.The value is a JSON object:{
    "tags": {
        "<tag>": "<value>"
    }
}Thetagssetting can be used to enter optional tag values for the metrics.syslog(defaultfalse)Determines whether syslog logging should be turned on or not.syslog_address(default"/dev/log")Determines syslog address to use in logging (requires syslog to be true as
well)syslog_facility(default"local2")Determines syslog log facility. (requires syslog to be true as well)transferWAL/basebackup transfer parametersthread_count(default max(cpu_count,5)) number of parallel uploads/downloadsupload_retries_warning_limit(default3)After this many failed upload attempts for a single file, create an
alert file.tar_executable(default"pghoard_gnutaremu")The tar command to use for restoring basebackups. This must be GNU tar because some
advanced switches like--transformare needed. If this value is not defined (or
is explicitly set to"pghoard_gnutaremu"), Python’s internal tarfile
implementation is used. The Python implementation is somewhat slower than the
actual tar command and in environments with fast disk IO (compared to available CPU
capacity) it is recommended to set this to"tar".Backup site configurationThe following options control the behavior of each backup site.  A backup
site means an individual PostgreSQL installation (“cluster” in PostgreSQL
terminology) from which to take backups.basebackup_chunks_in_progress(default5)How many basebackup chunks can there be simultaneously on disk while
it is being taken. For chunk size configuration seebasebackup_chunk_size.basebackup_chunk_size(default2147483648)In how large backup chunks to take alocal-tarbasebackup. Disk
space needed for a successful backup is this variable multiplied bybasebackup_chunks_in_progress.basebackup_count(default2)How many basebackups should be kept around for restoration purposes.  The
more there are the more diskspace will be used.basebackup_interval_hours(default24)How often to take a new basebackup of a cluster.  The shorter the interval,
the faster your recovery will be, but the more CPU/IO usage is required from
the servers it takes the basebackup from.  If set to a null value basebackups
are not automatically taken at all.basebackup_mode(default"basic")The way basebackups should be created.  The default mode,basicrunspg_basebackupand waits for it to write an uncompressed tar file on the
disk before compressing and optionally encrypting it.  The alternative modepipepipes the data directly frompg_basebackupto PGHoard’s
compression and encryption processing reducing the amount of temporary disk
space that’s required.Neitherbasicnorpipemodes support multiple tablespaces.Settingbasebackup_modetolocal-taravoids usingpg_basebackupentirely whenpghoardis running on the same host as the database.
PGHoard reads the files directly from$PGDATAin this mode and
compresses and optionally encrypts them.  This mode allows backing up user
tablespaces.Note that thelocal-tarbackup mode can not be used on replica servers
prior to PostgreSQL 9.6 unless the pgespresso extension is installed.encryption_key_id(no default)Specifies the encryption key used when storing encrypted backups. If this
configuration directive is specified, you must also define the public key
for storing as well as private key for retrieving stored backups. These
keys are specified withencryption_keysdictionary.encryption_keys(no default)This key is a mapping from key id to keys. Keys in turn are mapping frompublicandprivateto PEM encoded RSA public and private keys
respectively. Public key needs to be specified for storing backups. Private
key needs to be in place for restoring encrypted backups.You can usepghoard_create_keysto generate and output encryption keys
in thepghoardconfiguration format.object_storage(no default)Configured inbackup_sitesunder a specific site.  If set, it must be an
object describing a remote object storage.  The object must contain a keystorage_typedescribing the type of the store, other keys and values are
specific to the storage type.The following object storage types are supported:localmakes backups to a local directory, seepghoard-local-minimal.jsonfor example. Required keys:directoryfor the path to the backup target (local) storage directorysftpmakes backups to a sftp server, required keys:serverportusernamepasswordorprivate_keygooglefor Google Cloud Storage, required configuration keys:project_idcontaining the Google Storage project identifierbucket_namebucket where you want to store the filescredential_filefor the path to the Google JSON credential files3for Amazon Web Services S3, required configuration keys:aws_access_key_idfor the AWS access key idaws_secret_access_keyfor the AWS secret access keyregionS3 region of the bucketbucket_namename of the S3 bucketOptional keys for Amazon Web Services S3:encryptedif True, use server-side encryption. Default is False.s3for other S3 compatible services such as Ceph, required
configuration keys:aws_access_key_idfor the AWS access key idaws_secret_access_keyfor the AWS secret access keybucket_namename of the S3 buckethostfor overriding host for non AWS-S3 implementationsportfor overriding port for non AWS-S3 implementationsis_securefor overriding the requirement for https for non AWS-S3is_verify_tlsfor configuring tls verify for non AWS-S3
implementationsazurefor Microsoft Azure Storage, required configuration keys:account_namefor the name of the Azure Storage accountaccount_keyfor the secret key of the Azure Storage accountbucket_namefor the name of Azure Storage container used to store
objectsazure_cloudAzure cloud selector,"public"(default) or"germany"swiftfor OpenStack Swift, required configuration keys:userfor the Swift user (‘subuser’ in Ceph RadosGW)keyfor the Swift secret_keyauth_urlfor Swift authentication URLcontainer_namename of the data containerOptional configuration keys for Swift:auth_version-2.0(default) or3.0for keystone, use1.0with
Ceph Rados GW.segment_size- defaults to1024**3(1 gigabyte).  Objects larger
than this will be split into multiple segments on upload.  Many Swift
installations require large files (usually 5 gigabytes) to be segmented.tenant_nameregion_nameuser_id- for auth_version 3.0user_domain_id- for auth_version 3.0user_domain_name- for auth_version 3.0tenant_id- for auth_version 3.0project_id- for auth_version 3.0project_name- for auth_version 3.0project_domain_id- for auth_version 3.0project_domain_name- for auth_version 3.0service_type- for auth_version 3.0endpoint_type- for auth_version 3.0nodes(no default)Array of one or more nodes from which the backups are taken.  A node can be
described as an object of libpq key: value connection info pairs or libpq
connection string or apostgres://connection uri. If for example you’d
like to use a streaming replication slot use the syntax {… “slot”: “slotname”}.pg_bin_directory(default: find binaries from well-known directories)Site-specific option for findingpg_basebackupandpg_receivexlogcommands matching the given backup site’s PostgreSQL version.  If a value is
not supplied PGHoard will attempt to find matching binaries from various
well-known locations.  In casepg_data_directoryis set and points to a
valid data directory the lookup is restricted to the version contained in
the given data directory.pg_data_directory(no default)This is used when thelocal-tarbasebackup_modeis used.  The data
directory must point to PostgreSQL’s$PGDATAand must be readable by thepghoarddaemon.prefix(default: site name)Path prefix to use for all backups related to this site.  Defaults to the
name of the site.Alert filesAlert files are created whenever an error condition that requires human
intervention to solve.  You’re recommended to add checks for the existence
of these files to your alerting system.authentication_errorThere has been a problem in the authentication of at least one of the
PostgreSQL connections.  This usually denotes a wrong username and/or
password.configuration_errorThere has been a problem in the authentication of at least one of the
PostgreSQL connections.  This usually denotes a missingpg_hba.confentry or
incompatible settings in postgresql.conf.upload_retries_warningUpload of a file has failed more times thanupload_retries_warning_limit. Needs human intervention to figure
out why and to delete the alert once the situation has been fixed.version_mismatch_errorYour local PostgreSQL client versions ofpg_basebackuporpg_receivexlogdo not match with the servers PostgreSQL version.  You
need to update them to be on the same version level.version_unsupported_errorServer PostgreSQL version is not supported.LicensePGHoard is licensed under the Apache License, Version 2.0. Full license text
is available in theLICENSEfile and athttp://www.apache.org/licenses/LICENSE-2.0.txtCreditsPGHoard was created by Hannu Valtonen <hannu.valtonen@ohmu.fi> forAiven Cloud Databaseand is now maintained byOhmu Ltdhackers and
Aiven developers <pghoard@ohmu.fi>.Recent contributors are listed on the GitHub project page,https://github.com/aiven/pghoard/graphs/contributorsContactBug reports and patches are very welcome, please post them as GitHub issues
and pull requests athttps://github.com/aiven/pghoard.  Any possible
vulnerabilities or other serious issues should be reported directly to the
maintainers <opensource@ohmu.fi>.CopyrightCopyright (C) 2015 Ohmu Ltd