jpwoodbu/ari-backup — Issue #2: when a user cancels a backup, cleanup does not occur
Author: jpwoodbu
Created: 2012-02-28 19:53:21+00:00
State: open

While executing a backup job I realized that the hooks had been configured to perform work in the wrong order so I cancelled the job with a cntl-c. Here's what my console printed:

jwoodbury@vger:/etc/ari-backup/jobs.d$ sudo ./orpheus
^CARIBackup (orpheus) [ERROR] backup job cancelled by user
Traceback (most recent call last):
File "./orpheus", line 30, in
backup.run_backup()
File "/usr/local/lib/python2.6/dist-packages/ari_backup/__init__.py", line 158, in run_backup
self.logger.error('backup job cancelled by user')
File "/usr/lib/python2.6/logging/__init__.py", line 1082, in error
self._log(ERROR, msg, args, **kwargs)
File "/usr/lib/python2.6/logging/__init__.py", line 1173, in _log
self.handle(record)
File "/usr/lib/python2.6/logging/__init__.py", line 1183, in handle
self.callHandlers(record)
File "/usr/lib/python2.6/logging/__init__.py", line 1220, in callHandlers
hdlr.handle(record)
File "/usr/lib/python2.6/logging/__init__.py", line 679, in handle
self.emit(record)
File "/usr/lib/python2.6/logging/handlers.py", line 793, in emit
self.socket.send(msg)
KeyboardInterrupt

The logging output:

Feb 28 14:34:16 vger ARIBackup (orpheus) [INFO] started
Feb 28 14:34:16 vger ARIBackup (orpheus) [INFO] processing pre-job hooks...
Feb 28 14:34:16 vger ARIBackup (orpheus) [INFO] creating LVM snapshots...
Feb 28 14:34:16 vger ARIBackup (orpheus) [DEBUG] _run_command ['/usr/bin/ssh', 'root@cronus', 'lvcreate', '-s', '-L', '1G', 'cronus-vg0/orpheus-root', '-n', 'orpheus-root-ari_backup']
Feb 28 14:34:17 vger ARIBackup (orpheus) [DEBUG] Logical volume "orpheus-root-ari_backup" created
Feb 28 14:34:17 vger ARIBackup (orpheus) [DEBUG] _run_command ['/usr/bin/ssh', 'root@cronus', 'lvcreate', '-s', '-L', '1G', 'cronus-vg0/orpheus-postgres', '-n', 'orpheus-postgres-ari_backup']
Feb 28 14:34:18 vger ARIBackup (orpheus) [DEBUG] Logical volume "orpheus-postgres-ari_backup" created
Feb 28 14:34:18 vger ARIBackup (orpheus) [DEBUG] _run_command ['/usr/bin/ssh', 'root@cronus', 'lvcreate', '-s', '-L', '1G', 'cronus-vg0/orpheus-mysql', '-n', 'orpheus-mysql-ari_backup']
Feb 28 14:34:18 vger ARIBackup (orpheus) [DEBUG] Logical volume "orpheus-mysql-ari_backup" created
Feb 28 14:34:18 vger ARIBackup (orpheus) [DEBUG] _run_command ['/usr/bin/ssh', 'root@cronus', 'lvcreate', '-s', '-L', '1G', 'cronus-vg0/orpheus-backups', '-n', 'orpheus-backups-ari_backup']
Feb 28 14:34:19 vger ARIBackup (orpheus) [DEBUG] Logical volume "orpheus-backups-ari_backup" created
Feb 28 14:34:19 vger ARIBackup (orpheus) [INFO] mounting LVM snapshots...
Feb 28 14:34:19 vger ARIBackup (orpheus) [DEBUG] _run_command ['/usr/bin/ssh', 'root@cronus', 'mkdir', '-p', '/tmp/orpheus/']
Feb 28 14:34:19 vger ARIBackup (orpheus) [DEBUG] _run_command ['/usr/bin/ssh', 'root@cronus', 'mount', '/dev/cronus-vg0/orpheus-root-ari_backup', '/tmp/orpheus/']
Feb 28 14:34:23 vger ARIBackup (orpheus) [DEBUG] _run_command ['/usr/bin/ssh', 'root@cronus', 'mkdir', '-p', '/tmp/orpheus/var/lib/postgresql']
Feb 28 14:34:24 vger ARIBackup (orpheus) [DEBUG] _run_command ['/usr/bin/ssh', 'root@cronus', 'mount', '/dev/cronus-vg0/orpheus-postgres-ari_backup', '/tmp/orpheus/var/lib/postgresql']
Feb 28 14:34:24 vger ARIBackup (orpheus) [DEBUG] _run_command ['/usr/bin/ssh', 'root@cronus', 'mkdir', '-p', '/tmp/orpheus/var/lib/mysql']
Feb 28 14:34:24 vger ARIBackup (orpheus) [DEBUG] _run_command ['/usr/bin/ssh', 'root@cronus', 'mount', '/dev/cronus-vg0/orpheus-mysql-ari_backup', '/tmp/orpheus/var/lib/mysql']
Feb 28 14:34:26 vger ARIBackup (orpheus) [DEBUG] _run_command ['/usr/bin/ssh', 'root@cronus', 'mkdir', '-p', '/tmp/orpheus/var/backups']
Feb 28 14:34:27 vger ARIBackup (orpheus) [DEBUG] _run_command ['/usr/bin/ssh', 'root@cronus', 'mount', '/dev/cronus-vg0/orpheus-backups-ari_backup', '/tmp/orpheus/var/backups']
Feb 28 14:34:27 vger ARIBackup (orpheus) [DEBUG] _run_command ['/usr/bin/ssh', 'root@orpheus', 'su', 'postgres', '-c', 'pg_dumpall', '>', '/var/backups/pgdump.sql']
Feb 28 14:34:30 vger ARIBackup (orpheus) [ERROR] backup job cancelled by user
Feb 28 14:34:30 vger ARIBackup (orpheus) [INFO] stopped

The job script:

#!/usr/bin/env python

from ari_backup.zfs import ZFSLVMBackup

label = 'orpheus'

backup = ZFSLVMBackup(
label=label,
source_hostname='cronus',
rsync_dst='alaska:/volumes/zpool-0/backups/ari-backup/' + label,
zfs_hostname='alaska',
dataset_name='zpool-0/backups/ari-backup/' + label,
snapshot_expiration_days=2
)

# add pre-job hook to dump the PostgreSQL database to a script file since we
# run it with fsync = off
# see rm2998
pg_dump_command = 'su postgres -c pg_dumpall > /var/backups/pgdump.sql'
backup.pre_job_hook_list.append((
backup._run_command,
{'command': pg_dump_command, 'host': 'orpheus'}
))

backup.lv_list.append(('cronus-vg0/orpheus-root', '/'))
backup.lv_list.append(('cronus-vg0/orpheus-postgres', '/var/lib/postgresql'))
backup.lv_list.append(('cronus-vg0/orpheus-mysql', '/var/lib/mysql'))
backup.lv_list.append(('cronus-vg0/orpheus-backups', '/var/backups'))

backup.run_backup()

Admittedly, this is the first job script I've made that adds its own pre-job hook. Maybe there's something to that. In this case, the LVs created and mounted remained as such.