Package: lf-backup
LF Backup stands for large file backup.InstallationRHEL / CentOS:sudo yum -y install epel-release
sudo yum -y install python34 python34-setuptools python34-devel gcc postgresql-devel
sudo easy_install-3.4 pipDebian/Ubuntu:sudo apt-get install -y python3-dev libpq-devand then install lf-backupsudo pip3 install --upgrade pip
sudo pip3 install lf-backupConfigurationcreate a new swift container called “large-file-backup”add the export statements for variables starting with ST_ and the postgres authentication to config
file .lf-backuprc and set the permissions to 600. Optionally export PGSQL to override the built-in
SQL query.> nano ~/.lf-backuprc
> chmod 600 ~/.lf-backuprc
> cat ~/.lf-backuprc
export ST_AUTH=https://swiftcluster.domain.org/auth/v1.0
export ST_USER=swift_account
export ST_KEY=RshBXXXXXXXXXXXXXXXXXXXXX
export PGHOST=pgdb.domain.org
export PGPORT=32048
export PGDATABASE=storcrawldb
export PGUSER=xxxxxxxx
export PGPASSWORD=create a cron job /etc/cron.d/ running as root starting ca 7pm:> cat /etc/cron.d/lf-backup
## enabled on hostname xxx on 11-01-2016
55 18 * * * root /usr/local/bin/lf-backup --prefix /fh/fast \
           --container large-file-backup-fast --sql >> /var/tmp/lf-backup-fast 2>&1Exampleslf-backup -C frobozz -c filelist.csvRead list of files from 1st column of ‘filename.csv’ and backup to Swift container ‘frobozz’ using
environment for authentication.lf-backup -C grue -sQuery the database specified in the environment for the files and backup to Swift container ‘grue’
using environment for authentication.lf-backup -C flathead -r 7 --prefix /fh/fast/restore42Restore all objects in Swift container ‘flathead’ newer than 7 days back to current environment. The
optional –prefix parameter specifies a destination path where objects will be restored.Test & DevFor modifications and change testing install a new system and install from local git folder> git clone https://github.com/FredHutch/lf-backup
> rm -rf /usr/local/lib/python3.5/dist-packages/*; rm -rf /usr/local/bin/*
> pip3 install -e ./lf-backup

make changes in lf-backup and run again:

> rm -rf /usr/local/lib/python3.5/dist-packages/*; rm -rf /usr/local/bin/*
> pip3 install -e ./lf-backupThe script had the following original feature requests:take a file list from CSV file or SQL DB and backup each file to object storage (e.g. swift)restore files from object storage newer than specified number of days (>1)if the file has an atime within the last x days (configurable) take an md5sum of that file and
store the md5sum in an attribute / meta data called md5sum (not yet implemented)check if the file is already in object store and do not upload if the file size and mtime is
identicalnotify a list of email-addresses after finishing. attach list of files that were uploaded; create
one file list per file owner (username)log every file that was uploaded to syslog, detailed logging of success and failure to enable
storage team to monitor success / failure via splunkbash script lf-backup is a wrapper for python script lfbackup.py, lf-backup sources and sets env
vars with credentials and lfbackup.py only reads environments varsmain script lfbackup.py only uses swift functions in lflib.py.segment size should be 1GB, segment container name should be .segments-containername, object type
is slo, not dlobackup with full path but replace prefix, for example a file
/fh/fast/lastname_f/project/file.bam would be copied to container/bucket bam-backup in account
Swift__ADM_IT_backup. The target path would be /bam-bucket/lastname_f/project/file.bam a
–prefix=/fh/fast removes the fs root path from the destination