meskyanichi/backup — Issue #241: Backup data pipelining
Author: phene
Created: 2011-12-02 18:23:36+00:00
State: open

I've come to realize how disconnected each stage of the backup currently is, and I wanted to open up a dialog on refactoring this gem so that we can stream data through each stage rather than perform each read/write separately to the disk.

Currently, of I specify the following configuration

``` ruby
Backup::Model.new(:database_and_assets, 'full database and asset backup') do

database MySQL do |database|
database.host = DB_CONFIG['host']
database.name = DB_CONFIG['database']
database.username = DB_CONFIG['username']
database.password = DB_CONFIG['password']
database.additional_options = ['--single-transaction']
end

archive :assets do |archive|
archive.add "/path/to/asset/data"
end

compress_with Gzip do |compression|
compression.best = true
compression.fast = false
end

encrypt_with OpenSSL do |encryption|
encryption.password = "my_password"
encryption.salt = true
encryption.base64 = false
end

store_with Local do |local|
local.path = "/some/mount/of/remote/device'
local.keep = 4
end
end
```

When I trigger the backup, the following occurs:

1. Mysql dump to .sql file
2. Tar asset data
3. tar results of 1 and 2 together
4. gzip tar file from 3
5. encrypt tar.gz from 4
6. copy tar.gz.enc to destination path

It seems to me that we could easily combine steps 3, 4, 5, and 6 together, so that instead of performing a full read and write of the data 4 times, we only do it once. This could be accomplished by:

1. Enabling the compress_with Gzip statement to inform the 'final tar' that the -z option should be used
2. Enabling the encrypt_with OpenSSL statement to inform the 'final tar' that it should be piped into the openssl command with the correct options
3. Enabling the store_with Local statement to inform the final tar where it should be saved to instead of writing to a tmp file and copying.

These enhancements may become critical as backups get larger in size and disk/network I/O time can not be fully controlled.

I don't have any specific code change proposals yet, but I wanted to get a the rest of the communities thoughts on this problem.