srlefevre/zfs-repl — pull request #4 by whorfin (2014-06-19 20:40:06+00:00)
Title: New features and a few bugfixes and optimizations

Greetings

I've made some changes you might want to merge back to mainline

1) Update target-config to deal with restricted ssh PATH and find
socat on OmniOS with Joyent/SmartOS overlay

2) socat support refinements
a) Run socat in unidirectional mode. Necessary in my
tests to avoid the dreaded "invalid backup stream" failure.
b) Add support for additional arguments to socat [such as
timeout -t or blocksize -b]. Use excplicit STDIO address for clarity.
c) Add support for arguments to tcp: and tcp-listen: addressess,
such as retry= and interval=. Done in such a way as to work as
originally provided [default retry=5 on sender]
Note/TODO: If we drop support for everything other than socat, the
elaborate "---" handshake for starting up the pipelines is not needed.
",retry=${REPL_INIT_TIMEOUT},interval=1" on the sender would have the current
default behavior of REPL_INIT_TIMEOUT seconds timeout to fire up. socat seems
to run on everything so it is quite portable. Moving mbuffer to be in the
pipeline, with socat always as network transport, would allow big buffer
support to still be supported - ie
zfs send ... | mbuffer | socat --->
socat | mbufer | ... | zfs recv
I don't have mbuffer so haven't tested, but don't seem to need as i can
saturate Gbit link with just socat.

3) Fix to snapshot processing on --clean-snaps
This could be important - the grep was constructed in such a way that
substrings were erroneously matched.

4) Pipeline optimization.
Both SSH and [more complicatedly] the other network transport protocols
are built and pruned for a minimal pipeline. This means no more
"| cat - |". I was seeing approximately 5-10% CPU hit for each no-op cat
entry; this removes that unnecessary usage. As a bonus the actual
pipeline execution code is simpler to read and we guarantee that
the DEBUG pipe is actually what gets run.

5) Two new cleanup modes.
These are necessary in my use case, where my server [the source of
replication] already has an elaborate snapshot and snapshot retention
policy. hourly/daily/weekly/etc I set SNAP_PREF=RemoteBackup and do
one of
a) --clean-noprefix-snaps
Removes all snapshots on target which don't contain SNAP_PREF.
Effectively this uses "dest-snap-retain" and the frequency of
invocation of zfs-repl as the only snapshot policy on the target; any
additional snapshots on source are stripped.

b) --clean-noprefix-nosrc-snaps
Consider the set of snapshots on target which don't contain SNAP_PREF.
For them, delete any on target not present on source. This is done to
the child level, so recursive replication doesn't assume full
snapshots chains from child to parent for all snapshot types.
Effectively this allows the source to define the snapshot and retention
policy for the target for all non-SNAP_PREF snapshots, in addition to
the SNAP_PREF snapshots.

Disclaimer: tested only on OmniOS, but tried to write portable bash