Package: pdiffcopy
The pdiffcopy program synchronizes large binary data files between Linux
servers at blazing speeds by performing delta transfers and spreading its work
over many CPU cores. It’s currently tested onPython2.7, 3.5+ and PyPy (2.7)
on Ubuntu Linux but is expected to work on most Linux systems.StatusInstallationCommand lineBenchmarksLow concurrencyHigh concurrencySilly concurrencyLimitationsHistoryAbout the nameContactLicenseStatusAlthough the first prototype of pdiffcopy was developed back in June 2019 it
wasn’t until March 2020 that the first release was published as an open source
project.NoteThis is analpha release, meaning it’s not considered mature and
you may encounter bugs. As such, if you’re going to use pdiffcopy,
I would suggest you to keep backups, be cautious and sanity check
your results.There are lots of features and improvements I’d love to add but more
importantly the project needs to actually be used for a while before
I’ll consider changing the alpha label to beta or mature.InstallationThe pdiffcopy package is available onPyPIwhich means installation should be
as simple as:$pipinstall'pdiffcopy[client,server]'There’s actually a multitude of ways to installPythonpackages (e.g. theper user site-packages directory,virtual environmentsor just installing
system wide) and I have no intention of getting into that discussion here, so
if this intimidates you then read up on your options before returning to these
instructions 😉.The names between the square brackets (clientandserver) are called
“extras” and they enable you to choose whether to install the client
dependencies, server dependencies or both.Command lineUsage:pdiffcopy [OPTIONS] [SOURCE, TARGET]Synchronize large binary data files between Linux servers at blazing speeds
by performing delta transfers and spreading the work over many CPU cores.One of the SOURCE and TARGET arguments is expected to be the pathname of a
local file and the other argument is expected to be a URL that provides the
location of a remote pdiffcopy server and a remote filename. File data will be
read from SOURCE and written to TARGET.If no positional arguments are given the server is started.Supported options:OptionDescription-b,--block-size=BYTESCustomize the block size of the delta transfer. Can be a plain
integer number (bytes) or an expression like 5K, 1MiB, etc.-m,--hash-method=NAMECustomize the hash method of the delta transfer (defaults to ‘sha1’
but supports all hash methods provided by the Python hashlib module).-W,--whole-fileDisable the delta transfer algorithm (skips computing
of hashing and downloads all blocks unconditionally).-c,--concurrency=COUNTChange the number of parallel block hash / copy operations.-n,--dry-runScan for differences between the source and target file and report the
similarity index, but don’t write any changed blocks to the target.-B,--benchmark=COUNTEvaluate the effectiveness of delta transfer by mutating the TARGET
file (which must be a local file) and resynchronizing its contents.
This process is repeatedCOUNTtimes, with varying similarity.
At the end an overview is printed.-l,--listen=ADDRESSListen on the specified IP:PORT or PORT.-v,--verboseIncrease logging verbosity (can be repeated).-q,--quietDecrease logging verbosity (can be repeated).-h,--helpShow this message and exit.BenchmarksThe command line interface provides a simple way to evaluate the effectiveness
of the delta transfer implementation and compare it againstrsync. The tables
in the following sections are based on that benchmark.Low concurrencyHigh concurrencySilly concurrencyLow concurrencyConcurrency:6 processes on 4 CPU coresDisks:Magnetic storage(slow)Filesize:1.79 GiBThe following table shows the results of the benchmark on a 1.79 GiB
datafile that’s synchronized between two bare metal servers that each
have four CPU cores and spinning disks, where pdiffcopy was run with
a concurrency of six[1]:DeltaData sizepdiffcopyrsync10%183 MiB3.20 seconds38.55 seconds20%366 MiB4.15 seconds44.33 seconds30%549 MiB5.17 seconds49.63 seconds40%732 MiB6.09 seconds53.74 seconds50%916 MiB6.99 seconds57.49 seconds60%1.07 GiB8.06 seconds1 minute and 0.97 seconds70%1.25 GiB9.06 seconds1 minute and 2.38 seconds80%1.43 GiB10.12 seconds1 minute and 4.20 seconds90%1.61 GiB10.89 seconds1 minute and 3.80 seconds100%1.79 GiB12.05 seconds1 minute and 4.14 seconds[1]Allocating more processes than there are CPU cores available can make
sense when the majority of the time spent by those processes is waiting
for I/O (this definitely applies to pdiffcopy).High concurrencyConcurrency:10 processes on 48 CPU coresDisks:NVMe(fast)Filesize:5.5 GiBHere’s a benchmark based on a 5.5 GB datafile that’s synchronized between two
bare metal servers that each have 48 CPU cores and high-endNVMedisks, where
pdiffcopy was run with a concurrency of ten:DeltaData sizepdiffcopyrsync10%562 MiB4.23 seconds49.96 seconds20%1.10 GiB6.76 seconds1 minute and  2.38 seconds30%1.65 GiB9.43 seconds1 minute and 13.73 seconds40%2.20 GiB12.41 seconds1 minute and 19.67 seconds50%2.75 GiB14.54 seconds1 minute and 25.86 seconds60%3.29 GiB17.21 seconds1 minute and 26.97 seconds70%3.84 GiB19.79 seconds1 minute and 27.46 seconds80%4.39 GiB23.10 seconds1 minute and 26.15 seconds90%4.94 GiB25.19 seconds1 minute and 21.96 seconds100%5.43 GiB27.82 seconds1 minute and 19.17 secondsThis benchmark shows how well pdiffcopy can scale up its performance by running
on a large number of CPU cores. Notice how the smaller the delta is, the bigger
the edge is that pdiffcopy has overrsync? This is because pdiffcopy computes
the differences between the local and remote file using many CPU cores at the
same time. This operation requires only reading, and that parallelizes
surprisingly well on modernNVMedisks.Silly concurrencyConcurrency:20 processes on 48 CPU coresDisks:NVMe(fast)Filesize:5.5 GiBIn case you looked at the high concurrency benchmark above, noticed the large
number of CPU cores available and wondered whether increasing the concurrency
further would make a difference, this section is for you 😉. Having taken the
effort of developing pdiffcopy and enabling it to run on many CPU cores I was
curious myself so I reran the high concurrency benchmark using 20 processes
instead of 10. Here are the results:DeltaData sizepdiffcopyrsync10%562 MiB3.80 seconds49.71 seconds20%1.10 GiB6.25 seconds1 minute and  3.37 seconds30%1.65 GiB8.90 seconds1 minute and 12.40 seconds40%2.20 GiB11.44 seconds1 minute and 19.57 seconds50%2.75 GiB14.21 seconds1 minute and 25.43 seconds60%3.29 GiB16.45 seconds1 minute and 28.12 seconds70%3.84 GiB19.05 seconds1 minute and 28.34 seconds80%4.39 GiB21.95 seconds1 minute and 25.49 seconds90%4.94 GiB24.60 seconds1 minute and 22.27 seconds100%5.43 GiB26.42 seconds1 minute and 18.73 secondsAs you can see increasing the concurrency from 10 to 20 does make the benchmark
a bit faster, however the margin is so small that it’s hardly worth bothering.
I interpret this to mean that theNVMedisks on these servers can be more or
less saturated using 8–12 writer processes.NoteIn the end the question is how many CPU cores it takes to saturate
your storage infrastructure. This can be determined through
experimentation, which the benchmark can assist with. There are no
fundamental reasons why 30 or even 50 processes couldn’t work well,
as long as your storage infrastructure can keep up…LimitationsWhile inspired byrsyncthe goal definitely isn’t feature parity withrsync.
Right now only single files can be transferred and only the file data is
copied, not the metadata. It’s a proof of concept that works but is limited.
While I’m tempted to add support for synchronization of directory trees and
file metadata just because its convenient, it’s definitely not my intention to
compete withrsyncin the domain of synchronizing large directory trees,
because I would most likely fail.Error handling is currently very limited and interrupting the program using
Control-C may get you stuck with an angry pool ofmultiprocessingworkers that
refuse to shut down 😝. In all seriousness, hitting Control-C a couple of times
should break out of it, otherwise try Control-\ (that’s a backslash, it should
send a QUIT signal).HistoryIn June 2019 I found myself in a situation where I wanted to quickly
synchronize large binary datafiles (a small set of very largeMySQL*.ibdfiles totaling several hundred gigabytes) using the abundant
computing resources available to me (48 CPU cores,NVMedisks,
bonded network interfaces, you name it 😉).I spent quite a bit of time experimenting with running manyrsyncprocesses in
parallel, but the small number of very large files was “clogging up the pipe”
so to speak, no matter what I did. This was how I realized thatrsyncwas a
really poor fit, which was a disappointment for me becausersynchas long been
one my go-to programs for ad hoc problem solving on Linux servers 🙂.In any case I decided to prove to myself that the hardware available to me
could do much more than whatrsyncwas getting me and after a weekend of
hacking on a prototype I had something that could outperformrsynceven though
it was written inPythonand usedHTTPas a transport 😁. During this weekend
I decided that my prototype was worthy of being published as an open source
project, however it wasn’t until months later that I actually found the time to
do so.About the nameThe name pdiffcopy is intended as a (possibly somewhat obscure) abbreviation of
“Parallel Differential Copy”:Parallel because it’s intended run on many CPU cores.Differential because of the delta transfer mechanism.But mostly I just needed a short, unique name likersyncso that searching for
this project will actually turn up this project instead of a dozen others 😇.ContactThe latest version of pdiffcopy is available onPyPIandGitHub. The
documentation is hosted onRead the Docsand includes achangelog. For bug
reports please create an issue onGitHub. If you have questions, suggestions,
etc. feel free to send me an e-mail atpeter@peterodding.com.LicenseThis software is licensed under theMIT license.© 2020 Peter Odding.