Repository: borgbackup/borg

Issue:
upgrade zstd to 1.3.4
https://github.com/facebook/zstd/blob/dev/NEWS

speed improvements!

mostly for decompress, so we'll give it a bit of testing and bundle it with 1.1.6.


Discussion:
Nice! Thanks a lot!
Any chance of giving individual parameters to `zstd` soon? :-) (Like the new `--long` option)
@smoneck we use some zstd library calls (not zstd via cli), so these options would be parameters somewhere.

Using something like `--long` to increase the zstd window size to 128MB would likely make no difference as borg with default chunk target size of 2MiB only has about that much to compress at a time.
> Using something like --long to increase the zstd window size to 128MB would likely make no difference as borg with default chunk target size of 2MiB only has about that much to compress at a time.

Thank you for the answer! I didn't know that! Yes, then it's useless unless one fumbles around with the chunk size ... hmm anyway I'm glad zstd is on board! Thank you for all the work! :-)
IIRC the max buffer size of borg is ~8MiB currently, so tweaking chunker params would not give a big enough buffer for zstd `--long` either.
Yah, I just found it minutes ago on the [mailing list](https://mail.python.org/pipermail/borgbackup/2016q4/000461.html). I thought the `--long` option would go nice together with large data sets but I might open a can of worms ... I think I'll just stick with the standard options here. Interesting to me would be the effect on the zstd compression level of such small chunks (zstd uses a [default of 128 MB (27)](https://facebook.github.io/zstd/zstd_manual.html)). 
I couldn't sleep and thought let's look at the chunks ...

I took my largest journal file, cut out the biggest possible chunk (2^26) and tested it for compression efficiency using zstd command line.
Then I took compression level 8 and tried zstd's chunking procedure (`--long`) on that file.
I then took 17 consecutive smaller file sizes (10..26) trying to reproduce borg's chunking and tested them for compression efficiency (in good belief that the file is homogeneous in compression but what seems to be the case) again with level 8.

It turned out that -- if I didn't misunderstood or did something wrong -- previous chunking seems to work way worse for small sizes using zstd and an increase from 22 to 25 would _double_ the compression (on level 8).

P.s.: There is a tiny tiny difference between `default` and `--long=21`. For the sake of completeness I hid it as a red curve.

![benchmark](https://user-images.githubusercontent.com/7861774/38474346-0f005622-3b9e-11e8-958f-a5f0db0ecd8f.png)
[As PDF](https://github.com/borgbackup/borg/files/1888088/benchmark.pdf)


----
**Preparation**
```bash
$ dd if=mylargestjournal.journal bs=512 count=131072 of=/home/user/Downloads/chunktest
$ du -s chunktest 
65536	chunktest
for i in {1..17};do dd if=mylargestfile.journal bs=512 count=$((2**i)) of=/home/user/Downloads/chunktest$((i+9));done
```
**Evaluation**
```bash
$ for n in {1..22}; do zstd -b$n --long=21 -r -T2 chunktest 2>&1 | grep -oP '\(\K[^\)]+' | head -1 | xargs echo $n; done
$ for n in {10..30}; do zstd -b8 --long=$n -r -T2 chunktest 2>&1 | grep -oP '\(\K[^\)]+' | head -1 | xargs echo $n; done
$ for n in {10..26}; do zstd -b8 -r -T2 chunktest$n 2>&1 | grep -oP '\(\K[^\)]+' | head -1 | xargs echo $n; done
```
P.p.s.: FYI, I just drop the following here. I already [had the code](https://gist.github.com/smoneck/f38ad8d220d1e06e43d1c44efcd035a8) and just needed to add `zstd` and press "go". Maybe it's helpful for someone. :-)

![eval](https://user-images.githubusercontent.com/7861774/38508268-07d8ba48-3c1f-11e8-99e9-223654108cdc.png)
[As PDF](https://github.com/borgbackup/borg/files/1890627/eval.pdf)


Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -20,7 +20,7 @@
 prefer_system_liblz4 = True
 
 # True: use the shared libzstd (>= 1.3.0) from the system, False: use the bundled zstd code
-prefer_system_libzstd = True
+prefer_system_libzstd = False
 
 # True: use the shared libb2 from the system, False: use the bundled blake2 code
 prefer_system_libb2 = True


Test patch: