Repository: borgbackup/borg

Issue:
borg2 throws error for valid(?) chunker parameters
## Have you checked borgbackup docs, FAQ, and open GitHub issues?

Yes
## Is this a BUG / ISSUE report or a QUESTION?

A bug, probably

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).

borg-2.0.0b5

#### Operating system (distribution) and version.

Arch Linux 

#### Hardware / network configuration, and filesystems used.

Linux, borg-2.0.0b5 precompiled binary, ext4, backup to HDD

#### How much data is handled by borg?

a few GB for testing borg2 beta

#### Full borg commandline that lead to the problem (leave away excludes and passwords)

```
borg-2.0.0b5-linuxnew64 create --chunker-params buzhash,19,24,21,4095 MyArchive /path/to/source
```
(I didn't try this with borg-1.2.x)

## Describe the problem you're observing.

tl;dr: running the above command, I get the following error message:

```
borg-2.0.0b5-linuxnew64 create: error: argument --chunker-params: invalid ChunkerParams value: 'buzhash,19,24,21,4095'
```
Note the value of 24 for CHUNK_MAX_EXP.

Long story: I have a backup source with only large (several hundred MiB each) and seldom-changing files and wanted to adjust the chunker parameters for that by going from the default `buzhash,19,23,21,4095` (512 kiB min, 8 MiB max, 2 MiB avg chunk size) to twice that size (i.e. 1 MiB / 16 MiB / 4 MiB)  with `buzhash,20,24,22,4095` (and maybe even to still larger chunk sizes). However, this throws the error message given above. Some try-and-error revealed that the CHUNK_MAX_EXP parameter is to blame. Selecting anything larger than 23 (i.e. 8 MiB)  for CHUNK_MAX_EXP will trigger the error.

I could not find out why `--chunker-params buzhash,20,24,22,4095` should be invalid. The documentation says nothing about limits and/or inter-dependencies for the chunker values. 

Is this a bug? Maybe a leftover from the "old attic times" where nobody could imagine chunk sizes > 8 MiB? 






Discussion:
Not sure why you consider this a bug, I'ld rather say this was like this like forever and is expected due to CHUNK_MAX_EXP being defined to 23 (see src/borg/constants.py).

Some buffers and data structures in borg have a specific size, so this can't grow arbitrarily - also the max object size in the repo is intentionally limited to ~20MB.

Pros of bigger chunks:
- less management overhead (e.g. disk and memory use for the misc. hashtables) IF you have lot of big files

Cons of bigger chunks:
- less fine granularity of deduplication

In the old attic times, chunks were ~64kiB (now they are ~2MiB by default).

If CHUNK_MAX_EXP=23 is fixed and the maximum, I can live with that. :-)

But then shouldn't it be documented somewhere (not only in source code) that CHUNK_MAX_EXP is fixed / has a hard limit?  [The docs](https://borgbackup.readthedocs.io/en/master/internals/data-structures.html#chunker-details) don't say anything about hard-codes maximum values. At least the error message could be a bit more detailed to not leave people in the dark what they did wrong, no?
I could imagine that maybe 24 (16MiB) could also still work, but guess nobody tried and reviewed the code for that yet.

About the error msg: master branch code has this:

https://github.com/borgbackup/borg/blob/2.0.0b5/src/borg/helpers/parseformat.py#L162

Did you maybe try that with an older borg version?
Strange, that code somehow does not work as expected, I just reproduced it.

Crap, `ValueError(msg)` doesn't yield the error msg given as argument, one needs to raise `argparse.ArgumentTypeError(msg)`.

Thanks for reporting this, **that is a bug**. 🤣

The stuff in `parseformat.py` (and all that is used as `type=...` in argparser setup) needs a global review.
This line should probably have parentheses to make the intended logic clear:
```
if algo == CH_BUZHASH and count == 5 or count == 4:
```


> This line should probably have parentheses to make the intended logic clear:
> 
> ```
> if algo == CH_BUZHASH and count == 5 or count == 4:
> ```

I agree... I had to look up operator precedence in the [Python docs](https://docs.python.org/3/reference/expressions.html#operator-precedence)... for the record: `and` binds stronger than `or`, so the logic is correct.
Just an additional data point.... the [Python docs for argparse](https://docs.python.org/3/library/argparse.html#type) say the following:

> The argument to type can be any callable that accepts a single string. If the function raises ArgumentTypeError, [TypeError](https://docs.python.org/3/library/exceptions.html#TypeError), or [ValueError](https://docs.python.org/3/library/exceptions.html#ValueError), the exception is caught and a nicely formatted error message is displayed. No other exception types are handled.

So ValueError should work and be ok?

Resolving patch:
diff --git a/src/borg/helpers/parseformat.py b/src/borg/helpers/parseformat.py
--- a/src/borg/helpers/parseformat.py
+++ b/src/borg/helpers/parseformat.py
@@ -137,7 +137,7 @@ def ChunkerParams(s):
     params = s.strip().split(",")
     count = len(params)
     if count == 0:
-        raise ValueError("no chunker params given")
+        raise argparse.ArgumentTypeError("no chunker params given")
     algo = params[0].lower()
     if algo == CH_FAIL and count == 3:
         block_size = int(params[1])
@@ -152,9 +152,11 @@ def ChunkerParams(s):
             # or in-memory chunk management.
             # choose the block (chunk) size wisely: if you have a lot of data and you cut
             # it into very small chunks, you are asking for trouble!
-            raise ValueError("block_size must not be less than 64 Bytes")
+            raise argparse.ArgumentTypeError("block_size must not be less than 64 Bytes")
         if block_size > MAX_DATA_SIZE or header_size > MAX_DATA_SIZE:
-            raise ValueError("block_size and header_size must not exceed MAX_DATA_SIZE [%d]" % MAX_DATA_SIZE)
+            raise argparse.ArgumentTypeError(
+                "block_size and header_size must not exceed MAX_DATA_SIZE [%d]" % MAX_DATA_SIZE
+            )
         return algo, block_size, header_size
     if algo == "default" and count == 1:  # default
         return CHUNKER_PARAMS
@@ -162,14 +164,18 @@ def ChunkerParams(s):
     if algo == CH_BUZHASH and count == 5 or count == 4:  # [buzhash, ]chunk_min, chunk_max, chunk_mask, window_size
         chunk_min, chunk_max, chunk_mask, window_size = (int(p) for p in params[count - 4 :])
         if not (chunk_min <= chunk_mask <= chunk_max):
-            raise ValueError("required: chunk_min <= chunk_mask <= chunk_max")
+            raise argparse.ArgumentTypeError("required: chunk_min <= chunk_mask <= chunk_max")
         if chunk_min < 6:
             # see comment in 'fixed' algo check
-            raise ValueError("min. chunk size exponent must not be less than 6 (2^6 = 64B min. chunk size)")
+            raise argparse.ArgumentTypeError(
+                "min. chunk size exponent must not be less than 6 (2^6 = 64B min. chunk size)"
+            )
         if chunk_max > 23:
-            raise ValueError("max. chunk size exponent must not be more than 23 (2^23 = 8MiB max. chunk size)")
+            raise argparse.ArgumentTypeError(
+                "max. chunk size exponent must not be more than 23 (2^23 = 8MiB max. chunk size)"
+            )
         return CH_BUZHASH, chunk_min, chunk_max, chunk_mask, window_size
-    raise ValueError("invalid chunker params")
+    raise argparse.ArgumentTypeError("invalid chunker params")
 
 
 def FilesCacheMode(s):
@@ -177,11 +183,13 @@ def FilesCacheMode(s):
     VALID_MODES = ("cis", "ims", "cs", "ms", "cr", "mr", "d", "s")  # letters in alpha order
     entries = set(s.strip().split(","))
     if not entries <= set(ENTRIES_MAP):
-        raise ValueError("cache mode must be a comma-separated list of: %s" % ",".join(sorted(ENTRIES_MAP)))
+        raise argparse.ArgumentTypeError(
+            "cache mode must be a comma-separated list of: %s" % ",".join(sorted(ENTRIES_MAP))
+        )
     short_entries = {ENTRIES_MAP[entry] for entry in entries}
     mode = "".join(sorted(short_entries))
     if mode not in VALID_MODES:
-        raise ValueError("cache mode short must be one of: %s" % ",".join(VALID_MODES))
+        raise argparse.ArgumentTypeError("cache mode short must be one of: %s" % ",".join(VALID_MODES))
     return mode
 
 
@@ -277,7 +285,7 @@ def SortBySpec(text):
 
     for token in text.split(","):
         if token not in AI_HUMAN_SORT_KEYS:
-            raise ValueError("Invalid sort key: %s" % token)
+            raise argparse.ArgumentTypeError("Invalid sort key: %s" % token)
     return text.replace("timestamp", "ts")
 
 
diff --git a/src/borg/testsuite/compress.py b/src/borg/testsuite/compress.py
--- a/src/borg/testsuite/compress.py
+++ b/src/borg/testsuite/compress.py
@@ -1,3 +1,4 @@
+import argparse
 import os
 import zlib
 
@@ -226,7 +227,7 @@ def test_obfuscate_meta():
 
 
 def test_compression_specs():
-    with pytest.raises(ValueError):
+    with pytest.raises(argparse.ArgumentTypeError):
         CompressionSpec("")
 
     assert isinstance(CompressionSpec("none").compressor, CNONE)
@@ -241,7 +242,7 @@ def test_compression_specs():
     zlib = CompressionSpec("zlib,9").compressor
     assert isinstance(zlib, ZLIB)
     assert zlib.level == 9
-    with pytest.raises(ValueError):
+    with pytest.raises(argparse.ArgumentTypeError):
         CompressionSpec("zlib,9,invalid")
 
     lzma = CompressionSpec("lzma").compressor
@@ -264,7 +265,7 @@ def test_compression_specs():
     assert isinstance(zstd, ZSTD)
     assert zstd.level == 22
 
-    with pytest.raises(ValueError):
+    with pytest.raises(argparse.ArgumentTypeError):
         CompressionSpec("lzma,9,invalid")
-    with pytest.raises(ValueError):
+    with pytest.raises(argparse.ArgumentTypeError):
         CompressionSpec("invalid")
diff --git a/src/borg/testsuite/helpers.py b/src/borg/testsuite/helpers.py
--- a/src/borg/testsuite/helpers.py
+++ b/src/borg/testsuite/helpers.py
@@ -368,19 +368,19 @@ def test_chunkerparams():
     assert ChunkerParams("fixed,4096") == ("fixed", 4096, 0)
     assert ChunkerParams("fixed,4096,200") == ("fixed", 4096, 200)
     # invalid values checking
-    with pytest.raises(ValueError):
+    with pytest.raises(ArgumentTypeError):
         ChunkerParams("crap,1,2,3,4")  # invalid algo
-    with pytest.raises(ValueError):
+    with pytest.raises(ArgumentTypeError):
         ChunkerParams("buzhash,5,7,6,4095")  # too small min. size
-    with pytest.raises(ValueError):
+    with pytest.raises(ArgumentTypeError):
         ChunkerParams("buzhash,19,24,21,4095")  # too big max. size
-    with pytest.raises(ValueError):
+    with pytest.raises(ArgumentTypeError):
         ChunkerParams("buzhash,23,19,21,4095")  # violates min <= mask <= max
-    with pytest.raises(ValueError):
+    with pytest.raises(ArgumentTypeError):
         ChunkerParams("fixed,63")  # too small block size
-    with pytest.raises(ValueError):
+    with pytest.raises(ArgumentTypeError):
         ChunkerParams("fixed,%d,%d" % (MAX_DATA_SIZE + 1, 4096))  # too big block size
-    with pytest.raises(ValueError):
+    with pytest.raises(ArgumentTypeError):
         ChunkerParams("fixed,%d,%d" % (4096, MAX_DATA_SIZE + 1))  # too big header size
 
 


Test patch: