Repository: borgbackup/borg

Issue:
test_size_on_disk_accurate on 1.2.3 breaks when run on ZFS
<!--
Thank you for reporting an issue.

*IMPORTANT* -  *before* creating a new issue please look around:
 - Borgbackup documentation: http://borgbackup.readthedocs.io/en/stable/index.html
 - FAQ: https://borgbackup.readthedocs.io/en/stable/faq.html
 and
 - open issues in Github tracker: https://github.com/borgbackup/borg/issues
  
If you cannot find a similar problem, then create a new issue.

Please fill in as much of the template as possible.
-->

## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

Bug/Regression

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).



#### Operating system (distribution) and version.

NixOS Unstable

#### Hardware / network configuration, and filesystems used.

ZFS Root

#### How much data is handled by borg?

n/a

#### Full borg commandline that lead to the problem (leave away excludes and passwords)

n/a

## Describe the problem you're observing.

#### Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

Running the test suite on ZFS breaks on `test_size_on_disk_accurate` since https://github.com/borgbackup/borg/commit/8ddfd94f0904449175151faed67f9640710b6454.

ZFS has a commit interval of 5 seconds, so a manual flush might be required for the size to be reflected instantly.

#### Include any warning/errors/backtraces from the system logs

```python
_______________ HashIndexSizeTestCase.test_size_on_disk_accurate _______________
[gw42] linux -- Python 3.10.9 /nix/store/ar76b3831ydkmfhf0r9v3wbma5i61iz2-python3-3.10.9/bin/python3.10
self = <borg.testsuite.hashindex.HashIndexSizeTestCase testMethod=test_size_on_disk_accurate>

    def test_size_on_disk_accurate(self):
        idx = ChunkIndex()
        for i in range(1234):
            idx[H(i)] = i, i**2, i**3
        with tempfile.NamedTemporaryFile() as file:
            idx.write(file)
            size = os.path.getsize(file.fileno())
>       assert idx.size() == size
E       assert 90350 == 0
E        +  where 90350 = <built-in method size of borg.hashindex.ChunkIndex object at 0xffffeea51b30>()
E        +    where <built-in method size of borg.hashindex.ChunkIndex object at 0xffffeea51b30> = <borg.hashindex.ChunkIndex object at 0xffffeea51b30>.size

/nix/store/0ppx77a4xk6nij2p1klgsnksmrfnx9lj-borgbackup-1.2.3/lib/python3.10/site-packages/borg/testsuite/hashindex.py:202: AssertionError
```

<!--

If this complaint relates to borg performance, please include CRUD benchmark
results and any steps you took to troubleshoot.
How to run benchmark: http://borgbackup.readthedocs.io/en/stable/usage/benchmark.html

*IMPORTANT* - Please mark logs and text output from terminal commands 
or else Github will not display them correctly. 
An example is provided below.

Example:
```
this is an example how log text should be marked (wrap it with ```)
```
-->

test_size_on_disk_accurate on 1.2.3 breaks when run on ZFS
<!--
Thank you for reporting an issue.

*IMPORTANT* -  *before* creating a new issue please look around:
 - Borgbackup documentation: http://borgbackup.readthedocs.io/en/stable/index.html
 - FAQ: https://borgbackup.readthedocs.io/en/stable/faq.html
 and
 - open issues in Github tracker: https://github.com/borgbackup/borg/issues
  
If you cannot find a similar problem, then create a new issue.

Please fill in as much of the template as possible.
-->

## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

Bug/Regression

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).



#### Operating system (distribution) and version.

NixOS Unstable

#### Hardware / network configuration, and filesystems used.

ZFS Root

#### How much data is handled by borg?

n/a

#### Full borg commandline that lead to the problem (leave away excludes and passwords)

n/a

## Describe the problem you're observing.

#### Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

Running the test suite on ZFS breaks on `test_size_on_disk_accurate` since https://github.com/borgbackup/borg/commit/8ddfd94f0904449175151faed67f9640710b6454.

ZFS has a commit interval of 5 seconds, so a manual flush might be required for the size to be reflected instantly.

#### Include any warning/errors/backtraces from the system logs

```python
_______________ HashIndexSizeTestCase.test_size_on_disk_accurate _______________
[gw42] linux -- Python 3.10.9 /nix/store/ar76b3831ydkmfhf0r9v3wbma5i61iz2-python3-3.10.9/bin/python3.10
self = <borg.testsuite.hashindex.HashIndexSizeTestCase testMethod=test_size_on_disk_accurate>

    def test_size_on_disk_accurate(self):
        idx = ChunkIndex()
        for i in range(1234):
            idx[H(i)] = i, i**2, i**3
        with tempfile.NamedTemporaryFile() as file:
            idx.write(file)
            size = os.path.getsize(file.fileno())
>       assert idx.size() == size
E       assert 90350 == 0
E        +  where 90350 = <built-in method size of borg.hashindex.ChunkIndex object at 0xffffeea51b30>()
E        +    where <built-in method size of borg.hashindex.ChunkIndex object at 0xffffeea51b30> = <borg.hashindex.ChunkIndex object at 0xffffeea51b30>.size

/nix/store/0ppx77a4xk6nij2p1klgsnksmrfnx9lj-borgbackup-1.2.3/lib/python3.10/site-packages/borg/testsuite/hashindex.py:202: AssertionError
```

<!--

If this complaint relates to borg performance, please include CRUD benchmark
results and any steps you took to troubleshoot.
How to run benchmark: http://borgbackup.readthedocs.io/en/stable/usage/benchmark.html

*IMPORTANT* - Please mark logs and text output from terminal commands 
or else Github will not display them correctly. 
An example is provided below.

Example:
```
this is an example how log text should be marked (wrap it with ```)
```
-->



Discussion:
I don't think the 5s commit really matters (borg doesn't care if the writes have hit physical media). But cpython bases its BufferedWriter write buffer size on `st_blksize` https://github.com/python/cpython/blob/024ac542d738f56b36bdeb3517a10e93da5acab9/Lib/_pyio.py#L247-L255, and it's possible that this is higher on zfs too:

> https://docs.python.org/3/library/io.html#io.BufferedWriter
> The buffer will be written out to the underlying [RawIOBase](https://docs.python.org/3/library/io.html#io.RawIOBase) object under various conditions, including:
> - when the buffer gets too small for all pending data
> -  when [flush()](https://docs.python.org/3/library/io.html#io.BufferedWriter.flush) is called;
> - when a seek() is requested (for [BufferedRandom](https://docs.python.org/3/library/io.html#io.BufferedRandom) objects);
> - when the [BufferedWriter](https://docs.python.org/3/library/io.html#io.BufferedWriter) object is closed or destroyed.

So if `st_blksize > idx.size()`, python won't have made an actual os write call yet, and the `os.path.getsize(file.fileno())` will be zero.

This worked prior to https://github.com/borgbackup/borg/commit/8ddfd94f0904449175151faed67f9640710b6454, because it was passing `str` to `.write()`, and that meant https://github.com/borgbackup/borg/blob/c29d4a096bba4018e14208b54fe24bdc886f4f3d/src/borg/hashindex.pyx#L123-L128

would open its own file, write, and close it (flusing python's buffers) before returning. But now the python file object is still open when we use its fileno(), getting a possibly-incomplete answer.

So I see two possible fixes, depending on what the test is wanting to cover. Either


I think the correct fix would be:

```diff
--- a/src/borg/testsuite/hashindex.py
+++ b/src/borg/testsuite/hashindex.py
@@ -198,6 +198,7 @@ def test_size_on_disk_accurate(self):
         with tempfile.NamedTemporaryFile() as file:
             idx.write(file)
+            file.flush()
             size = os.path.getsize(file.fileno())
         assert idx.size() == size
 
``` 
to really push the writes to the filesystem before we go out-of-band and do os.path.getsize(file.fileno()) on the bare `fileno`, or 

```diff
--- a/src/borg/testsuite/hashindex.py
+++ b/src/borg/testsuite/hashindex.py
@@ -198,7 +198,7 @@ def test_size_on_disk_accurate(self):
         with tempfile.NamedTemporaryFile() as file:
             idx.write(file)
-            size = os.path.getsize(file.fileno())
+            size = file.tell()
         assert idx.size() == size
 
 
``` 
if it's enough to know how many bytes hashindex_write() wrote, without caring whether the os has seen it yet or if it's still just buffered somewhere in python `io`.

The name of the test being `test_size_on_disk_accurate` makes me think this wants the former (really involve the filesystem hosting NamedTemporaryFile), though I'm not sure why.


https://docs.oracle.com/cd/E23824_01/html/821-1448/gazss.html says "The default block size for volumes is 8 KB. Any power of 2 from 512 bytes to 128 KB is valid." but it seems to be something set at volume creation time. 128kiB would be enough to make a whole ChunkIndex() fit in the buffer and fit with @mweinelt's symptoms (I haven't been able to reproduce this, but I don't have a machine where $TEMP is on ZFS).

He says the `file.flush()` fixes the failing machines where he found this: https://github.com/NixOS/nixpkgs/pull/207631#issuecomment-1375094224
Thanks for reporting this! Yeah, I agree that this code is problematic and that either flush or tell are a good solution.

Important: the code has multiple such patterns, so if we fix this, we should fix all places.
I don't think the 5s commit really matters (borg doesn't care if the writes have hit physical media). But cpython bases its BufferedWriter write buffer size on `st_blksize` https://github.com/python/cpython/blob/024ac542d738f56b36bdeb3517a10e93da5acab9/Lib/_pyio.py#L247-L255, and it's possible that this is higher on zfs too:

> https://docs.python.org/3/library/io.html#io.BufferedWriter
> The buffer will be written out to the underlying [RawIOBase](https://docs.python.org/3/library/io.html#io.RawIOBase) object under various conditions, including:
> - when the buffer gets too small for all pending data
> -  when [flush()](https://docs.python.org/3/library/io.html#io.BufferedWriter.flush) is called;
> - when a seek() is requested (for [BufferedRandom](https://docs.python.org/3/library/io.html#io.BufferedRandom) objects);
> - when the [BufferedWriter](https://docs.python.org/3/library/io.html#io.BufferedWriter) object is closed or destroyed.

So if `st_blksize > idx.size()`, python won't have made an actual os write call yet, and the `os.path.getsize(file.fileno())` will be zero.

This worked prior to https://github.com/borgbackup/borg/commit/8ddfd94f0904449175151faed67f9640710b6454, because it was passing `str` to `.write()`, and that meant https://github.com/borgbackup/borg/blob/c29d4a096bba4018e14208b54fe24bdc886f4f3d/src/borg/hashindex.pyx#L123-L128

would open its own file, write, and close it (flusing python's buffers) before returning. But now the python file object is still open when we use its fileno(), getting a possibly-incomplete answer.

So I see two possible fixes, depending on what the test is wanting to cover. Either


I think the correct fix would be:

```diff
--- a/src/borg/testsuite/hashindex.py
+++ b/src/borg/testsuite/hashindex.py
@@ -198,6 +198,7 @@ def test_size_on_disk_accurate(self):
         with tempfile.NamedTemporaryFile() as file:
             idx.write(file)
+            file.flush()
             size = os.path.getsize(file.fileno())
         assert idx.size() == size
 
``` 
to really push the writes to the filesystem before we go out-of-band and do os.path.getsize(file.fileno()) on the bare `fileno`, or 

```diff
--- a/src/borg/testsuite/hashindex.py
+++ b/src/borg/testsuite/hashindex.py
@@ -198,7 +198,7 @@ def test_size_on_disk_accurate(self):
         with tempfile.NamedTemporaryFile() as file:
             idx.write(file)
-            size = os.path.getsize(file.fileno())
+            size = file.tell()
         assert idx.size() == size
 
 
``` 
if it's enough to know how many bytes hashindex_write() wrote, without caring whether the os has seen it yet or if it's still just buffered somewhere in python `io`.

The name of the test being `test_size_on_disk_accurate` makes me think this wants the former (really involve the filesystem hosting NamedTemporaryFile), though I'm not sure why.


https://docs.oracle.com/cd/E23824_01/html/821-1448/gazss.html says "The default block size for volumes is 8 KB. Any power of 2 from 512 bytes to 128 KB is valid." but it seems to be something set at volume creation time. 128kiB would be enough to make a whole ChunkIndex() fit in the buffer and fit with @mweinelt's symptoms (I haven't been able to reproduce this, but I don't have a machine where $TEMP is on ZFS).

He says the `file.flush()` fixes the failing machines where he found this: https://github.com/NixOS/nixpkgs/pull/207631#issuecomment-1375094224
Thanks for reporting this! Yeah, I agree that this code is problematic and that either flush or tell are a good solution.

Important: the code has multiple such patterns, so if we fix this, we should fix all places.

Resolving patch:
diff --git a/src/borg/testsuite/hashindex.py b/src/borg/testsuite/hashindex.py
--- a/src/borg/testsuite/hashindex.py
+++ b/src/borg/testsuite/hashindex.py
@@ -196,9 +196,9 @@ def test_size_on_disk_accurate(self):
         idx = ChunkIndex()
         for i in range(1234):
             idx[H(i)] = i, i**2, i**3
-        with tempfile.NamedTemporaryFile() as file:
-            idx.write(file)
-            size = os.path.getsize(file.fileno())
+        with unopened_tempfile() as filepath:
+            idx.write(filepath)
+            size = os.path.getsize(filepath)
         assert idx.size() == size
 
 


Test patch: