Repository: borgbackup/borg

Issue:
"borg compact" reports a wrong "freed space" value
## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

BUG

## System information. For client/server mode post info for both machines.

Intel with 16GB of RAM

#### Your borg version (borg -V).

1.2.0b2

#### Operating system (distribution) and version.

Ubuntu 20.10

#### Hardware / network configuration, and filesystems used.

local repo on btrfs

#### How much data is handled by borg?

1.1TB

#### Full borg commandline that lead to the problem (leave away excludes and passwords)
```
$ borg-1.2b2 -v compact /myrepo
compaction freed about 244.55 MB repository space.

$ borg-1.2b2 -v compact /myrepo
compaction freed about 106.94 MB repository space.

$ borg-1.2b2 -v compact /myrepo
compaction freed about 106.93 MB repository space.
```

## Describe the problem you're observing.

borg compact needs several iterations to reach a final state and does not seem to compact as much as it should.  More precisely a lot of 17 bytes files remain after compacttion:
```
# find /myrepo | wc
 200649  200649 2351093

# find /myrepo -size 17c | wc
  20344   20344  244128
```

#### Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

I observed the same behavior on several machines/systems/repos.



Discussion:
Interesting, have to investigate...

In general, there is a default `--threshold` which is not 0, so it won't compact segments with little space to free. But even with that, it should be stable after first iteration.

```
(borg-env) user@development:~/w/borg$ borg -V
borg 1.2.0b2

(borg-env) user@development:~/w/borg$ borg init -e none repo

(borg-env) user@development:~/w/borg$ borg create repo::arch1 src/
(borg-env) user@development:~/w/borg$ du -sk repo
10704   repo

(borg-env) user@development:~/w/borg$ borg create repo::arch2 /bin 
(borg-env) user@development:~/w/borg$ du -sk repo
17532   repo

(borg-env) user@development:~/w/borg$ borg delete repo::arch1
(borg-env) user@development:~/w/borg$ du -sk repo
17584   repo

(borg-env) user@development:~/w/borg$ borg compact repo
(borg-env) user@development:~/w/borg$ du -sk repo
6896    repo

(borg-env) user@development:~/w/borg$ borg compact repo
(borg-env) user@development:~/w/borg$ du -sk repo
6896    repo

(borg-env) user@development:~/w/borg$ borg delete repo::arch2
(borg-env) user@development:~/w/borg$ du -sk repo
6920    repo

(borg-env) user@development:~/w/borg$ borg compact repo
(borg-env) user@development:~/w/borg$ du -sk repo
80      repo

(borg-env) user@development:~/w/borg$ ls -lR repo/data/
repo/data/:
total 4
drwx------ 2 user user 4096 Feb  8 23:23 0

repo/data/0:
total 8
-rw------- 1 user user 476 Feb  8 23:23 27
-rw------- 1 user user  17 Feb  8 23:23 29
```

So, for a simple case, it works as expected. Can you write a script that reproduces your issue?
Note: not sure about the amount, but some 17b files are expected with borg 1.2 - commits are intentionally made in separate files.
> So, for a simple case, it works as expected. 

Your test case is very simple, indeed. There is probably little in common between arch1 and arch2.

> Can you write a script that reproduces your issue?

My repo contains full weekly backups of Ubuntu systems (that change a lot) and were started more than 2 years ago.
Generating something similar would not be a trivial task...

btrfs has an interesting util named "btrfs-image".  It allows making bug reports about btrfs without making a full copy of a FS by extracting only the internal data structures.  A similar tool for borg would be very useful...
> In general, there is a default --threshold which is not 0, 

I am currently running `borg compact --threshold 0` but it takes quite a lot of time.

I will also try the --cleanup-commits option and report back when everything is finished...
> (borg-env) user@development:~/w/borg$ borg compact repo

You did not use the -v option.
Can you run the command on your repo again, this time with `--debug`?
Does `compact` only works on repos created with 1.2?
I tried 1.2.0b3 compact on a multi-TB repo I created with 1.1 and compact didn't do anything. Tried also with `--threshold 0` and `--cleanup-commits`. I would have expected to reduce at least 300 GB.

Then I created a new repo with 1.2 and put some files into it, deleted some archives and compact correctly cleaned up.
Borg maintains a `hints` file where it stored hints about which segment files have how much compaction potential (so it does not need to look at all segment files and read through them to find out what to compact).

Unlike 1.2, borg 1.1 immediately processes these hints at the end of any repo writing command, so there's usually not much left in the hints for borg 1.2's separate `borg compact`.
About the 17b files: I did a recent change improving that: https://github.com/borgbackup/borg/pull/6220
BTW, I recently shot myself in the foot by trying to debug disk usage of repos. zfs snapshots can really lead to unexpected results, so make sure your disk space is really held by borg and not by zfs snapshots. :-)
Just had this:
```
root@server:/usr/local/bin# borg-1.2.0rc1 compact -v --progress
Remote: compaction freed about 716.85 MB repository space.

root@server:/usr/local/bin# borg-1.2.0rc1 compact -v --progress
Remote: compaction freed about 272.99 MB repository space.

root@server:/usr/local/bin# borg-1.2.0rc1 compact -v --progress
Remote: compaction freed about 272.99 MB repository space.

root@server:/usr/local/bin# borg-1.2.0rc1 compact -v --progress
Remote: compaction freed about 272.99 MB repository space.
```

Considering it would be a strange coincidence finding 3 times in a row the same amount to compact, I think there is some bug.

Either it is not really compacting this amount (telling a wrong number) or it does not commit it correctly or ...

Start and end of the log with --debug:
```
using builtin fallback logging configuration
38 self tests completed in 0.06 seconds
SSH command line: ['ssh', '-4', '-p', '4321', 'borg@tw.nsupdate.info', 'borg', 'serve', '--debug']
Remote: using builtin fallback logging configuration
Remote: 38 self tests completed in 0.37 seconds
Remote: using builtin fallback logging configuration
Remote: Initialized logging system for JSON-based protocol
Remote: Resolving repository path b'/./repos/1/borg12'
Remote: Resolved repository path to '/home/borg/repos/1/borg12'
Remote: Verified integrity of /home/borg/repos/1/borg12/index.413223
Remote: Cleaned up 0 uncommitted segment files (== everything after segment 413223).
Remote: Verified integrity of /home/borg/repos/1/borg12/hints.413223
Remote: check_free_space: required bytes 1035145178, free bytes 1295373172736
Remote: Compaction started (threshold is 10%).
Remote: not compacting segment 404505 (freeable: 0.00% [246 bytes])
Remote: not compacting segment 404507 (freeable: 0.56% [293593 bytes])
Remote: not compacting segment 404581 (freeable: 0.39% [203155 bytes])
Remote: not compacting segment 404583 (freeable: 0.09%
...
Remote: compacting segment 413208 with usage count 290 (freeable: 14.10% [7440106 bytes])
Remote: compacting segment 413209 with usage count 0 (freeable: 52.94% [9 bytes])
Remote: compacting segment 413210 with usage count 362 (freeable: 11.60% [6105064 bytes])
Remote: complete_xfer: wrote intermediate commit at segment 413228
Remote: complete_xfer: deleting unused segment 413206
Remote: complete_xfer: deleting unused segment 413208
Remote: complete_xfer: deleting unused segment 413209
Remote: compacting segment 413211 with usage count 0 (freeable: 52.94% [9 bytes])
Remote: compacting segment 413212 with usage count 437 (freeable: 25.47% [13355627 bytes])
Remote: complete_xfer: wrote intermediate commit at segment 413230
Remote: complete_xfer: deleting unused segment 413210
Remote: complete_xfer: deleting unused segment 413211
Remote: compacting segment 413213 with usage count 0 (freeable: 52.94% [9 bytes])
Remote: compacting segment 413214 with usage count 0 (freeable: 100.00% [52428832 bytes])
Remote: complete_xfer: wrote intermediate commit at segment 413232
Remote: complete_xfer: deleting unused segment 413212
Remote: complete_xfer: deleting unused segment 413213
Remote: compacting segment 413215 with usage count 0 (freeable: 52.94% [9 bytes])
Remote: compacting segment 413216 with usage count 0 (freeable: 100.00% [52428832 bytes])
Remote: complete_xfer: wrote intermediate commit at segment 413234
Remote: complete_xfer: deleting unused segment 413214
Remote: complete_xfer: deleting unused segment 413215
Remote: compacting segment 413217 with usage count 0 (freeable: 52.94% [9 bytes])
Remote: compacting segment 413218 with usage count 0 (freeable: 100.00% [52428832 bytes])
Remote: complete_xfer: wrote intermediate commit at segment 413236
Remote: complete_xfer: deleting unused segment 413216
Remote: complete_xfer: deleting unused segment 413217
Remote: compacting segment 413219 with usage count 0 (freeable: 52.94% [9 bytes])
Remote: compacting segment 413220 with usage count 0 (freeable: 100.00% [52428832 bytes])
Remote: complete_xfer: wrote intermediate commit at segment 413238
Remote: complete_xfer: deleting unused segment 413218
Remote: complete_xfer: deleting unused segment 413219
Remote: compacting segment 413221 with usage count 0 (freeable: 52.94% [9 bytes])
Remote: compacting segment 413222 with usage count 197 (freeable: 77.78% [36353429 bytes])
Remote: complete_xfer: wrote intermediate commit at segment 413240
Remote: complete_xfer: deleting unused segment 413220
Remote: complete_xfer: deleting unused segment 413221
Remote: compacting segment 413223 with usage count 0 (freeable: 52.94% [9 bytes])
Remote: compacting segment 413224 with usage count 0 (freeable: 83.67% [41 bytes])
Remote: compacting segment 413226 with usage count 0 (freeable: 52.94% [9 bytes])
Remote: complete_xfer: wrote commit at segment 413242
Remote: complete_xfer: deleting unused segment 413222
Remote: complete_xfer: deleting unused segment 413223
Remote: complete_xfer: deleting unused segment 413224
Remote: complete_xfer: deleting unused segment 413226
Remote: compaction freed about 272.99 MB repository space.
Remote: compaction completed.
```

Notable:
- did not clean up any uncommitted segment, so the previous run committed ok and ended cleanly
- seems like it found all the segment files that were freshly written by the previous compaction run to be still up to 100% freeable (which is very likely not true, these should be ~100% compact and ~0% freeable) - seems like some accounting problem.

I did some experiments and there are some first results:

**"compaction freed about xxx.xx MB repository space."**

well, this is a lie. the correct phrasing would be something like "borg thought it maybe could free xxx.xx MB repository space". and it did not measure how much it really freed.

See also the comment in the source:
```
             freed_space += freeable_space  # this is what we THINK we can free
```

The `freeable_space` value comes from the `hints` file, which is, like it says, just a hint (about in which segment there could be something to compact).
I'll improve this by using the `storage_quota_use` value we already maintain anyway.

It's not 100% correct either (because it just accounts PUTs + their overhead, but there is also other overhead, e.g. for deletes), but at least it is closer and more consistent with the quota computation we do.
With that fix, it will stabilise on this in my case:
```
compaction freed about 18.05 kB repository space.
```

So, not 0, but much closer than the 272MB I had before.


Resolving patch:
diff --git a/src/borg/repository.py b/src/borg/repository.py
--- a/src/borg/repository.py
+++ b/src/borg/repository.py
@@ -725,6 +725,7 @@ def compact_segments(self):
         """
         if not self.compact:
             return
+        quota_use_before = self.storage_quota_use
         index_transaction_id = self.get_index_transaction_id()
         segments = self.segments
         unused = []  # list of segments, that are not used anymore
@@ -755,13 +756,15 @@ def complete_xfer(intermediate=True):
                 pi.show()
                 continue
             segment_size = self.io.segment_size(segment)
+            freeable_ratio = 1.0 * freeable_space / segment_size
             if segment_size > 0.2 * self.max_segment_size and freeable_space < 0.15 * segment_size:
-                logger.debug('not compacting segment %d (only %d bytes are sparse)', segment, freeable_space)
+                logger.debug('not compacting segment %d (maybe freeable: %2.2f%% [%d bytes])',
+                             segment, freeable_ratio * 100.0, freeable_space)
                 pi.show()
                 continue
             segments.setdefault(segment, 0)
-            logger.debug('compacting segment %d with usage count %d and %d freeable bytes',
-                         segment, segments[segment], freeable_space)
+            logger.debug('compacting segment %d with usage count %d (maybe freeable: %2.2f%% [%d bytes])',
+                         segment, segments[segment], freeable_ratio * 100.0, freeable_space)
             for tag, key, offset, data in self.io.iter_objects(segment, include_data=True):
                 if tag == TAG_COMMIT:
                     continue
@@ -847,6 +850,8 @@ def complete_xfer(intermediate=True):
             pi.show()
         pi.finish()
         complete_xfer(intermediate=False)
+        quota_use_after = self.storage_quota_use
+        logger.info('compaction freed about %s repository space.', format_file_size(quota_use_before - quota_use_after))
         logger.debug('compaction completed.')
 
     def replay_segments(self, index_transaction_id, segments_transaction_id):


Test patch: