Repository: borgbackup/borg

Issue:
compacting for little win
while working on #2982, found this:
```
$ borg --debug-topic compact_segments with-lock repo ls repo
config  data  hints.36  index.36  integrity.36  lock.exclusive  lock.roster  README
compaction started.
not compacting segment 2 (only 564 bytes are sparse)
compacting segment 35 with usage count 4 and 634 freeable bytes
compacting segment 37 with usage count 1 and 41 freeable bytes
complete_xfer: wrote commit at segment 40
complete_xfer: deleting unused segment 35
complete_xfer: deleting unused segment 37
compaction completed.
```

Seem a bit too keen on compacting if the segment size is small.

The condition to skip compaction currently is:
```
segment_size > 0.2 * self.max_segment_size and freeable_space < 0.15 * segment_size
```
So, obviously, stuff in small segments always gets compacted, no matter how much we save by doing it.

Not sure, maybe this is ok, as we usually have big segments and only the final one is small (and commit-only segments).


Discussion:
This is related to a request I found in https://mail.python.org/pipermail/borgbackup/2016q4/000501.html which mentions that line:

https://github.com/borgbackup/borg/blob/52be21c690e39cfa7c13596ae8542f6c45b212e1/src/borg/repository.py#L698

Allowing the user to set a threshold much higher than 15% could be very beneficial in some uses (i.e. to minimize S3 transactions when rcloning a Borg repo).
Note: i did substantial changes to compaction in master branch, compaction is now a separate command and does not automatically run with each repo-writing command.

Have to check again, whether it helps enough with this issue or more changes are needed.

Resolving patch:
diff --git a/src/borg/repository.py b/src/borg/repository.py
--- a/src/borg/repository.py
+++ b/src/borg/repository.py
@@ -690,11 +690,11 @@ def compact_segments(self):
         """Compact sparse segments by copying data into new segments
         """
         if not self.compact:
+            logger.debug('nothing to do: compact empty')
             return
         index_transaction_id = self.get_index_transaction_id()
         segments = self.segments
         unused = []  # list of segments, that are not used anymore
-        logger = create_logger('borg.debug.compact_segments')
 
         def complete_xfer(intermediate=True):
             # complete the current transfer (when some target segment is full)
@@ -723,13 +723,17 @@ def complete_xfer(intermediate=True):
                 pi.show()
                 continue
             segment_size = self.io.segment_size(segment)
-            if segment_size > 0.2 * self.max_segment_size and freeable_space < 0.15 * segment_size:
-                logger.debug('not compacting segment %d (only %d bytes are sparse)', segment, freeable_space)
+            freeable_ratio = 1.0 * freeable_space / segment_size
+            # we want to compact if:
+            # - we can free a considerable relative amount of space (freeable_ratio over some threshold)
+            if not (freeable_ratio > 0.1):
+                logger.debug('not compacting segment %d (freeable: %2.2f%% [%d bytes])',
+                             segment, freeable_ratio * 100.0, freeable_space)
                 pi.show()
                 continue
             segments.setdefault(segment, 0)
-            logger.debug('compacting segment %d with usage count %d and %d freeable bytes',
-                         segment, segments[segment], freeable_space)
+            logger.debug('compacting segment %d with usage count %d (freeable: %2.2f%% [%d bytes])',
+                         segment, segments[segment], freeable_ratio * 100.0, freeable_space)
             for tag, key, offset, data in self.io.iter_objects(segment, include_data=True):
                 if tag == TAG_COMMIT:
                     continue


Test patch: