Repository: borgbackup/borg

Issue:
Interleave modifications, additions, and removals in 'borg diff'
I thought about this when implementing #783. The current version of `borg diff` always shows first the file modifications, then file additions, and finally removals. For me it would be easier to follow the list of changes if all of the different change types would be interleaved obeying the archive order.

See the example below. Note that the real-life change lists may be a lot longer with lots of changed in one folder.

```
old:
+234.5 kB -319.7 kB folder1/changed_file
+234.5 kB -319.7 kB folder2/changed_file
+234.5 kB -319.7 kB folder3/changed_file
added     455.61 kB folder1/new_file
added     455.61 kB folder2/new_file
added     455.61 kB folder3/new_file
removed     2.71 kB folder1/removed_file
removed     2.71 kB folder2/removed_file
removed     2.71 kB folder3/removed_file
```

```
new:
+234.5 kB -319.7 kB folder1/changed_file
added     455.61 kB folder1/new_file
removed     2.71 kB folder1/removed_file
+234.5 kB -319.7 kB folder2/changed_file
added     455.61 kB folder2/new_file
removed     2.71 kB folder2/removed_file
+234.5 kB -319.7 kB folder3/changed_file
added     455.61 kB folder3/new_file
removed     2.71 kB folder3/removed_file
```
#783 should be merged before implementing this to avoid conflicts.



Discussion:
the problem here might be that borg has no "directory entry list" data in the archive, so while processing items, you do not know in advance what items are coming for a specific directory. Nor can you easily know when you have finished one directory.

there is a hack in the code that assumes a specific traversal method at create time and then detects the directory level change at extract time. this was one of the things breaking after the multithreading changes.

so I'ld prefer not interleaving rather than adding more such hacks that just cause troubles.

Perhaps we could process all of the items and only after that start producing output. The diff command runs pretty quickly on my machine even with over 60 GB of data in the snapshots. This feature could be optional with a command line switch.

You can work with the assumption that paths are in-order and grouped by parent directory. The approach would look something like this:

```
for item1, item2 in zip(item iters):
  if path1 > path2:
    while path1 > path2:
      output(path1 deleted in path2)
      advance_only_iter1
  # analogous for path1 < path 2
  if path1 == path2:
    # detailed compare
```

IOW instead of using the "matching orphans" approach keep the two item-iterators synchronized; if they become out of sync a file has been deleted or added and can be output this way in the correct order.

(When I wrote the orphan-matching stuff I wasn't as aware of the implicit data models in Borg)

@enkore when doing multithreading, that grouping and ordering might go away. don't rely on it.

When doing MT in a way that doesn't uphold these assumptions, we'll need to rework the data model or a good amount of code anyway (stuff like extract relies on this). E.g. instead of a flat list a nested list that maps the directory structure.

It's maybe better not to increase this "code debt" any further than we must, I agree...


Resolving patch:
diff --git a/borg/archiver.py b/borg/archiver.py
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -514,7 +514,7 @@ def compare_mode(item1, item2):
             if item1[b'mode'] != item2[b'mode']:
                 return '[{} -> {}]'.format(get_mode(item1), get_mode(item2))
 
-        def compare_items(path, item1, item2, hardlink_masters, deleted=False):
+        def compare_items(output, path, item1, item2, hardlink_masters, deleted=False):
             """
             Compare two items with identical paths.
             :param deleted: Whether one of the items has been deleted
@@ -545,43 +545,56 @@ def compare_items(path, item1, item2, hardlink_masters, deleted=False):
 
             changes = [x for x in changes if x]
             if changes:
-                print("{:<19} {}".format(' '.join(changes), remove_surrogates(path)))
+                output_line = (remove_surrogates(path), ' '.join(changes))
+
+                if args.sort:
+                    output.append(output_line)
+                else:
+                    print_output(output_line)
+
+        def print_output(line):
+            print("{:<19} {}".format(line[1], line[0]))
 
         def compare_archives(archive1, archive2, matcher):
             orphans_archive1 = collections.OrderedDict()
             orphans_archive2 = collections.OrderedDict()
             hardlink_masters = {}
+            output = []
+
             for item1, item2 in zip_longest(
                     archive1.iter_items(lambda item: matcher.match(item[b'path'])),
                     archive2.iter_items(lambda item: matcher.match(item[b'path'])),
             ):
                 if item1 and item2 and item1[b'path'] == item2[b'path']:
-                    compare_items(item1[b'path'], item1, item2, hardlink_masters)
+                    compare_items(output, item1[b'path'], item1, item2, hardlink_masters)
                     continue
                 if item1:
                     matching_orphan = orphans_archive2.pop(item1[b'path'], None)
                     if matching_orphan:
-                        compare_items(item1[b'path'], item1, matching_orphan, hardlink_masters)
+                        compare_items(output, item1[b'path'], item1, matching_orphan, hardlink_masters)
                     else:
                         orphans_archive1[item1[b'path']] = item1
                 if item2:
                     matching_orphan = orphans_archive1.pop(item2[b'path'], None)
                     if matching_orphan:
-                        compare_items(item2[b'path'], matching_orphan, item2, hardlink_masters)
+                        compare_items(output, item2[b'path'], matching_orphan, item2, hardlink_masters)
                     else:
                         orphans_archive2[item2[b'path']] = item2
             # At this point orphans_* contain items that had no matching partner in the other archive
             for added in orphans_archive2.values():
-                compare_items(added[b'path'], {
+                compare_items(output, added[b'path'], {
                     b'deleted': True,
                     b'chunks': [],
                 }, added, hardlink_masters, deleted=True)
             for deleted in orphans_archive1.values():
-                compare_items(deleted[b'path'], deleted, {
+                compare_items(output, deleted[b'path'], deleted, {
                     b'deleted': True,
                     b'chunks': [],
                 }, hardlink_masters, deleted=True)
 
+            for line in sorted(output):
+                print_output(line)
+
         archive1 = archive
         archive2 = Archive(repository, key, manifest, args.archive2)
 
@@ -1308,6 +1321,9 @@ def build_parser(self, args=None, prog=None):
         subparser.add_argument('--same-chunker-params', dest='same_chunker_params',
                                action='store_true', default=False,
                                help='Override check of chunker parameters.')
+        subparser.add_argument('--sort', dest='sort',
+                               action='store_true', default=False,
+                               help='Sort the output lines by file path.')
         subparser.add_argument('location', metavar='ARCHIVE1',
                                type=location_validator(archive=True),
                                help='archive')
diff --git a/borg/testsuite/archiver.py b/borg/testsuite/archiver.py
--- a/borg/testsuite/archiver.py
+++ b/borg/testsuite/archiver.py
@@ -1351,6 +1351,37 @@ def do_asserts(output, archive):
         # We expect exit_code=1 due to the chunker params warning
         do_asserts(self.cmd('diff', self.repository_location + '::test0', 'test1b', exit_code=1), '1b')
 
+    def test_sort_option(self):
+        self.cmd('init', self.repository_location)
+
+        self.create_regular_file('a_file_removed', size=8)
+        self.create_regular_file('f_file_removed', size=16)
+        self.create_regular_file('c_file_changed', size=32)
+        self.create_regular_file('e_file_changed', size=64)
+        self.cmd('create', self.repository_location + '::test0', 'input')
+
+        os.unlink('input/a_file_removed')
+        os.unlink('input/f_file_removed')
+        os.unlink('input/c_file_changed')
+        os.unlink('input/e_file_changed')
+        self.create_regular_file('c_file_changed', size=512)
+        self.create_regular_file('e_file_changed', size=1024)
+        self.create_regular_file('b_file_added', size=128)
+        self.create_regular_file('d_file_added', size=256)
+        self.cmd('create', self.repository_location + '::test1', 'input')
+
+        output = self.cmd('diff', '--sort', self.repository_location + '::test0', 'test1')
+        expected = [
+            'a_file_removed',
+            'b_file_added',
+            'c_file_changed',
+            'd_file_added',
+            'e_file_changed',
+            'f_file_removed',
+        ]
+
+        assert all(x in line for x, line in zip(expected, output.splitlines()))
+
 
 def test_get_args():
     archiver = Archiver()


Test patch: