Repository: borgbackup/borg

Issue:
borg create --list: when to show a path/filename
borg 1.1.x prints the path/filename after completing the back up of that file.

for rather big files, this might take rather long and the user only sees the previous filenames, not the filename of the big file.

thus, the user might think, borg is hanging / not doing anything. Example: https://github.com/borgbackup/borg/issues/4762#issuecomment-538785655

also, if some file-reading related error or warning is output, it might be unclear to which file the error message refers.

this ticket is about changing this behaviour, to print the path/filename before starting to back up the respective file.

Bounty: https://www.bountysource.com/issues/93406062-borg-create-list-when-to-show-a-path-filename



Discussion:
There is an issue with showing filename first:

We also show the status and while reading the file, the status might change to E)rror, like:
```
A some_added_file
M some_modified_file
E some_file_that_had_a_read_error
```
Idea: before borg decides to call `chunk_file` (which reads, chunks, hashes, ... the file's content) we could emit the latest status (usually A or M for regular files) plus filename and signal via status = None (usually the return value of the file item handler) that no additional output is wanted after completion (needs some other code changes, status None is currently treated as invalid state).

So, if the file is read without errors, there will only be output before reading it.
If the file is read with error, there will be also `E filename` output when it errors processing it.

3 places to change: regular file, stdin, recreate - all call chunk_file.

Will affect normal log output as well as json log output.
Does anybody see an issue with this slight behaviour change?
So the file with error will have double output (M somefile, E somefile)?
Yes, in case of an error.
Wouldn't it be possible in this specific case to rewrite the line using \r?
This way, we avoid deduplication, and make it easier to check the number of back-upped files?
Hmm, not sure if that is possible, have to check how that output is generated.
@ThomasWaldmann Can you point me to the place where chunk_file is called ? I cannot find it in the source only in the builds :/
Then I can write the patch with the two lines, or the \r if possible

NVM found it the name has changed since your comment, that's why I was unable to find it.
This change should be delayed until after PR #5492 (+ potential backport).
Sure, I'll wait then!
I have started working on this one, still WIP but close to an end
@Gu1nness i've just changed the milestone for this, we first want to have that in master for a bit.

Later we can maybe backport to 1.1-maint.
@Gu1nness please open a PR as soon as you have something, to get early feedback.
Sure, I'll do this.
@Gu1nness are you still working on this?
Yes, I just was offline for quite a few days.
Patch incoming soon
@Gu1nness told me that this will take a while and can be taken by someone else.
Will work on this now.

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -1222,13 +1222,14 @@ class FilesystemObjectProcessors:
     def __init__(self, *, metadata_collector, cache, key,
                  add_item, process_file_chunks,
                  chunker_params, show_progress, sparse,
-                 log_json, iec):
+                 log_json, iec, file_status_printer=None):
         self.metadata_collector = metadata_collector
         self.cache = cache
         self.key = key
         self.add_item = add_item
         self.process_file_chunks = process_file_chunks
         self.show_progress = show_progress
+        self.print_file_status = file_status_printer or (lambda *args: None)
 
         self.hard_links = {}
         self.stats = Statistics(output_json=log_json, iec=iec)  # threading: done by cache (including progress)
@@ -1301,6 +1302,9 @@ def process_symlink(self, *, path, parent_fd, name, st):
             return status
 
     def process_pipe(self, *, path, cache, fd, mode, user, group):
+        status = 'i'  # stdin (or other pipe)
+        self.print_file_status(status, path)
+        status = None  # we already printed the status
         uid = user2uid(user)
         if uid is None:
             raise Error("no such user: %s" % user)
@@ -1319,7 +1323,7 @@ def process_pipe(self, *, path, cache, fd, mode, user, group):
         item.get_size(memorize=True)
         self.stats.nfiles += 1
         self.add_item(item, stats=self.stats)
-        return 'i'  # stdin
+        return status
 
     def process_file(self, *, path, parent_fd, name, st, cache, flags=flags_normal):
         with self.create_helper(path, st, None) as (item, status, hardlinked, hardlink_master):  # no status yet
@@ -1356,6 +1360,8 @@ def process_file(self, *, path, parent_fd, name, st, cache, flags=flags_normal):
                             status = 'U'  # regular file, unchanged
                     else:
                         status = 'M' if known else 'A'  # regular file, modified or added
+                    self.print_file_status(status, path)
+                    status = None  # we already printed the status
                     item.hardlink_master = hardlinked
                     # Only chunkify the file if needed
                     if chunks is not None:
@@ -2018,11 +2024,14 @@ def item_is_hardlink_master(item):
             target.stats.show_progress(final=True)
 
     def process_item(self, archive, target, item):
+        status = file_status(item.mode)
         if 'chunks' in item:
+            self.print_file_status(status, item.path)
+            status = None
             self.process_chunks(archive, target, item)
             target.stats.nfiles += 1
         target.add_item(item, stats=target.stats)
-        self.print_file_status(file_status(item.mode), item.path)
+        self.print_file_status(status, item.path)
 
     def process_chunks(self, archive, target, item):
         if not self.recompress and not target.recreate_rechunkify:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -235,7 +235,8 @@ def print_warning(self, msg, *args):
         logger.warning(msg)
 
     def print_file_status(self, status, path):
-        if self.output_list and (self.output_filter is None or status in self.output_filter):
+        # if we get called with status == None, the final file status was already printed
+        if self.output_list and status is not None and (self.output_filter is None or status in self.output_filter):
             if self.log_json:
                 print(json.dumps({
                     'type': 'file_status',
@@ -562,8 +563,6 @@ def create_inner(archive, cache, fso):
                         status = 'E'
                     if status == 'C':
                         self.print_warning('%s: file changed while we backed it up', path)
-                    if status is None:
-                        status = '?'
                     self.print_file_status(status, path)
                 if args.paths_from_command:
                     rc = proc.wait()
@@ -664,7 +663,7 @@ def create_inner(archive, cache, fso):
                 fso = FilesystemObjectProcessors(metadata_collector=metadata_collector, cache=cache, key=key,
                     process_file_chunks=cp.process_file_chunks, add_item=archive.add_item,
                     chunker_params=args.chunker_params, show_progress=args.progress, sparse=args.sparse,
-                    log_json=args.log_json, iec=args.iec)
+                    log_json=args.log_json, iec=args.iec, file_status_printer=self.print_file_status)
                 create_inner(archive, cache, fso)
         else:
             create_inner(None, None, None)
@@ -820,8 +819,6 @@ def _rec_walk(self, *, path, parent_fd, name, fso, cache, matcher,
             status = 'E'
         if status == 'C':
             self.print_warning('%s: file changed while we backed it up', path)
-        if status is None:
-            status = '?'  # need to add a status code somewhere
         if not recurse_excluded_dir:
             self.print_file_status(status, path)
 


Test patch: