Repository: borgbackup/borg

Issue:
borg create: Everything 0 archive_progress at end of backup
<!--
Thank you for reporting an issue.

*IMPORTANT* -  *before* creating a new issue please look around:
 - Borgbackup documentation: http://borgbackup.readthedocs.io/en/stable/index.html
 - FAQ: https://borgbackup.readthedocs.io/en/stable/faq.html
 and
 - open issues in Github tracker: https://github.com/borgbackup/borg/issues
  
If you cannot find a similar problem, then create a new issue.

Please fill in as much of the template as possible.
-->

## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

BUG

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).

1.2.0

#### Operating system (distribution) and version.

#### Hardware / network configuration, and filesystems used.

#### How much data is handled by borg?

Tested with very small test repos

#### Full borg commandline that lead to the problem (leave away excludes and passwords)

## Describe the problem you're observing.

During borg create I get the usual json output

> {"original_size": 107384183, "compressed_size": 103279086, "deduplicated_size": 759473, "nfiles": 15, "time": 1649719099.1076434, "type": "archive_progress", "path": "somepath"}

But at the end I get

> {"original_size": 0, "compressed_size": 0, "deduplicated_size": 0, "nfiles": 0, "time": 1649719099.2390628, "type": "archive_progress", "path": ""}

and the next message is

> {"message": "Saving files cache", "operation": 2, "msgid": "cache.commit", "type": "progress_message", "finished": false, "time": 1649719106.242265}

I'm not sure how I should handle the archive_progress output with everything zero. Does it have a special meaning or is it a bug?

#### Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

#### Include any warning/errors/backtraces from the system logs

<!--

If this complaint relates to borg performance, please include CRUD benchmark
results and any steps you took to troubleshoot.
How to run benchmark: http://borgbackup.readthedocs.io/en/stable/usage/benchmark.html

*IMPORTANT* - Please mark logs and text output from terminal commands 
or else Github will not display them correctly. 
An example is provided below.

Example:
```
this is an example how log text should be marked (wrap it with ```)
```
-->



Discussion:
That comes from `Statistics.show_progress`:

- every dt this emits some progress info
- at the end, it gets called with `final=True` and
  - for the console this clears the progress display (and does not show any data)
  - for json there is no special treatment of the `final` flag (which might be a bug)

TODO:
- check the docs, if there is documented related behaviour
- decide whether to introduce special treatment of final json

The docs say nothing about a "final" json for the `archive_progress`, they only describe the normal output.

The (somehow related) `progress_percent` / `progress_message` json includes `finished=true/false` for the same purpose and that is documented.

So I think `finished=true/false` should be added for all `archive_progress` messages and the empty stuff should be removed from the final `finished=True` output. Also the docs (`frontend.rst`) should be updated accordingly.

The final message can be used by GUI for similar purposes as for the text-based UI: e.g. clear the progress display.

@sophie-h @m3nu do you agree?

Bounty: https://app.bountysource.com/issues/106988150-borg-create-everything-0-archive_progress-at-end-of-backup
> do you agree?

Thanks! Sounds good to me. I think it's much cleaner than adding a special case for 'everything 0' in the frontend code.

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -143,11 +143,15 @@ def show_progress(self, item=None, final=False, stream=None, dt=None):
         if dt is None or now - self.last_progress > dt:
             self.last_progress = now
             if self.output_json:
-                data = self.as_dict()
+                if not final:
+                    data = self.as_dict()
+                    data['path'] = remove_surrogates(item.path if item else '')
+                else:
+                    data = {}
                 data.update({
                     'time': time.time(),
                     'type': 'archive_progress',
-                    'path': remove_surrogates(item.path if item else ''),
+                    'finished': final,
                 })
                 msg = json.dumps(data)
                 end = '\n'
diff --git a/src/borg/testsuite/archive.py b/src/borg/testsuite/archive.py
--- a/src/borg/testsuite/archive.py
+++ b/src/borg/testsuite/archive.py
@@ -1,3 +1,4 @@
+import json
 from collections import OrderedDict
 from datetime import datetime, timezone
 from io import StringIO
@@ -61,6 +62,34 @@ def test_stats_format(stats):
     assert repr(stats) == f'<Statistics object at {id(stats):#x} (20, 10, 10)>'
 
 
+def test_stats_progress_json(stats):
+    stats.output_json = True
+
+    out = StringIO()
+    stats.show_progress(item=Item(path='foo'), stream=out)
+    result = json.loads(out.getvalue())
+    assert result['type'] == 'archive_progress'
+    assert isinstance(result['time'], float)
+    assert result['finished'] is False
+    assert result['path'] == 'foo'
+    assert result['original_size'] == 20
+    assert result['compressed_size'] == 10
+    assert result['deduplicated_size'] == 10
+    assert result['nfiles'] == 0  # this counter gets updated elsewhere
+
+    out = StringIO()
+    stats.show_progress(stream=out, final=True)
+    result = json.loads(out.getvalue())
+    assert result['type'] == 'archive_progress'
+    assert isinstance(result['time'], float)
+    assert result['finished'] is True  # see #6570
+    assert 'path' not in result
+    assert 'original_size' not in result
+    assert 'compressed_size' not in result
+    assert 'deduplicated_size' not in result
+    assert 'nfiles' not in result
+
+
 class MockCache:
 
     class MockRepo:


Test patch: