Repository: borgbackup/borg

Issue:
borg create: Everything 0 archive_progress at end of backup
<!--
Thank you for reporting an issue.

*IMPORTANT* -  *before* creating a new issue please look around:
 - Borgbackup documentation: http://borgbackup.readthedocs.io/en/stable/index.html
 - FAQ: https://borgbackup.readthedocs.io/en/stable/faq.html
 and
 - open issues in Github tracker: https://github.com/borgbackup/borg/issues
  
If you cannot find a similar problem, then create a new issue.

Please fill in as much of the template as possible.
-->

## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

BUG

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).

1.2.0

#### Operating system (distribution) and version.

#### Hardware / network configuration, and filesystems used.

#### How much data is handled by borg?

Tested with very small test repos

#### Full borg commandline that lead to the problem (leave away excludes and passwords)

## Describe the problem you're observing.

During borg create I get the usual json output

> {"original_size": 107384183, "compressed_size": 103279086, "deduplicated_size": 759473, "nfiles": 15, "time": 1649719099.1076434, "type": "archive_progress", "path": "somepath"}

But at the end I get

> {"original_size": 0, "compressed_size": 0, "deduplicated_size": 0, "nfiles": 0, "time": 1649719099.2390628, "type": "archive_progress", "path": ""}

and the next message is

> {"message": "Saving files cache", "operation": 2, "msgid": "cache.commit", "type": "progress_message", "finished": false, "time": 1649719106.242265}

I'm not sure how I should handle the archive_progress output with everything zero. Does it have a special meaning or is it a bug?

#### Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

#### Include any warning/errors/backtraces from the system logs

<!--

If this complaint relates to borg performance, please include CRUD benchmark
results and any steps you took to troubleshoot.
How to run benchmark: http://borgbackup.readthedocs.io/en/stable/usage/benchmark.html

*IMPORTANT* - Please mark logs and text output from terminal commands 
or else Github will not display them correctly. 
An example is provided below.

Example:
```
this is an example how log text should be marked (wrap it with ```)
```
-->


Discussion:

Patch:
diff --git a/docs/faq.rst b/docs/faq.rst
index b500ae46..8ef1b82a 100644
--- a/docs/faq.rst
+++ b/docs/faq.rst
@@ -1080,10 +1080,6 @@ will be slow because it would chunk all the files each time. If you set
 BORG_FILES_CACHE_TTL to at least 26 (or maybe even a small multiple of that),
 it would be much faster.
 
-Besides using a higher BORG_FILES_CACHE_TTL (which also increases memory usage),
-there is also BORG_FILES_CACHE_SUFFIX which can be used to have separate (smaller)
-files caches for each backup set instead of the default one (big) unified files cache.
-
 Another possible reason is that files don't always have the same path, for
 example if you mount a filesystem without stable mount points for each backup
 or if you are running the backup from a filesystem snapshot whose name is not
diff --git a/src/borg/archive.py b/src/borg/archive.py
index 49edb816..a9c145d4 100644
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -143,11 +143,15 @@ def show_progress(self, item=None, final=False, stream=None, dt=None):
         if dt is None or now - self.last_progress > dt:
             self.last_progress = now
             if self.output_json:
-                data = self.as_dict()
+                if not final:
+                    data = self.as_dict()
+                    data['path'] = remove_surrogates(item.path if item else '')
+                else:
+                    data = {}
                 data.update({
                     'time': time.time(),
                     'type': 'archive_progress',
-                    'path': remove_surrogates(item.path if item else ''),
+                    'finished': final,
                 })
                 msg = json.dumps(data)
                 end = '\n'


Test patch:
diff --git a/src/borg/selftest.py b/src/borg/selftest.py
index 006e85a5..bacdcea2 100644
--- a/src/borg/selftest.py
+++ b/src/borg/selftest.py
@@ -1,6 +1,3 @@
-# Note: these tests are part of the self test, do not use or import pytest functionality here.
-#       See borg.selftest for details. If you add/remove test methods, update SELFTEST_COUNT
-
 """
 Self testing module
 ===================
@@ -15,7 +12,6 @@
 variable. SELFTEST_COUNT must be updated if new tests are added or removed to or from any of the tests
 used here.
 """
-
 import os
 import sys
 import time
@@ -66,10 +62,6 @@ def selftest(logger):
     result = SelfTestResult()
     test_suite = TestSuite()
     for test_case in SELFTEST_CASES:
-        module = sys.modules[test_case.__module__]
-        # a normal borg user does not have pytest installed, we must not require it in the test modules used here.
-        # note: this only detects the usual toplevel import
-        assert 'pytest' not in dir(module), "pytest must not be imported in %s" % module.__name__
         test_suite.addTest(defaultTestLoader.loadTestsFromTestCase(test_case))
     test_suite.run(result)
     result.log_results(logger)
diff --git a/src/borg/testsuite/archive.py b/src/borg/testsuite/archive.py
index db9ed573..3f8535ff 100644
--- a/src/borg/testsuite/archive.py
+++ b/src/borg/testsuite/archive.py
@@ -1,3 +1,4 @@
+import json
 from collections import OrderedDict
 from datetime import datetime, timezone
 from io import StringIO
@@ -61,6 +62,34 @@ def test_stats_format(stats):
     assert repr(stats) == f'<Statistics object at {id(stats):#x} (20, 10, 10)>'
 
 
+def test_stats_progress_json(stats):
+    stats.output_json = True
+
+    out = StringIO()
+    stats.show_progress(item=Item(path='foo'), stream=out)
+    result = json.loads(out.getvalue())
+    assert result['type'] == 'archive_progress'
+    assert isinstance(result['time'], float)
+    assert result['finished'] is False
+    assert result['path'] == 'foo'
+    assert result['original_size'] == 20
+    assert result['compressed_size'] == 10
+    assert result['deduplicated_size'] == 10
+    assert result['nfiles'] == 0  # this counter gets updated elsewhere
+
+    out = StringIO()
+    stats.show_progress(stream=out, final=True)
+    result = json.loads(out.getvalue())
+    assert result['type'] == 'archive_progress'
+    assert isinstance(result['time'], float)
+    assert result['finished'] is True  # see #6570
+    assert 'path' not in result
+    assert 'original_size' not in result
+    assert 'compressed_size' not in result
+    assert 'deduplicated_size' not in result
+    assert 'nfiles' not in result
+
+
 class MockCache:
 
     class MockRepo:
diff --git a/src/borg/testsuite/chunker.py b/src/borg/testsuite/chunker.py
index ce45b7c9..1b275978 100644
--- a/src/borg/testsuite/chunker.py
+++ b/src/borg/testsuite/chunker.py
@@ -1,12 +1,12 @@
-# Note: these tests are part of the self test, do not use or import pytest functionality here.
-#       See borg.selftest for details. If you add/remove test methods, update SELFTEST_COUNT
-
 from io import BytesIO
 
 from ..chunker import ChunkerFixed, Chunker, get_chunker, buzhash, buzhash_update
 from ..constants import *  # NOQA
 from . import BaseTestCase
 
+# Note: these tests are part of the self test, do not use or import py.test functionality here.
+#       See borg.selftest for details. If you add/remove test methods, update SELFTEST_COUNT
+
 
 def cf(chunks):
     """chunk filter"""
diff --git a/src/borg/testsuite/crypto.py b/src/borg/testsuite/crypto.py
index e691c321..dac3efeb 100644
--- a/src/borg/testsuite/crypto.py
+++ b/src/borg/testsuite/crypto.py
@@ -1,10 +1,8 @@
-# Note: these tests are part of the self test, do not use or import pytest functionality here.
-#       See borg.selftest for details. If you add/remove test methods, update SELFTEST_COUNT
-
 from binascii import hexlify
 from unittest.mock import MagicMock
 import unittest
 
+
 from ..crypto.low_level import AES256_CTR_HMAC_SHA256, AES256_OCB, CHACHA20_POLY1305, UNENCRYPTED, IntegrityError
 from ..crypto.low_level import bytes_to_long, bytes_to_int, long_to_bytes
 from ..crypto.low_level import hkdf_hmac_sha512
@@ -14,6 +12,9 @@
 
 from . import BaseTestCase
 
+# Note: these tests are part of the self test, do not use or import py.test functionality here.
+#       See borg.selftest for details. If you add/remove test methods, update SELFTEST_COUNT
+
 
 class CryptoTestCase(BaseTestCase):
 
diff --git a/src/borg/testsuite/hashindex.py b/src/borg/testsuite/hashindex.py
index 89a793a9..764de114 100644
--- a/src/borg/testsuite/hashindex.py
+++ b/src/borg/testsuite/hashindex.py
@@ -1,6 +1,3 @@
-# Note: these tests are part of the self test, do not use or import pytest functionality here.
-#       See borg.selftest for details. If you add/remove test methods, update SELFTEST_COUNT
-
 import base64
 import hashlib
 import io
@@ -13,6 +10,9 @@
 from ..crypto.file_integrity import IntegrityCheckedFile, FileIntegrityError
 from . import BaseTestCase, unopened_tempfile
 
+# Note: these tests are part of the self test, do not use or import py.test functionality here.
+#       See borg.selftest for details. If you add/remove test methods, update SELFTEST_COUNT
+
 
 def H(x):
     # make some 32byte long thing that depends on x