Repository: borgbackup/borg

Issue:
Inconsistent non-UTF8 character JSON output in list vs. diff
<!--
Thank you for reporting an issue.

*IMPORTANT* -  *before* creating a new issue please look around:
 - Borgbackup documentation: http://borgbackup.readthedocs.io/en/stable/index.html
 - FAQ: https://borgbackup.readthedocs.io/en/stable/faq.html
 and
 - open issues in GitHub tracker: https://github.com/borgbackup/borg/issues
  
If you cannot find a similar problem, then create a new issue.

Please fill in as much of the template as possible.
-->

## Have you checked borgbackup docs, FAQ, and open GitHub issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

BUG

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).

borg-linux64 1.2.4 from GitHub / .3 from nixpkgs

#### Operating system (distribution) and version.

Debian 11 on Qubes OS

#### Hardware / network configuration, and filesystems used.

ext4

#### How much data is handled by borg?

single test file

#### Full borg commandline that lead to the problem (leave away excludes and passwords)

<details>
  <summary>Script that demonstrates the problem</summary>

```sh
#!/bin/bash

set -eu

cd $(mktemp -d)
pwd

# setup
borg init -e none repo
mkdir data

( cd data && borg create ../repo::empty . ; )

echo x > $'data/foo\344'
borg create repo::archive data

echo
echo funny character is ? in regular list output
borg list repo::archive
echo
echo funny character unfortunately is ? in JSON too
borg list repo::archive --json-lines

echo
echo funny character is not ? in diff JSON
borg diff repo::empty archive --json-lines
```
</details>

produces

```
/tmp/tmp.HpDmFaX53g

funny character is ? regular list output
drwxr-xr-x user   user          0 Thu, 2023-04-20 23:07:48 data
-rw-r--r-- user   user          2 Thu, 2023-04-20 23:07:48 data/foo?

funny character unfortunately is ? in JSON too
{"type": "d", "mode": "drwxr-xr-x", "user": "user", "group": "user", "uid": 1000, "gid": 1000, "path": "data", "healthy": true, "source": "", "linktarget": "", "flags": 0, "size": 0, "mtime": "2023-04-20T23:07:48.893000"}
{"type": "-", "mode": "-rw-r--r--", "user": "user", "group": "user", "uid": 1000, "gid": 1000, "path": "data/foo?", "healthy": true, "source": "", "linktarget": "", "flags": 0, "size": 2, "mtime": "2023-04-20T23:07:48.893000"}

funny character is not ? in diff JSON
{"changes": [{"type": "added directory"}], "path": "data"}
{"changes": [{"size": 2, "type": "added"}], "path": "data/foo\udce4"}
{"changes": [{"type": "removed directory"}], "path": "."}
```

## Describe the problem you're observing.

Funny characters (such as a latin1 `ä`, octal 344) are not rendered as ? in the `borg diff` output JSON but are replaced by `?` in the `borg list` output JSON. Whatever the agreed upon "correct" way might be it should be consistent.

#### Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

See above, note that 1.1.16 (from Debian 11) printed a `?` in the diff output as well but extracted correctly (i.e. with the funny character preserved).

#### Include any warning/errors/backtraces from the system logs

none


Discussion:
Interesting find, I'll have a look.

In borg2, such potential "non-text" values will get a special treatment (involving base64 to produce an exact representation in json and a "as good as it gets" pure text representation).
OK, guess borg 1.2.x should treat paths for `borg diff --json-lines` in the same way as already implemented for `borg list --json-lines`.

## borg list
Uses `ItemFormatter.format_item_json`, `.get_item_data` there uses `remove_surrogates(item.path)`.

## borg diff
Directly uses `item.path`. (bug)

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -1154,7 +1154,7 @@ def print_text_output(diff, path):
             diffs = sorted(diffs)
 
         for path, diff in diffs:
-            print_output(diff, path)
+            print_output(diff, remove_surrogates(path))
 
         for pattern in matcher.get_unmatched_include_patterns():
             self.print_warning("Include pattern '%s' never matched.", pattern)


Test patch: