Repository: borgbackup/borg

Issue:
unicode filenames mess up progress display
while recreating one of my old backups i noticed that Unicode filenames cause it to print dozens and dozens of lines instead of rewriting the progress line



Discussion:
does "unicode" here mean any non-ascii stuff or CJK double-width chars specifically?
what borg version? precise commandline? OS and version?

not entirely sure, its some some kind of asian character set 

borg version 765b8f38d68aa2b623fe1fac0030c0c478f4a863
on nixos 16.09 relatively recent

CJK = chinese, japanese, korean

they use double-width terminal char cells and we use wcswidth in some borg versions to compute correct width. https://github.com/borgbackup/borg/search?utf8=%E2%9C%93&q=wcswidth

if it doesn't work for you, you now know what to look at.

so, any insights? do you have an example filename that formats badly?

no insights yet, no data i want to share in public

it happens even for short names

```
set -x
export BORG_PASSPHRASE=abc
borg init repo
PATHNAME=filetree/나윤선
mkdir -p $PATHNAME

for i in {1..100}
do
    [ -f $PATHNAME/$i ] ||
    dd if=/dev/urandom of=$PATHNAME/$i bs=1M count=5

done
borg create repo::filetree filetree --progress
rm -rf repo filetree
```

and the last bits of display on the terminal i see is

```
+ borg create repo::filetree filetree --progress
12.52 MB O 12.53 MB C 12.53 MB D 2 N filetree/나윤선/100                                                                                                                                                           26.21 MB O 26.21 MB C 26.21 MB D 4 N filetree/나윤선/12                                                                                                                                                            38.09 MB O 38.09 MB C 38.09 MB D 7 N filetree/나윤선/15                                                                                                                                                            52.43 MB O 52.43 MB C 52.43 MB D 9 N filetree/나윤선/17                                                                                                                                                            68.16 MB O 68.16 MB C 68.16 MB D 12 N filetree/나윤선/2                                                                                                                                                            78.64 MB O 78.64 MB C 78.64 MB D 15 N filetree/나윤선/21                                                                                                                                                           92.56 MB O 92.57 MB C 92.57 MB D 17 N filetree/나윤선/24                                                                                                                                                           110.10 MB O 110.10 MB C 110.10 MB D 20 N filetree/나윤선/27                                                                                                                                                        123.31 MB O 123.31 MB C 123.31 MB D 23 N filetree/나윤선/3                                                                                                                                                         136.31 MB O 136.32 MB C 136.32 MB D 25 N filetree/나윤선/31                                                                                                                                                        146.80 MB O 146.80 MB C 146.80 MB D 27 N filetree/나윤선/33                                                                                                                                                        157.85 MB O 157.85 MB C 157.85 MB D 30 N filetree/나윤선/36                                                                                                                                                        169.92 MB O 169.93 MB C 169.93 MB D 32 N filetree/나윤선/38                                                                                                                                                        183.50 MB O 183.50 MB C 183.50 MB D 34 N filetree/나윤선/4                                                                                                                                                         195.27 MB O 195.28 MB C 195.28 MB D 37 N filetree/나윤선/42                                                                                                                                                        207.73 MB O 207.73 MB C 207.73 MB D 39 N filetree/나윤선/44                                                                                                                                                        223.91 MB O 223.92 MB C 223.92 MB D 42 N filetree/나윤선/47                                                                                                                                                        236.95 MB O 236.96 MB C 236.96 MB D 45 N filetree/나윤선/5                                                                                                                                                         251.19 MB O 251.19 MB C 251.19 MB D 47 N filetree/나윤선/51                                                                                                                                                        267.36 MB O 267.37 MB C 267.37 MB D 50 N filetree/나윤선/54                                                                                                                                                        279.11 MB O 279.12 MB C 279.12 MB D 53 N filetree/나윤선/57                                                                                                                                                        290.21 MB O 290.22 MB C 290.22 MB D 55 N filetree/나윤선/59                                                                                                                                                        304.09 MB O 304.09 MB C 304.09 MB D 57 N filetree/나윤선/60                                                                                                                                                        319.82 MB O 319.82 MB C 319.82 MB D 60 N filetree/나윤선/63                                                                                                                                                        330.30 MB O 330.31 MB C 330.31 MB D 62 N filetree/나윤선/65                                                                                                                                                        341.77 MB O 341.78 MB C 341.78 MB D 65 N filetree/나윤선/68                                                                                                                                                        356.52 MB O 356.52 MB C 356.52 MB D 67 N filetree/나윤선/7                                                                                                                                                         371.30 MB O 371.31 MB C 371.31 MB D 70 N filetree/나윤선/72                                                                                                                                                        387.97 MB O 387.98 MB C 387.98 MB D 73 N filetree/나윤선/75                                                                                                                                                        399.11 MB O 399.12 MB C 399.12 MB D 76 N filetree/나윤선/78                                                                                                                                                        414.18 MB O 414.19 MB C 414.19 MB D 78 N filetree/나윤선/8                                                                                                                                                         424.67 MB O 424.68 MB C 424.68 MB D 80 N filetree/나윤선/81                                                                                                                                                        437.41 MB O 437.42 MB C 437.42 MB D 83 N filetree/나윤선/84                                                                                                                                                        448.18 MB O 448.19 MB C 448.19 MB D 85 N filetree/나윤선/86                                                                                                                                                        460.97 MB O 460.98 MB C 460.98 MB D 87 N filetree/나윤선/88                                                                                                                                                        471.86 MB O 471.87 MB C 471.87 MB D 89 N filetree/나윤선/9                                                                                                                                                         482.34 MB O 482.36 MB C 482.36 MB D 91 N filetree/나윤선/91                                                                                                                                                        492.83 MB O 492.84 MB C 492.84 MB D 93 N filetree/나윤선/93                                                                                                                                                        505.19 MB O 505.21 MB C 505.21 MB D 96 N filetree/나윤선/96                                                                                                                                                        519.05 MB O 519.06 MB C 519.06 MB D 98 N filetree/나윤선/98                                                                                                                                                        + rm -rf repo filetree  
```

interestingly the pasted value is a mess, on the terminal its displayed as multiple lines


Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -28,7 +28,7 @@
 from .helpers import uid2user, user2uid, gid2group, group2gid
 from .helpers import parse_timestamp, to_localtime
 from .helpers import format_time, format_timedelta, format_file_size, file_status
-from .helpers import safe_encode, safe_decode, make_path_safe, remove_surrogates
+from .helpers import safe_encode, safe_decode, make_path_safe, remove_surrogates, swidth_slice
 from .helpers import decode_dict, StableDict
 from .helpers import int_to_bigint, bigint_to_int, bin_to_hex
 from .helpers import ProgressIndicatorPercent, log_multi
@@ -94,8 +94,10 @@ def show_progress(self, item=None, final=False, stream=None, dt=None):
                     space = columns - swidth(msg)
                 if space >= 8:
                     if space < swidth('...') + swidth(path):
-                        path = '%s...%s' % (path[:(space // 2) - swidth('...')], path[-space // 2:])
-                    msg += "{0:<{space}}".format(path, space=space)
+                        path = '%s...%s' % (swidth_slice(path, space // 2 - swidth('...')),
+                                            swidth_slice(path, -space // 2))
+                    space -= swidth(path)
+                    msg += path + ' ' * space
             else:
                 msg = ' ' * columns
             print(msg, file=stream or sys.stderr, end="\r", flush=True)
diff --git a/src/borg/helpers.py b/src/borg/helpers.py
--- a/src/borg/helpers.py
+++ b/src/borg/helpers.py
@@ -1696,3 +1696,29 @@ def write(self, s):
                 except OSError:
                     pass
         return len(s)
+
+
+def swidth_slice(string, max_width):
+    """
+    Return a slice of *max_width* cells from *string*.
+
+    Negative *max_width* means from the end of string.
+
+    *max_width* is in units of character cells (or "columns").
+    Latin characters are usually one cell wide, many CJK characters are two cells wide.
+    """
+    from .platform import swidth
+    reverse = max_width < 0
+    max_width = abs(max_width)
+    if reverse:
+        string = reversed(string)
+    current_swidth = 0
+    result = []
+    for character in string:
+        current_swidth += swidth(character)
+        if current_swidth > max_width:
+            break
+        result.append(character)
+    if reverse:
+        result.reverse()
+    return ''.join(result)
diff --git a/src/borg/testsuite/helpers.py b/src/borg/testsuite/helpers.py
--- a/src/borg/testsuite/helpers.py
+++ b/src/borg/testsuite/helpers.py
@@ -9,6 +9,7 @@
 import msgpack
 import msgpack.fallback
 
+from .. import platform
 from ..helpers import Location
 from ..helpers import Buffer
 from ..helpers import partial_format, format_file_size, parse_file_size, format_timedelta, format_line, PlaceholderError, replace_placeholders
@@ -23,6 +24,7 @@
 from ..helpers import load_excludes
 from ..helpers import CompressionSpec, CompressionDecider1, CompressionDecider2
 from ..helpers import parse_pattern, PatternMatcher, RegexPattern, PathPrefixPattern, FnmatchPattern, ShellPattern
+from ..helpers import swidth_slice
 
 from . import BaseTestCase, environment_variable, FakeInputs
 
@@ -1041,3 +1043,23 @@ def test_replace_placeholders():
     now = datetime.now()
     assert " " not in replace_placeholders('{now}')
     assert int(replace_placeholders('{now:%Y}')) == now.year
+
+
+def working_swidth():
+    return platform.swidth('선') == 2
+
+
+@pytest.mark.skipif(not working_swidth(), reason='swidth() is not supported / active')
+def test_swidth_slice():
+    string = '나윤선나윤선나윤선나윤선나윤선'
+    assert swidth_slice(string, 1) == ''
+    assert swidth_slice(string, -1) == ''
+    assert swidth_slice(string, 4) == '나윤'
+    assert swidth_slice(string, -4) == '윤선'
+
+
+@pytest.mark.skipif(not working_swidth(), reason='swidth() is not supported / active')
+def test_swidth_slice_mixed_characters():
+    string = '나윤a선나윤선나윤선나윤선나윤선'
+    assert swidth_slice(string, 5) == '나윤a'
+    assert swidth_slice(string, 6) == '나윤a'


Test patch: