Repository: borgbackup/borg

Issue:
Progress output generates extra lines with some filenames
I'm running `borg create --no-files-cache --progress --numeric-owner` in an interactive terminal, and with most filenames, the command shows only one line of progress with some counters and the current filename. However, with some filenames, it seems to go on to the next line instead of overwriting the current line. Basically, in the screenshot below, I expect to see only the last line, not all the previous lines.

![image](https://user-images.githubusercontent.com/15314936/30412513-063b9a3e-98e6-11e7-85ff-efd61a011259.png)


Discussion:
In case it helps, the filenames are utf-8, the non-ascii characters are `女子十二乐坊`, and here's some more information about my environment:

```
dseomn@solaria:~$ locale
LANG=en_US.UTF-8
LANGUAGE=en_US
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
dseomn@solaria:~$ echo $TERM
screen.xterm-256color
```
We have some code in borg that should deal with that.

So, what borg version, python version, operating system and version do you use?
Here you go. Sorry I forgot to check if the bug is still present in the latest version.

```
dseomn@solaria:~$ /usr/bin/borg --version
borg 1.0.7
dseomn@solaria:~$ head -n 1 /usr/bin/borg
#!/usr/bin/python3
dseomn@solaria:~$ python3 --version
Python 3.5.2
dseomn@solaria:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.3 LTS
Release:        16.04
Codename:       xenial
```
1.0.7 is not quite the latest stuff, try the 1.0.11 and 1.1.0rc3 borg-linux64 binaries from there:

https://github.com/borgbackup/borg/releases

I just tried with borg 1.0.11, python 3.5.3, FreeBSD 10.3, LC_ALL=en_US.UTF-8, and TERM=xterm-256color, and the issue was still there. For what it's worth, many other non-ascii characters seem to work fine. As far as I can tell, it's only CJK characters that are causing trouble.
I think I found the cause. https://github.com/borgbackup/borg/blob/master/src/borg/helpers/progress.py uses `str.ljust`, which doesn't seem to take character width into account: https://www.recurse.com/blog/74-a-string-of-unexpected-lengths (scroll down to "The Unicode jungle"). I don't think I'm going to use `--progress` often enough for it to matter much to me, so feel free to close this bug if you don't think it's worth the effort to fix.
This is fixed in 1.1 with no backport.

Edit: borg.helpers.progress isn't actually responsible for the --progress output of borg create. That's done by archive.Statistics. https://github.com/borgbackup/borg/blob/master/src/borg/archive.py#L111
The same issue might still kind-of apply to progress displays involving archive names, if double-cell width CJK characters are used for archive names.
Thanks for the correction. That code looks like it uses `swidth` in https://github.com/borgbackup/borg/blob/master/src/borg/platform/base.py#L178, which doesn't seem to implement what the docstring says, and the docstring is also slightly wrong I think. (Some code points have 0 width.)
The posix platform modules delegate swidth to wcswidth(3).

Resolving patch:
diff --git a/src/borg/helpers/progress.py b/src/borg/helpers/progress.py
--- a/src/borg/helpers/progress.py
+++ b/src/borg/helpers/progress.py
@@ -144,10 +144,12 @@ def show(self, current=None, increase=1, info=None):
             # truncate the last argument, if no space is available
             if info is not None:
                 if not self.json:
+                    from ..platform import swidth  # avoid circular import
+
                     # no need to truncate if we're not outputting to a terminal
                     terminal_space = get_terminal_size(fallback=(-1, -1))[0]
                     if terminal_space != -1:
-                        space = terminal_space - len(self.msg % tuple([pct] + info[:-1] + [""]))
+                        space = terminal_space - swidth(self.msg % tuple([pct] + info[:-1] + [""]))
                         info[-1] = ellipsis_truncate(info[-1], space)
                 return self.output(self.msg % tuple([pct] + info), justify=False, info=info)
 
diff --git a/src/borg/testsuite/helpers.py b/src/borg/testsuite/helpers.py
--- a/src/borg/testsuite/helpers.py
+++ b/src/borg/testsuite/helpers.py
@@ -45,7 +45,7 @@
 from ..helpers import safe_unlink
 from ..helpers import text_to_json, binary_to_json
 from ..helpers.passphrase import Passphrase, PasswordRetriesExceeded
-from ..platform import is_cygwin, is_win32, is_darwin
+from ..platform import is_cygwin, is_win32, is_darwin, swidth
 
 from . import BaseTestCase, FakeInputs, are_hardlinks_supported
 
@@ -1017,6 +1017,29 @@ def test_progress_percentage_sameline(capfd, monkeypatch):
     assert err == " " * 4 + "\r"
 
 
+@pytest.mark.skipif(is_win32, reason="no working swidth() implementation on this platform")
+def test_progress_percentage_widechars(capfd, monkeypatch):
+    st = "スター・トレック"  # "startrek" :-)
+    assert swidth(st) == 16
+    path = "/カーク船長です。"  # "Captain Kirk"
+    assert swidth(path) == 17
+    spaces = " " * 4  # to avoid usage of "..."
+    width = len("100%") + 1 + swidth(st) + 1 + swidth(path) + swidth(spaces)
+    monkeypatch.setenv("COLUMNS", str(width))
+    monkeypatch.setenv("LINES", "1")
+    pi = ProgressIndicatorPercent(100, step=5, start=0, msg=f"%3.0f%% {st} %s")
+    pi.logger.setLevel("INFO")
+    pi.show(0, info=[path])
+    out, err = capfd.readouterr()
+    assert err == f"  0% {st} {path}{spaces}\r"
+    pi.show(100, info=[path])
+    out, err = capfd.readouterr()
+    assert err == f"100% {st} {path}{spaces}\r"
+    pi.finish()
+    out, err = capfd.readouterr()
+    assert err == " " * width + "\r"
+
+
 def test_progress_percentage_step(capfd, monkeypatch):
     # run the test as if it was in a 4x1 terminal
     monkeypatch.setenv("COLUMNS", "4")


Test patch: