Repository: borgbackup/borg

Issue:
Trailing and leading whitespace and control characters are allowed as archive names
It is possible for borg archive names to have trailing and leading whitespace and control characters.

``` bash
$ borg create "/path/to/repo::archive       " /files/to/backup
$ borg create "/path/to/repo::   archive" /files/to/backup
$ borg create "/path/to/repo::new
> line" /files/to/backup
$ borg create /path/to/repo::$'\1' /files/to/backup
$ borg list /path/to/repo
archive                              Tue, 2016-06-14 02:10:48
   archive                           Tue, 2016-06-14 02:11:20
new
line                             Tue, 2016-06-14 02:11:23
                                    Tue, 2016-06-14 02:12:30
```

Obviously, this creates several issues, escpecially trailing whitespaces.
## Trailing whitespace

Due to `borg list`'s format **It is impossible to know whether an archive name has trailing spaces, let alone how many spaces it has**.

For example let's create an archive with a normal name.

``` bash
$ borg create /path/to/repo::archive /files/to/backup
```

Then create an archive with 1 space after the name, and another with 3 spaces after the name.

``` bash
$ borg create "/path/to/repo::archive " /files/to/backup
$ borg create "/path/to/repo::archive   " /files/to/backup
```

Listing the repository gives an awkward result.

``` bash
$ borg list /path/to/repo
archive                              Tue, 2016-06-14 03:20:01
archive                              Tue, 2016-06-14 03:24:06
archive                              Tue, 2016-06-14 03:28:13
```

The following problems occur:
- All archives appeares to have the same name, which is impossible.
- It is impossible to know that there are trailing whitespace characters.
- It is impossible to know how many trailing spaces there are.
## Control characters and leading whitespaces

Control characters and leading whitespaces may make `borg list` harder to read and harder to be parsed by scripts.
## Possible solutions
- Reject control characters and leading and trailing whitespaces when creating a new archive.
- Truncate leading and trailing whitespace when creating a new archive.
- Change format of `borg list`, or allow to change it via `--list-format`.

_Note: times in the shell's output are not actual times._

Trailing and leading whitespace and control characters are allowed as archive names
It is possible for borg archive names to have trailing and leading whitespace and control characters.

``` bash
$ borg create "/path/to/repo::archive       " /files/to/backup
$ borg create "/path/to/repo::   archive" /files/to/backup
$ borg create "/path/to/repo::new
> line" /files/to/backup
$ borg create /path/to/repo::$'\1' /files/to/backup
$ borg list /path/to/repo
archive                              Tue, 2016-06-14 02:10:48
   archive                           Tue, 2016-06-14 02:11:20
new
line                             Tue, 2016-06-14 02:11:23
                                    Tue, 2016-06-14 02:12:30
```

Obviously, this creates several issues, escpecially trailing whitespaces.
## Trailing whitespace

Due to `borg list`'s format **It is impossible to know whether an archive name has trailing spaces, let alone how many spaces it has**.

For example let's create an archive with a normal name.

``` bash
$ borg create /path/to/repo::archive /files/to/backup
```

Then create an archive with 1 space after the name, and another with 3 spaces after the name.

``` bash
$ borg create "/path/to/repo::archive " /files/to/backup
$ borg create "/path/to/repo::archive   " /files/to/backup
```

Listing the repository gives an awkward result.

``` bash
$ borg list /path/to/repo
archive                              Tue, 2016-06-14 03:20:01
archive                              Tue, 2016-06-14 03:24:06
archive                              Tue, 2016-06-14 03:28:13
```

The following problems occur:
- All archives appeares to have the same name, which is impossible.
- It is impossible to know that there are trailing whitespace characters.
- It is impossible to know how many trailing spaces there are.
## Control characters and leading whitespaces

Control characters and leading whitespaces may make `borg list` harder to read and harder to be parsed by scripts.
## Possible solutions
- Reject control characters and leading and trailing whitespaces when creating a new archive.
- Truncate leading and trailing whitespace when creating a new archive.
- Change format of `borg list`, or allow to change it via `--list-format`.

_Note: times in the shell's output are not actual times._



Discussion:
> - It is impossible to know that there are trailing whitespace characters.
> - It is impossible to know how many trailing spaces there are.

borg-info gives the verbatim name with no added formatting, so you'd be able to get it from there (by limited trial-and-error).

@enkore Trial and error may work though, still it's silly for borg to allow such a thing, users and scripts would not expect that this would be possible.

@enkore `--short` may also reveal the trailing whitespaces but does not handle new lines well.

This is a bit of a crap-in crap-out problem.

Maybe just do not feed crap (strange archive names) into borg?

Alright then, if you do not consider this as a bug, then you can close this issue. Otherwise you can review my pull request (#1174)

Well, it is somehow a solution for a problem I usually do not have.

Why are you having crap characters in archive names? Can you tell more about the use case?

I agree that this is indeed a trivial problem.

In my case, I'm attempting to make a GUI for borg that would open existing repositories and parse `borg list`, It would list other people's repositories. The problem is I can't guarantee it would work 100% of the time, as there could be some crazy guy out there who might have a repository which uses this crap.

In the end it's a design choice that it's up to you:
- Do you want to have the linux filesystem wild west, where you can allow all characters except NULL and slash, and see that this makes no problem for you in the future of this project, then that is fine. Decline the pull request and I'll work on the third solution provided, where I apply `--list-format` when listing repositories. Just be careful when dealing with these archive names as they may break repositories.
- Do you want to remove the headache and just it ban it all together, so you can guarantee correct formatting when printing archive names and that archive names won't break repositories in some awkward way, then that is fine. I'll still work on the third solution though as it would make parsing `borg list` way easier.

I've applied the third solution (#1179). Explanation of how this will fix the issue is in the PR. If this PR is merged, you can close the issue.

> - It is impossible to know that there are trailing whitespace characters.
> - It is impossible to know how many trailing spaces there are.

borg-info gives the verbatim name with no added formatting, so you'd be able to get it from there (by limited trial-and-error).

@enkore Trial and error may work though, still it's silly for borg to allow such a thing, users and scripts would not expect that this would be possible.

@enkore `--short` may also reveal the trailing whitespaces but does not handle new lines well.

This is a bit of a crap-in crap-out problem.

Maybe just do not feed crap (strange archive names) into borg?

Alright then, if you do not consider this as a bug, then you can close this issue. Otherwise you can review my pull request (#1174)

Well, it is somehow a solution for a problem I usually do not have.

Why are you having crap characters in archive names? Can you tell more about the use case?

I agree that this is indeed a trivial problem.

In my case, I'm attempting to make a GUI for borg that would open existing repositories and parse `borg list`, It would list other people's repositories. The problem is I can't guarantee it would work 100% of the time, as there could be some crazy guy out there who might have a repository which uses this crap.

In the end it's a design choice that it's up to you:
- Do you want to have the linux filesystem wild west, where you can allow all characters except NULL and slash, and see that this makes no problem for you in the future of this project, then that is fine. Decline the pull request and I'll work on the third solution provided, where I apply `--list-format` when listing repositories. Just be careful when dealing with these archive names as they may break repositories.
- Do you want to remove the headache and just it ban it all together, so you can guarantee correct formatting when printing archive names and that archive names won't break repositories in some awkward way, then that is fine. I'll still work on the third solution though as it would make parsing `borg list` way easier.

I've applied the third solution (#1179). Explanation of how this will fix the issue is in the PR. If this PR is merged, you can close the issue.


Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -29,7 +29,7 @@
 from .helpers import EXIT_SUCCESS, EXIT_WARNING, EXIT_ERROR
 from .helpers import Error, NoManifestError
 from .helpers import location_validator, archivename_validator, ChunkerParams, CompressionSpec
-from .helpers import ItemFormatter, format_time, format_file_size, format_archive
+from .helpers import BaseFormatter, ItemFormatter, ArchiveFormatter, format_time, format_file_size, format_archive
 from .helpers import safe_encode, remove_surrogates, bin_to_hex
 from .helpers import prune_within, prune_split
 from .helpers import to_localtime, timestamp
@@ -738,6 +738,14 @@ def do_mount(self, args, repository, manifest, key):
     @with_repository()
     def do_list(self, args, repository, manifest, key):
         """List archive or repository contents"""
+        if not hasattr(sys.stdout, 'buffer'):
+            # This is a shim for supporting unit tests replacing sys.stdout with e.g. StringIO,
+            # which doesn't have an underlying buffer (= lower file object).
+            def write(bytestring):
+                sys.stdout.write(bytestring.decode('utf-8', errors='replace'))
+        else:
+            write = sys.stdout.buffer.write
+
         if args.location.archive:
             matcher, _ = self.build_matcher(args.excludes, args.paths)
             with Cache(repository, key, manifest, lock_wait=self.lock_wait) as cache:
@@ -751,23 +759,22 @@ def do_list(self, args, repository, manifest, key):
                     format = "{mode} {user:6} {group:6} {size:8} {isomtime} {path}{extra}{NL}"
                 formatter = ItemFormatter(archive, format)
 
-                if not hasattr(sys.stdout, 'buffer'):
-                    # This is a shim for supporting unit tests replacing sys.stdout with e.g. StringIO,
-                    # which doesn't have an underlying buffer (= lower file object).
-                    def write(bytestring):
-                        sys.stdout.write(bytestring.decode('utf-8', errors='replace'))
-                else:
-                    write = sys.stdout.buffer.write
                 for item in archive.iter_items(lambda item: matcher.match(item.path)):
                     write(safe_encode(formatter.format_item(item)))
         else:
+            if args.format:
+                format = args.format
+            elif args.short:
+                format = "{archive}{NL}"
+            else:
+                format = "{archive:<36} {time} [{id}]{NL}"
+            formatter = ArchiveFormatter(format)
+
             for archive_info in manifest.list_archive_infos(sort_by='ts'):
                 if args.prefix and not archive_info.name.startswith(args.prefix):
                     continue
-                if args.short:
-                    print(archive_info.name)
-                else:
-                    print(format_archive(archive_info))
+                write(safe_encode(formatter.format_item(archive_info)))
+
         return self.exit_code
 
     @with_repository(cache=True)
@@ -1618,8 +1625,13 @@ def build_parser(self, prog=None):
 
         See the "borg help patterns" command for more help on exclude patterns.
 
-        The following keys are available for --format when listing files:
+        The following keys are available for --format:
+        """) + BaseFormatter.keys_help() + textwrap.dedent("""
+
+        -- Keys for listing repository archives:
+        """) + ArchiveFormatter.keys_help() + textwrap.dedent("""
 
+        -- Keys for listing archive files:
         """) + ItemFormatter.keys_help()
         subparser = subparsers.add_parser('list', parents=[common_parser], add_help=False,
                                           description=self.do_list.__doc__,
diff --git a/src/borg/helpers.py b/src/borg/helpers.py
--- a/src/borg/helpers.py
+++ b/src/borg/helpers.py
@@ -1132,7 +1132,7 @@ def log_multi(*msgs, level=logging.INFO, logger=logger):
         logger.log(level, line)
 
 
-class ItemFormatter:
+class BaseFormatter:
     FIXED_KEYS = {
         # Formatting aids
         'LF': '\n',
@@ -1143,19 +1143,54 @@ class ItemFormatter:
         'NEWLINE': os.linesep,
         'NL': os.linesep,
     }
+
+    def get_item_data(self, item):
+        raise NotImplementedError
+
+    def format_item(self, item):
+        return self.format.format_map(self.get_item_data(item))
+
+    @staticmethod
+    def keys_help():
+        return " - NEWLINE: OS dependent line separator\n" \
+               " - NL: alias of NEWLINE\n" \
+               " - NUL: NUL character for creating print0 / xargs -0 like output, see barchive/bpath\n" \
+               " - SPACE\n" \
+               " - TAB\n" \
+               " - CR\n" \
+               " - LF"
+
+
+class ArchiveFormatter(BaseFormatter):
+
+    def __init__(self, format):
+        self.format = partial_format(format, self.FIXED_KEYS)
+
+    def get_item_data(self, archive):
+        return {
+            'barchive': archive.name,
+            'archive': remove_surrogates(archive.name),
+            'id': bin_to_hex(archive.id),
+            'time': format_time(to_localtime(archive.ts)),
+        }
+
+    @staticmethod
+    def keys_help():
+        return " - archive: archive name interpreted as text (might be missing non-text characters, see barchive)\n" \
+               " - barchive: verbatim archive name, can contain any character except NUL\n" \
+               " - time: time of creation of the archive\n" \
+               " - id: internal ID of the archive"
+
+
+class ItemFormatter(BaseFormatter):
     KEY_DESCRIPTIONS = {
         'bpath': 'verbatim POSIX path, can contain any character except NUL',
         'path': 'path interpreted as text (might be missing non-text characters, see bpath)',
         'source': 'link target for links (identical to linktarget)',
         'extra': 'prepends {source} with " -> " for soft links and " link to " for hard links',
-
         'csize': 'compressed size',
         'num_chunks': 'number of chunks in this file',
         'unique_chunks': 'number of unique chunks in this file',
-
-        'NEWLINE': 'OS dependent line separator',
-        'NL': 'alias of NEWLINE',
-        'NUL': 'NUL character for creating print0 / xargs -0 like ouput, see bpath',
     }
     KEY_GROUPS = (
         ('type', 'mode', 'uid', 'gid', 'user', 'group', 'path', 'bpath', 'source', 'linktarget', 'flags'),
@@ -1163,7 +1198,6 @@ class ItemFormatter:
         ('mtime', 'ctime', 'atime', 'isomtime', 'isoctime', 'isoatime'),
         tuple(sorted(hashlib.algorithms_guaranteed)),
         ('archiveid', 'archivename', 'extra'),
-        ('NEWLINE', 'NL', 'NUL', 'SPACE', 'TAB', 'CR', 'LF'),
     )
 
     @classmethod
@@ -1183,6 +1217,9 @@ class FakeArchive:
     def keys_help(cls):
         help = []
         keys = cls.available_keys()
+        for key in cls.FIXED_KEYS:
+            keys.remove(key)
+
         for group in cls.KEY_GROUPS:
             for key in group:
                 keys.remove(key)
@@ -1254,9 +1291,6 @@ def get_item_data(self, item):
             item_data[key] = self.call_keys[key](item)
         return item_data
 
-    def format_item(self, item):
-        return self.format.format_map(self.get_item_data(item))
-
     def calculate_num_chunks(self, item):
         return len(item.get('chunks', []))
 
diff --git a/src/borg/testsuite/archiver.py b/src/borg/testsuite/archiver.py
--- a/src/borg/testsuite/archiver.py
+++ b/src/borg/testsuite/archiver.py
@@ -1144,6 +1144,18 @@ def test_list_format(self):
         self.assertEqual(output_1, output_2)
         self.assertNotEqual(output_1, output_3)
 
+    def test_list_repository_format(self):
+        self.cmd('init', self.repository_location)
+        self.cmd('create', self.repository_location + '::test-1', src_dir)
+        self.cmd('create', self.repository_location + '::test-2', src_dir)
+        output_1 = self.cmd('list', self.repository_location)
+        output_2 = self.cmd('list', '--format', '{archive:<36} {time} [{id}]{NL}', self.repository_location)
+        self.assertEqual(output_1, output_2)
+        output_1 = self.cmd('list', '--short', self.repository_location)
+        self.assertEqual(output_1, 'test-1\ntest-2\n')
+        output_1 = self.cmd('list', '--format', '{barchive}/', self.repository_location)
+        self.assertEqual(output_1, 'test-1/test-2/')
+
     def test_list_hash(self):
         self.create_regular_file('empty_file', size=0)
         self.create_regular_file('amb', contents=b'a' * 1000000)


Test patch: