Repository: borgbackup/borg

Issue:
support json-lines output for stream-processing of list
http://jsonlines.org/ is a format for streaming json values as lines in a stream,

to me it seems that could be vastly useful to handle list outputs in a streamed manner (given that there are no stream style json parser, but `for value in map(json.loads, fp)` is pretty easy to use/handle


Discussion:
There are streaming JSON parsers available for most languages (e.g. yajl, gson, ...).
@enkore have you surveyed any of those for ease of use, at least the python ones i tried are not something i would want to program with
I used yajl in C which is quite alright. For i3pystatus I [rolled my own](https://github.com/enkore/i3pystatus/blob/master/i3pystatus/core/io.py#L146).

I totally see your point about ease of use here and there is also a semantic difference between having a large `{}` with everything in it vs. an explicit stream (in the former case there is the theoretical possibility of keys appearing twice ; probably the reason why the default interface in JSON is serdes, not events/streaming).

`{}` and the standard JSON parsers kind-of suggest loading into memory is okay. An explicit streamed format does not. On the other hand a streamed format means a bit of special-casing due to some additional metadata. I.e. First `{}` is the metadata (repository info etc.), following `{}`s are items.

Ok.

---

**Tasks:**

1. Make the _file_ listing output of `borg list --json` conform to a JSONlines-style output (and update the docs). The first object should have the metadata that is output now (everything but the `items` key), the following objects are the files (what is now found under the `items` key).
2. Rename the --json flag (of `borg list`) to account for the output not being pure JSON anymore, e.g. `--json-list` or `--json-lines`.

**Discuss:** Drop the metadata that is usually output in this case?
@enkore note that i would strongly suggest to keep json output and json-lines output separate including the flags,

one is about getting output that is a json document, the other is a stream of one json document per line
I disagree: JSON lines makes sense for large list-documents and while JSON can do the same, the difference is minor semantics and more importantly external support. Providing both for the same output thus has no advantage.

This is (as written above) specifically about the file listing in borg-list. In other instances (archive listings etc.) the rationale does not apply and thus no change should be made.
@enkore you dont have to provide the same, im just opposed to providing the formats under the same name
`--json-list`?
> **Discuss:** Drop the metadata that is usually output in this case?

Special-casing the first item is kinda ugly. So, don't do it? This data if of interest  is just a `borg list --json <repo>` away.

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -1048,8 +1048,14 @@ def write(bytestring):
             write = sys.stdout.buffer.write
 
         if args.location.archive:
+            if args.json:
+                self.print_error('The --json option is only valid for listing archives, not archive contents.')
+                return self.exit_code
             return self._list_archive(args, repository, manifest, key, write)
         else:
+            if args.json_lines:
+                self.print_error('The --json-lines option is only valid for listing archive contents, not archives.')
+                return self.exit_code
             return self._list_repository(args, manifest, write)
 
     def _list_archive(self, args, repository, manifest, key, write):
@@ -1065,11 +1071,9 @@ def _list_inner(cache):
             archive = Archive(repository, key, manifest, args.location.archive, cache=cache,
                               consider_part_files=args.consider_part_files)
 
-            formatter = ItemFormatter(archive, format, json=args.json)
-            write(safe_encode(formatter.begin()))
+            formatter = ItemFormatter(archive, format, json_lines=args.json_lines)
             for item in archive.iter_items(lambda item: matcher.match(item.path)):
                 write(safe_encode(formatter.format_item(item)))
-            write(safe_encode(formatter.end()))
 
         # Only load the cache if it will be used
         if ItemFormatter.format_needs_cache(format):
@@ -2616,9 +2620,17 @@ def process_epilog(epilog):
                                help="""specify format for file listing
                                 (default: "{mode} {user:6} {group:6} {size:8d} {isomtime} {path}{extra}{NL}")""")
         subparser.add_argument('--json', action='store_true',
-                               help='format output as JSON. The form of --format is ignored, but keys used in it '
-                                    'are added to the JSON output. Some keys are always present. Note: JSON can only '
-                                    'represent text. A "bpath" key is therefore not available.')
+                               help='Only valid for listing repository contents. Format output as JSON. '
+                                    'The form of --format is ignored, '
+                                    'but keys used in it are added to the JSON output. '
+                                    'Some keys are always present. Note: JSON can only represent text. '
+                                    'A "barchive" key is therefore not available.')
+        subparser.add_argument('--json-lines', action='store_true',
+                               help='Only valid for listing archive contents. Format output as JSON Lines. '
+                                    'The form of --format is ignored, '
+                                    'but keys used in it are added to the JSON output. '
+                                    'Some keys are always present. Note: JSON can only represent text. '
+                                    'A "bpath" key is therefore not available.')
         subparser.add_argument('location', metavar='REPOSITORY_OR_ARCHIVE', nargs='?', default='',
                                type=location_validator(),
                                help='repository/archive to list contents of')
diff --git a/src/borg/helpers.py b/src/borg/helpers.py
--- a/src/borg/helpers.py
+++ b/src/borg/helpers.py
@@ -1505,9 +1505,9 @@ def format_needs_cache(cls, format):
         format_keys = {f[1] for f in Formatter().parse(format)}
         return any(key in cls.KEYS_REQUIRING_CACHE for key in format_keys)
 
-    def __init__(self, archive, format, *, json=False):
+    def __init__(self, archive, format, *, json_lines=False):
         self.archive = archive
-        self.json = json
+        self.json_lines = json_lines
         static_keys = {
             'archivename': archive.name,
             'archiveid': archive.fpr,
@@ -1532,33 +1532,14 @@ def __init__(self, archive, format, *, json=False):
         for hash_function in hashlib.algorithms_guaranteed:
             self.add_key(hash_function, partial(self.hash_item, hash_function))
         self.used_call_keys = set(self.call_keys) & self.format_keys
-        if self.json:
+        if self.json_lines:
             self.item_data = {}
             self.format_item = self.format_item_json
-            self.first = True
         else:
             self.item_data = static_keys
 
-    def begin(self):
-        if not self.json:
-            return ''
-        begin = json_dump(basic_json_data(self.archive.manifest))
-        begin, _, _ = begin.rpartition('\n}')  # remove last closing brace, we want to extend the object
-        begin += ',\n'
-        begin += '    "items": [\n'
-        return begin
-
-    def end(self):
-        if not self.json:
-            return ''
-        return "]}"
-
     def format_item_json(self, item):
-        if self.first:
-            self.first = False
-            return json.dumps(self.get_item_data(item))
-        else:
-            return ',' + json.dumps(self.get_item_data(item))
+        return json.dumps(self.get_item_data(item)) + '\n'
 
     def add_key(self, key, callable_with_item):
         self.call_keys[key] = callable_with_item
@@ -1585,7 +1566,7 @@ def get_item_data(self, item):
         item_data['uid'] = item.uid
         item_data['gid'] = item.gid
         item_data['path'] = remove_surrogates(item.path)
-        if self.json:
+        if self.json_lines:
             item_data['healthy'] = 'chunks_healthy' not in item
         else:
             item_data['bpath'] = item.path
diff --git a/src/borg/testsuite/archiver.py b/src/borg/testsuite/archiver.py
--- a/src/borg/testsuite/archiver.py
+++ b/src/borg/testsuite/archiver.py
@@ -1615,22 +1615,44 @@ def test_list_json(self):
         assert list_repo['encryption']['mode'] == 'repokey'
         assert 'keyfile' not in list_repo['encryption']
 
-        list_archive = json.loads(self.cmd('list', '--json', self.repository_location + '::test'))
-        assert list_repo['repository'] == list_archive['repository']
-        items = list_archive['items']
+        list_archive = self.cmd('list', '--json-lines', self.repository_location + '::test')
+        items = [json.loads(s) for s in list_archive.splitlines()]
         assert len(items) == 2
         file1 = items[1]
         assert file1['path'] == 'input/file1'
         assert file1['size'] == 81920
 
-        list_archive = json.loads(self.cmd('list', '--json', '--format={sha256}', self.repository_location + '::test'))
-        assert list_repo['repository'] == list_archive['repository']
-        items = list_archive['items']
+        list_archive = self.cmd('list', '--json-lines', '--format={sha256}', self.repository_location + '::test')
+        items = [json.loads(s) for s in list_archive.splitlines()]
         assert len(items) == 2
         file1 = items[1]
         assert file1['path'] == 'input/file1'
         assert file1['sha256'] == 'b2915eb69f260d8d3c25249195f2c8f4f716ea82ec760ae929732c0262442b2b'
 
+    def test_list_json_args(self):
+        self.cmd('init', '--encryption=repokey', self.repository_location)
+        self.cmd('list', '--json-lines', self.repository_location, exit_code=2)
+        self.cmd('list', '--json', self.repository_location + '::archive', exit_code=2)
+
+    def test_log_json(self):
+        self.create_test_files()
+        self.cmd('init', '--encryption=repokey', self.repository_location)
+        log = self.cmd('create', '--log-json', self.repository_location + '::test', 'input', '--list', '--debug')
+        messages = {}  # type -> message, one of each kind
+        for line in log.splitlines():
+            msg = json.loads(line)
+            messages[msg['type']] = msg
+
+        file_status = messages['file_status']
+        assert 'status' in file_status
+        assert file_status['path'].startswith('input')
+
+        log_message = messages['log_message']
+        assert isinstance(log_message['time'], float)
+        assert log_message['levelname'] == 'DEBUG'  # there should only be DEBUG messages
+        assert log_message['name'].startswith('borg.')
+        assert isinstance(log_message['message'], str)
+
     def _get_sizes(self, compression, compressible, size=10000):
         if compressible:
             contents = b'X' * size


Test patch: