Repository: borgbackup/borg

Issue:
repository / rpc api: sequential scan
The current `repo.list()` method lists repo objects in repo index order (means: random segment order).
If one just wants the keys, this is no problem, everything is read from the index, which is in memory.

But if you want (all) the values also, this order causes a lot of random access - a real stress test for your disk(s).

If we want to iterate over all repo objects, same-segment offset (1.) and segment (2.) order would be much better. Maybe it could also support just getting the keys and alternatively key/value pairs.



Discussion:
Algorithm could be like:

```
for all segment files:
    for all objects in segment file:
        i_pos = index[key]  # seg, offs
        if i_pos == current_pos:
            yield key, data
```

Maybe we could even consider to just change `repo.list()` to yield keys that way - while not quite as efficient as the above, it would lead to some linear 2-pass behaviour that likely could make good use of disk caching and thus be almost as fast as above.

I think check is the only op using it anyway. For the reconstruction cache population it doesn't matter how they're ordered, so changing the existing implementation to returning linearly ordered (by `(segment, offset)`) keys makes sense. Since it's purely in-memory it should be plenty fast. Maybe even implement it straight in Cython.

Memory use of a simple implementation is constrained by the number of objects in each segment, so not too much.

borg check (init_chunks, no data access, order does not matter) and some debug cmd (reading data, so order matters) uses it currently. In future, --verify-data would use it, reading all data, so order matters.

What I was proposing above is a disk-based algorithm that just checks against the repo index to make sure it is no superseded object, but results in a completely linear ordering for all keys.

So, considering that the repo index itself might already take some hundred MBs .. some GBs of memory, how would your purely-in-memory algorithm look like?

Like a nested-loop JOIN, so it'd be ~O(n_segments \* n_index), but doesn't do any IO. Hm.

Maybe do it a bit differently; like you proposed an IO based approach, and --verify-data consumes chunks of list() as it goes. This way the segment(s) read by list() would most likely still be in the page cache and the only cost in re-reading them shortly afterwards is another pass of CRC and copying data around.

Since the cookie returned by list() has to be an object ID, we can just use the last yielded object again as it is now. This immediately tells us in which segment and offset the listing will continue.

Yeah, that approach is way more practical. Lets forget about nested loops.

(this is referring to the in-memory approach)

OK, so have a low-level op in C / Cython that does a hashtable scan, filtering out all entries for segment N, then sort (much smaller) result by offset.

Could even filter out entries for segment N..M, so high level code could estimate the segment count according to the amount of data wanted. This would reduce amount of hashtable scans by some factor.

see PR #1607

We removed the sequential scan API from #1607 again due to some complications.

What's currently on the table:
- Add new Repository method + RPC call called eg. `scan` or `linear_scan`
- Which returns a number of object IDs (with the usual limit and marker args)
- It works as follows:
  - If `marker`: get (segment, offset) from index
  - Elif not `marker`: segment = first segment, offset = 0
  - iter_objects from (segment, offset) on; ignore everything except PUTs, yield PUTs that are in the index and current
  - Take care not to evict the read segments from the page cache (this bit is important)
  - After yielding `limit` IDs, return.
- This enables
  - Efficient linear scanning
  - No extra disk access when objects are get()ted, assuming the page cache is large enough for the objects contained within `limit`

---

A different take on this is to have a `next`-like function, which yields the next (or a couple) objects, complete with data. Runtime-wise this would likely not be much different from a `scan`-like approach. A con is that this has to be handled very differently from `list` on the client.

Thoughts?

If we make a new rpc call anyway, we could also offer to yield not just keys, but also key/value tuples, if requested.

Making that a toggle would make sense, even if not used right now.

We could also make a low-level API and a high-level API here.

With the low-level API being a marker-based interface like those above, while the high-level API gives you a generator over the whole repo (essentially a high level `LoggedIO.iter_objects`). The latter can be implemented in a mix-in and be shared between Repository and RemoteRepository. It can also check object count vs the index after all objects where yielded. (OTOH this would make hardly a difference for --verify-data, since it still has to juggle the generators in case of errors).

So maybe leave it at the "low-level" API for now. Is also simpler. KISS.

Implemented by #1682 (Repository.scan()) and used for `borg check --verify-data`. (master branch)

This implementation is better than the random access before but still has very poor performance. Average I/O utilization is below 1 %, while the CPU is hit _hard_. (Repository check has 85 % I/O and only 15 % CPU on this repo)

Possible reaons:
1. https://github.com/borgbackup/borg/blob/master/src/borg/repository.py#L780
   
   For every scan() call the segment list is walked up to the current segment. Probably no biggie.
2. https://github.com/borgbackup/borg/blob/master/src/borg/repository.py#L783
   
   For every scan() call the current segment is re-read up to the marker object. I think this is where the poor performance comes from: it reads 100 objects (0-100); processes them. Reads the next 100 objects (100-200), for which the first 100 objects are read again etc. Effectively this squares the cost of this to the number of objects per segment.
   
   Note: Test was done with 500 MB segments on local disk.

Thanks for testing this. Yes, for big segments, 2. is likely an issue. If one has relatively small input files, a 500MB segment can hold a lot of them. I'll work on that.


Resolving patch:
diff --git a/src/borg/repository.py b/src/borg/repository.py
--- a/src/borg/repository.py
+++ b/src/borg/repository.py
@@ -775,12 +775,10 @@ def scan(self, limit=None, marker=None):
             self.index = self.open_index(transaction_id)
         at_start = marker is None
         # smallest valid seg is <uint32> 0, smallest valid offs is <uint32> 8
-        marker_segment, marker_offset = (0, 0) if at_start else self.index[marker]
+        start_segment, start_offset = (0, 0) if at_start else self.index[marker]
         result = []
-        for segment, filename in self.io.segment_iterator():
-            if segment < marker_segment:
-                continue
-            obj_iterator = self.io.iter_objects(segment, read_data=False, include_data=False)
+        for segment, filename in self.io.segment_iterator(start_segment):
+            obj_iterator = self.io.iter_objects(segment, start_offset, read_data=False, include_data=False)
             while True:
                 try:
                     tag, id, offset, size = next(obj_iterator)
@@ -788,7 +786,11 @@ def scan(self, limit=None, marker=None):
                     # either end-of-segment or an error - we can not seek to objects at
                     # higher offsets than one that has an error in the header fields
                     break
-                if segment == marker_segment and offset <= marker_offset:
+                if start_offset > 0:
+                    # we are using a marker and the marker points to the last object we have already
+                    # returned in the previous scan() call - thus, we need to skip this one object.
+                    # also, for the next segment, we need to start at offset 0.
+                    start_offset = 0
                     continue
                 if tag == TAG_PUT and (segment, offset) == self.index.get(id):
                     # we have found an existing and current object
@@ -886,14 +888,25 @@ def close_fd(self, fd):
             os.posix_fadvise(fd.fileno(), 0, 0, os.POSIX_FADV_DONTNEED)
         fd.close()
 
-    def segment_iterator(self, reverse=False):
+    def segment_iterator(self, segment=None, reverse=False):
+        if segment is None:
+            segment = 0 if not reverse else 2 ** 32 - 1
         data_path = os.path.join(self.path, 'data')
-        dirs = sorted((dir for dir in os.listdir(data_path) if dir.isdigit()), key=int, reverse=reverse)
+        start_segment_dir = segment // self.segments_per_dir
+        dirs = os.listdir(data_path)
+        if not reverse:
+            dirs = [dir for dir in dirs if dir.isdigit() and int(dir) >= start_segment_dir]
+        else:
+            dirs = [dir for dir in dirs if dir.isdigit() and int(dir) <= start_segment_dir]
+        dirs = sorted(dirs, key=int, reverse=reverse)
         for dir in dirs:
             filenames = os.listdir(os.path.join(data_path, dir))
-            sorted_filenames = sorted((filename for filename in filenames
-                                       if filename.isdigit()), key=int, reverse=reverse)
-            for filename in sorted_filenames:
+            if not reverse:
+                filenames = [filename for filename in filenames if filename.isdigit() and int(filename) >= segment]
+            else:
+                filenames = [filename for filename in filenames if filename.isdigit() and int(filename) <= segment]
+            filenames = sorted(filenames, key=int, reverse=reverse)
+            for filename in filenames:
                 yield int(filename), os.path.join(data_path, dir, filename)
 
     def get_latest_segment(self):
@@ -999,7 +1012,7 @@ def segment_exists(self, segment):
     def segment_size(self, segment):
         return os.path.getsize(self.segment_filename(segment))
 
-    def iter_objects(self, segment, include_data=False, read_data=True):
+    def iter_objects(self, segment, offset=0, include_data=False, read_data=True):
         """
         Return object iterator for *segment*.
 
@@ -1009,10 +1022,14 @@ def iter_objects(self, segment, include_data=False, read_data=True):
         The iterator returns four-tuples of (tag, key, offset, data|size).
         """
         fd = self.get_fd(segment)
-        fd.seek(0)
-        if fd.read(MAGIC_LEN) != MAGIC:
-            raise IntegrityError('Invalid segment magic [segment {}, offset {}]'.format(segment, 0))
-        offset = MAGIC_LEN
+        fd.seek(offset)
+        if offset == 0:
+            # we are touching this segment for the first time, check the MAGIC.
+            # Repository.scan() calls us with segment > 0 when it continues an ongoing iteration
+            # from a marker position - but then we have checked the magic before already.
+            if fd.read(MAGIC_LEN) != MAGIC:
+                raise IntegrityError('Invalid segment magic [segment {}, offset {}]'.format(segment, 0))
+            offset = MAGIC_LEN
         header = fd.read(self.header_fmt.size)
         while header:
             size, tag, key, data = self._read(fd, self.header_fmt, header, segment, offset,


Test patch: