Repository: borgbackup/borg

Issue:
Restore zero'd chunks (`borg check --repair`) by the next `borg create`
/kind enhancement

Borg currently repairs broken repos (e.g. due to failing hardware) by zero'ing missing or otherwise damaged chunks (`borg check --repair`). This is fine because it makes an otherwise broken repo usable again at the cost of loosing some data. The data is irrecoverably lost. In practice it means that some files, possibly across a large number of archives, are either completely, or partially zero'd. Borg generally handles this really great by "keeping" the file in the archive, but reporting that some parts of this file (to be more precise: one of the file's chunks) were zero'd when accessing such file (via `borg extract` or `borg mount`).

I'd like to draw on the "irrecoverably lost" part of this procedure: some data might indeed be recoverable - not by some obscure data recovery process, but by using the original data. Whenever possible Borg should keep information about the zero'd chunks to check whether a subsequent `borg create` might allow recreating the lost chunk - and therefore, practically speaking, restoring this previously lost file in older archives, too.

I don't really know much about Borg's data structure and the chunker, so I can't really assess *how* this might be possible. It might not be possible for any lost data. It might require some imperfections when matching data (like how Borg chooses not to read the contents of a file because it assumes - based on the file's metadata - that the file wasn't changed). It might be so CPU-intensive or error-prone that this "recovery by original data" needs to be requested manually by some CLI option. It might be a lot of work to implement it and thus not worth the effort. But I'm rather confident (like 90% :laughing:) that it should be possible - and allowing something like this would make Borg even more amazing.

Here are some steps to better illustrate what I mean:

1. Setup test environment
   ```console
   $ cd some_dir
   $ export BORG_BASE_DIR="$PWD"
   $ borg init -e none repo
   $ mkdir data
   ```
2. Create some junk data and create a backup
   ```console
   $ for i in {1..100}; do s=$(shuf -i 0-10485760 -n 1); dd if=/dev/urandom of=$(mktemp -u -p data XXXXXXXXXX) bs=$s count=1 status=none; done
   $ borg create repo/::test1 data/
   ```
3. Delete some repo data (let's choose the largest file), check repo (it will report errors) and repair it (remember one of the affected files)
   ```console
   $ rm $(find repo/data/ -type f -printf '%s\t%p\n' | sort -n | tail -n1 | cut -f2)
   $ borg check repo
   Index object count mismatch.
   committed index: 307 objects
   rebuilt index:   14 objects
   ID: cc7781ae02aaa25eb2d18b31b40e15376d1f08dfff90eb7f46d4a83dfe7c3e5c rebuilt index: <not found>      committed index: (2, 80919919)   
   …
   $ borg check repo --repair
   borg check repo --repair
   This is a potentially dangerous function.
   check --repair might lead to data loss (for kinds of corruption it is not
   capable of dealing with). BE VERY CAREFUL!
   
   Type 'YES' if you understand this and want to continue: YES
   test1: data/LXFJGLqmgW: New missing file chunk detected (Byte 0-1414617). Replacing with all-zero chunk.
   …
   ```
4. Create another backup of the same original data (!)
   ```console
   $ borg create repo/::test2 data/
   ```
5. Try to extract one of the affected files
   ```console
   $ borg extract repo/::test1 data/LXFJGLqmgW
   data/LXFJGLqmgW: File has damaged (all-zero) chunks. Try running borg check --repair.
   $ borg extract repo/::test2 data/LXFJGLqmgW
   ```

As you can see Borg fails to restore the affected file (`data/LXFJGLqmgW` in this example) from the pre-data-loss `test1` archive, but happily restores the exact same file from the post-data-loss `test2` archive. Right now this is expected behaviour, but I suggest trying more to detect that both archives contain the same data and that - theoretically - the file can be restored from the `test1` archive, too.

By the way: Why is `borg extract` mentioning `borg check --repair`? Isn't `borg check --repair` the only way for a chunk to get zero'd, so that having an all-zero chunk means that `borg check --repair` was used already?

Discussion:

Patch:
diff --git a/docs/development.rst b/docs/development.rst
index 7f38545d..5db1f6d5 100644
--- a/docs/development.rst
+++ b/docs/development.rst
@@ -146,9 +146,12 @@ follow their `guide about avoiding ruining git blame`_:
 Continuous Integration
 ----------------------
 
-All pull requests go through `GitHub Actions`_, which runs the tests on misc.
-Python versions and on misc. platforms as well as some additional checks.
+All pull requests go through `GitHub Actions`_, which runs the tests on Linux
+and Mac OS X as well as the flake8 style checker. Windows builds run on AppVeyor_,
+while additional Unix-like platforms are tested on Golem_.
 
+.. _AppVeyor: https://ci.appveyor.com/project/borgbackup/borg/
+.. _Golem: https://golem.enkore.de/view/Borg/
 .. _GitHub Actions: https://github.com/borgbackup/borg/actions
 
 Output and Logging
@@ -349,8 +352,6 @@ Checklist:
     scripts/upload-pypi X.Y.Z test
     scripts/upload-pypi X.Y.Z
 
-  Note: the signature is not uploaded to PyPi any more, but we upload it to
-  github releases.
 - Put binaries into dist/borg-OSNAME and sign them:
 
   ::
@@ -369,10 +370,9 @@ Checklist:
 
 - Create a GitHub release, include:
 
-  * pypi dist package and signature
   * Standalone binaries (see above for how to create them).
 
-    + For macOS, document the macFUSE version in the README of the binaries.
-      macFUSE uses a kernel extension that needs to be compatible with the
+    + For OS X, document the OS X Fuse version in the README of the binaries.
+      OS X FUSE uses a kernel extension that needs to be compatible with the
       code contained in the binary.
   * A link to ``CHANGES.rst``.
diff --git a/docs/installation.rst b/docs/installation.rst
index 731c88a6..fd6eeca1 100644
--- a/docs/installation.rst
+++ b/docs/installation.rst
@@ -264,7 +264,7 @@ the installed ``openssl`` formula, point pkg-config to the correct path::
 
     PKG_CONFIG_PATH="/usr/local/opt/openssl@1.1/lib/pkgconfig" pip install borgbackup[llfuse]
 
-Be aware that for all recent macOS releases you must authorize full disk access.
+For OS X Catalina and later, be aware that you must authorize full disk access.
 It is no longer sufficient to run borg backups as root. If you have not yet
 granted full disk access, and you run Borg backup from cron, you will see
 messages such as::
diff --git a/docs/internals/data-structures.rst b/docs/internals/data-structures.rst
index 76147876..f0fc9323 100644
--- a/docs/internals/data-structures.rst
+++ b/docs/internals/data-structures.rst
@@ -384,7 +384,8 @@ The *tam* key is part of the :ref:`tertiary authentication mechanism <tam_descri
 the manifest, since an ID check is not possible.
 
 *config* is a general-purpose location for additional metadata. All versions
-of Borg preserve its contents.
+of Borg preserve its contents (it may have been a better place for *item_keys*,
+which is not preserved by unaware Borg versions, releases predating 1.0.4).
 
 Feature flags
 +++++++++++++
@@ -792,7 +793,7 @@ For small hash tables, we start with a growth factor of 2, which comes down to
 
 E.g. backing up a total count of 1 Mi (IEC binary prefix i.e. 2^20) files with a total size of 1TiB.
 
-a) with ``create --chunker-params buzhash,10,23,16,4095`` (custom):
+a) with ``create --chunker-params buzhash,10,23,16,4095`` (custom, like borg < 1.0):
 
   mem_usage  =  2.8GiB
 
diff --git a/docs/usage/create.rst b/docs/usage/create.rst
index f2ff4966..8bc4c89a 100644
--- a/docs/usage/create.rst
+++ b/docs/usage/create.rst
@@ -36,7 +36,8 @@ Examples
     $ fusermount -u sshfs-mount
 
     # Make a big effort in fine granular deduplication (big chunk management
-    # overhead, needs a lot of RAM and disk space, see formula in internals docs):
+    # overhead, needs a lot of RAM and disk space, see formula in internals
+    # docs - same parameters as borg < 1.0):
     $ borg create --chunker-params buzhash,10,23,16,4095 small /smallstuff
 
     # Backup a raw device (must not be active/in use/mounted at that time)
diff --git a/docs/usage/general/file-metadata.rst.inc b/docs/usage/general/file-metadata.rst.inc
index c2921819..c2694d18 100644
--- a/docs/usage/general/file-metadata.rst.inc
+++ b/docs/usage/general/file-metadata.rst.inc
@@ -29,7 +29,7 @@ On some platforms additional features are supported:
 +=========================+==========+===========+============+
 | Linux                   | Yes      | Yes       | Yes [1]_   |
 +-------------------------+----------+-----------+------------+
-| macOS                   | Yes      | Yes       | Yes (all)  |
+| Mac OS X                | Yes      | Yes       | Yes (all)  |
 +-------------------------+----------+-----------+------------+
 | FreeBSD                 | Yes      | Yes       | Yes (all)  |
 +-------------------------+----------+-----------+------------+
diff --git a/docs/usage/general/file-systems.rst.inc b/docs/usage/general/file-systems.rst.inc
index 1fbe4724..7a022899 100644
--- a/docs/usage/general/file-systems.rst.inc
+++ b/docs/usage/general/file-systems.rst.inc
@@ -21,7 +21,7 @@ and readable after one of the failures mentioned above occurred, run
 - At least three directory levels with short names
 - Typically, file sizes up to a few hundred MB.
   Large repositories may require large files (>2 GB).
-- Up to 1000 files per directory.
+- Up to 1000 files per directory (10000 for repositories initialized with Borg 1.0)
 - rename(2) / MoveFile(Ex) should work as specified, i.e. on the same file system
   it should be a move (not a copy) operation, and in case of a directory
   it should fail if the destination exists and is not an empty directory,
diff --git a/docs/usage/notes.rst b/docs/usage/notes.rst
index 520043c5..79d42254 100644
--- a/docs/usage/notes.rst
+++ b/docs/usage/notes.rst
@@ -175,7 +175,9 @@ Separate compaction
 ~~~~~~~~~~~~~~~~~~~
 
 Borg does not auto-compact the segment files in the repository at commit time
-(at the end of each repository-writing command) any more (since borg 1.2.0).
+(at the end of each repository-writing command) any more.
+
+This is new since borg 1.2.0 and requires borg >= 1.2.0 on client and server.
 
 This causes a similar behaviour of the repository as if it was in append-only
 mode (see below) most of the time (until ``borg compact`` is invoked or an
@@ -234,7 +236,7 @@ in ``.ssh/authorized_keys``:
     command="borg serve --append-only ..." ssh-rsa <key used for not-always-trustable backup clients>
     command="borg serve ..." ssh-rsa <key used for backup management>
 
-Running ``borg rcreate`` via a ``borg serve --append-only`` server will *not* create
+Running ``borg init`` via a ``borg serve --append-only`` server will *not* create
 an append-only repository. Running ``borg rcreate --append-only`` creates an append-only
 repository regardless of server settings.
 
@@ -274,7 +276,7 @@ with file 6::
 
 That's all to do in the repository.
 
-If you want to access this rolled back repository from a client that already has
+If you want to access this rollbacked repository from a client that already has
 a cache for this repository, the cache will reflect a newer repository state
 than what you actually have in the repository now, after the rollback.
 
diff --git a/setup.cfg b/setup.cfg
index 78762794..327a38c7 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -159,7 +159,7 @@ per_file_ignores =
     src/borg/testsuite/__init__.py:E501,F401
     src/borg/testsuite/archive.py:E128,W504
     src/borg/testsuite/archiver/__init__.py:E128,E501,E722,F401,F405,F811
-    src/borg/testsuite/archiver/debug_cmds.py:E501,F405
+    src/borg/testsuite/archiver/debug_cmds.py:E501
     src/borg/testsuite/archiver/disk_full.py:F401,F405,F811
     src/borg/testsuite/archiver/extract_cmd.py:F405
     src/borg/testsuite/archiver/mount_cmds.py:E501,E722
diff --git a/setup_docs.py b/setup_docs.py
index 0861140e..41a41f35 100644
--- a/setup_docs.py
+++ b/setup_docs.py
@@ -6,7 +6,7 @@
 import sys
 import textwrap
 from collections import OrderedDict
-from datetime import datetime, timezone
+from datetime import datetime
 import time
 
 from setuptools import Command
@@ -470,8 +470,10 @@ def write_man_header(self, write, title, description):
         self.write_heading(write, description, double_sided=True)
         # man page metadata
         write(":Author: The Borg Collective")
-        source_date_epoch = int(os.environ.get("SOURCE_DATE_EPOCH", time.time()))
-        write(":Date:", datetime.fromtimestamp(source_date_epoch, timezone.utc).date().isoformat())
+        write(
+            ":Date:",
+            datetime.utcfromtimestamp(int(os.environ.get("SOURCE_DATE_EPOCH", time.time()))).date().isoformat(),
+        )
         write(":Manual section: 1")
         write(":Manual group: borg backup tool")
         write()
diff --git a/src/borg/archiver/check_cmd.py b/src/borg/archiver/check_cmd.py
index e63e8652..52563fc5 100644
--- a/src/borg/archiver/check_cmd.py
+++ b/src/borg/archiver/check_cmd.py
@@ -71,78 +71,118 @@ def build_parser_check(self, subparsers, common_parser, mid_common_parser):
 
         check_epilog = process_epilog(
             """
-        The check command verifies the consistency of a repository and the corresponding archives.
+        The check command verifies the consistency of a repository and its archives.
+        It consists of two major steps:
 
-        check --repair is a potentially dangerous function and might lead to data loss
-        (for kinds of corruption it is not capable of dealing with). BE VERY CAREFUL!
+        1. Checking the consistency of the repository itself. This includes checking
+           the segment magic headers, and both the metadata and data of all objects in
+           the segments. The read data is checked by size and CRC. Bit rot and other
+           types of accidental damage can be detected this way. Running the repository
+           check can be split into multiple partial checks using ``--max-duration``.
+           When checking a remote repository, please note that the checks run on the
+           server and do not cause significant network traffic.
+
+        2. Checking consistency and correctness of the archive metadata and optionally
+           archive data (requires ``--verify-data``). This includes ensuring that the
+           repository manifest exists, the archive metadata chunk is present, and that
+           all chunks referencing files (items) in the archive exist. This requires
+           reading archive and file metadata, but not data. To cryptographically verify
+           the file (content) data integrity pass ``--verify-data``, but keep in mind
+           that this requires reading all data and is hence very time consuming. When
+           checking archives of a remote repository, archive checks run on the client
+           machine because they require decrypting data and therefore the encryption
+           key.
+
+        Both steps can also be run independently. Pass ``--repository-only`` to run the
+        repository checks only, or pass ``--archives-only`` to run the archive checks
+        only.
+
+        The ``--max-duration`` option can be used to split a long-running repository
+        check into multiple partial checks. After the given number of seconds the check
+        is interrupted. The next partial check will continue where the previous one
+        stopped, until the full repository has been checked. Assuming a complete check
+        would take 7 hours, then running a daily check with ``--max-duration=3600``
+        (1 hour) would result in one full repository check per week. Doing a full
+        repository check aborts any previous partial check; the next partial check will
+        restart from the beginning. With partial repository checks you can run neither
+        archive checks, nor enable repair mode. Consequently, if you want to use
+        ``--max-duration`` you must also pass ``--repository-only``, and must not pass
+        ``--archives-only``, nor ``--repair``.
+
+        **Warning:** Please note that partial repository checks (i.e. running it with
+        ``--max-duration``) can only perform non-cryptographic checksum checks on the
+        segment files. A full repository check (i.e. without ``--max-duration``) can
+        also do a repository index check. Enabling partial repository checks excepts
+        archive checks for the same reason. Therefore partial checks may be useful with
+        very large repositories only where a full check would take too long.
+
+        The ``--verify-data`` option will perform a full integrity verification (as
+        opposed to checking the CRC32 of the segment) of data, which means reading the
+        data from the repository, decrypting and decompressing it. It is a complete
+        cryptographic verification and hence very time consuming, but will detect any
+        accidental and malicious corruption. Tamper-resistance is only guaranteed for
+        encrypted repositories against attackers without access to the keys. You can
+        not use ``--verify-data`` with ``--repository-only``.
+
+        About repair mode
+        +++++++++++++++++
+
+        The check command is a readonly task by default. If any corruption is found,
+        Borg will report the issue and proceed with checking. To actually repair the
+        issues found, pass ``--repair``.
+
+        .. note::
+
+            ``--repair`` is a **POTENTIALLY DANGEROUS FEATURE** and might lead to data
+            loss! This does not just include data that was previously lost anyway, but
+            might include more data for kinds of corruption it is not capable of
+            dealing with. **BE VERY CAREFUL!**
 
         Pursuant to the previous warning it is also highly recommended to test the
-        reliability of the hardware running this software with stress testing software
-        such as memory testers. Unreliable hardware can also lead to data loss especially
-        when this command is run in repair mode.
-
-        First, the underlying repository data files are checked:
-
-        - For all segments, the segment magic header is checked.
-        - For all objects stored in the segments, all metadata (e.g. CRC and size) and
-          all data is read. The read data is checked by size and CRC. Bit rot and other
-          types of accidental damage can be detected this way.
-        - In repair mode, if an integrity error is detected in a segment, try to recover
-          as many objects from the segment as possible.
-        - In repair mode, make sure that the index is consistent with the data stored in
-          the segments.
-        - If checking a remote repo via ``ssh:``, the repo check is executed on the server
-          without causing significant network traffic.
-        - The repository check can be skipped using the ``--archives-only`` option.
-        - A repository check can be time consuming. Partial checks are possible with the
-          ``--max-duration`` option.
-
-        Second, the consistency and correctness of the archive metadata is verified:
-
-        - Is the repo manifest present? If not, it is rebuilt from archive metadata
-          chunks (this requires reading and decrypting of all metadata and data).
-        - Check if archive metadata chunk is present; if not, remove archive from manifest.
-        - For all files (items) in the archive, for all chunks referenced by these
-          files, check if chunk is present. In repair mode, if a chunk is not present,
-          replace it with a same-size replacement chunk of zeroes. If a previously lost
-          chunk reappears (e.g. via a later backup), in repair mode the all-zero replacement
-          chunk will be replaced by the correct chunk. This requires reading of archive and
-          file metadata, but not data.
-        - In repair mode, when all the archives were checked, orphaned chunks are deleted
-          from the repo. One cause of orphaned chunks are input file related errors (like
-          read errors) in the archive creation process.
-        - In verify-data mode, a complete cryptographic verification of the archive data
-          integrity is performed. This conflicts with ``--repository-only`` as this mode
-          only makes sense if the archive checks are enabled. The full details of this mode
-          are documented below.
-        - If checking a remote repo via ``ssh:``, the archive check is executed on the
-          client machine because it requires decryption, and this is always done client-side
-          as key access is needed.
-        - The archive checks can be time consuming; they can be skipped using the
-          ``--repository-only`` option.
-
-        The ``--max-duration`` option can be used to split a long-running repository check
-        into multiple partial checks. After the given number of seconds the check is
-        interrupted. The next partial check will continue where the previous one stopped,
-        until the complete repository has been checked. Example: Assuming a complete check took 7
-        hours, then running a daily check with --max-duration=3600 (1 hour) resulted in one
-        completed check per week.
-
-        Attention: A partial --repository-only check can only do way less checking than a full
-        --repository-only check: only the non-cryptographic checksum checks on segment file
-        entries are done, while a full --repository-only check would also do a repo index check.
-        A partial check cannot be combined with the ``--repair`` option. Partial checks
-        may therefore be useful only with very large repositories where a full check would take
-        too long.
-        Doing a full repository check aborts a partial check; the next partial check will restart
-        from the beginning.
-
-        The ``--verify-data`` option will perform a full integrity verification (as opposed to
-        checking the CRC32 of the segment) of data, which means reading the data from the
-        repository, decrypting and decompressing it. This is a cryptographic verification,
-        which will detect (accidental) corruption. For encrypted repositories it is
-        tamper-resistant as well, unless the attacker has access to the keys. It is also very
-        slow.
+        reliability of the hardware running Borg with stress testing software. This
+        especially includes storage and memory testers. Unreliable hardware might lead
+        to additional data loss.
+
+        It is highly recommended to create a backup of your repository before running
+        in repair mode (i.e. running it with ``--repair``).
+
+        Repair mode will attempt to fix any corruptions found. Fixing corruptions does
+        not mean recovering lost data: Borg can not magically restore data lost due to
+        e.g. a hardware failure. Repairing a repository means sacrificing some data
+        for the sake of the repository as a whole and the remaining data. Hence it is,
+        by definition, a potentially lossy task.
+
+        In practice, repair mode hooks into both the repository and archive checks:
+
+        1. When checking the repository's consistency, repair mode will try to recover
+           as many objects from segments with integrity errors as possible, and ensure
+           that the index is consistent with the data stored in the segments.
+
+        2. When checking the consistency and correctness of archives, repair mode might
+           remove whole archives from the manifest if their archive metadata chunk is
+           corrupt or lost. On a chunk level (i.e. the contents of files), repair mode
+           will replace corrupt or lost chunks with a same-size replacement chunk of
+           zeroes. If a previously zeroed chunk reappears, repair mode will restore
+           this lost chunk using the new chunk. Lastly, repair mode will also delete
+           orphaned chunks (e.g. caused by read errors while creating the archive).
+
+        Most steps taken by repair mode have a one-time effect on the repository, like
+        removing a lost archive from the repository. However, replacing a corrupt or
+        lost chunk with an all-zero replacement will have an ongoing effect on the
+        repository: When attempting to extract a file referencing an all-zero chunk,
+        the ``extract`` command will distinctly warn about it. The FUSE filesystem
+        created by the ``mount`` command will reject reading such a "zero-patched"
+        file unless a special mount option is given.
+
+        As mentioned earlier, Borg might be able to "heal" a "zero-patched" file in
+        repair mode, if all its previously lost chunks reappear (e.g. via a later
+        backup). This is achieved by Borg not only keeping track of the all-zero
+        replacement chunks, but also by keeping metadata about the lost chunks. In
+        repair mode Borg will check whether a previously lost chunk reappeared and will
+        replace the all-zero replacement chunk by the reappeared chunk. If all lost
+        chunks of a "zero-patched" file reappear, this effectively "heals" the file.
+        Consequently, if lost chunks were repaired earlier, it is advised to run
+        ``--repair`` a second time after creating some new backups.
         """
         )
         subparser = subparsers.add_parser(
diff --git a/src/borg/helpers/fs.py b/src/borg/helpers/fs.py
index 14af5893..68f0cd69 100644
--- a/src/borg/helpers/fs.py
+++ b/src/borg/helpers/fs.py
@@ -1,7 +1,7 @@
 import errno
 import hashlib
 import os
-import posixpath
+import os.path
 import re
 import stat
 import subprocess
@@ -231,7 +231,7 @@ def make_path_safe(path):
     path = path.lstrip("/")
     if path.startswith("../") or "/../" in path or path.endswith("/..") or path == "..":
         raise ValueError(f"unexpected '..' element in path {path!r}")
-    path = posixpath.normpath(path)
+    path = os.path.normpath(path)
     return path
 
 
@@ -245,11 +245,6 @@ def remove_dotdot_prefixes(path):
 
     `path` is expected to be normalized already (e.g. via `os.path.normpath()`).
     """
-    if is_win32:
-        if len(path) > 1 and path[1] == ":":
-            path = path.replace(":", "", 1)
-        path = path.replace("\\", "/")
-
     path = path.lstrip("/")
     path = _dotdot_re.sub("", path)
     if path in ["", ".."]:
diff --git a/src/borg/item.pyx b/src/borg/item.pyx
index 5da91c2e..fac6ea49 100644
--- a/src/borg/item.pyx
+++ b/src/borg/item.pyx
@@ -562,7 +562,7 @@ cdef class ManifestItem(PropDict):
     archives = PropDictProperty(dict, 'dict of str -> dict')  # name -> dict
     timestamp = PropDictProperty(str)
     config = PropDictProperty(dict)
-    item_keys = PropDictProperty(tuple, 'tuple of str')  # legacy. new location is inside config.
+    item_keys = PropDictProperty(tuple, 'tuple of str')
 
     def update_internal(self, d):
         # legacy support for migration (data from old msgpacks comes in as bytes always, but sometimes we want str)
@@ -650,7 +650,7 @@ class ItemDiff:
         self._can_compare_chunk_ids = can_compare_chunk_ids
         self._chunk_1 = chunk_1
         self._chunk_2 = chunk_2
-
+        
         self._changes = {}
 
         if self._item1.is_link() or self._item2.is_link():
diff --git a/src/borg/manifest.py b/src/borg/manifest.py
index 189caf6a..c6409f4b 100644
--- a/src/borg/manifest.py
+++ b/src/borg/manifest.py
@@ -264,9 +264,7 @@ def load(cls, repository, operations, key=None, force_tam_not_required=False, *,
         manifest.timestamp = m.get("timestamp")
         manifest.config = m.config
         # valid item keys are whatever is known in the repo or every key we know
-        manifest.item_keys = ITEM_KEYS
-        manifest.item_keys |= frozenset(m.config.get("item_keys", []))  # new location of item_keys since borg2
-        manifest.item_keys |= frozenset(m.get("item_keys", []))  # legacy: borg 1.x: item_keys not in config yet
+        manifest.item_keys = ITEM_KEYS | frozenset(m.get("item_keys", []))
 
         if manifest.tam_verified:
             manifest_required = manifest.config.get("tam_required", False)
@@ -323,12 +321,12 @@ def write(self):
         assert len(self.archives) <= MAX_ARCHIVES
         assert all(len(name) <= 255 for name in self.archives)
         assert len(self.item_keys) <= 100
-        self.config["item_keys"] = tuple(sorted(self.item_keys))
         manifest = ManifestItem(
-            version=2,
+            version=1,
             archives=StableDict(self.archives.get_raw_dict()),
             timestamp=self.timestamp,
             config=StableDict(self.config),
+            item_keys=tuple(sorted(self.item_keys)),
         )
         self.tam_verified = True
         data = self.key.pack_and_authenticate_metadata(manifest.as_dict())
diff --git a/src/borg/xattr.py b/src/borg/xattr.py
index b914c504..f3bd2761 100644
--- a/src/borg/xattr.py
+++ b/src/borg/xattr.py
@@ -1,4 +1,4 @@
-"""A basic extended attributes (xattr) implementation for Linux, FreeBSD and macOS."""
+"""A basic extended attributes (xattr) implementation for Linux, FreeBSD and MacOS X."""
 
 import errno
 import os


Test patch:
diff --git a/src/borg/testsuite/archiver/create_cmd.py b/src/borg/testsuite/archiver/create_cmd.py
index bf9f9c71..cd80ea48 100644
--- a/src/borg/testsuite/archiver/create_cmd.py
+++ b/src/borg/testsuite/archiver/create_cmd.py
@@ -103,14 +103,15 @@ def filter(output):
         # the interesting parts of info_output2 and info_output should be same
         self.assert_equal(filter(info_output), filter(info_output2))
 
+    @pytest.mark.skipif(is_win32, reason="still broken on windows")
     def test_archived_paths(self):
         # As borg comes from the POSIX (Linux, UNIX) world, a lot of stuff assumes path separators
         # to be slashes "/", e.g.: in archived items, for pattern matching.
         # To make our lives easier and to support cross-platform extraction we always use slashes.
         # Similarly, archived paths are expected to be full, but relative (have no leading slash).
         full_path = os.path.abspath(os.path.join(self.input_path, "test"))
-        # remove colon from windows drive letter, if any:
-        posix_path = full_path.replace(":", "") if full_path[1] == ":" else full_path
+        # remove windows drive letter, if any:
+        posix_path = full_path[2:] if full_path[1] == ":" else full_path
         # only needed on windows in case there are backslashes:
         posix_path = posix_path.replace("\\", "/")
         # no leading slash in borg archives:
diff --git a/src/borg/testsuite/archiver/debug_cmds.py b/src/borg/testsuite/archiver/debug_cmds.py
index 6f7c251e..ee22da55 100644
--- a/src/borg/testsuite/archiver/debug_cmds.py
+++ b/src/borg/testsuite/archiver/debug_cmds.py
@@ -133,8 +133,7 @@ def test_debug_dump_manifest(self):
             result = json.load(f)
         assert "archives" in result
         assert "config" in result
-        assert "item_keys" in result["config"]
-        assert frozenset(result["config"]["item_keys"]) == ITEM_KEYS
+        assert "item_keys" in result
         assert "timestamp" in result
         assert "version" in result
 
diff --git a/src/borg/testsuite/helpers.py b/src/borg/testsuite/helpers.py
index ff2cc630..0f96ad6c 100644
--- a/src/borg/testsuite/helpers.py
+++ b/src/borg/testsuite/helpers.py
@@ -800,12 +800,7 @@ def test_get_runtime_dir(monkeypatch):
     else:
         monkeypatch.delenv("XDG_RUNTIME_DIR", raising=False)
         monkeypatch.delenv("BORG_RUNTIME_DIR", raising=False)
-        uid = str(os.getuid())
-        assert get_runtime_dir() in [
-            os.path.join("/run/user", uid, "borg"),
-            os.path.join("/var/run/user", uid, "borg"),
-            os.path.join(f"/tmp/runtime-{uid}", "borg"),
-        ]
+        assert get_runtime_dir() == os.path.join("/run/user", str(os.getuid()), "borg")
         monkeypatch.setenv("XDG_RUNTIME_DIR", "/var/tmp/.cache")
         assert get_runtime_dir() == os.path.join("/var/tmp/.cache", "borg")
         monkeypatch.setenv("BORG_RUNTIME_DIR", "/var/tmp")
@@ -1207,11 +1202,6 @@ def test_swidth_slice_mixed_characters():
     assert swidth_slice(string, 6) == "나윤a"
 
 
-def utcfromtimestamp(timestamp):
-    """Returns a naive datetime instance representing the timestamp in the UTC timezone"""
-    return datetime.fromtimestamp(timestamp, timezone.utc).replace(tzinfo=None)
-
-
 def test_safe_timestamps():
     if SUPPORT_32BIT_PLATFORMS:
         # ns fit into int64
@@ -1223,9 +1213,9 @@ def test_safe_timestamps():
         # datetime won't fall over its y10k problem
         beyond_y10k = 2**100
         with pytest.raises(OverflowError):
-            utcfromtimestamp(beyond_y10k)
-        assert utcfromtimestamp(safe_s(beyond_y10k)) > datetime(2038, 1, 1)
-        assert utcfromtimestamp(safe_ns(beyond_y10k) / 1000000000) > datetime(2038, 1, 1)
+            datetime.utcfromtimestamp(beyond_y10k)
+        assert datetime.utcfromtimestamp(safe_s(beyond_y10k)) > datetime(2038, 1, 1)
+        assert datetime.utcfromtimestamp(safe_ns(beyond_y10k) / 1000000000) > datetime(2038, 1, 1)
     else:
         # ns fit into int64
         assert safe_ns(2**64) <= 2**63 - 1
@@ -1236,9 +1226,9 @@ def test_safe_timestamps():
         # datetime won't fall over its y10k problem
         beyond_y10k = 2**100
         with pytest.raises(OverflowError):
-            utcfromtimestamp(beyond_y10k)
-        assert utcfromtimestamp(safe_s(beyond_y10k)) > datetime(2262, 1, 1)
-        assert utcfromtimestamp(safe_ns(beyond_y10k) / 1000000000) > datetime(2262, 1, 1)
+            datetime.utcfromtimestamp(beyond_y10k)
+        assert datetime.utcfromtimestamp(safe_s(beyond_y10k)) > datetime(2262, 1, 1)
+        assert datetime.utcfromtimestamp(safe_ns(beyond_y10k) / 1000000000) > datetime(2262, 1, 1)
 
 
 class TestPopenWithErrorHandling:
diff --git a/src/borg/testsuite/platform.py b/src/borg/testsuite/platform.py
index 598a8b5b..1f03c9a8 100644
--- a/src/borg/testsuite/platform.py
+++ b/src/borg/testsuite/platform.py
@@ -161,7 +161,7 @@ def test_utils(self):
         self.assert_equal(acl_use_local_uid_gid(b"group:root:rw-:0"), b"group:0:rw-")
 
 
-@unittest.skipUnless(sys.platform.startswith("darwin"), "macOS only test")
+@unittest.skipUnless(sys.platform.startswith("darwin"), "OS X only test")
 @unittest.skipIf(fakeroot_detected(), "not compatible with fakeroot")
 class PlatformDarwinTestCase(BaseTestCase):
     def setUp(self):