internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: borgbackup/borg

Issue:
crc32_slice_by_8 alignment issue on armhf
seen in ubuntu bionic testing by @LocutusOfBorg, reproduced by Elladan (IRC):

```
Nov 11 19:17:36 pumpkin kernel: [25372.479551] Alignment trap: python3.6 (9704)
                 PC=0x75427c78 Instr=0xe955b402 Address=0x76f65039 FSR 0x001
```
Alignment trap confirmed :-)

Bus error raised by:
```
#4  crc32_slice_by_8
    (data=0x76fa2039, length=<optimized out>, previousCrc32=<optimized out>)
    at src/borg/algorithms/crc32_slice_by_8.c:358
#5  0x76462e4e in __pyx_pf_4borg_10algorithms_9checksums_crc32_slice_by_8
    (__pyx_self=<optimized out>, __pyx_v_value=<optimized out>, __pyx_v_data=<optimized out>)
    at src/borg/algorithms/checksums.c:1341
```
Bug was no repro with armhf xenial w/Python 3.5, reprod with bionic w/Python 3.6.

Legible stack trace here: http://rn0.me/show/8jem6kLzNkvHWldCYfzq/



Discussion:
See the `data=<unaligned_address>` in `#4`.

ARM does not like it when memory accesses are unaligned.

Depending on the CPU configuration (`/proc/cpu/alignment`) and ARM generation, it might crash, fixup the issue in software, or just compute with wrong values.

The problem is that the crc-slice-by-8 code starts fetching 32 bit integers right from the buffer start address. It has some code for tail management, processing single bytes remaining at the end of the buffer.
Some head management code needs to get added there to process singles bytes at the start of the buffer until some divisible-by-4 address is reached.


Resolving patch:
diff --git a/src/borg/testsuite/checksums.py b/src/borg/testsuite/checksums.py
--- a/src/borg/testsuite/checksums.py
+++ b/src/borg/testsuite/checksums.py
@@ -14,13 +14,18 @@
 
 @pytest.mark.parametrize('implementation', crc32_implementations)
 def test_crc32(implementation):
-    # This includes many critical values, like zero length, 3/4/5, 6/7/8 and so on which are near and on
-    # alignment boundaries. This is of course just a sanity check ie. "did it compile all right?".
-    data = os.urandom(256)
+    # This includes many critical values, like misc. length and misc. aligned start addresses.
+    data = os.urandom(300)
+    mv = memoryview(data)
     initial_crc = 0x12345678
-    for i in range(0, 256):
-        d = data[:i]
-        assert zlib.crc32(d, initial_crc) == implementation(d, initial_crc)
+    for start in range(0, 4):  # 4B / int32 alignment, head processing
+        for length in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10,
+                       31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41,
+                       63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73,
+                       127, 128, 129, 130, 131, 132, 133, 134, 135,
+                       255, 256, 257, ]:
+            d = mv[start:start+length]
+            assert zlib.crc32(d, initial_crc) == implementation(d, initial_crc)
 
 
 def test_xxh64():


Test patch:
Repository: borgbackup/borg

Issue:
improve borg list --format doc (alias attributes)
https://borgbackup.readthedocs.io/en/stable/usage/list.html name and archive have the same description.

if they are the same you may want to refer one of them as an alias of the other?


Discussion:
About telling that one is the alias of the other: yes, we could do that. Also applies to some other value descriptions there.

Resolving patch:
diff --git a/src/borg/helpers/parseformat.py b/src/borg/helpers/parseformat.py
--- a/src/borg/helpers/parseformat.py
+++ b/src/borg/helpers/parseformat.py
@@ -500,20 +500,20 @@ def keys_help():
 
 class ArchiveFormatter(BaseFormatter):
     KEY_DESCRIPTIONS = {
-        'name': 'archive name interpreted as text (might be missing non-text characters, see barchive)',
         'archive': 'archive name interpreted as text (might be missing non-text characters, see barchive)',
+        'name': 'alias of "archive"',
         'barchive': 'verbatim archive name, can contain any character except NUL',
         'comment': 'archive comment interpreted as text (might be missing non-text characters, see bcomment)',
         'bcomment': 'verbatim archive comment, can contain any character except NUL',
-        'time': 'time (start) of creation of the archive',
         # *start* is the key used by borg-info for this timestamp, this makes the formats more compatible
         'start': 'time (start) of creation of the archive',
+        'time': 'alias of "start"',
         'end': 'time (end) of creation of the archive',
         'id': 'internal ID of the archive',
     }
     KEY_GROUPS = (
-        ('name', 'archive', 'barchive', 'comment', 'bcomment', 'id'),
-        ('time', 'start', 'end'),
+        ('archive', 'name', 'barchive', 'comment', 'bcomment', 'id'),
+        ('start', 'time', 'end'),
     )
 
     @classmethod


Test patch:
Repository: borgbackup/borg

Issue:
improve borg list --format doc (alias attributes)
https://borgbackup.readthedocs.io/en/stable/usage/list.html name and archive have the same description.

if they are the same you may want to refer one of them as an alias of the other?


Discussion:
About telling that one is the alias of the other: yes, we could do that. Also applies to some other value descriptions there.

Resolving patch:
diff --git a/src/borg/helpers.py b/src/borg/helpers.py
--- a/src/borg/helpers.py
+++ b/src/borg/helpers.py
@@ -1647,20 +1647,20 @@ def keys_help():
 
 class ArchiveFormatter(BaseFormatter):
     KEY_DESCRIPTIONS = {
-        'name': 'archive name interpreted as text (might be missing non-text characters, see barchive)',
         'archive': 'archive name interpreted as text (might be missing non-text characters, see barchive)',
+        'name': 'alias of "archive"',
         'barchive': 'verbatim archive name, can contain any character except NUL',
         'comment': 'archive comment interpreted as text (might be missing non-text characters, see bcomment)',
         'bcomment': 'verbatim archive comment, can contain any character except NUL',
-        'time': 'time (start) of creation of the archive',
         # *start* is the key used by borg-info for this timestamp, this makes the formats more compatible
         'start': 'time (start) of creation of the archive',
+        'time': 'alias of "start"',
         'end': 'time (end) of creation of the archive',
         'id': 'internal ID of the archive',
     }
     KEY_GROUPS = (
-        ('name', 'archive', 'barchive', 'comment', 'bcomment', 'id'),
-        ('time', 'start', 'end'),
+        ('archive', 'name', 'barchive', 'comment', 'bcomment', 'id'),
+        ('start', 'time', 'end'),
     )
 
     @classmethod


Test patch:
Repository: borgbackup/borg

Issue:
borg borgfs?
http://borgbackup.readthedocs.io/en/stable/usage/borgfs.html

- strange headline
- strangeusage

```
borg borgfs

borg [common options] borgfs [options] REPOSITORY_OR_ARCHIVE MOUNTPOINT [PATH...]
```

borg borgfs?
http://borgbackup.readthedocs.io/en/stable/usage/borgfs.html

- strange headline
- strangeusage

```
borg borgfs

borg [common options] borgfs [options] REPOSITORY_OR_ARCHIVE MOUNTPOINT [PATH...]
```



Discussion:
Hmm, I thought I explicitly prevented that in my borgfs PR. I'll take another look...
The page is not listed in the TOC, but does show up in a search, which I didn't account for. I thought borgfs didn't need its own docs page when `borg mount` is the same thing, but I didn't actually prevent the page from being generated.
Hmm, I thought I explicitly prevented that in my borgfs PR. I'll take another look...
The page is not listed in the TOC, but does show up in a search, which I didn't account for. I thought borgfs didn't need its own docs page when `borg mount` is the same thing, but I didn't actually prevent the page from being generated.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -242,9 +242,11 @@ def run(self):
         # allows us to build docs without the C modules fully loaded during help generation
         from borg.archiver import Archiver
         parser = Archiver(prog='borg').build_parser()
-        borgfs_parser = Archiver(prog='borgfs').build_parser()
+        # borgfs has a separate man page to satisfy debian's "every program from a package
+        # must have a man page" requirement, but it doesn't need a separate HTML docs page
+        #borgfs_parser = Archiver(prog='borgfs').build_parser()
 
-        self.generate_level("", parser, Archiver, {'borgfs': borgfs_parser})
+        self.generate_level("", parser, Archiver)
 
     def generate_level(self, prefix, parser, Archiver, extra_choices=None):
         is_subcommand = False


Test patch:
Repository: borgbackup/borg

Issue:
borg borgfs?
http://borgbackup.readthedocs.io/en/stable/usage/borgfs.html

- strange headline
- strangeusage

```
borg borgfs

borg [common options] borgfs [options] REPOSITORY_OR_ARCHIVE MOUNTPOINT [PATH...]
```



Discussion:
Hmm, I thought I explicitly prevented that in my borgfs PR. I'll take another look...
The page is not listed in the TOC, but does show up in a search, which I didn't account for. I thought borgfs didn't need its own docs page when `borg mount` is the same thing, but I didn't actually prevent the page from being generated.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -238,9 +238,11 @@ def run(self):
         # allows us to build docs without the C modules fully loaded during help generation
         from borg.archiver import Archiver
         parser = Archiver(prog='borg').build_parser()
-        borgfs_parser = Archiver(prog='borgfs').build_parser()
+        # borgfs has a separate man page to satisfy debian's "every program from a package
+        # must have a man page" requirement, but it doesn't need a separate HTML docs page
+        #borgfs_parser = Archiver(prog='borgfs').build_parser()
 
-        self.generate_level("", parser, Archiver, {'borgfs': borgfs_parser})
+        self.generate_level("", parser, Archiver)
 
     def generate_level(self, prefix, parser, Archiver, extra_choices=None):
         is_subcommand = False


Test patch:
Repository: borgbackup/borg

Issue:
check docs about key export
i remember multiple people asking about whether the data exported by `borg key export` contains the key in encrypted or unencrypted form.

this should be in the docs, also some related statement about the passphrase.


Discussion:

Resolving patch:
diff --git a/borg/archiver.py b/borg/archiver.py
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -1285,6 +1285,9 @@ def build_parser(self, args=None, prog=None):
         key_export_epilog = textwrap.dedent("""
         If repository encryption is used, the repository is inaccessible
         without the key. This command allows to backup this essential key.
+        Note that the backup produced does not include the passphrase itself
+        (i.e. the exported key stays encrypted). In order to regain access to a
+        repository, one needs both the exported key and the original passphrase.
 
         There are two backup formats. The normal backup format is suitable for
         digital storage as a file. The ``--paper`` backup format is optimized


Test patch:
Repository: borgbackup/borg

Issue:
remaining build_man issue
```
(borg-env) tw@tux:~/w/borg$ python setup.py build_man
running build_man
...
building man page borg-prune(1)
docs/virtmanpage.rst:145: (ERROR/3) Unknown directive type "highlight".
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. highlight:: none
```

The problem is that this will put some ugly error text into the resulting man page also.

The fix likely is not just removing the highlight directive as we likely need it for the html docs.

---

:moneybag: [there is a bounty for this](https://www.bountysource.com/issues/52118992-remaining-build_man-issue)


Discussion:
See #3364 for context

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -655,6 +655,8 @@ def write_examples(self, write, command):
             examples = examples.replace(usage_include, '')
             examples = examples.replace('Examples\n~~~~~~~~', '')
             examples = examples.replace('Miscellaneous Help\n------------------', '')
+            examples = examples.replace('``docs/misc/prune-example.txt``:', '``docs/misc/prune-example.txt``.')
+            examples = examples.replace('.. highlight:: none\n', '')  # we don't support highlight
             examples = re.sub('^(~+)$', lambda matches: '+' * len(matches.group(0)), examples, flags=re.MULTILINE)
             examples = examples.strip()
         if examples:


Test patch:
Repository: borgbackup/borg

Issue:
borg check --repair runs very slow (and corrupts archives)
from @bebehei on IRC (IRC log shortened):

borg 1.1.3 on client and server, running borg check --repair on a corrupt repository hangs for >5h after emitting a msg about missing metadata.


```
23:11  bebehei$ Now it happened: Oh, the metadata chunk is missing.
23:12  bebehei$ strace output slowed down immediately
23:16  bebehei$ item metadata chunk missing [chunk: 001056_bdee87d...a3e50d]
23:16  bebehei$ But the borg serve process is dead.
23:22  ThomasWaldmann$ archive.py:1497 is the place where that msg gets emitted
```

Bounty: https://www.bountysource.com/issues/52774936-borg-check-repair-hangs-infinitely


Discussion:
The bounty is for the bug being found and fixed.

To prove that it is this bug and it is fixed, maybe a unit test is needed - reproducing the bug first (hanging) and then succeeding after the fix.

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -1457,7 +1457,7 @@ def robust_iterator(archive):
             """
             item_keys = frozenset(key.encode() for key in self.manifest.item_keys)
             required_item_keys = frozenset(key.encode() for key in REQUIRED_ITEM_KEYS)
-            unpacker = RobustUnpacker(lambda item: isinstance(item, dict) and 'path' in item,
+            unpacker = RobustUnpacker(lambda item: isinstance(item, StableDict) and b'path' in item,
                                       self.manifest.item_keys)
             _state = 0
 
diff --git a/src/borg/testsuite/archiver.py b/src/borg/testsuite/archiver.py
--- a/src/borg/testsuite/archiver.py
+++ b/src/borg/testsuite/archiver.py
@@ -3006,7 +3006,7 @@ def test_missing_file_chunk(self):
     def test_missing_archive_item_chunk(self):
         archive, repository = self.open_archive('archive1')
         with repository:
-            repository.delete(archive.metadata.items[-5])
+            repository.delete(archive.metadata.items[0])
             repository.commit()
         self.cmd('check', self.repository_location, exit_code=1)
         self.cmd('check', '--repair', self.repository_location, exit_code=0)


Test patch:
Repository: borgbackup/borg

Issue:
borg check --repair runs very slow (and corrupts archives)
from @bebehei on IRC (IRC log shortened):

borg 1.1.3 on client and server, running borg check --repair on a corrupt repository hangs for >5h after emitting a msg about missing metadata.


```
23:11  bebehei$ Now it happened: Oh, the metadata chunk is missing.
23:12  bebehei$ strace output slowed down immediately
23:16  bebehei$ item metadata chunk missing [chunk: 001056_bdee87d...a3e50d]
23:16  bebehei$ But the borg serve process is dead.
23:22  ThomasWaldmann$ archive.py:1497 is the place where that msg gets emitted
```

Bounty: https://www.bountysource.com/issues/52774936-borg-check-repair-hangs-infinitely


Discussion:
The bounty is for the bug being found and fixed.

To prove that it is this bug and it is fixed, maybe a unit test is needed - reproducing the bug first (hanging) and then succeeding after the fix.

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -1585,7 +1585,7 @@ def robust_iterator(archive):
             """
             item_keys = frozenset(key.encode() for key in self.manifest.item_keys)
             required_item_keys = frozenset(key.encode() for key in REQUIRED_ITEM_KEYS)
-            unpacker = RobustUnpacker(lambda item: isinstance(item, dict) and 'path' in item,
+            unpacker = RobustUnpacker(lambda item: isinstance(item, StableDict) and b'path' in item,
                                       self.manifest.item_keys)
             _state = 0
 
diff --git a/src/borg/testsuite/archiver.py b/src/borg/testsuite/archiver.py
--- a/src/borg/testsuite/archiver.py
+++ b/src/borg/testsuite/archiver.py
@@ -3000,7 +3000,7 @@ def test_missing_file_chunk(self):
     def test_missing_archive_item_chunk(self):
         archive, repository = self.open_archive('archive1')
         with repository:
-            repository.delete(archive.metadata.items[-5])
+            repository.delete(archive.metadata.items[0])
             repository.commit()
         self.cmd('check', self.repository_location, exit_code=1)
         self.cmd('check', '--repair', self.repository_location, exit_code=0)


Test patch:
Repository: borgbackup/borg

Issue:
borg delete <repo> doesn’t delete BORG_SECURITY_DIR/<repodir>
I just started using borgbackup and tested a few things with throwaway repositories. I noticed that deleting a repository (using `borg delete`) does not delete its subdirectory below BORG_SECURITY_DIR. Is that intentional?




Discussion:
No, guess that was forgotten when adding the security dir.

@enkore ^

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -38,7 +38,7 @@
 from .archive import Archive, ArchiveChecker, ArchiveRecreater, Statistics, is_special
 from .archive import BackupOSError, backup_io
 from .archive import FilesystemObjectProcessors, MetadataCollector, ChunksProcessor
-from .cache import Cache, assert_secure
+from .cache import Cache, assert_secure, SecurityManager
 from .constants import *  # NOQA
 from .compress import CompressionSpec
 from .crypto.key import key_creator, key_argument_names, tam_required_file, tam_required, RepoKey, PassphraseKey
@@ -1108,6 +1108,7 @@ def _delete_repository(self, args, repository):
                 return self.exit_code
             repository.destroy()
             logger.info("Repository deleted.")
+            SecurityManager.destroy(repository)
         Cache.destroy(repository)
         logger.info("Cache deleted.")
         return self.exit_code
diff --git a/src/borg/cache.py b/src/borg/cache.py
--- a/src/borg/cache.py
+++ b/src/borg/cache.py
@@ -66,6 +66,13 @@ def __init__(self, repository):
         self.location_file = os.path.join(self.dir, 'location')
         self.manifest_ts_file = os.path.join(self.dir, 'manifest-timestamp')
 
+    @staticmethod
+    def destroy(repository, path=None):
+        """destroy the security dir for ``repository`` or at ``path``"""
+        path = path or get_security_dir(repository.id_str)
+        if os.path.exists(path):
+            shutil.rmtree(path)
+
     def known(self):
         return os.path.exists(self.key_type_file)
 


Test patch:
Repository: borgbackup/borg

Issue:
borg delete <repo> doesn’t delete BORG_SECURITY_DIR/<repodir>
I just started using borgbackup and tested a few things with throwaway repositories. I noticed that deleting a repository (using `borg delete`) does not delete its subdirectory below BORG_SECURITY_DIR. Is that intentional?




Discussion:
No, guess that was forgotten when adding the security dir.

@enkore ^

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -38,7 +38,7 @@
 from .algorithms.checksums import crc32
 from .archive import Archive, ArchiveChecker, ArchiveRecreater, Statistics, is_special
 from .archive import BackupOSError, backup_io
-from .cache import Cache, assert_secure
+from .cache import Cache, assert_secure, SecurityManager
 from .constants import *  # NOQA
 from .compress import CompressionSpec
 from .crypto.key import key_creator, key_argument_names, tam_required_file, tam_required, RepoKey, PassphraseKey
@@ -1303,6 +1303,7 @@ def _delete_repository(self, args, repository):
                 return self.exit_code
             repository.destroy()
             logger.info("Repository deleted.")
+            SecurityManager.destroy(repository)
         Cache.destroy(repository)
         logger.info("Cache deleted.")
         return self.exit_code
diff --git a/src/borg/cache.py b/src/borg/cache.py
--- a/src/borg/cache.py
+++ b/src/borg/cache.py
@@ -66,6 +66,13 @@ def __init__(self, repository):
         self.location_file = os.path.join(self.dir, 'location')
         self.manifest_ts_file = os.path.join(self.dir, 'manifest-timestamp')
 
+    @staticmethod
+    def destroy(repository, path=None):
+        """destroy the security dir for ``repository`` or at ``path``"""
+        path = path or get_security_dir(repository.id_str)
+        if os.path.exists(path):
+            shutil.rmtree(path)
+
     def known(self):
         return os.path.exists(self.key_type_file)
 


Test patch:
Repository: borgbackup/borg

Issue:
`borg init --storage-quota` not working properly?
I ran `borg init --encryption keyfile --storage-quota 300G --progress --show-version --show-rc /repo`, but the `/repo/config` file has `storage_quota = 0`. Is this expected? If not, what format should the number in the config file be in? (looks like bytes from reading the source, but I'd like to check)


Discussion:
Confirmed, affects at least 1.1.3 and master.
1.1.3, installed from PyPI
Also getting `storage_quota = 0` on 1.1.0 and even 1.1.0b6 (first release with quota support).

With some print() debugging, it appears that the quota is not getting passed to `Repository` – I don't really understand how the flow of the program works, but it looks like the `with_repository` decorator in `archiver.py` doesn't pass any quota when it creates a `Repository`. Could that be (part of) the issue?

I believe this fixes it: add `storage_quota=args.storage_quota` to the call to `Repository()` at around line 133 in `archiver.py` (but I don't know if the remote repository handling needs anything like that). Happy to make a PR if this is the right fix.
@anowlcalledjosh yes, your analysis seems correct. I already started working on it before reading your latest comment, but I'ld appreciate if you review my PR (soon).

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -124,6 +124,7 @@ def decorator(method):
         def wrapper(self, args, **kwargs):
             location = args.location  # note: 'location' must be always present in args
             append_only = getattr(args, 'append_only', False)
+            storage_quota = getattr(args, 'storage_quota', None)
             if argument(args, fake) ^ invert_fake:
                 return method(self, args, repository=None, **kwargs)
             elif location.proto == 'ssh':
@@ -131,8 +132,8 @@ def wrapper(self, args, **kwargs):
                                               lock_wait=self.lock_wait, lock=lock, append_only=append_only, args=args)
             else:
                 repository = Repository(location.path, create=create, exclusive=argument(args, exclusive),
-                                        lock_wait=self.lock_wait, lock=lock,
-                                        append_only=append_only)
+                                        lock_wait=self.lock_wait, lock=lock, append_only=append_only,
+                                        storage_quota=storage_quota)
             with repository:
                 if manifest or cache:
                     kwargs['manifest'], kwargs['key'] = Manifest.load(repository, compatibility)


Test patch: