internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: borgbackup/borg

Issue:
remove support for openssl < 1.1.1
master branch, after 1.2-maint has forked.


Discussion:

Resolving patch:
diff --git a/src/borg/testsuite/crypto.py b/src/borg/testsuite/crypto.py
--- a/src/borg/testsuite/crypto.py
+++ b/src/borg/testsuite/crypto.py
@@ -1,7 +1,7 @@
 from binascii import hexlify, unhexlify
 
 from ..crypto.low_level import AES256_CTR_HMAC_SHA256, AES256_OCB, CHACHA20_POLY1305, UNENCRYPTED, \
-                               IntegrityError, blake2b_128, blake2b_256, hmac_sha256, openssl10
+                               IntegrityError, blake2b_128, blake2b_256, hmac_sha256, is_libressl
 from ..crypto.low_level import bytes_to_long, bytes_to_int, long_to_bytes
 from ..crypto.low_level import hkdf_hmac_sha512
 
@@ -98,15 +98,15 @@ def test_AE(self):
         header = b'\x23'
         tests = [
             # (ciphersuite class, exp_mac, exp_cdata)
+            (CHACHA20_POLY1305,
+             b'fd08594796e0706cde1e8b461e3e0555',
+             b'a093e4b0387526f085d3c40cca84a35230a5c0dd766453b77ba38bcff775',)
         ]
-        if not openssl10:
+        if not is_libressl:
             tests += [
                 (AES256_OCB,
                  b'b6909c23c9aaebd9abbe1ff42097652d',
                  b'877ce46d2f62dee54699cebc3ba41d9ab613f7c486778c1b3636664b1493', ),
-                (CHACHA20_POLY1305,
-                 b'fd08594796e0706cde1e8b461e3e0555',
-                 b'a093e4b0387526f085d3c40cca84a35230a5c0dd766453b77ba38bcff775', )
             ]
         for cs_cls, exp_mac, exp_cdata in tests:
             # print(repr(cs_cls))
@@ -142,15 +142,15 @@ def test_AEAD(self):
         header = b'\x12\x34\x56'
         tests = [
             # (ciphersuite class, exp_mac, exp_cdata)
+            (CHACHA20_POLY1305,
+             b'b7e7c9a79f2404e14f9aad156bf091dd',
+             b'a093e4b0387526f085d3c40cca84a35230a5c0dd766453b77ba38bcff775',)
         ]
-        if not openssl10:
+        if not is_libressl:
             tests += [
                 (AES256_OCB,
                  b'f2748c412af1c7ead81863a18c2c1893',
                  b'877ce46d2f62dee54699cebc3ba41d9ab613f7c486778c1b3636664b1493', ),
-                (CHACHA20_POLY1305,
-                 b'b7e7c9a79f2404e14f9aad156bf091dd',
-                 b'a093e4b0387526f085d3c40cca84a35230a5c0dd766453b77ba38bcff775', )
             ]
         for cs_cls, exp_mac, exp_cdata in tests:
             # print(repr(cs_cls))


Test patch:
Repository: borgbackup/borg

Issue:
require python >= 3.9
borg 1.2 requires python >= 3.8.

so, for helium milestone (borg 1.3?), we can increase requirements considering the development will take a while and stable dists will have catched up by whenever it will get released and accepted into stable dists.

https://docs.python.org/3/whatsnew/3.9.html



Discussion:
Just wanted to mention that the upcoming RHEL 9 will have Python 3.9 so please keep in mind that by dropping 3.9 you'll cut off RHEL 9 (unless users use the pyinstaller-based binaries).
The plan here is to support everything >= 3.9, so if RHEL9 has 3.9, that's fine.
Note: Ubuntu 22.04 LTS also has py39.
Yeah, just wanted to mention the RHEL version early on as they have pretty long life cycle and I can not spend as much time as I'd like to with borg so I'm noticing new limitations only very late in the cycle :-)

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -260,7 +260,6 @@ def members_appended(*ds):
         'Operating System :: POSIX :: Linux',
         'Programming Language :: Python',
         'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.8',
         'Programming Language :: Python :: 3.9',
         'Programming Language :: Python :: 3.10',
         'Topic :: Security :: Cryptography',
@@ -287,5 +286,5 @@ def members_appended(*ds):
     setup_requires=['setuptools_scm>=1.7'],
     install_requires=install_requires,
     extras_require=extras_require,
-    python_requires='>=3.8',
+    python_requires='>=3.9',
 )
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -2457,11 +2457,11 @@ def do_break_lock(self, args, repository):
 
         {now}
             The current local date and time, by default in ISO-8601 format.
-            You can also supply your own `format string <https://docs.python.org/3.8/library/datetime.html#strftime-and-strptime-behavior>`_, e.g. {now:%Y-%m-%d_%H:%M:%S}
+            You can also supply your own `format string <https://docs.python.org/3.9/library/datetime.html#strftime-and-strptime-behavior>`_, e.g. {now:%Y-%m-%d_%H:%M:%S}
 
         {utcnow}
             The current UTC date and time, by default in ISO-8601 format.
-            You can also supply your own `format string <https://docs.python.org/3.8/library/datetime.html#strftime-and-strptime-behavior>`_, e.g. {utcnow:%Y-%m-%d_%H:%M:%S}
+            You can also supply your own `format string <https://docs.python.org/3.9/library/datetime.html#strftime-and-strptime-behavior>`_, e.g. {utcnow:%Y-%m-%d_%H:%M:%S}
 
         {user}
             The user name (or UID, if no name is available) of the user running borg.
@@ -4300,7 +4300,7 @@ def define_borg_mount(parser):
         +++++++++++++++++++++++++++
 
         The ``--format`` option uses python's `format string syntax
-        <https://docs.python.org/3.8/library/string.html#formatstrings>`_.
+        <https://docs.python.org/3.9/library/string.html#formatstrings>`_.
 
         Examples:
         ::


Test patch:
Repository: borgbackup/borg

Issue:
borg import-tar hangs 
## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

BUG

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).
1.2.0
#### Operating system (distribution) and version.
arch linux (up to date as of 2022-03-04)
#### Hardware / network configuration, and filesystems used.
VM using local files on btrfs fs
#### How much data is handled by borg?
~500MB
#### Full borg commandline that lead to the problem (leave away excludes and passwords)
borg import-tar -p -s path_to_repo::archive_name filename.tar.zstd

## Describe the problem you're observing.
The command was hanging without any output. I *think* it is an error due to the archive already existing but not properly propagating back to user. If I unzstd the file before, and try it with an uncompressed tar file it correctly gives `Archive archive_name already exists`
#### Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.
Try running import-tar with a zstd compressed tar file when the archive_name already exists in the repo


Discussion:

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -1447,7 +1447,7 @@ def __init__(self, *, cache, key,
     def create_helper(self, tarinfo, status=None, type=None):
         item = Item(path=make_path_safe(tarinfo.name), mode=tarinfo.mode | type,
                     uid=tarinfo.uid, gid=tarinfo.gid, user=tarinfo.uname or None, group=tarinfo.gname or None,
-                    mtime=tarinfo.mtime * 1000**3)
+                    mtime=safe_ns(int(tarinfo.mtime * 1000**3)))
         yield item, status
         # if we get here, "with"-block worked ok without error/exception, the item was processed ok...
         self.add_item(item, stats=self.stats)
diff --git a/src/borg/helpers/process.py b/src/borg/helpers/process.py
--- a/src/borg/helpers/process.py
+++ b/src/borg/helpers/process.py
@@ -329,6 +329,14 @@ def create_filter_process(cmd, stream, stream_close, inbound=True):
     try:
         yield stream
 
+    except Exception:
+        # something went wrong with processing the stream by borg
+        logger.debug('Exception, killing the filter...')
+        proc.kill()
+        borg_succeeded = False
+        raise
+    else:
+        borg_succeeded = True
     finally:
         if stream_close:
             stream.close()
@@ -339,5 +347,6 @@ def create_filter_process(cmd, stream, stream_close, inbound=True):
             logger.debug('filter cmd exited with code %d', rc)
             if filter_stream_close:
                 filter_stream.close()
-            if rc:
+            if borg_succeeded and rc:
+                # if borg did not succeed, we know that we killed the filter process
                 raise Error('filter %s failed, rc=%d' % (cmd, rc))


Test patch:
Repository: borgbackup/borg

Issue:
use hmac.compare_digest when decrypting the key
https://github.com/borgbackup/borg/blob/1.2.0/src/borg/crypto/key.py#L636

guess it is not strictly necessary in that context, but maybe raises less questions / eyebrows.

note: key processing will change to encrypt-then-mac / mac-then-decrypt with v2 keys, this is off-topic in THIS ticket.


Discussion:

Resolving patch:
diff --git a/src/borg/crypto/key.py b/src/borg/crypto/key.py
--- a/src/borg/crypto/key.py
+++ b/src/borg/crypto/key.py
@@ -434,7 +434,7 @@ def decrypt_key_file(self, data, passphrase):
         assert enc_key.algorithm == 'sha256'
         key = passphrase.kdf(enc_key.salt, enc_key.iterations, 32)
         data = AES(key, b'\0'*16).decrypt(enc_key.data)
-        if hmac_sha256(key, data) == enc_key.hash:
+        if hmac.compare_digest(hmac_sha256(key, data), enc_key.hash):
             return data
 
     def encrypt_key_file(self, data, passphrase):


Test patch:
Repository: borgbackup/borg

Issue:
use hmac.compare_digest when decrypting the key
https://github.com/borgbackup/borg/blob/1.2.0/src/borg/crypto/key.py#L636

guess it is not strictly necessary in that context, but maybe raises less questions / eyebrows.

note: key processing will change to encrypt-then-mac / mac-then-decrypt with v2 keys, this is off-topic in THIS ticket.


Discussion:

Resolving patch:
diff --git a/src/borg/crypto/key.py b/src/borg/crypto/key.py
--- a/src/borg/crypto/key.py
+++ b/src/borg/crypto/key.py
@@ -633,7 +633,7 @@ def decrypt_key_file(self, data, passphrase):
         assert enc_key.algorithm == 'sha256'
         key = passphrase.kdf(enc_key.salt, enc_key.iterations, 32)
         data = AES(key, b'\0'*16).decrypt(enc_key.data)
-        if hmac_sha256(key, data) == enc_key.hash:
+        if hmac.compare_digest(hmac_sha256(key, data), enc_key.hash):
             return data
 
     def encrypt_key_file(self, data, passphrase):


Test patch:
Repository: borgbackup/borg

Issue:
add kdf benchmarks to borg benchmark cpu
we'll soon have some benchmarks after #6233 is merged.

it already has crypto benchmarks, but no benchmark for the KDF(s) yet.

there's also planned work on the KDFs (see #747), so we can add the benchmarks for that afterwards.


Discussion:

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -611,6 +611,15 @@ def chunkit(chunker_name, *args, **kwargs):
         for spec, func in tests:
             print(f"{spec:<24} {size:<10} {timeit(func, number=100):.3f}s")
 
+        from borg.helpers.passphrase import Passphrase
+        print("KDFs (slow is GOOD, use argon2!) ===============================")
+        count = 5
+        for spec, func in [
+            ("pbkdf2", lambda: Passphrase('mypassphrase').kdf(b'salt'*8, PBKDF2_ITERATIONS, 32)),
+            ("argon2", lambda: Passphrase('mypassphrase').argon2(32, b'salt'*8, 3, 65536, 1, 'id')),
+        ]:
+            print(f"{spec:<24} {count:<10} {timeit(func, number=count):.3f}s")
+
         from borg.compress import CompressionSpec
         print("Compression ====================================================")
         for spec in [


Test patch:
Repository: borgbackup/borg

Issue:
LZ4 micro opt: memoryview -> bytes transform
https://github.com/borgbackup/borg/blob/1.1.3/src/borg/compress.pyx#L135

there it transforms to bytes rather early, before invoking `super().decompress(data)` (which would happily work with memoryviews also). the bytes requirement is only for the lz4 decompressor.

we could save creation/destruction of one (potentially large) bytes object by doing the bytes transform after the super call.


Discussion:
```        # strip ID bytes
return data[2:]``` will copy for bytes.
The point was that data before that operation can be a memoryview, so it does not need to be transformed to a bytes object before that.

Resolving patch:
diff --git a/src/borg/crypto/key.py b/src/borg/crypto/key.py
--- a/src/borg/crypto/key.py
+++ b/src/borg/crypto/key.py
@@ -355,7 +355,7 @@ def decrypt(self, id, data, decompress=True):
             raise IntegrityError(f"Chunk {bin_to_hex(id)}: Could not decrypt [{str(e)}]")
         if not decompress:
             return payload
-        data = self.decompress(payload)
+        data = self.decompress(memoryview(payload))
         self.assert_id(id, data)
         return data
 
@@ -745,7 +745,7 @@ def decrypt(self, id, data, decompress=True):
             raise IntegrityError(f"Chunk {bin_to_hex(id)}: Could not decrypt [{str(e)}]")
         if not decompress:
             return payload
-        data = self.decompress(payload)
+        data = self.decompress(memoryview(payload))
         self.assert_id(id, data)
         return data
 


Test patch:
Repository: borgbackup/borg

Issue:
add pyproject.toml to 1.2-maint branch?
We have it already in master and it seems to solve the build requirements setup (like installing `pkgconfig` before `setup.py` runs).

1.2.0 release did not have `pyproject.toml` yet, shall we add it to `1.2.1`?


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -19,6 +19,7 @@
 except ImportError:
     cythonize = None
 
+sys.path += [os.path.dirname(__file__)]
 import setup_checksums
 import setup_compress
 import setup_crypto


Test patch:
Repository: borgbackup/borg

Issue:
borg compact docs
```
elho$ the borg compact docs should also state whether compact --cleanup-commits discards the
      "empty" commits *only*, or does both a regular compact *and* that cleanup
elho$ and if the former, whether that is excempt from being prevented with append-only in place for
      doing no harm, or not, to be paranoidly safe
```



Discussion:
`borg compact --cleanup-commits` does:
- first: cleanup 17byte commit files that were erroneously created by borg 1.1, see #2850
- second: run compact_segments (same as with `borg compact` without `--cleanup-commits`)

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -3424,7 +3424,8 @@ def define_borg_mount(parser):
 
         After upgrading borg (server) to 1.2+, you can use ``borg compact --cleanup-commits``
         to clean up the numerous 17byte commit-only segments that borg 1.1 did not clean up
-        due to a bug. It is enough to do that once per repository.
+        due to a bug. It is enough to do that once per repository. After cleaning up the
+        commits, borg will also do a normal compaction.
 
         See :ref:`separate_compaction` in Additional Notes for more details.
         """)


Test patch:
Repository: borgbackup/borg

Issue:
borg compact docs
```
elho$ the borg compact docs should also state whether compact --cleanup-commits discards the
      "empty" commits *only*, or does both a regular compact *and* that cleanup
elho$ and if the former, whether that is excempt from being prevented with append-only in place for
      doing no harm, or not, to be paranoidly safe
```



Discussion:
`borg compact --cleanup-commits` does:
- first: cleanup 17byte commit files that were erroneously created by borg 1.1, see #2850
- second: run compact_segments (same as with `borg compact` without `--cleanup-commits`)

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -3226,7 +3226,8 @@ def define_borg_mount(parser):
 
         After upgrading borg (server) to 1.2+, you can use ``borg compact --cleanup-commits``
         to clean up the numerous 17byte commit-only segments that borg 1.1 did not clean up
-        due to a bug. It is enough to do that once per repository.
+        due to a bug. It is enough to do that once per repository. After cleaning up the
+        commits, borg will also do a normal compaction.
 
         See :ref:`separate_compaction` in Additional Notes for more details.
         """)


Test patch:
Repository: borgbackup/borg

Issue:
invalid repository
if an old borg detects a repo version > 1 it just says "invalid repository".

it could be a bit more helpful and e.g. tell "you need a new borg release to support repo version X".


Discussion:
fix this in 1.1-maint and also 1.2-maint.
@hexagonrecursion yet another such place. ^^^
Maybe could rather use `InvalidRepositoryConfig` exception.
For people (like me) wishing to know what "repo version > 1" means: https://github.com/borgbackup/borg/issues/1704

Resolving patch:
diff --git a/src/borg/repository.py b/src/borg/repository.py
--- a/src/borg/repository.py
+++ b/src/borg/repository.py
@@ -437,9 +437,16 @@ def open(self, path, exclusive, lock_wait=None, lock=True):
         except FileNotFoundError:
             self.close()
             raise self.InvalidRepository(self.path)
-        if 'repository' not in self.config.sections() or self.config.getint('repository', 'version') != 1:
+        if 'repository' not in self.config.sections():
             self.close()
-            raise self.InvalidRepository(path)
+            raise self.InvalidRepositoryConfig(path, 'no repository section found')
+        repo_version = self.config.getint('repository', 'version')
+        if repo_version != 1:
+            self.close()
+            raise self.InvalidRepositoryConfig(
+                path,
+                'repository version %d is not supported by this borg version' % repo_version
+            )
         self.max_segment_size = parse_file_size(self.config.get('repository', 'max_segment_size'))
         if self.max_segment_size >= MAX_SEGMENT_SIZE_LIMIT:
             self.close()


Test patch:
Repository: borgbackup/borg

Issue:
invalid repository
if an old borg detects a repo version > 1 it just says "invalid repository".

it could be a bit more helpful and e.g. tell "you need a new borg release to support repo version X".


Discussion:
fix this in 1.1-maint and also 1.2-maint.
@hexagonrecursion yet another such place. ^^^
Maybe could rather use `InvalidRepositoryConfig` exception.
For people (like me) wishing to know what "repo version > 1" means: https://github.com/borgbackup/borg/issues/1704

Resolving patch:
diff --git a/src/borg/repository.py b/src/borg/repository.py
--- a/src/borg/repository.py
+++ b/src/borg/repository.py
@@ -438,9 +438,16 @@ def open(self, path, exclusive, lock_wait=None, lock=True):
         except FileNotFoundError:
             self.close()
             raise self.InvalidRepository(self.path)
-        if 'repository' not in self.config.sections() or self.config.getint('repository', 'version') != 1:
+        if 'repository' not in self.config.sections():
             self.close()
-            raise self.InvalidRepository(path)
+            raise self.InvalidRepositoryConfig(path, 'no repository section found')
+        repo_version = self.config.getint('repository', 'version')
+        if repo_version != 1:
+            self.close()
+            raise self.InvalidRepositoryConfig(
+                path,
+                'repository version %d is not supported by this borg version' % repo_version
+            )
         self.max_segment_size = parse_file_size(self.config.get('repository', 'max_segment_size'))
         if self.max_segment_size >= MAX_SEGMENT_SIZE_LIMIT:
             self.close()


Test patch:
Repository: borgbackup/borg

Issue:
invalid repository
if an old borg detects a repo version > 1 it just says "invalid repository".

it could be a bit more helpful and e.g. tell "you need a new borg release to support repo version X".


Discussion:
fix this in 1.1-maint and also 1.2-maint.
@hexagonrecursion yet another such place. ^^^
Maybe could rather use `InvalidRepositoryConfig` exception.
For people (like me) wishing to know what "repo version > 1" means: https://github.com/borgbackup/borg/issues/1704

Resolving patch:
diff --git a/src/borg/repository.py b/src/borg/repository.py
--- a/src/borg/repository.py
+++ b/src/borg/repository.py
@@ -446,11 +446,14 @@ def open(self, path, exclusive, lock_wait=None, lock=True):
             raise self.InvalidRepository(self.path)
         if 'repository' not in self.config.sections():
             self.close()
-            raise self.InvalidRepository(path)
+            raise self.InvalidRepositoryConfig(path, 'no repository section found')
         self.version = self.config.getint('repository', 'version')
         if self.version not in (2, ):  # for now, only work on new repos
             self.close()
-            raise self.InvalidRepository(path)
+            raise self.InvalidRepositoryConfig(
+                path,
+                'repository version %d is not supported by this borg version' % self.version
+            )
         self.max_segment_size = parse_file_size(self.config.get('repository', 'max_segment_size'))
         if self.max_segment_size >= MAX_SEGMENT_SIZE_LIMIT:
             self.close()


Test patch:
Repository: borgbackup/borg

Issue:
borg info gives wrong info about encryption for "authenticated" mode
## Have you checked borgbackup docs, FAQ, and open Github issues?  
Yes  

## Is this a BUG / ISSUE report or a QUESTION?  
BUG

#### Your borg version (borg -V).  
borg 1.1.15

#### Full borg commandline that lead to the problem (leave away excludes and passwords)  
```
borg init --encryption=authenticated /path/to/repository
borg info /path/to/repository
```

## Describe the problem you're observing.
The [documentation here](https://borgbackup.readthedocs.io/en/stable/usage/init.html) state "If you do not want to encrypt the contents of your backups, but still want to detect malicious tampering use --encryption authenticated."

But `borg info` reports that:
```
Encrypted: Yes (authenticated)
```

What is the real status about that? All informations welcome!

EDIT: following [this link](https://github.com/borgbackup/borg/issues/4042) it looks to not be encrypted.



Discussion:
`authenticated` is not encrypted. That's a bug in `borg info` due to a bad check for unencrypted (only `plaintext` is considered not encrypted, `authenticated` missing there).

Also, guess for the Yes/No we could better use `KEYCLASS.logically_encrypted` which also considers the case that an empty passphrase repokey should be considered as logically not encrypted. And we should give `KEYCLASS.NAME` in any case to make clear what mode we have specifically.

I'd like to work on this issue!
@bybwang Thanks! Just open a PR against master branch with your changes.
@bybwang do you need help?
Hi there! I'm a novice at Python and I was wondering if this would be a good open source for me to contribute to at my skill level? :)
@cmcgee0308 It depends. Some parts are easy, some parts are hard.

Maybe check the issues for tag "easy" or "good first issue".
This issue had those tags so I was hoping it would be right up my alley. 😁
@ThomasWaldmann I'm ok thanks for asking! Sorry I had some stuff come up but I will be taking a look soon
i want to release 1.1.18 soon, can we speed up a bit?
I'll fix this myself.

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -1608,7 +1608,7 @@ def _info_repository(self, args, repository, manifest, key, cache):
             json_print(info)
         else:
             encryption = 'Encrypted: '
-            if key.NAME == 'plaintext':
+            if key.NAME in ('plaintext', 'authenticated'):
                 encryption += 'No'
             else:
                 encryption += 'Yes (%s)' % key.NAME


Test patch: