internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: borgbackup/borg

Issue:
verify_file_chunks(): verify replacement chunks also
check the case of *lost* all-zero replacement chunks (== chunk ids in chunks list that are not in chunks_healthy list). guess they go undetected right now.

borg 1.0 / 1.1.


Discussion:

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -1246,6 +1246,13 @@ def verify_file_chunks(item):
             Missing file chunks will be replaced with new chunks of the same length containing all zeros.
             If a previously missing file chunk re-appears, the replacement chunk is replaced by the correct one.
             """
+            def replacement_chunk(size):
+                data = bytes(size)
+                chunk_id = self.key.id_hash(data)
+                cdata = self.key.encrypt(Chunk(data))
+                csize = len(cdata)
+                return chunk_id, size, csize, cdata
+
             offset = 0
             chunk_list = []
             chunks_replaced = False
@@ -1261,16 +1268,20 @@ def verify_file_chunks(item):
                         logger.error('{}: New missing file chunk detected (Byte {}-{}). '
                                      'Replacing with all-zero chunk.'.format(item.path, offset, offset + size))
                         self.error_found = chunks_replaced = True
-                        data = bytes(size)
-                        chunk_id = self.key.id_hash(data)
-                        cdata = self.key.encrypt(Chunk(data))
-                        csize = len(cdata)
+                        chunk_id, size, csize, cdata = replacement_chunk(size)
                         add_reference(chunk_id, size, csize, cdata)
                     else:
                         logger.info('{}: Previously missing file chunk is still missing (Byte {}-{}). It has a '
                                     'all-zero replacement chunk already.'.format(item.path, offset, offset + size))
                         chunk_id, size, csize = chunk_current
-                        add_reference(chunk_id, size, csize)
+                        if chunk_id in self.chunks:
+                            add_reference(chunk_id, size, csize)
+                        else:
+                            logger.warning('{}: Missing all-zero replacement chunk detected (Byte {}-{}). '
+                                           'Generating new replacement chunk.'.format(item.path, offset, offset + size))
+                            self.error_found = chunks_replaced = True
+                            chunk_id, size, csize, cdata = replacement_chunk(size)
+                            add_reference(chunk_id, size, csize, cdata)
                 else:
                     if chunk_current == chunk_healthy:
                         # normal case, all fine.


Test patch:
Repository: borgbackup/borg

Issue:
bug in --verify-data
https://github.com/borgbackup/borg/pull/2220/files#r103111041

if data is corrupted in a way that the decompression code fails, it won't get as far as to verify the cryptographic hash over the (decompressed) plaintext data, but will crash with an exception first.


Discussion:
~Decompressors should raise IntegrityError not some library-specific exception
```
# in borg.testsuite.archiver.ArchiverCheckTestCase testMethod=test_verify_data_unencrypted
...
.tox/py34/lib/python3.4/site-packages/borg/archive.py:1169: in verify_data
    _, data = self.key.decrypt(_chunk_id, encrypted_data)
.tox/py34/lib/python3.4/site-packages/borg/key.py:269: in decrypt
    data = self.compressor.decompress(payload)
src/borg/compress.pyx:200: in borg.compress.Compressor.decompress (src/borg/compress.c:4880)
    return compressor_cls(**self.params).decompress(data)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   raise Exception('lz4 decompress failed')
E   Exception: lz4 decompress failed

src/borg/compress.pyx:115: Exception
```


Resolving patch:
diff --git a/src/borg/helpers.py b/src/borg/helpers.py
--- a/src/borg/helpers.py
+++ b/src/borg/helpers.py
@@ -81,6 +81,10 @@ class IntegrityError(ErrorWithTraceback):
     """Data integrity error: {}"""
 
 
+class DecompressionError(IntegrityError):
+    """Decompression error: {}"""
+
+
 class ExtensionModuleError(Error):
     """The Borg binary extension modules do not seem to be properly installed"""
 
@@ -99,7 +103,7 @@ def check_extension_modules():
         raise ExtensionModuleError
     if chunker.API_VERSION != '1.1_01':
         raise ExtensionModuleError
-    if compress.API_VERSION != '1.1_01':
+    if compress.API_VERSION != '1.1_02':
         raise ExtensionModuleError
     if crypto.API_VERSION != '1.1_01':
         raise ExtensionModuleError
diff --git a/src/borg/testsuite/archiver.py b/src/borg/testsuite/archiver.py
--- a/src/borg/testsuite/archiver.py
+++ b/src/borg/testsuite/archiver.py
@@ -262,7 +262,7 @@ def cmd(self, *args, **kw):
         return output
 
     def create_src_archive(self, name):
-        self.cmd('create', '--compression=none', self.repository_location + '::' + name, src_dir)
+        self.cmd('create', '--compression=lz4', self.repository_location + '::' + name, src_dir)
 
     def open_archive(self, name):
         repository = Repository(self.repository_path, exclusive=True)


Test patch:
Repository: borgbackup/borg

Issue:
files cache: update inode numbers?
scenario:
- you move big amounts of stuff to a new filesystem on a new disk (like copying from ext4 -> zfs).
- but in the end, everything gets mounted together like it was before
- first backup after the migration: inodes are expected to be all different, so --ignore-inodes is advised to avoid having it chunk everything again
- but after the first backup, we want to have inodes checking back - what happens if --ignore-inodes is removed again? will it have updated all the inode numbers in the cache to their currently seen value in the filesystem in the first backup or did it not touch them and would chunk everything in the second backup if we enabled the inode check again?



Discussion:
A: no we do not update (yet) the inode numbers in the files cache in ignore_inode mode.

Shall we just do that? Do we need an option for this or can it be hardcoded?

See: borg.cache.file_known_and_unchanged

As this is a simple change, but fixes a major performance issue: apply to 1.0-maint also?

Resolving patch:
diff --git a/src/borg/cache.py b/src/borg/cache.py
--- a/src/borg/cache.py
+++ b/src/borg/cache.py
@@ -575,7 +575,15 @@ def file_known_and_unchanged(self, path_hash, st, ignore_inode=False):
         entry = FileCacheEntry(*msgpack.unpackb(entry))
         if (entry.size == st.st_size and entry.mtime == st.st_mtime_ns and
                 (ignore_inode or entry.inode == st.st_ino)):
-            self.files[path_hash] = msgpack.packb(entry._replace(age=0))
+            # we ignored the inode number in the comparison above or it is still same.
+            # if it is still the same, replacing it in the tuple doesn't change it.
+            # if we ignored it, a reason for doing that is that files were moved to a new
+            # disk / new fs (so a one-time change of inode number is expected) and we wanted
+            # to avoid everything getting chunked again. to be able to re-enable the inode
+            # number comparison in a future backup run (and avoid chunking everything
+            # again at that time), we need to update the inode number in the cache with what
+            # we see in the filesystem.
+            self.files[path_hash] = msgpack.packb(entry._replace(inode=st.st_ino, age=0))
             return entry.chunk_ids
         else:
             return None


Test patch:
Repository: borgbackup/borg

Issue:
Prune help text: --keep-last parameter doesn't exist
Borg doesn't know any keep-last parameter. Tested with 1.0.9/10


> borg  prune -n --list /volumeUSB1/backup
> 
> At least one of the "keep-within", **"keep-last"**, "keep-hourly", "keep-daily", "keep-weekly", "keep-monthly" or "keep-yearly" settings must be specified.

> borg  prune -n --list --keep-last 1 /volumeUSB1/backup
> borg: error: unrecognized arguments: --keep-last


Discussion:
Indeed it doesn't, that is new in 1.1.x, so the 1.0.x message should not mention it.

d52b5cf5

Resolving patch:
diff --git a/borg/archiver.py b/borg/archiver.py
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -683,7 +683,7 @@ def do_prune(self, args, repository, manifest, key):
         """Prune repository archives according to specified rules"""
         if not any((args.hourly, args.daily,
                     args.weekly, args.monthly, args.yearly, args.within)):
-            self.print_error('At least one of the "keep-within", "keep-last", '
+            self.print_error('At least one of the "keep-within", '
                              '"keep-hourly", "keep-daily", '
                              '"keep-weekly", "keep-monthly" or "keep-yearly" settings must be specified.')
             return self.exit_code


Test patch:
Repository: borgbackup/borg

Issue:
borg key crashes
```
$ BORG_REPO=abc borg key
Traceback (most recent call last):
  File "/home/tw/w/borg-env/bin/borg", line 11, in <module>
    load_entry_point('borgbackup', 'console_scripts', 'borg')()
  File "/home/tw/w/borg/src/borg/archiver.py", line 3423, in main
    args = archiver.get_args(sys.argv, os.environ.get('SSH_ORIGINAL_COMMAND'))
  File "/home/tw/w/borg/src/borg/archiver.py", line 3296, in get_args
    result = self.parse_args(argv[1:])
  File "/home/tw/w/borg/src/borg/archiver.py", line 3315, in parse_args
    if args.func == self.do_create:
AttributeError: 'Namespace' object has no attribute 'func'
```


Discussion:
same in 1.0-maint:
``` 
(borg-env)tw@tux:~/w/borg$ BORG_REPO=ccc BORG_PASSPHRASE=1 borg key
Traceback (most recent call last):
  File "/home/tw/w/borg-env/bin/borg", line 11, in <module>
    load_entry_point('borgbackup', 'console_scripts', 'borg')()
  File "/home/tw/w/borg/borg/archiver.py", line 2135, in main
    args = archiver.get_args(sys.argv, os.environ.get('SSH_ORIGINAL_COMMAND'))
  File "/home/tw/w/borg/borg/archiver.py", line 2046, in get_args
    result = self.parse_args(argv[1:])
  File "/home/tw/w/borg/borg/archiver.py", line 2065, in parse_args
    if args.func == self.do_create:
AttributeError: 'Namespace' object has no attribute 'func'
```
Didn't we fix this already once?

Edit: This is a regression. 30a5c5e44 / patterns

Also affects "borg debug".

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -3334,10 +3334,11 @@ def parse_args(self, args=None):
             args = self.preprocess_args(args)
         parser = self.build_parser()
         args = parser.parse_args(args or ['-h'])
-        if args.func == self.do_create:
+        # This works around http://bugs.python.org/issue9351
+        func = getattr(args, 'func', None) or getattr(args, 'fallback_func')
+        if func == self.do_create and not args.paths:
             # need at least 1 path but args.paths may also be populated from patterns
-            if not args.paths:
-                parser.error('Need at least one PATH argument.')
+            parser.error('Need at least one PATH argument.')
         return args
 
     def prerun_checks(self, logger):


Test patch:
Repository: borgbackup/borg

Issue:
mount requires num(archives) * 8 MiB of unnecessary memory
```
Currently mapped 1814888465 bytes; peak was 1814939609 bytes
Tracer used 1472072 bytes (before stopping)
Top 20 allocators
#1: borg/archive.py:331: 1761286.9 KiB
    self.zeros = b'\0' * (1 << chunker_params[1])
#2: borg/helpers.py:948: 9011.3 KiB
    self._thread_local.buffer = self.allocator(size)
#3: python3.6/linecache.py:137: 836.6 KiB  [this is part of the instrumentation code]
    lines = fp.readlines()
#4: borg/archive.py:335: 652.0 KiB
    metadata = ArchiveItem(internal_dict=msgpack.unpackb(data, unicode_errors='surrogateescape'))
```

(About 99.3 % of all memory allocated [excluding the repository index since Python does not know about that])

Suggested fix: ~~Global zero block.~~ ~~If we just mmap it we can make it sufficiently large enough for all cases, and won't even be using any relevant memory for it until used.~~ (does not work due to Python API limitation) => allocate when needed.


Discussion:

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -325,7 +325,7 @@ def __init__(self, repository, key, manifest, name, cache=None, create=False,
             if info is None:
                 raise self.DoesNotExist(name)
             self.load(info.id)
-            self.zeros = b'\0' * (1 << chunker_params[1])
+            self.zeros = None
 
     def _load_meta(self, id):
         _, data = self.key.decrypt(id, self.repository.get(id))
@@ -578,6 +578,8 @@ def extract_item(self, item, restore_attrs=True, dry_run=False, stdout=False, sp
                 # Extract chunks, since the item which had the chunks was not extracted
             with backup_io('open'):
                 fd = open(path, 'wb')
+            if sparse and self.zeros is None:
+                self.zeros = b'\0' * (1 << self.chunker_params[1])
             with fd:
                 ids = [c.id for c in item.chunks]
                 for _, data in self.pipeline.fetch_many(ids, is_preloaded=True):


Test patch:
Repository: borgbackup/borg

Issue:
do 1.0-maint backports
https://github.com/borgbackup/borg/issues?utf8=%E2%9C%93&q=label%3Abackport%2F1.0-maint%20

do 3 simple/small or 1 bigger/complex backport from there.

---

:moneybag: [there is a bounty for this](https://www.bountysource.com/issues/43626145-do-1-0-maint-backports)



Discussion:

Resolving patch:
diff --git a/borg/archive.py b/borg/archive.py
--- a/borg/archive.py
+++ b/borg/archive.py
@@ -204,6 +204,7 @@ def __init__(self, repository, key, manifest, name, cache=None, create=False,
         if start is None:
             start = datetime.utcnow()
             start_monotonic = time.monotonic()
+        self.chunker_params = chunker_params
         self.start = start
         self.start_monotonic = start_monotonic
         if end is None:
@@ -227,7 +228,7 @@ def __init__(self, repository, key, manifest, name, cache=None, create=False,
                 raise self.DoesNotExist(name)
             info = self.manifest.archives[name]
             self.load(info[b'id'])
-            self.zeros = b'\0' * (1 << chunker_params[1])
+            self.zeros = None
 
     def _load_meta(self, id):
         data = self.key.decrypt(id, self.repository.get(id))
@@ -405,6 +406,8 @@ def make_parent(path):
                 with backup_io():
                     os.link(source, path)
             else:
+                if sparse and self.zeros is None:
+                    self.zeros = b'\0' * (1 << self.chunker_params[1])
                 with backup_io():
                     fd = open(path, 'wb')
                 with fd:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -213,43 +213,8 @@ def generate_level(self, prefix, parser, Archiver):
         return is_subcommand
 
 
-class build_api(Command):
-    description = "generate a basic api.rst file based on the modules available"
-
-    user_options = [
-        ('output=', 'O', 'output directory'),
-    ]
-
-    def initialize_options(self):
-        pass
-
-    def finalize_options(self):
-        pass
-
-    def run(self):
-        print("auto-generating API documentation")
-        with open("docs/api.rst", "w") as doc:
-            doc.write("""
-.. IMPORTANT: this file is auto-generated by "setup.py build_api", do not edit!
-
-
-API Documentation
-=================
-""")
-            for mod in sorted(glob('borg/*.py') + glob('borg/*.pyx')):
-                print("examining module %s" % mod)
-                mod = mod.replace('.pyx', '').replace('.py', '').replace('/', '.')
-                if "._" not in mod:
-                    doc.write("""
-.. automodule:: %s
-    :members:
-    :undoc-members:
-""" % mod)
-
-
 cmdclass = {
     'build_ext': build_ext,
-    'build_api': build_api,
     'build_usage': build_usage,
     'sdist': Sdist
 }


Test patch:
Repository: borgbackup/borg

Issue:
borg rename using template doesn't eval
I realized I created an archive using ENVIRONMENT_VARIABLE_NAME instead of $ENVIRONMENT_VARIABLE_NAME, so I tried running
     borg rename ::ENVIRONMENT_VARIABLE_NAME '{hostname}-{utcnow:%Y-%m-%d}'

But somehow the command changed the archive name to the literal string  '{hostname}-{utcnow:%Y-%m-%d}'! I'm trying to rename it again but it seems like the eval only works on the existing archive name, not the targeted name. And now that my archive has that name, I don't know how to rename it. This seems like a bug, and also maybe something that should be covere in the docs. Can you please help me out? Using borg 1.0.10 on macOS via homebrew. 


Discussion:
Labelling "corruption" since the resulting archive is not accessible using `list`, `extract`... (should be accessible through `mount`).
Try this

```
borg rename "::'{{hostname}}-{{utcnow:%Y-%m-%d}}'" target-name
                ^ double braces to keep them from being interpreted
            ^ nested quotes to keep the shell from removing '
              (not needed if archive name is not enclosed in ')
```
It works! Thanks for that debugging tip. In hindsight it's obvious but it wasn't intuitive as a user without pouring over source code.

Is this still a bug that the rename target doesn't seem to apply the interpolation?
I'd say there are two things to do here:

- Rename target is not interpolated, which it should be (for consistency)
- Documentation should explain how to escape interpolation

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -59,6 +59,7 @@
 from .helpers import ErrorIgnoringTextIOWrapper
 from .helpers import ProgressIndicatorPercent
 from .helpers import basic_json_data, json_print
+from .helpers import replace_placeholders
 from .item import Item
 from .key import key_creator, tam_required_file, tam_required, RepoKey, PassphraseKey
 from .keymanager import KeyManager
@@ -905,7 +906,8 @@ def compare_or_defer(item1, item2):
     @with_archive
     def do_rename(self, args, repository, manifest, key, cache, archive):
         """Rename an existing archive"""
-        archive.rename(args.name)
+        name = replace_placeholders(args.name)
+        archive.rename(name)
         manifest.write()
         repository.commit()
         cache.commit()


Test patch:
Repository: borgbackup/borg

Issue:
borg rename using template doesn't eval
I realized I created an archive using ENVIRONMENT_VARIABLE_NAME instead of $ENVIRONMENT_VARIABLE_NAME, so I tried running
     borg rename ::ENVIRONMENT_VARIABLE_NAME '{hostname}-{utcnow:%Y-%m-%d}'

But somehow the command changed the archive name to the literal string  '{hostname}-{utcnow:%Y-%m-%d}'! I'm trying to rename it again but it seems like the eval only works on the existing archive name, not the targeted name. And now that my archive has that name, I don't know how to rename it. This seems like a bug, and also maybe something that should be covere in the docs. Can you please help me out? Using borg 1.0.10 on macOS via homebrew. 


Discussion:
Labelling "corruption" since the resulting archive is not accessible using `list`, `extract`... (should be accessible through `mount`).
Try this

```
borg rename "::'{{hostname}}-{{utcnow:%Y-%m-%d}}'" target-name
                ^ double braces to keep them from being interpreted
            ^ nested quotes to keep the shell from removing '
              (not needed if archive name is not enclosed in ')
```
It works! Thanks for that debugging tip. In hindsight it's obvious but it wasn't intuitive as a user without pouring over source code.

Is this still a bug that the rename target doesn't seem to apply the interpolation?
I'd say there are two things to do here:

- Rename target is not interpolated, which it should be (for consistency)
- Documentation should explain how to escape interpolation

Resolving patch:
diff --git a/borg/archiver.py b/borg/archiver.py
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -24,7 +24,8 @@
     PathPrefixPattern, to_localtime, timestamp, safe_timestamp, bin_to_hex, get_cache_dir, prune_within, prune_split, \
     Manifest, NoManifestError, remove_surrogates, format_archive, check_extension_modules, Statistics, \
     dir_is_tagged, bigint_to_int, ChunkerParams, CompressionSpec, PrefixSpec, is_slow_msgpack, yes, sysinfo, \
-    EXIT_SUCCESS, EXIT_WARNING, EXIT_ERROR, log_multi, PatternMatcher, ErrorIgnoringTextIOWrapper, set_ec
+    EXIT_SUCCESS, EXIT_WARNING, EXIT_ERROR, log_multi, PatternMatcher, ErrorIgnoringTextIOWrapper, set_ec, \
+    replace_placeholders
 from .helpers import signal_handler, raising_signal_handler, SigHup, SigTerm
 from .logger import create_logger, setup_logging
 logger = create_logger()
@@ -494,7 +495,8 @@ def do_extract(self, args, repository, manifest, key, archive):
     @with_archive
     def do_rename(self, args, repository, manifest, key, cache, archive):
         """Rename an existing archive"""
-        archive.rename(args.name)
+        name = replace_placeholders(args.name)
+        archive.rename(name)
         manifest.write()
         repository.commit()
         cache.commit()


Test patch:
Repository: borgbackup/borg

Issue:
verify_data: fix defect chunk processing
While re-reading that code, I noticed it does not catch IntegrityError there:

https://github.com/borgbackup/borg/blob/1.1.0b4/src/borg/archive.py#L1242

compare to:

https://github.com/borgbackup/borg/blob/1.1.0b4/src/borg/archive.py#L1202

Guess it should catch IntegrityError there also?

Note: it does not need to catch ObjectNotFound as missing object ids do not get added to the defect chunks list.


Discussion:

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -1254,8 +1254,8 @@ def verify_data(self):
                     # local repo (fs): as chunks.iteritems loop usually pumps a lot of data through,
                     # a defect chunk is likely not in the fs cache any more and really gets re-read
                     # from the underlying media.
-                    encrypted_data = self.repository.get(defect_chunk)
                     try:
+                        encrypted_data = self.repository.get(defect_chunk)
                         _chunk_id = None if defect_chunk == Manifest.MANIFEST_ID else defect_chunk
                         self.key.decrypt(_chunk_id, encrypted_data)
                     except IntegrityError:


Test patch: