internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: borgbackup/borg

Issue:
improve FUSE data read speed
when running e.g. sha256sum on a file in a FUSE mount (`borg mount`) it runs much slower than if the file is extracted and then the tool is run on the extracted file.



Discussion:

Resolving patch:
diff --git a/borg/archiver.py b/borg/archiver.py
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -1533,6 +1533,11 @@ def build_parser(self, args=None, prog=None):
 
         To allow a regular user to use fstab entries, add the ``user`` option:
         ``/path/to/repo /mnt/point fuse.borgfs defaults,noauto,user 0 0``
+
+        The BORG_MOUNT_DATA_CACHE_ENTRIES environment variable is meant for advanced users
+        to tweak the performance. It sets the number of cached data chunks; additional
+        memory usage can be up to ~8 MiB times this number. The default is the number
+        of CPU cores.
         """)
         subparser = subparsers.add_parser('mount', parents=[common_parser], add_help=False,
                                           description=self.do_mount.__doc__,
diff --git a/borg/fuse.py b/borg/fuse.py
--- a/borg/fuse.py
+++ b/borg/fuse.py
@@ -8,9 +8,13 @@
 import time
 from .archive import Archive
 from .helpers import daemonize, bigint_to_int
+from .logger import create_logger
+from .lrucache import LRUCache
 from distutils.version import LooseVersion
 import msgpack
 
+logger = create_logger()
+
 # Does this version of llfuse support ns precision?
 have_fuse_xtime_ns = hasattr(llfuse.EntryAttributes, 'st_mtime_ns')
 
@@ -54,6 +58,9 @@ def __init__(self, key, repository, manifest, archive, cached_repo):
         self.pending_archives = {}
         self.accounted_chunks = {}
         self.cache = ItemCache()
+        data_cache_capacity = int(os.environ.get('BORG_MOUNT_DATA_CACHE_ENTRIES', os.cpu_count() or 1))
+        logger.debug('mount data cache capacity: %d chunks', data_cache_capacity)
+        self.data_cache = LRUCache(capacity=data_cache_capacity, dispose=lambda _: None)
         if archive:
             self.process_archive(archive)
         else:
@@ -229,7 +236,16 @@ def read(self, fh, offset, size):
                 offset -= s
                 continue
             n = min(size, s - offset)
-            _, data = self.key.decrypt(id, self.repository.get(id))
+            if id in self.data_cache:
+                data = self.data_cache[id]
+                if offset + n == len(data):
+                    # evict fully read chunk from cache
+                    del self.data_cache[id]
+            else:
+                _, data = self.key.decrypt(id, self.repository.get(id))
+                if offset + n < len(data):
+                    # chunk was only partially read, cache it
+                    self.data_cache[id] = data
             parts.append(data[offset:offset + n])
             offset = 0
             size -= n


Test patch:
Repository: borgbackup/borg

Issue:
update readthedocs url to *.readthedocs.io
also (after moving borgweb docs), we could directly point there without going over borgbackup.github.io.



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -236,7 +236,7 @@ def run(self):
     },
     author='The Borg Collective (see AUTHORS file)',
     author_email='borgbackup@python.org',
-    url='https://borgbackup.readthedocs.org/',
+    url='https://borgbackup.readthedocs.io/',
     description='Deduplicated, encrypted, authenticated and compressed backups',
     long_description=long_description,
     license='BSD',


Test patch:
Repository: borgbackup/borg

Issue:
Document archive name auto-generation
Commit: f7c1632a

It's only documented in the usage example, not the command epilog.

```
# Format tags available for archive name:
# {now}, {utcnow}, {fqdn}, {hostname}, {user}, {pid}
# add short hostname, backup username and current unixtime (seconds from epoch)
$ borg create  /mnt/backup::{hostname}-{user}-{now:%s} ~
```



Discussion:

Resolving patch:
diff --git a/borg/archiver.py b/borg/archiver.py
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -997,6 +997,9 @@ def build_parser(self, args=None, prog=None):
         '.checkpoint.N' (with N being a number), because these names are used for
         checkpoints and treated in special ways.
 
+        In the archive name, you may use the following format tags:
+        {now}, {utcnow}, {fqdn}, {hostname}, {user}, {pid}
+
         To speed up pulling backups over sshfs and similar network file systems which do
         not provide correct inode information the --ignore-inode flag can be used. This
         potentially decreases reliability of change detection, while avoiding always reading


Test patch:
Repository: borgbackup/borg

Issue:
check vs extract --dry-run
Does extract --dry-run test an archive any more than check does?  The drive that my repositories are on is reporting unreadable sectors, so I'm replacing it under warranty.  I'm waiting for the replacement drive, and I'd like to verify the repositories both before and after I transfer them to the new drive.

I already run check over each of the repositories at the end of every set of backups (ie, I create A archives spread among R repositories every night, then run borg check for each R).

I know that there used to be a borg verify command, now replaced with extract --dry-run.  Does running that on each archive give me any extra assurance of uncorrupted data over just running check?  Should I run check on each archive instead, or additionally?

I looked at the doc for the commands, and the FAQ.  I think the doc for check says that I don't need to run check on each archive, that a repo level check will do that if I don't use --repository-only.  Beyond that I'm not sure.



Discussion:
`borg extract --dry-run repo::arch` will extract all items (verify, decrypt, decompress) like it does without `--dry-run`, it will just not write the resulting data to disk files.

`borg check` does not (yet) extract all data chunks, but there is ongoing work to add that.

So extract --dry-run will do some validation that is not currently done by check?

Yes, it will check the (hmac-)sha256 of the file data chunks and it will actually decrypt / decompress them.


Resolving patch:
diff --git a/borg/archiver.py b/borg/archiver.py
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -1401,6 +1401,10 @@ def build_parser(self, args=None, prog=None):
         be restricted by using the ``--exclude`` option.
 
         See the output of the "borg help patterns" command for more help on exclude patterns.
+
+        By using ``--dry-run``, you can do all extraction steps except actually writing the
+        output data: reading metadata and data chunks from the repo, checking the hash/hmac,
+        decrypting, decompressing.
         """)
         subparser = subparsers.add_parser('extract', parents=[common_parser], add_help=False,
                                           description=self.do_extract.__doc__,


Test patch:
Repository: borgbackup/borg

Issue:
crash related to (non-borg) binary files in keys directory
If i try to list/mount a remote repository i get this error:

`Local Exception.`
`Traceback (most recent call last):`
`File "borg/archiver.py", line 1498, in main`
`File "borg/archiver.py", line 1452, in run`
`File "borg/archiver.py", line 74, in wrapper`
`File "borg/helpers.py", line 95, in load`
`File "borg/key.py", line 53, in key_factory`
`File "borg/key.py", line 289, in detect`
`File "borg/key.py", line 402, in find_key`
`File "/home/vagrant/.pyenv/versions/3.5.1/lib/python3.5/codecs.py", line 321, in decode`
`UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9b in position 12: invalid start byte`

`Platform: Linux xxx 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08) x86_64`
`Linux: debian 8.4`
`Borg: 1.0.2  Python: CPython 3.5.1`

I'm using the standalone binary version 1.0.2 on debian 8.4.0 (server and client) with locales configured as en_US.UTF-8.
The generated key file seems to be correctly encoded:
`file -i` outputs "us-ascii" which according to http://stackoverflow.com/a/11303410 is correct.

Has anybody a clue what could be the issue here?



Discussion:
In a Python 3 shell...

```
>>> open("/path/to/your/keyfile").read()
```

Does that fail as well?

is there some file in the keys directory that does not belong there?

the directory is usually ~/.config/borg/keys

Note: we should open the file in binary mode first and do the magic check before trying to decode it.

Yes i had a zip-file lying around there...  if that's deleted it works.
Thank you so much!

Sounds like you backuped your keys, which is good. :)


Resolving patch:
diff --git a/borg/key.py b/borg/key.py
--- a/borg/key.py
+++ b/borg/key.py
@@ -394,13 +394,15 @@ class KeyfileKey(KeyfileKeyBase):
     FILE_ID = 'BORG_KEY'
 
     def find_key(self):
-        id = hexlify(self.repository.id).decode('ascii')
+        file_id = self.FILE_ID.encode()
+        first_line = file_id + b' ' + hexlify(self.repository.id)
         keys_dir = get_keys_dir()
         for name in os.listdir(keys_dir):
             filename = os.path.join(keys_dir, name)
-            with open(filename, 'r') as fd:
-                line = fd.readline().strip()
-                if line.startswith(self.FILE_ID) and line[len(self.FILE_ID) + 1:] == id:
+            # we do the magic / id check in binary mode to avoid stumbling over
+            # decoding errors if somebody has binary files in the keys dir for some reason.
+            with open(filename, 'rb') as fd:
+                if fd.read(len(first_line)) == first_line:
                     return filename
         raise KeyfileNotFoundError(self.repository._location.canonical_path(), get_keys_dir())
 


Test patch:
Repository: borgbackup/borg

Issue:
chunker performance
the chunker seems to perform at 400-500MB/s on a in-memory file (BytesIO or on ramdisk), including reading, chunk buffer management and the chunking itself.

just reading (fd.read()) the file performs at ~6000MB/s.

can we make the chunker significantly faster without making the code more complex?



Discussion:
Force-inlining buzhash and buzhash_update seemed to make a difference when I tried it. If you have a good, deterministic benchmark it's worth a shot.

I don't know if GCC is able to vectorize the code, or if it's even something that can be vectorized by hand; if so it's also worth a shot, although it of course only does something for the arch you write the code for...

[borg_chunker_test.py.txt](https://github.com/borgbackup/borg/files/252576/borg_chunker_test.py.txt)

Hmm, i changed `static` to `static inline` for buzhash/buzhash_update, but did not see a difference in speed.

iirc `__attribute__((always_inline))`, edit: doesn't make much of a difference with the linked test case. The difference is about 5 times smaller than the run-to-run variance.

gcc does inline small functions with -O2 (as it is run by default by cythonize call).

note: replacing .read() by .readinto() (and thus avoiding the memcpy) in the python-file part of the chunker doesn't give a noticeable speedup.

btw, i had a look into the assembler code and gcc -O2 seems clever enough to translate that "BARREL_SHIFT" code into a single "rol" x86 instruction. \o/

Replacing lenmod in a buzhash_update with a constant (because window is always 4095) gives 30 mb/s increase in chunker speed.

@rakajack ah, good find! it could be also computed outside the loop and passed in instead the length.

A chunk have a minimum size (min_size). Are we skipping (min_size - window_size) bytes after finding a boundary?

No. Also, we are not yet skipping files that are too small.


Resolving patch:
diff --git a/borg/testsuite/archiver.py b/borg/testsuite/archiver.py
--- a/borg/testsuite/archiver.py
+++ b/borg/testsuite/archiver.py
@@ -1491,9 +1491,9 @@ def test_recreate_rechunkify_interrupt(self):
         self.cmd('create', self.repository_location + '::test', 'input')
         archive_before = self.cmd('list', self.repository_location + '::test', '--format', '{sha512}')
         with patch.object(Cache, 'add_chunk', self._test_recreate_chunker_interrupt_patch()):
-            self.cmd('recreate', '-pv', '--chunker-params', '10,12,11,4095', self.repository_location)
+            self.cmd('recreate', '-pv', '--chunker-params', '10,13,11,4095', self.repository_location)
         assert 'test.recreate' in self.cmd('list', self.repository_location)
-        output = self.cmd('recreate', '-svp', '--debug', '--chunker-params', '10,12,11,4095', self.repository_location)
+        output = self.cmd('recreate', '-svp', '--debug', '--chunker-params', '10,13,11,4095', self.repository_location)
         assert 'Found test.recreate, will resume' in output
         assert 'Copied 1 chunks from a partially processed item' in output
         archive_after = self.cmd('list', self.repository_location + '::test', '--format', '{sha512}')


Test patch:
Repository: borgbackup/borg

Issue:
Clear up the environment when launching subprocesses
Remove things like BORG_PASSPHRASE from the environmen of subprocesses.

E.g. RemoteRepository.init "env = dict(os.environ)" should be fixed.



Discussion:

Resolving patch:
diff --git a/borg/remote.py b/borg/remote.py
--- a/borg/remote.py
+++ b/borg/remote.py
@@ -153,6 +153,7 @@ def __init__(self, location, create=False, lock_wait=None, lock=True, args=None)
             # pyinstaller binary adds LD_LIBRARY_PATH=/tmp/_ME... but we do not want
             # that the system's ssh binary picks up (non-matching) libraries from there
             env.pop('LD_LIBRARY_PATH', None)
+        env.pop('BORG_PASSPHRASE', None)  # security: do not give secrets to subprocess
         self.p = Popen(borg_cmd, bufsize=0, stdin=PIPE, stdout=PIPE, stderr=PIPE, env=env)
         self.stdin_fd = self.p.stdin.fileno()
         self.stdout_fd = self.p.stdout.fileno()


Test patch:
Repository: borgbackup/borg

Issue:
Is prune using start time or end time of archives?
In the help for prune it says "The time that each backup completes is used for pruning purposes". Is that really so? It seems to me that it is the start time that is used - not the completion time.

```
$ borg prune --list -v -H 2 /media/jan/OS/borgbackup
Keeping archive: xps15-jan-2016-06-06T12:05:47        Mon, 2016-06-06 14:05:47
Keeping archive: xps15-jan-2016-06-06T11:54:40        Mon, 2016-06-06 13:54:40
```

The archive ' xps15-jan-2016-06-06T11:54:40' took 9 minutees to create and hence completed in the same hour as the other archive. If completion time was used for pruning shouldn't that archive be deleted?

If it is really the complete time that should be used, I also think it would make more sense to display that instead of the start time in the output.

I created the archive with --stats option and got

```
Archive name: xps15-jan-2016-06-06T11:54:40
Archive fingerprint: 9f39548959b628305ad34917cc1d2c772f5135e63068a4b174157653f77f5b0f
Time (start): Mon, 2016-06-06 13:54:41
Time (end):   Mon, 2016-06-06 14:03:48
Duration: 9 minutes 6.96 seconds
Number of files: 100993
```



Discussion:
Yes, looks like a documentation bug.

prune uses the start times of the archives to compute what to keep (and what to delete).


Resolving patch:
diff --git a/borg/archiver.py b/borg/archiver.py
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -1244,7 +1244,7 @@ def build_parser(self, args=None, prog=None):
                                help='repository for which to break the locks')
 
         prune_epilog = textwrap.dedent("""
-        The prune command prunes a repository by deleting archives not matching
+        The prune command prunes a repository by deleting all archives not matching
         any of the specified retention options. This command is normally used by
         automated backup scripts wanting to keep a certain number of historic backups.
 
@@ -1252,7 +1252,7 @@ def build_parser(self, args=None, prog=None):
         most recent days with backups (days without backups do not count).
         The rules are applied from hourly to yearly, and backups selected by previous
         rules do not count towards those of later rules. The time that each backup
-        completes is used for pruning purposes. Dates and times are interpreted in
+        starts is used for pruning purposes. Dates and times are interpreted in
         the local timezone, and weeks go from Monday to Sunday. Specifying a
         negative number of archives to keep means that there is no limit.
 


Test patch:
Repository: borgbackup/borg

Issue:
fix the sparse test
as seen in #1150 this test has issues on zfs. and also on xfs as seen in some borg 1.0.3 SuSE build log.

as all these filesystems also exist on linux (or freebsd), disabling by OS is not good enough.



Discussion:
found out that xfs is doing stuff behind the scenes: it is pre-allocating 16MB to prevent fragmentation (in my case, value depends on misc factors).

fixed the test so it just checks that the extracted sparse file uses less (not necessary much less) space than a non-sparse file would.

another problem showed up when i tried to verify the holes in the sparse file via SEEK_HOLE, SEEK_DATA: after the few bytes of real data in the file, there was another 16MB preallocated space. So I ended up checking just the hole at the start of the file.


Resolving patch:
diff --git a/borg/testsuite/archiver.py b/borg/testsuite/archiver.py
--- a/borg/testsuite/archiver.py
+++ b/borg/testsuite/archiver.py
@@ -412,8 +412,21 @@ def test_sparse_file(self):
             self.assert_equal(fd.read(hole_size), b'\0' * hole_size)
         st = os.stat(filename)
         self.assert_equal(st.st_size, total_len)
-        if sparse_support and hasattr(st, 'st_blocks'):
-            self.assert_true(st.st_blocks * 512 < total_len / 9)  # is output sparse?
+        if sparse_support:
+            if hasattr(st, 'st_blocks'):
+                # do only check if it is less, do NOT check if it is much less
+                # as that causes troubles on xfs and zfs:
+                self.assert_true(st.st_blocks * 512 < total_len)
+            if hasattr(os, 'SEEK_HOLE') and hasattr(os, 'SEEK_DATA'):
+                with open(filename, 'rb') as fd:
+                    # only check if the first hole is as expected, because the 2nd hole check
+                    # is problematic on xfs due to its "dynamic speculative EOF preallocation
+                    try:
+                        self.assert_equal(fd.seek(0, os.SEEK_HOLE), 0)
+                        self.assert_equal(fd.seek(0, os.SEEK_DATA), hole_size)
+                    except OSError:
+                        # does not really support SEEK_HOLE/SEEK_DATA
+                        pass
 
     def test_unusual_filenames(self):
         filenames = ['normal', 'with some blanks', '(with_parens)', ]


Test patch:
Repository: borgbackup/borg

Issue:
avoid F405 failures caused by new flake8 version
flake8 is unhappy about our `from x import *` imports.

that currently fails all test runs on travis.



Discussion:
simplest solution is to switch off F405.


Resolving patch:
diff --git a/borg/helpers.py b/borg/helpers.py
--- a/borg/helpers.py
+++ b/borg/helpers.py
@@ -559,7 +559,7 @@ def format_line(format, data):
     except (KeyError, ValueError) as e:
         # this should catch format errors
         print('Error in lineformat: "{}" - reason "{}"'.format(format, str(e)))
-    except:
+    except Exception as e:
         # something unexpected, print error and raise exception
         print('Error in lineformat: "{}" - reason "{}"'.format(format, str(e)))
         raise


Test patch:
Repository: borgbackup/borg

Issue:
TypeError: acl_access value must be surrogate-escaped str or None
`acl_get` in `platform/linux.pyx` returns type of `bytes` for `acl_access`, `acl_default`, `acl_extended`, `acl_nfs4` on Linux x86_64. Those `bytes` are not converted to `str` afterwards.

Relevant stacktrace:

```
File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 332, in _process
status = archive.process_file(path, st, cache, self.ignore_inode)
File "/usr/lib/python3.5/site-packages/borg/archive.py", line 669, in process_file
item.update(self.stat_attrs(st, path))
File "/usr/lib/python3.5/site-packages/borg/item.py", line 40, in update
setattr(self, self._check_key(k), v)
File "/usr/lib/python3.5/site-packages/borg/item.py", line 95, in _set
raise TypeError(type_error_msg)
TypeError: acl_access value must be surrogate-escaped str or None
```



Discussion:

Resolving patch:
diff --git a/src/borg/item.py b/src/borg/item.py
--- a/src/borg/item.py
+++ b/src/borg/item.py
@@ -130,10 +130,10 @@ class Item(PropDict):
     user = PropDict._make_property('user', (str, type(None)), 'surrogate-escaped str or None', encode=safe_encode, decode=safe_decode)
     group = PropDict._make_property('group', (str, type(None)), 'surrogate-escaped str or None', encode=safe_encode, decode=safe_decode)
 
-    acl_access = PropDict._make_property('acl_access', str, 'surrogate-escaped str', encode=safe_encode, decode=safe_decode)
-    acl_default = PropDict._make_property('acl_default', str, 'surrogate-escaped str', encode=safe_encode, decode=safe_decode)
-    acl_extended = PropDict._make_property('acl_extended', str, 'surrogate-escaped str', encode=safe_encode, decode=safe_decode)
-    acl_nfs4 = PropDict._make_property('acl_nfs4', str, 'surrogate-escaped str', encode=safe_encode, decode=safe_decode)
+    acl_access = PropDict._make_property('acl_access', bytes)
+    acl_default = PropDict._make_property('acl_default', bytes)
+    acl_extended = PropDict._make_property('acl_extended', bytes)
+    acl_nfs4 = PropDict._make_property('acl_nfs4', bytes)
 
     mode = PropDict._make_property('mode', int)
     uid = PropDict._make_property('uid', int)


Test patch:
Repository: borgbackup/borg

Issue:
read-special mode behaves inconsistent
Moved from #1003:

@gpoudrel said:

I have a case where I backup a special device 
/dev/vg/lv-server-snap which is a symlink to /dev/dm-10

I use borg to backup /dev/vg/lv-server it follows the symlink and backup the special device.

```
# borg create --read-special /repo::today lvdisplay.txt /dev/vg/lv-server-snap
```

I mount the archive, it is odd.

```
# borg mount /repo::today /mnt
# ls /mnt/dev/
dev tmp
# ls -l /mnt/dev/vg
ls: cannot read symbolic link /mnt/dev/vg/lv-server-snap: Input/output error
ls: reading directory /mnt/dev/vg/: Software caused connection abort
total 20971520
lrwxrwxrwx 1 root root 21474836480 Jun 28 05:05 lv-server-snap
```

When I use a device which is not a symlink it works.



Discussion:
It is likely because we `stat()` the symlink, but in `--read-special` mode, we `open()` a regular file (using the symlink as path).

`open()` then follows the symlink and reads the file (or in this case: device) it points to.

But: the `stat()` call we do, does not follow the symlink, but stats the symlink itself, which is inconsistent in this case.

So, in `--read-special` mode, we should follow the symlink for `stat()` also.


Resolving patch:
diff --git a/borg/archiver.py b/borg/archiver.py
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -256,7 +256,15 @@ def _process(self, archive, cache, matcher, exclude_caches, exclude_if_present,
             return
 
         try:
-            st = os.lstat(path)
+            # usually, do not follow symlinks (if we have a symlink, we want to
+            # backup it as such).
+            # but if we are in --read-special mode, we later process <path> as
+            # a regular file (we open and read the symlink target file's content).
+            # thus, in read_special mode, we also want to stat the symlink target
+            # file, for consistency. if we did not, we also have issues extracting
+            # this file, as it would be in the archive as a symlink, not as the
+            # target's file type (which could be e.g. a block device).
+            st = os.stat(path, follow_symlinks=read_special)
         except OSError as e:
             self.print_warning('%s: %s', path, e)
             return


Test patch: