internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: borgbackup/borg

Issue:
goodbye wheezy
the binaries for 1.0.x and 1.1.x are currently built using pyinstaller on debian 7 "wheezy" (64 and 32bit).

but, wheezy will run out of LTS support 2018-05-31 and this is a problem because we bundle python and libraries from the dist into the borg binary. https://wiki.debian.org/LTS

so, the plan currently is:
- stop releasing binaries for borg 1.0.x (unless severe [security or other] bugs are discovered, maybe 1.0.12 is the last release anyway) 
- migrate 1.1-maint binary release building to debian 8 "jessie"
- try to fix #2652 to not require a too new glibc version without need
- drop wheezy from release testing (vagrant)


Discussion:
about debian stretch: see #4046

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -7,7 +7,6 @@
 from datetime import datetime
 from glob import glob
 
-from distutils.command.build import build
 from distutils.core import Command
 
 import textwrap
@@ -870,6 +869,7 @@ def run(self):
         'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: 3.6',
         'Programming Language :: Python :: 3.7',
+        'Programming Language :: Python :: 3.8',
         'Topic :: Security :: Cryptography',
         'Topic :: System :: Archiving :: Backup',
     ],


Test patch:
Repository: borgbackup/borg

Issue:
part files are not regular files for --read-special
https://github.com/borgbackup/borg/blob/master/src/borg/archive.py#L1301

when using `--read-special` it makes the item look like a regular file there, but that only is effective for the final item.

if there are .part files created at checkpoints, these will still have the mode of the special file.


Discussion:

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -1249,6 +1249,11 @@ def process_file(self, *, path, parent_fd, name, st, cache, flags=flags_normal):
                     st = stat_update_check(st, os.fstat(fd))
                 item.update(self.metadata_collector.stat_simple_attrs(st))
                 is_special_file = is_special(st.st_mode)
+                if is_special_file:
+                    # we process a special file like a regular file. reflect that in mode,
+                    # so it can be extracted / accessed in FUSE mount like a regular file.
+                    # this needs to be done early, so that part files also get the patched mode.
+                    item.mode = stat.S_IFREG | stat.S_IMODE(item.mode)
                 if not hardlinked or hardlink_master:
                     if not is_special_file:
                         path_hash = self.key.id_hash(safe_encode(os.path.join(self.cwd, path)))
@@ -1298,10 +1303,6 @@ def process_file(self, *, path, parent_fd, name, st, cache, flags=flags_normal):
                     self.stats.nfiles += 1
                 item.update(self.metadata_collector.stat_ext_attrs(st, path, fd=fd))
                 item.get_size(memorize=True)
-                if is_special_file:
-                    # we processed a special file like a regular file. reflect that in mode,
-                    # so it can be extracted / accessed in FUSE mount like a regular file:
-                    item.mode = stat.S_IFREG | stat.S_IMODE(item.mode)
                 return status
 
 


Test patch:
Repository: borgbackup/borg

Issue:
part files are not regular files for --read-special
https://github.com/borgbackup/borg/blob/master/src/borg/archive.py#L1301

when using `--read-special` it makes the item look like a regular file there, but that only is effective for the final item.

if there are .part files created at checkpoints, these will still have the mode of the special file.


Discussion:

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -1038,6 +1038,11 @@ def process_file(self, path, st, cache):
         with self.create_helper(path, st, None) as (item, status, hardlinked, hardlink_master):  # no status yet
             item.update(self.stat_simple_attrs(st))
             is_special_file = is_special(st.st_mode)
+            if is_special_file:
+                # we process a special file like a regular file. reflect that in mode,
+                # so it can be extracted / accessed in FUSE mount like a regular file.
+                # this needs to be done early, so that part files also get the patched mode.
+                item.mode = stat.S_IFREG | stat.S_IMODE(item.mode)
             if not hardlinked or hardlink_master:
                 if not is_special_file:
                     path_hash = self.key.id_hash(safe_encode(os.path.join(self.cwd, path)))
@@ -1076,10 +1081,6 @@ def process_file(self, path, st, cache):
                 self.stats.nfiles += 1
             item.update(self.stat_ext_attrs(st, path))
             item.get_size(memorize=True)
-            if is_special_file:
-                # we processed a special file like a regular file. reflect that in mode,
-                # so it can be extracted / accessed in FUSE mount like a regular file:
-                item.mode = stat.S_IFREG | stat.S_IMODE(item.mode)
             return status
 
     @staticmethod


Test patch:
Repository: borgbackup/borg

Issue:
master: drop py35 support
py35 is EOL now, so we'll drop support for it in master branch.


Discussion:
https://docs.python.org/3/whatsnew/3.6.html

notable, we can now use:
- f-strings
- variable annotations
- secrets module
- hashlib has blake2 (and scrypt)


Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -266,7 +266,6 @@ def members_appended(*ds):
         'Operating System :: POSIX :: Linux',
         'Programming Language :: Python',
         'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: 3.6',
         'Programming Language :: Python :: 3.7',
         'Programming Language :: Python :: 3.8',
@@ -294,5 +293,5 @@ def members_appended(*ds):
     setup_requires=['setuptools_scm>=1.7'],
     install_requires=install_requires,
     extras_require=extras_require,
-    python_requires='>=3.5',
+    python_requires='>=3.6',
 )


Test patch:
Repository: borgbackup/borg

Issue:
check the docs + faq about files cache
there are frequent questions like "why is borg slow for me, considering all my files modified" when the underlying filesystem has issues with:
- stable inode numbers
- when people do not always have the same full/absolute paths
- when they have ctime issues (like cron scripts running chown -R/chmod -R)

there are quite some related posts on this issue tracker (also see closed issues) and on the borg mailing list.

the usual way to debug / fix this is using `borg create --list ...` to see file status and whether a lot of files are considered modified although they should not be plus using the `stat` command on these files to see why borg considers them changed or has a files cache miss (if not obvious, e.g. due to full path change).

fixing the issue is sometimes out of borg scope (e.g. mounting always to same mountpoint, not doing chown/chmod by cron) and sometimes tweaking borg's `--files-cache` (e.g. if the source fs does not have stable inode numbers).

check our docs (esp. borg create and faq) and make them more complete / better discoverable, so less people ask these questions.



Discussion:

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -3281,6 +3281,7 @@ def define_archive_filters_group(subparser, *, sort_by=True, first_last=True):
           as it can not be set from userspace. But, a metadata-only change will already
           update the ctime, so there might be some unnecessary chunking/hashing even
           without content changes. Some filesystems do not support ctime (change time).
+          E.g. doing a chown or chmod to a file will change its ctime.
         - mtime usually works and only updates if file contents were changed. But mtime
           can be arbitrarily set from userspace, e.g. to set mtime back to the same value
           it had before a content change happened. This can be used maliciously as well as


Test patch:
Repository: borgbackup/borg

Issue:
Tar export on Ubuntu 20.04 outputs @paxheader files
## Have you checked borgbackup docs, FAQ, and open Github issues?
Yes

## Is this a BUG / ISSUE report or a QUESTION?
ISSUE

## System information. For client/server mode post info for both machines.
borg 1.1.13
Ubuntu 20.04.1 LTS

#### Full borg commandline that lead to the problem (leave away excludes and passwords)
borg export-tar --tar-filter="gzip -3" --strip-components 6 /var/borg/$App::$DownloadName $DownloadName.tar.gz

## Describe the problem you're observing.
Gzip/tar file contains lots of @PaxHeader files and missing file extensions.

Is there a way to force GNU export over POSIX from tar in borg? Tar is set to GNU by default on the system but I haven't found how to change this within the borg files, there seems to be something overriding this. I have borg installed and exporting ok on older Ubuntu LTS versions, and can only replicate this issue on 20.04





Discussion:
Sorry, I don't get what the problem is...
I'm still researching this myself. The export-tar command through borg appears to be using the posix format instead of gnu which is adding thousands of additional @paxheader files to the gzip file along with incomplete file extensions on some files, which makes the gzip file unusable.

I have been able to replicate this by using the --format=posix option on a tar request eg:
tar -czvf archive.tar.gz test-folder/ --format=posix

Using --format=gnu with tar should fix the above issue but I'm unable to add that option to the export-tar command, nor can I pinpoint why tar is using that format through borg. Perhaps I can create a work around with piping the extract command to gzip in the meantime. 
It's Python's fault. Borg uses the default format of the tarfile module, which changed defaults in 3.8:

> Changed in version 3.8: The default format for new archives was changed to PAX_FORMAT from GNU_FORMAT.

export-tar probably needs a --format switch now, or a maintenance release making the format GNU_FORMAT always.

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -933,7 +933,7 @@ def do_export_tar(self, args, repository, manifest, key, archive):
 
         # The | (pipe) symbol instructs tarfile to use a streaming mode of operation
         # where it never seeks on the passed fileobj.
-        tar = tarfile.open(fileobj=tarstream, mode='w|')
+        tar = tarfile.open(fileobj=tarstream, mode='w|', format=tarfile.GNU_FORMAT)
 
         self._export_tar(args, archive, tar)
 


Test patch:
Repository: borgbackup/borg

Issue:
Tar export on Ubuntu 20.04 outputs @paxheader files
## Have you checked borgbackup docs, FAQ, and open Github issues?
Yes

## Is this a BUG / ISSUE report or a QUESTION?
ISSUE

## System information. For client/server mode post info for both machines.
borg 1.1.13
Ubuntu 20.04.1 LTS

#### Full borg commandline that lead to the problem (leave away excludes and passwords)
borg export-tar --tar-filter="gzip -3" --strip-components 6 /var/borg/$App::$DownloadName $DownloadName.tar.gz

## Describe the problem you're observing.
Gzip/tar file contains lots of @PaxHeader files and missing file extensions.

Is there a way to force GNU export over POSIX from tar in borg? Tar is set to GNU by default on the system but I haven't found how to change this within the borg files, there seems to be something overriding this. I have borg installed and exporting ok on older Ubuntu LTS versions, and can only replicate this issue on 20.04





Discussion:
Sorry, I don't get what the problem is...
I'm still researching this myself. The export-tar command through borg appears to be using the posix format instead of gnu which is adding thousands of additional @paxheader files to the gzip file along with incomplete file extensions on some files, which makes the gzip file unusable.

I have been able to replicate this by using the --format=posix option on a tar request eg:
tar -czvf archive.tar.gz test-folder/ --format=posix

Using --format=gnu with tar should fix the above issue but I'm unable to add that option to the export-tar command, nor can I pinpoint why tar is using that format through borg. Perhaps I can create a work around with piping the extract command to gzip in the meantime. 
It's Python's fault. Borg uses the default format of the tarfile module, which changed defaults in 3.8:

> Changed in version 3.8: The default format for new archives was changed to PAX_FORMAT from GNU_FORMAT.

export-tar probably needs a --format switch now, or a maintenance release making the format GNU_FORMAT always.

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -884,7 +884,7 @@ def do_export_tar(self, args, repository, manifest, key, archive):
 
         # The | (pipe) symbol instructs tarfile to use a streaming mode of operation
         # where it never seeks on the passed fileobj.
-        tar = tarfile.open(fileobj=tarstream, mode='w|')
+        tar = tarfile.open(fileobj=tarstream, mode='w|', format=tarfile.GNU_FORMAT)
 
         self._export_tar(args, archive, tar)
 


Test patch:
Repository: borgbackup/borg

Issue:
check the docs + faq about files cache
there are frequent questions like "why is borg slow for me, considering all my files modified" when the underlying filesystem has issues with:
- stable inode numbers
- when people do not always have the same full/absolute paths
- when they have ctime issues (like cron scripts running chown -R/chmod -R)

there are quite some related posts on this issue tracker (also see closed issues) and on the borg mailing list.

the usual way to debug / fix this is using `borg create --list ...` to see file status and whether a lot of files are considered modified although they should not be plus using the `stat` command on these files to see why borg considers them changed or has a files cache miss (if not obvious, e.g. due to full path change).

fixing the issue is sometimes out of borg scope (e.g. mounting always to same mountpoint, not doing chown/chmod by cron) and sometimes tweaking borg's `--files-cache` (e.g. if the source fs does not have stable inode numbers).

check our docs (esp. borg create and faq) and make them more complete / better discoverable, so less people ask these questions.



Discussion:

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -3079,6 +3079,7 @@ def define_borg_mount(parser):
           as it can not be set from userspace. But, a metadata-only change will already
           update the ctime, so there might be some unnecessary chunking/hashing even
           without content changes. Some filesystems do not support ctime (change time).
+          E.g. doing a chown or chmod to a file will change its ctime.
         - mtime usually works and only updates if file contents were changed. But mtime
           can be arbitrarily set from userspace, e.g. to set mtime back to the same value
           it had before a content change happened. This can be used maliciously as well as


Test patch:
Repository: borgbackup/borg

Issue:
spurious failure in test_cache_files
current master branch on travis-ci:
```
_________________ ArchiverCorruptionTestCase.test_cache_files __________________

[gw0] linux -- Python 3.8.6 /home/travis/build/borgbackup/borg/.tox/py38/bin/python

self = <borg.testsuite.archiver.ArchiverCorruptionTestCase testMethod=test_cache_files>
    def test_cache_files(self):
        self.cmd('create', self.repository_location + '::test', 'input')
        self.corrupt(os.path.join(self.cache_path, 'files'))
        out = self.cmd('create', self.repository_location + '::test1', 'input')
        # borg warns about the corrupt files cache, but then continues without files cache.
>       assert 'files cache is corrupted' in out
E       AssertionError: assert 'files cache is corrupted' in ''
.tox/py38/lib/python3.8/site-packages/borg/testsuite/archiver.py:3714: AssertionError
```

Bounty: https://www.bountysource.com/issues/93666717-spurious-failure-in-test_cache_files

spurious failure in test_cache_files
current master branch on travis-ci:
```
_________________ ArchiverCorruptionTestCase.test_cache_files __________________

[gw0] linux -- Python 3.8.6 /home/travis/build/borgbackup/borg/.tox/py38/bin/python

self = <borg.testsuite.archiver.ArchiverCorruptionTestCase testMethod=test_cache_files>
    def test_cache_files(self):
        self.cmd('create', self.repository_location + '::test', 'input')
        self.corrupt(os.path.join(self.cache_path, 'files'))
        out = self.cmd('create', self.repository_location + '::test1', 'input')
        # borg warns about the corrupt files cache, but then continues without files cache.
>       assert 'files cache is corrupted' in out
E       AssertionError: assert 'files cache is corrupted' in ''
.tox/py38/lib/python3.8/site-packages/borg/testsuite/archiver.py:3714: AssertionError
```

Bounty: https://www.bountysource.com/issues/93666717-spurious-failure-in-test_cache_files



Discussion:

Resolving patch:
diff --git a/src/borg/testsuite/archiver.py b/src/borg/testsuite/archiver.py
--- a/src/borg/testsuite/archiver.py
+++ b/src/borg/testsuite/archiver.py
@@ -3686,10 +3686,12 @@ def setUp(self):
         self.cmd('init', '--encryption=repokey', self.repository_location)
         self.cache_path = json.loads(self.cmd('info', self.repository_location, '--json'))['cache']['path']
 
-    def corrupt(self, file):
+    def corrupt(self, file, amount=1):
         with open(file, 'r+b') as fd:
-            fd.seek(-1, io.SEEK_END)
-            fd.write(b'1')
+            fd.seek(-amount, io.SEEK_END)
+            corrupted = bytes(255-c for c in fd.read(amount))
+            fd.seek(-amount, io.SEEK_END)
+            fd.write(corrupted)
 
     def test_cache_chunks(self):
         self.corrupt(os.path.join(self.cache_path, 'chunks'))


Test patch:
Repository: borgbackup/borg

Issue:
spurious failure in test_cache_files
current master branch on travis-ci:
```
_________________ ArchiverCorruptionTestCase.test_cache_files __________________

[gw0] linux -- Python 3.8.6 /home/travis/build/borgbackup/borg/.tox/py38/bin/python

self = <borg.testsuite.archiver.ArchiverCorruptionTestCase testMethod=test_cache_files>
    def test_cache_files(self):
        self.cmd('create', self.repository_location + '::test', 'input')
        self.corrupt(os.path.join(self.cache_path, 'files'))
        out = self.cmd('create', self.repository_location + '::test1', 'input')
        # borg warns about the corrupt files cache, but then continues without files cache.
>       assert 'files cache is corrupted' in out
E       AssertionError: assert 'files cache is corrupted' in ''
.tox/py38/lib/python3.8/site-packages/borg/testsuite/archiver.py:3714: AssertionError
```

Bounty: https://www.bountysource.com/issues/93666717-spurious-failure-in-test_cache_files



Discussion:

Resolving patch:
diff --git a/src/borg/testsuite/archiver.py b/src/borg/testsuite/archiver.py
--- a/src/borg/testsuite/archiver.py
+++ b/src/borg/testsuite/archiver.py
@@ -3495,10 +3495,12 @@ def setUp(self):
         self.cmd('init', '--encryption=repokey', self.repository_location)
         self.cache_path = json.loads(self.cmd('info', self.repository_location, '--json'))['cache']['path']
 
-    def corrupt(self, file):
+    def corrupt(self, file, amount=1):
         with open(file, 'r+b') as fd:
-            fd.seek(-1, io.SEEK_END)
-            fd.write(b'1')
+            fd.seek(-amount, io.SEEK_END)
+            corrupted = bytes(255-c for c in fd.read(amount))
+            fd.seek(-amount, io.SEEK_END)
+            fd.write(corrupted)
 
     def test_cache_chunks(self):
         self.corrupt(os.path.join(self.cache_path, 'chunks'))


Test patch:
Repository: borgbackup/borg

Issue:
Example path for --exclude-from
<!--
Thank you for reporting an issue.

*IMPORTANT* -  *before* creating a new issue please look around:
 - Borgbackup documentation: http://borgbackup.readthedocs.io/en/stable/index.html
 - FAQ: https://borgbackup.readthedocs.io/en/stable/faq.html
 and
 - open issues in Github tracker: https://github.com/borgbackup/borg/issues
  
If you cannot find a similar problem, then create a new issue.

Please fill in as much of the template as possible.
-->

## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

ISSUE

## Describe the problem you're observing.

I found it quite hard to find out how to use/allow spaces in --exclude-from file, one extra example in manual/help might help a lot:

# No need to escape spaces 
/mnt/c/System Volume Information

(not having to escape spaces is quite counterintuitive :)




Discussion:
Well, if you consider that parsing of arguments and splitting at spaces is a feature of your shell (and quoting is for working around that feature), it is quite intuitive.

The contents of --exclude-from or --patterns-from files are not processed by your shell, but directly read by borg.

OTOH, when entering `borg ... <arguments>` on the command line or in a shell script, the arguments *are* processed by the shell.

Taking this one, documenting both patterns files and exclude files.
From what I understand (and from my tests) adding a space is not a problem since everything is handled by borg which reads one entry per line.

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -2227,6 +2227,8 @@ def do_break_lock(self, args, repository):
             fm:aa:something/*
             re:^home/[^/]\\.tmp/
             sh:home/*/.thumbnails
+            # Example with spaces, no need to escape as it is processed by borg
+            some file with spaces.txt
             EOF
             $ borg create --exclude-from exclude.txt backup /
 
@@ -3106,6 +3108,7 @@ def define_borg_mount(parser):
         exclusive because the data is not actually compressed and deduplicated during a dry run.
 
         See the output of the "borg help patterns" command for more help on exclude patterns.
+
         See the output of the "borg help placeholders" command for more help on placeholders.
 
         .. man NOTES


Test patch: