internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: borgbackup/borg

Issue:
`--dry-run` does not show exclude tag file being present in the list of archived files as per `--keep-exclude-tags`
## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

ISSUE

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).

```
borg 1.1.16
```

#### Operating system (distribution) and version.

Arch Linux

#### Hardware / network configuration, and filesystems used.

EXT4

#### Full borg commandline that lead to the problem (leave away excludes and passwords)

```
borg ... --dry-run ... --exclude-if-present .nobackup --keep-exclude-tags ...
```

## Describe the problem you're observing.

The `.nobackup` file is not present in the list of output files to be backed up.


Discussion:

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -791,8 +791,9 @@ def _rec_walk(self, *, path, parent_fd, name, fso, cache, matcher,
                             # if we are already recursing in an excluded dir, we do not need to do anything else than
                             # returning (we do not need to archive or recurse into tagged directories), see #3991:
                             if not recurse_excluded_dir:
-                                if keep_exclude_tags and not dry_run:
-                                    fso.process_dir_with_fd(path=path, fd=child_fd, st=st)
+                                if keep_exclude_tags:
+                                    if not dry_run:
+                                        fso.process_dir_with_fd(path=path, fd=child_fd, st=st)
                                     for tag_name in tag_names:
                                         tag_path = os.path.join(path, tag_name)
                                         self._rec_walk(


Test patch:
Repository: borgbackup/borg

Issue:
require py37 (master branch / borg 1.2)
considering that py36 runs out of support 2021-12-23 and creating the next release (1.2) is still todo, i guess we could also drop support for py36 and require py37 as the minimum for borg >= 1.2.

anyone seeing a problem with this?

everybody who is still on a such old python version seems to prefer older / stable stuff, so can as well use borg 1.1.x. 


Discussion:
https://docs.python.org/3/whatsnew/3.7.html

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -271,7 +271,6 @@ def members_appended(*ds):
         'Operating System :: POSIX :: Linux',
         'Programming Language :: Python',
         'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.6',
         'Programming Language :: Python :: 3.7',
         'Programming Language :: Python :: 3.8',
         'Programming Language :: Python :: 3.9',
@@ -300,5 +299,5 @@ def members_appended(*ds):
     setup_requires=['setuptools_scm>=1.7'],
     install_requires=install_requires,
     extras_require=extras_require,
-    python_requires='>=3.6',
+    python_requires='>=3.7',
 )
diff --git a/src/borg/testsuite/archiver.py b/src/borg/testsuite/archiver.py
--- a/src/borg/testsuite/archiver.py
+++ b/src/borg/testsuite/archiver.py
@@ -4322,13 +4322,7 @@ def basic_parser(self):
 
     @pytest.fixture
     def subparsers(self, basic_parser):
-        if sys.version_info >= (3, 7):
-            # py37 pre-release defaults to unwanted required=True, in 3.7.0+ it was fixed to =False
-            return basic_parser.add_subparsers(title='required arguments', metavar='<command>', required=False)
-        else:
-            # py36 does not support required=... argument (but behaves like required=False).
-            # note: use below call for 3.6 and 3.7 when there are no alphas/betas/RCs of 3.7.0 around any more.
-            return basic_parser.add_subparsers(title='required arguments', metavar='<command>')
+        return basic_parser.add_subparsers(title='required arguments', metavar='<command>')
 
     @pytest.fixture
     def parser(self, basic_parser):


Test patch:
Repository: borgbackup/borg

Issue:
Example path for --exclude-from
<!--
Thank you for reporting an issue.

*IMPORTANT* -  *before* creating a new issue please look around:
 - Borgbackup documentation: http://borgbackup.readthedocs.io/en/stable/index.html
 - FAQ: https://borgbackup.readthedocs.io/en/stable/faq.html
 and
 - open issues in Github tracker: https://github.com/borgbackup/borg/issues
  
If you cannot find a similar problem, then create a new issue.

Please fill in as much of the template as possible.
-->

## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

ISSUE

## Describe the problem you're observing.

I found it quite hard to find out how to use/allow spaces in --exclude-from file, one extra example in manual/help might help a lot:

# No need to escape spaces 
/mnt/c/System Volume Information

(not having to escape spaces is quite counterintuitive :)




Discussion:
Well, if you consider that parsing of arguments and splitting at spaces is a feature of your shell (and quoting is for working around that feature), it is quite intuitive.

The contents of --exclude-from or --patterns-from files are not processed by your shell, but directly read by borg.

OTOH, when entering `borg ... <arguments>` on the command line or in a shell script, the arguments *are* processed by the shell.

Taking this one, documenting both patterns files and exclude files.
From what I understand (and from my tests) adding a space is not a problem since everything is handled by borg which reads one entry per line.

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -2369,7 +2369,7 @@ def do_break_lock(self, args, repository):
             fm:aa:something/*
             re:^home/[^/]\\.tmp/
             sh:home/*/.thumbnails
-             # Example with spaces, no need to escape as it is processed by borg
+            # Example with spaces, no need to escape as it is processed by borg
             some file with spaces.txt
             EOF
             $ borg create --exclude-from exclude.txt backup /


Test patch:
Repository: borgbackup/borg

Issue:
docs: clarify borg check --verify-data
@andrewddavis in https://github.com/borgbackup/borg/discussions/5782#discussioncomment-734601 :
```
IMO, the man page for borg-check makes it clear that --verify-data applies
only to the archives part under the option's description:

  --verify-data: perform cryptographic archive data integrity verification (conflicts with --repository-only)

However, this could be made more clear in the main "description" section of
the man page, and the online documentation for borg, by making the paragraph
about --verify-data a bullet point of the archives check section, and including the
part about "conflicts with --repository-only".
```


Discussion:

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -3072,6 +3072,10 @@ def define_borg_mount(parser):
         - In repair mode, when all the archives were checked, orphaned chunks are deleted
           from the repo. One cause of orphaned chunks are input file related errors (like
           read errors) in the archive creation process.
+        - In verify-data mode, a complete cryptographic verification of the archive data
+          integrity is performed. This conflicts with ``--repository-only`` as this mode
+          only makes sense if the archive checks are enabled. The full details of this mode
+          are documented below.
         - If checking a remote repo via ``ssh:``, the archive check is executed on the
           client machine because it requires decryption, and this is always done client-side
           as key access is needed.


Test patch:
Repository: borgbackup/borg

Issue:
reduce startup time
borg backup space providers may want to reduce startup time / cpu / I/O because they run a lot of borg processes in parallel.

`borg.selftest` runs some unit-tests to make sure some fundamental stuff is working ok.
at bigger sites, this could be just done at install / testing time and skipped in production.

skipping should be possible without modifying the borg source, maybe via a BORG_SKIP_SELFTESTS env variable.


Discussion:
Never thought about this, but could be useful. Since many connections I see are very short-lived.

Resolving patch:
diff --git a/src/borg/selftest.py b/src/borg/selftest.py
--- a/src/borg/selftest.py
+++ b/src/borg/selftest.py
@@ -12,8 +12,7 @@
 variable. SELFTEST_COUNT must be updated if new tests are added or removed to or from any of the tests
 used here.
 """
-
-
+import os
 import sys
 import time
 from unittest import TestResult, TestSuite, defaultTestLoader
@@ -56,6 +55,9 @@ def successful_test_count(self):
 
 
 def selftest(logger):
+    if os.environ.get('BORG_SELFTEST') == 'disabled':
+        logger.debug("borg selftest disabled via BORG_SELFTEST env variable")
+        return
     selftest_started = time.perf_counter()
     result = SelfTestResult()
     test_suite = TestSuite()
diff --git a/src/borg/testsuite/archiver.py b/src/borg/testsuite/archiver.py
--- a/src/borg/testsuite/archiver.py
+++ b/src/borg/testsuite/archiver.py
@@ -251,6 +251,7 @@ def setUp(self):
         os.environ['BORG_CHECK_I_KNOW_WHAT_I_AM_DOING'] = 'YES'
         os.environ['BORG_DELETE_I_KNOW_WHAT_I_AM_DOING'] = 'YES'
         os.environ['BORG_PASSPHRASE'] = 'waytooeasyonlyfortests'
+        os.environ['BORG_SELFTEST'] = 'disabled'
         self.archiver = not self.FORK_DEFAULT and Archiver() or None
         self.tmpdir = tempfile.mkdtemp()
         self.repository_path = os.path.join(self.tmpdir, 'repository')


Test patch:
Repository: borgbackup/borg

Issue:
find_library() needs shell
borg users are patching away the find_library call so that they don't need a shell in the chroot they use.

there should be a way to do this without patching borg source code.

maybe via a BORG_LIBC env variable?


Discussion:

Resolving patch:
diff --git a/src/borg/xattr.py b/src/borg/xattr.py
--- a/src/borg/xattr.py
+++ b/src/borg/xattr.py
@@ -62,21 +62,22 @@ def get_all(path, follow_symlinks=True):
             return {}
 
 
-libc_name = find_library('c')
+HINT_MSG = "Try installing ldconfig, gcc/cc or objdump or use BORG_LIBC."
+LIBC_NOT_FOUND_NO_FALLBACK_MSG = "Can't find C library. No fallback known. " + HINT_MSG
+LIBC_NOT_FOUND_FNAME_MSG = "Can't find C library [%s]. " + HINT_MSG
+
+libc_name = os.environ.get('BORG_LIBC') or find_library('c')
 if libc_name is None:
     # find_library didn't work, maybe we are on some minimal system that misses essential
     # tools used by find_library, like ldconfig, gcc/cc, objdump.
     # so we can only try some "usual" names for the C library:
     if sys.platform.startswith('linux'):
         libc_name = 'libc.so.6'
-    elif sys.platform.startswith(('freebsd', 'netbsd')):
-        libc_name = 'libc.so'
     elif sys.platform == 'darwin':
         libc_name = 'libc.dylib'
     else:
-        msg = "Can't find C library. No fallback known. Try installing ldconfig, gcc/cc or objdump."
-        print(msg, file=sys.stderr)  # logger isn't initialized at this stage
-        raise Exception(msg)
+        print(LIBC_NOT_FOUND_NO_FALLBACK_MSG, file=sys.stderr)  # logger isn't initialized at this stage
+        raise Exception(LIBC_NOT_FOUND_NO_FALLBACK_MSG)
 
 # If we are running with fakeroot on Linux, then use the xattr functions of fakeroot. This is needed by
 # the 'test_extract_capabilities' test, but also allows xattrs to work with fakeroot on Linux in normal use.
@@ -102,8 +103,7 @@ def get_all(path, follow_symlinks=True):
 try:
     libc = CDLL(libc_name, use_errno=True)
 except OSError as e:
-    msg = "Can't find C library [%s]. Try installing ldconfig, gcc/cc or objdump." % e
-    raise Exception(msg)
+    raise Exception(LIBC_NOT_FOUND_FNAME_MSG % e)
 
 
 def split_string0(buf):


Test patch:
Repository: borgbackup/borg

Issue:
Document renaming repository
## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

Request for documentation

## Describe the problem you're observing.

Could not find how to rename a repository, what i found is that all it takes is:

**rename the repository on the server
change path in ~/.cache/borg/*/config**






Discussion:
if you just rename the directory (or move the whole thing to a different ssh location), borg will warn you next time when you talk to the repo at the new location that the repo has moved. if you confirm, you should be done.

please try that. not sure if we need to document something if you can just do it, confirm and you're done!?
The fact I ended up messing about kinda suggests it's worth documenting. It seems sensible to not just try renaming directories without first checking how it's supposed to be done.
Figured out a useful thing is trying a borg list after renaming, to get prompted and have the rename checked as well, without interfering with the (cron schedule).
I also forgot about BORG_RELOCATED_REPO_ACCESS_IS_OK.




Hello!, I would like to work on this issue today on the EuroPython sprint.

sure, just do it! :-)
note:

that env variable is good when doing a mass rename or when needing automation. if you do not set that env var, borg will just interactively ask whether it is ok.

about generation of docs you have to know that some part is auto-generated from text in archiver.py and some other parts are taken from the docs/ directory.

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -2367,8 +2367,8 @@ def do_break_lock(self, args, repository):
             /home/*/junk
             *.tmp
             fm:aa:something/*
-            re:^home/[^/]\\.tmp/
-            sh:home/*/.thumbnails
+            re:^/home/[^/]+\\.tmp/
+            sh:/home/*/.thumbnails
             # Example with spaces, no need to escape as it is processed by borg
             some file with spaces.txt
             EOF


Test patch:
Repository: borgbackup/borg

Issue:
reduce startup time
borg backup space providers may want to reduce startup time / cpu / I/O because they run a lot of borg processes in parallel.

`borg.selftest` runs some unit-tests to make sure some fundamental stuff is working ok.
at bigger sites, this could be just done at install / testing time and skipped in production.

skipping should be possible without modifying the borg source, maybe via a BORG_SKIP_SELFTESTS env variable.


Discussion:
Never thought about this, but could be useful. Since many connections I see are very short-lived.

Resolving patch:
diff --git a/src/borg/selftest.py b/src/borg/selftest.py
--- a/src/borg/selftest.py
+++ b/src/borg/selftest.py
@@ -12,8 +12,7 @@
 variable. SELFTEST_COUNT must be updated if new tests are added or removed to or from any of the tests
 used here.
 """
-
-
+import os
 import sys
 import time
 from unittest import TestResult, TestSuite, defaultTestLoader
@@ -56,6 +55,9 @@ def successful_test_count(self):
 
 
 def selftest(logger):
+    if os.environ.get('BORG_SELFTEST') == 'disabled':
+        logger.debug("borg selftest disabled via BORG_SELFTEST env variable")
+        return
     selftest_started = time.perf_counter()
     result = SelfTestResult()
     test_suite = TestSuite()
diff --git a/src/borg/testsuite/archiver.py b/src/borg/testsuite/archiver.py
--- a/src/borg/testsuite/archiver.py
+++ b/src/borg/testsuite/archiver.py
@@ -249,6 +249,7 @@ def setUp(self):
         os.environ['BORG_CHECK_I_KNOW_WHAT_I_AM_DOING'] = 'YES'
         os.environ['BORG_DELETE_I_KNOW_WHAT_I_AM_DOING'] = 'YES'
         os.environ['BORG_PASSPHRASE'] = 'waytooeasyonlyfortests'
+        os.environ['BORG_SELFTEST'] = 'disabled'
         self.archiver = not self.FORK_DEFAULT and Archiver() or None
         self.tmpdir = tempfile.mkdtemp()
         self.repository_path = os.path.join(self.tmpdir, 'repository')


Test patch:
Repository: borgbackup/borg

Issue:
[doc] typo / inaccuracy on 'borg benchmark' usage page
[EDIT - originally I misunderstood the docs. They are correct but could benefit from maybe adding an example command ? ]

When using wrong arguments or with permission issues, `borg benchmark crud` can exit without deleting the `borg-test-data/` it created, leading to some head scratching. It would be a nice addition to cleanup that folder systematically on exit (clean or error), unless there's some use case for the current bheavior ?


Discussion:
Maybe we can use some `try / finally` or so in the code to make sure the cleanup happens.
@ThomasWaldmann can I be assigned to this issue?
@brunobehnken sure! :-)

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -456,16 +456,18 @@ def measurement_run(repo, path):
 
         @contextmanager
         def test_files(path, count, size, random):
-            path = os.path.join(path, 'borg-test-data')
-            os.makedirs(path)
-            z_buff = None if random else memoryview(zeros)[:size] if size <= len(zeros) else b'\0' * size
-            for i in range(count):
-                fname = os.path.join(path, 'file_%d' % i)
-                data = z_buff if not random else os.urandom(size)
-                with SyncFile(fname, binary=True) as fd:  # used for posix_fadvise's sake
-                    fd.write(data)
-            yield path
-            shutil.rmtree(path)
+            try:
+                path = os.path.join(path, 'borg-test-data')
+                os.makedirs(path)
+                z_buff = None if random else memoryview(zeros)[:size] if size <= len(zeros) else b'\0' * size
+                for i in range(count):
+                    fname = os.path.join(path, 'file_%d' % i)
+                    data = z_buff if not random else os.urandom(size)
+                    with SyncFile(fname, binary=True) as fd:  # used for posix_fadvise's sake
+                        fd.write(data)
+                yield path
+            finally:
+                shutil.rmtree(path)
 
         if '_BORG_BENCHMARK_CRUD_TEST' in os.environ:
             tests = [


Test patch:
Repository: borgbackup/borg

Issue:
pyinstaller ssh/openssh library loading issue / OpenSSL version mismatch
From the mailing list, reported by cherio:
```
Remote: OpenSSL version mismatch. Built against 1010106f, you have 101000cf
```
It seems issue #514 is there again.
This was observed using `borg-linux64` and `borg-linux64.tgz` 1.1.15 and 1.1.16, but not with 1.1.14.

1.1.14 was last borg built with pyinstaller 3.2.1.
1.1.15 was built with 4.0, 1.1.16 was built with 4.2.

The user uses borg binaries on Ubuntu 20.04. 
```
> openssl version
OpenSSL 1.1.1f  31 Mar 2020

> ssh -V
OpenSSH_8.2p1 Ubuntu-4ubuntu0.2, OpenSSL 1.1.1f  31 Mar 2020

> python
Python 3.8.5 (default, May 27 2021, 13:30:53)
[GCC 9.3.0] on linux
>>> import ssl
>>> ssl.OPENSSL_VERSION
'OpenSSL 1.1.1f  31 Mar 2020'
```
So, system openssl is 1.1.1f, system ssh wants 1.1.1f.
Very likely the pyi bundle has 1.1.0 (for bundled python).

Related borg code:
- https://github.com/borgbackup/borg/blob/1.1.16/src/borg/helpers.py#L2407
- https://github.com/borgbackup/borg/blob/1.1.16/src/borg/remote.py#L553


Discussion:
I think the bug is slightly different than described above (update: cherio confirmed it is only in single-dir mode):

The problem is that LD_LIBRARY_PATH_ORIG is only provided by pyinstaller, if there was a LD_LIBRARY_PATH in the original env. If not, things get complicated:

https://github.com/borgbackup/borg/blob/1.1.16/src/borg/helpers.py#L2426

We check there if the LDLP contains `_MEI`, but that is only the case for the single-file pyi bundle.
For the single-dir bundle, it is just the absolute path of the unpacked bundle (not _MEI temp path).

Workaround for using the single-dir bundle, avoiding the problematic part of the code:
```
LD_LIBRARY_PATH=/invalid borg ... ssh://user@host/repo ...
```
see also there: https://github.com/pyinstaller/pyinstaller/issues/2892

Resolving patch:
diff --git a/src/borg/helpers.py b/src/borg/helpers.py
--- a/src/borg/helpers.py
+++ b/src/borg/helpers.py
@@ -2445,10 +2445,9 @@ def prepare_subprocess_env(system, env=None):
             #    in the original env (in this case, the pyinstaller bootloader
             #    does *not* put ..._ORIG into the env either).
             #    in this case, we must kill LDLP.
-            # The directory used by pyinstaller is created by mkdtemp("_MEIXXXXXX"),
-            # we can use that to differentiate between the cases.
+            #    We can recognize this via sys.frozen and sys._MEIPASS being set.
             lp = env.get(lp_key)
-            if lp is not None and re.search(r'/_MEI......', lp):
+            if lp is not None and getattr(sys, 'frozen', False) and hasattr(sys, '_MEIPASS'):
                 env.pop(lp_key)
     # security: do not give secrets to subprocess
     env.pop('BORG_PASSPHRASE', None)


Test patch:
Repository: borgbackup/borg

Issue:
[doc] typo / inaccuracy on 'borg benchmark' usage page
[EDIT - originally I misunderstood the docs. They are correct but could benefit from maybe adding an example command ? ]

When using wrong arguments or with permission issues, `borg benchmark crud` can exit without deleting the `borg-test-data/` it created, leading to some head scratching. It would be a nice addition to cleanup that folder systematically on exit (clean or error), unless there's some use case for the current bheavior ?


Discussion:
Maybe we can use some `try / finally` or so in the code to make sure the cleanup happens.
@ThomasWaldmann can I be assigned to this issue?
@brunobehnken sure! :-)

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -463,15 +463,17 @@ def measurement_run(repo, path):
 
         @contextmanager
         def test_files(path, count, size, random):
-            path = os.path.join(path, 'borg-test-data')
-            os.makedirs(path)
-            for i in range(count):
-                fname = os.path.join(path, 'file_%d' % i)
-                data = b'\0' * size if not random else os.urandom(size)
-                with SyncFile(fname, binary=True) as fd:  # used for posix_fadvise's sake
-                    fd.write(data)
-            yield path
-            shutil.rmtree(path)
+            try:
+                path = os.path.join(path, 'borg-test-data')
+                os.makedirs(path)
+                for i in range(count):
+                    fname = os.path.join(path, 'file_%d' % i)
+                    data = b'\0' * size if not random else os.urandom(size)
+                    with SyncFile(fname, binary=True) as fd:  # used for posix_fadvise's sake
+                        fd.write(data)
+                yield path
+            finally:
+                shutil.rmtree(path)
 
         if '_BORG_BENCHMARK_CRUD_TEST' in os.environ:
             tests = [


Test patch: