internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: borgbackup/borg

Issue:
do back/forward ports (16)
https://github.com/borgbackup/borg/issues?utf8=%E2%9C%93&q=label%3Abackport%2F1.0-maint
https://github.com/borgbackup/borg/issues?utf8=%E2%9C%93&q=label%3Abackport%2F1.1-maint
https://github.com/borgbackup/borg/issues?utf8=%E2%9C%93&q=label%3Aforwardport%2Fmaster

do 3 simple/small or 1 bigger/complex forward/backport from there.

---

:moneybag: [there is a bounty for this](https://www.bountysource.com/issues/69659840-do-back-forward-ports-16)



Discussion:

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -1990,7 +1990,7 @@ def do_break_lock(self, args, repository):
             any number of characters, '?' matching any single character, '[...]'
             matching any single character specified, including ranges, and '[!...]'
             matching any character not specified. For the purpose of these patterns,
-            the path separator ('\\' for Windows and '/' on other systems) is not
+            the path separator (backslash for Windows and '/' on other systems) is not
             treated specially. Wrap meta-characters in brackets for a literal
             match (i.e. `[?]` to match the literal character `?`). For a path
             to match a pattern, the full path must match, or it must match
@@ -2011,7 +2011,7 @@ def do_break_lock(self, args, repository):
             shell patterns regular expressions are not required to match the full
             path and any substring match is sufficient. It is strongly recommended to
             anchor patterns to the start ('^'), to the end ('$') or both. Path
-            separators ('\\' for Windows and '/' on other systems) in paths are
+            separators (backslash for Windows and '/' on other systems) in paths are
             always normalized to a forward slash ('/') before applying a pattern. The
             regular expression syntax is described in the `Python documentation for
             the re module <https://docs.python.org/3/library/re.html>`_.
diff --git a/src/borg/helpers/checks.py b/src/borg/helpers/checks.py
--- a/src/borg/helpers/checks.py
+++ b/src/borg/helpers/checks.py
@@ -33,7 +33,7 @@ def check_extension_modules():
         raise ExtensionModuleError
     if borg.crypto.low_level.API_VERSION != '1.1_02':
         raise ExtensionModuleError
-    if platform.API_VERSION != platform.OS_API_VERSION or platform.API_VERSION != '1.2_02':
+    if platform.API_VERSION != platform.OS_API_VERSION or platform.API_VERSION != '1.2_03':
         raise ExtensionModuleError
     if item.API_VERSION != '1.1_03':
         raise ExtensionModuleError
diff --git a/src/borg/platform/base.py b/src/borg/platform/base.py
--- a/src/borg/platform/base.py
+++ b/src/borg/platform/base.py
@@ -17,7 +17,7 @@
 are correctly composed into the base functionality.
 """
 
-API_VERSION = '1.2_02'
+API_VERSION = '1.2_03'
 
 fdatasync = getattr(os, 'fdatasync', os.fsync)
 


Test patch:
Repository: borgbackup/borg

Issue:
xattr test fails for non-root test runs
master branch: if you just start `tox`, you'll get 3 test fails caused by permission issues for xattr stuff.


Discussion:

Resolving patch:
diff --git a/src/borg/xattr.py b/src/borg/xattr.py
--- a/src/borg/xattr.py
+++ b/src/borg/xattr.py
@@ -2,15 +2,41 @@
 
 import errno
 import os
+import re
+import subprocess
+import sys
 import tempfile
 
+from distutils.version import LooseVersion
+
+from .helpers import prepare_subprocess_env
+
 from .logger import create_logger
 
 logger = create_logger()
 
 from .platform import listxattr, getxattr, setxattr, ENOATTR
 
-XATTR_FAKEROOT = True  # fakeroot with xattr support required (>= 1.20.2?)
+# If we are running with fakeroot on Linux, then use the xattr functions of fakeroot. This is needed by
+# the 'test_extract_capabilities' test, but also allows xattrs to work with fakeroot on Linux in normal use.
+# TODO: Check whether fakeroot supports xattrs on all platforms supported below.
+# TODO: If that's the case then we can make Borg fakeroot-xattr-compatible on these as well.
+XATTR_FAKEROOT = False
+if sys.platform.startswith('linux'):
+    LD_PRELOAD = os.environ.get('LD_PRELOAD', '')
+    preloads = re.split("[ :]", LD_PRELOAD)
+    for preload in preloads:
+        if preload.startswith("libfakeroot"):
+            env = prepare_subprocess_env(system=True)
+            fakeroot_output = subprocess.check_output(['fakeroot', '-v'], env=env)
+            fakeroot_version = LooseVersion(fakeroot_output.decode('ascii').split()[-1])
+            if fakeroot_version >= LooseVersion("1.20.2"):
+                # 1.20.2 has been confirmed to have xattr support
+                # 1.18.2 has been confirmed not to have xattr support
+                # Versions in-between are unknown
+                libc_name = preload
+                XATTR_FAKEROOT = True
+            break
 
 
 def is_enabled(path=None):


Test patch:
Repository: borgbackup/borg

Issue:
borg key export confusing error msg
from IRC:
```
borg key export REPO PATH
```

if the second argument is a directory it will throw an error.

I understand it needs a file, but the error it throws may be confusing to other users. :)

borg key export confusing error msg
from IRC:
```
borg key export REPO PATH
```

if the second argument is a directory it will throw an error.

I understand it needs a file, but the error it throws may be confusing to other users. :)



Discussion:

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -325,10 +325,14 @@ def do_key_export(self, args, repository):
             if not args.path:
                 self.print_error("output file to export key to expected")
                 return EXIT_ERROR
-            if args.qr:
-                manager.export_qr(args.path)
-            else:
-                manager.export(args.path)
+            try:
+                if args.qr:
+                    manager.export_qr(args.path)
+                else:
+                    manager.export(args.path)
+            except IsADirectoryError:
+                self.print_error("'{}' must be a file, not a directory".format(args.path))
+                return EXIT_ERROR
         return EXIT_SUCCESS
 
     @with_repository(lock=False, exclusive=False, manifest=False, cache=False)
diff --git a/src/borg/testsuite/archiver.py b/src/borg/testsuite/archiver.py
--- a/src/borg/testsuite/archiver.py
+++ b/src/borg/testsuite/archiver.py
@@ -2639,6 +2639,14 @@ def test_key_export_qr(self):
         assert export_contents.startswith('<!doctype html>')
         assert export_contents.endswith('</html>')
 
+    def test_key_export_directory(self):
+        export_directory = self.output_path + '/exported'
+        os.mkdir(export_directory)
+
+        self.cmd('init', self.repository_location, '--encryption', 'repokey')
+
+        self.cmd('key', 'export', self.repository_location, export_directory, exit_code=EXIT_ERROR)
+
     def test_key_import_errors(self):
         export_file = self.output_path + '/exported'
         self.cmd('init', self.repository_location, '--encryption', 'keyfile')


Test patch:
Repository: borgbackup/borg

Issue:
update bundled zstd code
we have 1.3.4 bundled, current see there: https://github.com/facebook/zstd/releases


Discussion:
hmm, it doesn't seem like there are substantial improvements or fixes relevant to borg yet, so we can do that later.

Resolving patch:
diff --git a/setup_zstd.py b/setup_zstd.py
--- a/setup_zstd.py
+++ b/setup_zstd.py
@@ -12,6 +12,7 @@
 # zstd files, structure as seen in zstd project repository:
 
 zstd_sources = [
+    'lib/common/debug.c',
     'lib/common/entropy_common.c',
     'lib/common/error_private.c',
     'lib/common/fse_decompress.c',
@@ -20,6 +21,7 @@
     'lib/common/xxhash.c',
     'lib/common/zstd_common.c',
     'lib/compress/fse_compress.c',
+    'lib/compress/hist.c',
     'lib/compress/huf_compress.c',
     'lib/compress/zstd_compress.c',
     'lib/compress/zstd_double_fast.c',
@@ -29,9 +31,12 @@
     'lib/compress/zstd_opt.c',
     'lib/compress/zstdmt_compress.c',
     'lib/decompress/huf_decompress.c',
+    'lib/decompress/zstd_ddict.c',
     'lib/decompress/zstd_decompress.c',
+    'lib/decompress/zstd_decompress_block.c',
     'lib/dictBuilder/cover.c',
     'lib/dictBuilder/divsufsort.c',
+    'lib/dictBuilder/fastcover.c',
     'lib/dictBuilder/zdict.c',
 ]
 


Test patch:
Repository: borgbackup/borg

Issue:
Feature Request: --list for delete command
While cleaning up my repo to remove some misnamed archives (from a script snafu on my part) I had occasion to use the `delete` command. While using delete to remove some archives with a specific starting string, I found the command to be lacking a `--list` flag like the `prune` command has. I would suggest that it would be good UI for delete to have this too, so that Users can use the `--dry-run` flag and see what they will be matching.

The current STDOUT results of both `borg delete --dry-run -P "some_string"` (repo set by ENV VAR) and `borg delete -P "some_string"` is NULL currently.






Discussion:
If the issue is still available, I'll work on it.
@joey4 sure, go for it, make a PR. :)
@demure does `--info` (aka `-v`) already what you need?

Found this while reviewing #4382, see there for comments.

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -1079,14 +1079,14 @@ def _delete_archives(self, args, repository):
             deleted = False
             for i, archive_name in enumerate(archive_names, 1):
                 try:
-                    del manifest.archives[archive_name]
+                    current_archive = manifest.archives.pop(archive_name)
                 except KeyError:
                     self.exit_code = EXIT_WARNING
                     logger.warning('Archive {} not found ({}/{}).'.format(archive_name, i, len(archive_names)))
                 else:
                     deleted = True
                     msg = 'Would delete: {} ({}/{})' if dry_run else 'Deleted archive: {} ({}/{})'
-                    logger.info(msg.format(archive_name, i, len(archive_names)))
+                    logger.info(msg.format(format_archive(current_archive), i, len(archive_names)))
             if dry_run:
                 logger.info('Finished dry-run.')
             elif deleted:
@@ -1102,7 +1102,7 @@ def _delete_archives(self, args, repository):
         with Cache(repository, key, manifest, progress=args.progress, lock_wait=self.lock_wait) as cache:
             for i, archive_name in enumerate(archive_names, 1):
                 msg = 'Would delete archive: {} ({}/{})' if dry_run else 'Deleting archive: {} ({}/{})'
-                logger.info(msg.format(archive_name, i, len(archive_names)))
+                logger.info(msg.format(format_archive(manifest.archives[archive_name]), i, len(archive_names)))
                 if not dry_run:
                     Archive(repository, key, manifest, archive_name, cache=cache).delete(
                         stats, progress=args.progress, forced=args.forced)


Test patch:
Repository: borgbackup/borg

Issue:
Feature Request: --list for delete command
While cleaning up my repo to remove some misnamed archives (from a script snafu on my part) I had occasion to use the `delete` command. While using delete to remove some archives with a specific starting string, I found the command to be lacking a `--list` flag like the `prune` command has. I would suggest that it would be good UI for delete to have this too, so that Users can use the `--dry-run` flag and see what they will be matching.

The current STDOUT results of both `borg delete --dry-run -P "some_string"` (repo set by ENV VAR) and `borg delete -P "some_string"` is NULL currently.






Discussion:
If the issue is still available, I'll work on it.
@joey4 sure, go for it, make a PR. :)
@demure does `--info` (aka `-v`) already what you need?

Found this while reviewing #4382, see there for comments.

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -1261,14 +1261,14 @@ def _delete_archives(self, args, repository):
             deleted = False
             for i, archive_name in enumerate(archive_names, 1):
                 try:
-                    del manifest.archives[archive_name]
+                    current_archive = manifest.archives.pop(archive_name)
                 except KeyError:
                     self.exit_code = EXIT_WARNING
                     logger.warning('Archive {} not found ({}/{}).'.format(archive_name, i, len(archive_names)))
                 else:
                     deleted = True
                     msg = 'Would delete: {} ({}/{})' if dry_run else 'Deleted archive: {} ({}/{})'
-                    logger.info(msg.format(archive_name, i, len(archive_names)))
+                    logger.info(msg.format(format_archive(current_archive), i, len(archive_names)))
             if dry_run:
                 logger.info('Finished dry-run.')
             elif deleted:
@@ -1284,7 +1284,7 @@ def _delete_archives(self, args, repository):
         with Cache(repository, key, manifest, progress=args.progress, lock_wait=self.lock_wait) as cache:
             for i, archive_name in enumerate(archive_names, 1):
                 msg = 'Would delete archive: {} ({}/{})' if dry_run else 'Deleting archive: {} ({}/{})'
-                logger.info(msg.format(archive_name, i, len(archive_names)))
+                logger.info(msg.format(format_archive(manifest.archives[archive_name]), i, len(archive_names)))
                 if not dry_run:
                     Archive(repository, key, manifest, archive_name, cache=cache).delete(
                         stats, progress=args.progress, forced=args.forced)


Test patch:
Repository: borgbackup/borg

Issue:
borg key export confusing error msg
from IRC:
```
borg key export REPO PATH
```

if the second argument is a directory it will throw an error.

I understand it needs a file, but the error it throws may be confusing to other users. :)



Discussion:

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -352,10 +352,14 @@ def do_key_export(self, args, repository):
             if not args.path:
                 self.print_error("output file to export key to expected")
                 return EXIT_ERROR
-            if args.qr:
-                manager.export_qr(args.path)
-            else:
-                manager.export(args.path)
+            try:
+                if args.qr:
+                    manager.export_qr(args.path)
+                else:
+                    manager.export(args.path)
+            except IsADirectoryError:
+                self.print_error("'{}' must be a file, not a directory".format(args.path))
+                return EXIT_ERROR
         return EXIT_SUCCESS
 
     @with_repository(lock=False, exclusive=False, manifest=False, cache=False)
diff --git a/src/borg/testsuite/archiver.py b/src/borg/testsuite/archiver.py
--- a/src/borg/testsuite/archiver.py
+++ b/src/borg/testsuite/archiver.py
@@ -2650,6 +2650,14 @@ def test_key_export_qr(self):
         assert export_contents.startswith('<!doctype html>')
         assert export_contents.endswith('</html>')
 
+    def test_key_export_directory(self):
+        export_directory = self.output_path + '/exported'
+        os.mkdir(export_directory)
+
+        self.cmd('init', self.repository_location, '--encryption', 'repokey')
+
+        self.cmd('key', 'export', self.repository_location, export_directory, exit_code=EXIT_ERROR)
+
     def test_key_import_errors(self):
         export_file = self.output_path + '/exported'
         self.cmd('init', self.repository_location, '--encryption', 'keyfile')


Test patch:
Repository: borgbackup/borg

Issue:
Compatibility issue (minor), error message in case of invalid repo.
Compatibility issue (minor), error message in case of invalid repo.

Old version

	jk@paju:~$ borg  --version
	borg 1.1.8

Repo dir is not valid repo:

	jk@paju:~$ borg create ~/tmp::foo ~/tmp
	/home/jk/tmp is not a valid repository. Check repo config.


Repo dir does not exist:

	jk@paju:~$ borg create ~/tmp/foo::foo ~/tmp
	Repository /home/jk/tmp/foo does not exist.

Latest version

	jk@tiira:~$ borg --version
	borg 1.1.9

Repo dir is not valid repo:

	jk@tiira:~$ borg create ~/tmp::foo ~/tmp
	Exception ignored in: <bound method Repository.__del__ of <Repository /home/jk/tmp>>
	Traceback (most recent call last):
	  File "/usr/lib/python3/dist-packages/borg/repository.py", line 179, in __del__
	    assert False, "cleanup happened in Repository.__del__"
	AssertionError: cleanup happened in Repository.__del__
	Local Exception
	Traceback (most recent call last):
	  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4455, in main
	    exit_code = archiver.run(args)
	  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4387, in run
    	return set_ec(func(args))
	  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 139, in wrapper
    	with repository:
	  File "/usr/lib/python3/dist-packages/borg/repository.py", line 189, in __enter__
	    self.open(self.path, bool(self.exclusive), lock_wait=self.lock_wait, lock=self.do_lock)
	  File "/usr/lib/python3/dist-packages/borg/repository.py", line 396, in open
	    with open(os.path.join(self.path, 'config')) as fd:
	FileNotFoundError: [Errno 2] No such file or directory: '/home/jk/tmp/config'
	
	Platform: Linux tiira 4.4.0-142-generic #168-Ubuntu SMP Wed Jan 16 21:00:45 UTC 2019 x86_64
	Linux: Ubuntu 16.04 xenial
	Borg: 1.1.9  Python: CPython 3.5.2
	PID: 23536  CWD: /home/jk
	sys.argv: ['/usr/bin/borg', 'create', '/home/jk/tmp::foo', '/home/jk/tmp']
	SSH_ORIGINAL_COMMAND: None

Repo dir does not exist:
	
	jk@tiira:~$ borg create ~/tmp/foo::foo ~/tmp
	Repository /home/jk/tmp/foo does not exist.


Please restore the v1.1.8 type of error message in case of invalid repo dir.


Discussion:

Resolving patch:
diff --git a/src/borg/repository.py b/src/borg/repository.py
--- a/src/borg/repository.py
+++ b/src/borg/repository.py
@@ -393,8 +393,12 @@ def open(self, path, exclusive, lock_wait=None, lock=True):
         else:
             self.lock = None
         self.config = ConfigParser(interpolation=None)
-        with open(os.path.join(self.path, 'config')) as fd:
-            self.config.read_file(fd)
+        try:
+            with open(os.path.join(self.path, 'config')) as fd:
+                self.config.read_file(fd)
+        except FileNotFoundError:
+            self.close()
+            raise self.InvalidRepository(self.path)
         if 'repository' not in self.config.sections() or self.config.getint('repository', 'version') != 1:
             self.close()
             raise self.InvalidRepository(path)


Test patch:
Repository: borgbackup/borg

Issue:
Incorrect borg prune --stats output with --dry-run
With `borg 1.1.7` on `ubuntu 18.10`:
```
borg prune <borg_dir> --dry-run --list --stats --keep-last 1 
...
                       Original size      Compressed size    Deduplicated size
Deleted data:                    0 B                  0 B                  0 B
All archives:              981.25 GB            793.40 GB            112.32 GB

                       Unique chunks         Total chunks
Chunk index:                  250054              2164226
```
```
borg prune <borg_dir> --list --stats --keep-last 1 
...
                       Original size      Compressed size    Deduplicated size
Deleted data:             -820.45 GB           -669.72 GB             -9.60 GB
All archives:              160.80 GB            123.69 GB            102.72 GB

                       Unique chunks         Total chunks
Chunk index:                  237044               371408
```
Pruning worked as I wanted it to, but the dry run stats do not give an accurate preview.


Discussion:
Likely, that is simply not implemented (just guessing, needs to be checked in the src).

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -4230,6 +4230,9 @@ def run(self, args):
         args.progress |= is_serve
         self._setup_implied_logging(vars(args))
         self._setup_topic_debugging(args)
+        if getattr(args, 'stats', False) and getattr(args, 'dry_run', False):
+            logger.error("--stats does not work with --dry-run.")
+            return self.exit_code
         if args.show_version:
             logging.getLogger('borg.output.show-version').info('borgbackup version %s' % __version__)
         self.prerun_checks(logger)
diff --git a/src/borg/testsuite/archiver.py b/src/borg/testsuite/archiver.py
--- a/src/borg/testsuite/archiver.py
+++ b/src/borg/testsuite/archiver.py
@@ -1870,10 +1870,9 @@ def test_prune_repository_save_space(self):
         self.cmd('init', '--encryption=repokey', self.repository_location)
         self.cmd('create', self.repository_location + '::test1', src_dir)
         self.cmd('create', self.repository_location + '::test2', src_dir)
-        output = self.cmd('prune', '--list', '--stats', '--dry-run', self.repository_location, '--keep-daily=2')
+        output = self.cmd('prune', '--list', '--dry-run', self.repository_location, '--keep-daily=2')
         assert re.search(r'Keeping archive \(rule: daily #1\):\s+test2', output)
         assert re.search(r'Would prune:\s+test1', output)
-        self.assert_in('Deleted data:', output)
         output = self.cmd('list', self.repository_location)
         self.assert_in('test1', output)
         self.assert_in('test2', output)


Test patch:
Repository: borgbackup/borg

Issue:
only do glibc checks on client?
Copying my reply to an issue posted on the mailing list to here:

```
     $ borg init --e none ./borgtest
     Remote: FATAL: this Python was compiled for a too old (g)libc and
     misses required functionality.
```
This is the code which triggers this exception / error message:

```
    required_funcs = {os.stat, os.utime, os.chown}
    if not os.supports_follow_symlinks.issuperset(required_funcs):
        raise PythonLibcTooOld
```

So, as you see, quite basic functionality like not following symlinks is not supported for all function where borg needs that.

So far for the bad news.

The good news are that these functions might be only required for using borg as a client, but not for "borg serve" (repo server part in client/server mode, if repo access is done via ssh:).

borg currently always does this check, but I guess it could be maybe omitted for the "borg serve" server part.


Discussion:
I had a look at the code and yes, as suspected, the `follow_symlinks = False` is only used clientside (for borg create and extract).

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -4322,8 +4322,10 @@ def parse_args(self, args=None):
             parser.error('Need at least one PATH argument.')
         return args
 
-    def prerun_checks(self, logger):
-        check_python()
+    def prerun_checks(self, logger, is_serve):
+        if not is_serve:
+            # this is the borg *client*, we need to check the python:
+            check_python()
         check_extension_modules()
         selftest(logger)
 
@@ -4362,7 +4364,7 @@ def run(self, args):
         self._setup_topic_debugging(args)
         if args.show_version:
             logging.getLogger('borg.output.show-version').info('borgbackup version %s' % __version__)
-        self.prerun_checks(logger)
+        self.prerun_checks(logger, is_serve)
         if not is_supported_msgpack():
             logger.error("You do not have a supported msgpack[-python] version installed. Terminating.")
             logger.error("This should never happen as specific, supported versions are required by our setup.py.")


Test patch: