internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: borgbackup/borg

Issue:
py.test has lost the nice assertion failure display
when fixing #1930, I noticed that when the assertion fails, it just says "AssertionError", it does not display the values of the terms in the assertion expression.



Discussion:
E.g. like here:
```
__________________________________________________________ TestLocationWithoutEnv.test_file ___________________________________________________________

self = <borg.testsuite.helpers.TestLocationWithoutEnv object at 0x7fced6086940>

    def test_file(self):
        #monkeypatch.delenv('BORG_REPO', raising=False)
        assert repr(Location('xfile:///some/path::archive')) == \
>           "Location(proto='file', user=None, host=None, port=None, path='/some/path', archive='archive')"
E       AssertionError

borg/testsuite/helpers.py:48: AssertionError
```
The lhs value would have been interesting...
i believe with the particular layout for the helpers, you need 

http://doc.pytest.org/en/latest/writing_plugins.html#pytest.register_assert_rewrite
fixed by #1951.
Reopening, see comments in #1951.

Resolving patch:
diff --git a/conftest.py b/conftest.py
--- a/conftest.py
+++ b/conftest.py
@@ -5,7 +5,7 @@
 import pytest
 
 # needed to get pretty assertion failures in unit tests:
-pytest.register_assert_rewrite('borg')
+pytest.register_assert_rewrite('borg.testsuite')
 
 # This is a hack to fix path problems because "borg" (the package) is in the source root.
 # When importing the conftest an "import borg" can incorrectly import the borg from the


Test patch:
Repository: borgbackup/borg

Issue:
check remote.py "~user/..." path processing
compare master against 1.0-maint and see how /~username is processed.

i suspect it is incorrect in master:
https://github.com/borgbackup/borg/blob/0ff2352691ee82ebf3c828092a68d1558a64f872/src/borg/remote.py#L153

1.0-maint:
https://github.com/borgbackup/borg/blob/1.0-maint/borg/remote.py#L136



---

:moneybag: [there is a bounty for this](https://www.bountysource.com/issues/38636201-check-remote-py-user-path-processing)


Discussion:
git blames #759

@nachtgeist can you have a look?


Resolving patch:
diff --git a/src/borg/remote.py b/src/borg/remote.py
--- a/src/borg/remote.py
+++ b/src/borg/remote.py
@@ -263,14 +263,26 @@ def negotiate(self, client_data):
         # not a known old format, send newest negotiate this version knows
         return {'server_version': BORG_VERSION}
 
-    def open(self, path, create=False, lock_wait=None, lock=True, exclusive=None, append_only=False):
+    def _resolve_path(self, path):
         if isinstance(path, bytes):
             path = os.fsdecode(path)
-        if path.startswith('/~'):  # /~/x = path x relative to home dir, /~username/x = relative to "user" home dir
-            path = os.path.join(get_home_dir(), path[2:])  # XXX check this (see also 1.0-maint), is it correct for ~u?
+        # Leading slash is always present with URI (ssh://), but not with short-form (who@host:path).
+        if path.startswith('/~/'):  # /~/x = path x relative to home dir
+            path = os.path.join(get_home_dir(), path[3:])
+        elif path.startswith('~/'):
+            path = os.path.join(get_home_dir(), path[2:])
+        elif path.startswith('/~'):  # /~username/x = relative to "user" home dir
+            path = os.path.expanduser(path[1:])
+        elif path.startswith('~'):
+            path = os.path.expanduser(path)
         elif path.startswith('/./'):  # /./x = path x relative to cwd
             path = path[3:]
-        path = os.path.realpath(path)
+        return os.path.realpath(path)
+
+    def open(self, path, create=False, lock_wait=None, lock=True, exclusive=None, append_only=False):
+        logging.debug('Resolving repository path %r', path)
+        path = self._resolve_path(path)
+        logging.debug('Resolved repository path to %r', path)
         if self.restrict_to_paths:
             # if --restrict-to-path P is given, we make sure that we only operate in/below path P.
             # for the prefix check, it is important that the compared pathes both have trailing slashes,


Test patch:
Repository: borgbackup/borg

Issue:
add non-encrypted repo check for --tam repo upgrades (or at least better docs)
Hi, while reading the release notes for the new 1.0.9 version, I saw I need to run a command similar to --tam upgrade or sth like that. It upgrades the repo to make it more secure.

The problem is the docs don't say specifically that the repo needs to have some encryption. For non-encrypted repo-s the upgrade command fails with some python error that looks ugly.

Two simple things could remedy this situation:
1. Add one sentence to docs/release notes that specifies we need to upgrade ONLY repos with encryption.
2. Add a simple if-check for the --tam option that would print "this repo is not encrypted, you don't need to run this upgrade at all. Bye bye..."

Agree?



Discussion:
the python error I saw on upgrade left me thinking I might have screwed my repo somehow, while I think I didn't ...
Yes, it errors out in an ugly way but no harm done to repo/key. Thanks for the report! :)

Note: tam isn't enabled for unencrypted repos that are created with 1.0.9, and since the upgrade doesn't work there is no way to get "locked into" it. (--disable-tam also works). So this is more a cosmetic / docs issue.

Resolving patch:
diff --git a/borg/archiver.py b/borg/archiver.py
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -177,8 +177,14 @@ def do_check(self, args, repository):
     @with_repository()
     def do_change_passphrase(self, args, repository, manifest, key):
         """Change repository key file passphrase"""
+        if not hasattr(key, 'change_passphrase'):
+            print('This repository is not encrypted, cannot change the passphrase.')
+            return EXIT_ERROR
         key.change_passphrase()
         logger.info('Key updated')
+        if hasattr(key, 'find_key'):
+            # print key location to make backing it up easier
+            logger.info('Key location: %s', key.find_key())
         return EXIT_SUCCESS
 
     @with_repository(lock=False, exclusive=False, manifest=False, cache=False)
@@ -730,6 +736,10 @@ def do_upgrade(self, args, repository, manifest=None, key=None):
         if args.tam:
             manifest, key = Manifest.load(repository, force_tam_not_required=args.force)
 
+            if not hasattr(key, 'change_passphrase'):
+                print('This repository is not encrypted, cannot enable TAM.')
+                return EXIT_ERROR
+
             if not manifest.tam_verified or not manifest.config.get(b'tam_required', False):
                 # The standard archive listing doesn't include the archive ID like in borg 1.1.x
                 print('Manifest contents:')


Test patch:
Repository: borgbackup/borg

Issue:
borg check does not exit with failure status when archive is absent
when running a command such as:
borg check -v repo::archive1           (archive1 is not actually part of the repo)
...if archive1 does not exist in the repo, the command should report an error ( that archive "archive1" does not exist ), then exit with a non-zero status.
Instead, it says the archive consistency check is complete(see below), and exits with a 0 status.

Starting repository check
Completed repository check, no problems found.
Starting archive consistency check...
Orphaned objects check skipped (needs all archives checked).
Archive consistency check complete, no problems found.
[root@chlxfs1 trc]# echo $?
0





Discussion:

Resolving patch:
diff --git a/borg/archive.py b/borg/archive.py
--- a/borg/archive.py
+++ b/borg/archive.py
@@ -1100,13 +1100,21 @@ def valid_item(obj):
                                    key=lambda name_info: name_info[1][b'time'])
             if prefix is not None:
                 archive_items = [item for item in archive_items if item[0].startswith(prefix)]
+                if not archive_items:
+                    logger.warning('--prefix %s does not match any archives', prefix)
             num_archives = len(archive_items)
             end = None if last is None else min(num_archives, last)
+            if last is not None and end < last:
+                logger.warning('--last %d archives: only found %d archives', last, end)
         else:
             # we only want one specific archive
             archive_items = [item for item in self.manifest.archives.items() if item[0] == archive]
             num_archives = 1
             end = 1
+            if not archive_items:
+                logger.error('Archive %s does not exist', archive)
+                self.error_found = True
+                return
 
         with cache_if_remote(self.repository) as repository:
             for i, (name, info) in enumerate(archive_items[:end]):


Test patch:
Repository: borgbackup/borg

Issue:
API versions number-space overlap issue
Up to now, we just used 1..N in 1.0-maint and master (soon 1.1) branch and incremented the version when we did changes needing a recompile of the .pyx stuff / added new functionality.

Always at 1.0 -> master merge time, I noticed that both branches are overlapping and therefore that the increment is not really reliable when switching from one branch to the other.

Thus, I suggest doing it like this:

1.0-maint: use versions starting from 01_00_00
master (1.1): use versions starting from 01_01_00


Discussion:
Since `_` syntax is new in 3.6 we could also just make it a string, since we do an exact comparison anyway.

```py
API_VERSION = '1.1_01'
```
I did not really intend to use _ in the source, just used them in the ticket to be more clear.
But good idea, I'll use strings to make it more clear in the source alo.

Resolving patch:
diff --git a/borg/helpers.py b/borg/helpers.py
--- a/borg/helpers.py
+++ b/borg/helpers.py
@@ -88,13 +88,13 @@ class PlaceholderError(Error):
 
 def check_extension_modules():
     from . import platform
-    if hashindex.API_VERSION != 3:
+    if hashindex.API_VERSION != '1.0_01':
         raise ExtensionModuleError
-    if chunker.API_VERSION != 2:
+    if chunker.API_VERSION != '1.0_01':
         raise ExtensionModuleError
-    if crypto.API_VERSION != 3:
+    if crypto.API_VERSION != '1.0_01':
         raise ExtensionModuleError
-    if platform.API_VERSION != 3:
+    if platform.API_VERSION != '1.0_01':
         raise ExtensionModuleError
 
 
diff --git a/borg/platform.py b/borg/platform.py
--- a/borg/platform.py
+++ b/borg/platform.py
@@ -30,7 +30,7 @@ def umount(mountpoint):
 elif sys.platform == 'darwin':  # pragma: darwin only
     from .platform_darwin import acl_get, acl_set, API_VERSION
 else:  # pragma: unknown platform only
-    API_VERSION = 3
+    API_VERSION = '1.0_01'
 
     def acl_get(path, item, st, numeric_owner=False):
         pass


Test patch:
Repository: borgbackup/borg

Issue:
read backup data from stdin
Hi. Borg seems awesome. I was using rdiff-backup and rdiff-snapshot (fortunately not needed backups often) and seems so much more intuitive to recover.

I'm wondering if I can backup myslq _directly_ into borg without hitting the disk twice.

Percona has a stream option -- but it's in tar format (for the directory of files) and also I couldn't figure out why it still wants a file path.... 
https://www.percona.com/doc/percona-xtrabackup/2.4/innobackupex/streaming_backups_innobackupex.html

Alternatively, is there a "delete-on-backup" option? (I didn't notice one) -- like rsync's `--remove-source-files`

I would use that to "suck backups" into borg's backup.

Thanks!


Discussion:
Something along the lines of `rdbms_dump_tool (eg. mysqldump, pg_dumpall) | borg create repo::archive -`

This gives you an archive with a single file named `-` whose contents are whatever you put into stdin.

You can also extract to stdout, like so: `borg extract repo::archive --stdout`
The archive file name created is `stdin` iirc.
TODO: Document "-" for "PATHS"¹ and the file name in the archive ("stdin")

¹ I believe it's shown in the examples, but it should also be in the reference.
We're successfully using borg for cloud image backups via stdin:
`curl --silent --location --user $APIUSER:$APIPASS https://provider.com/api/xxxxxxx/download/ | borg create --compression zlib,6 -v --stats ssh://borg@example.org/backup::my-snapshot-1234 -`

Resolving patch:
diff --git a/borg/archiver.py b/borg/archiver.py
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -1311,8 +1311,12 @@ def build_parser(self, args=None, prog=None):
 
         create_epilog = textwrap.dedent("""
         This command creates a backup archive containing all files found while recursively
-        traversing all paths specified. The archive will consume almost no disk space for
-        files or parts of files that have already been stored in other archives.
+        traversing all paths specified. When giving '-' as path, borg will read data
+        from standard input and create a file 'stdin' in the created archive from that
+        data.
+
+        The archive will consume almost no disk space for files or parts of files that
+        have already been stored in other archives.
 
         The archive name needs to be unique. It must not end in '.checkpoint' or
         '.checkpoint.N' (with N being a number), because these names are used for


Test patch:
Repository: borgbackup/borg

Issue:
crc32.c (cythonized crc32.pyx) is missing from pypi package



Discussion:
I suspect above is the root cause of this:
```
(env-temp)tw@tux:~/w/borg-clean/dist$ pip install borgbackup-1.1.0b3.tar.gz 
Unpacking ./borgbackup-1.1.0b3.tar.gz
  Running setup.py (path:/tmp/pip-a1otsf_7-build/setup.py) egg_info for package from file:///home/tw/w/borg-clean/dist/borgbackup-1.1.0b3.tar.gz
    Traceback (most recent call last):
      File "/tmp/pip-a1otsf_7-build/setup.py", line 74, in <module>
        from Cython.Distutils import build_ext
    ImportError: No module named 'Cython'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/tmp/pip-a1otsf_7-build/setup.py", line 118, in <module>
        raise ImportError('The GIT version of Borg needs Cython. Install Cython or use a released version.')
    ImportError: The GIT version of Borg needs Cython. Install Cython or use a released version.
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "/tmp/pip-a1otsf_7-build/setup.py", line 74, in <module>

    from Cython.Distutils import build_ext

ImportError: No module named 'Cython'



During handling of the above exception, another exception occurred:



Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/tmp/pip-a1otsf_7-build/setup.py", line 118, in <module>

    raise ImportError('The GIT version of Borg needs Cython. Install Cython or use a released version.')

ImportError: The GIT version of Borg needs Cython. Install Cython or use a released version.

----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /tmp/pip-a1otsf_7-build
Storing debug log for failure in /home/tw/.pip/pip.log
```

Hmm, but where is it missing?

https://github.com/borgbackup/borg/blob/master/setup.py#L90
https://github.com/borgbackup/borg/blob/master/setup.py#L374
https://github.com/borgbackup/borg/blob/master/setup.py#L116
In cython_sources.
Interesting, when I do sdist I get a .tar.gz which *also* contains the binaries.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -63,6 +63,7 @@
     chunker_source,
     hashindex_source,
     item_source,
+    crc32_source,
 
     platform_posix_source,
     platform_linux_source,


Test patch:
Repository: borgbackup/borg

Issue:
xattr tests race condition
Found on Travis: https://travis-ci.org/borgbackup/borg/jobs/191915043

```
__________________ XattrTestCase.test_getxattr_buffer_growth ___________________
[gw7] linux -- Python 3.5.2 /home/travis/build/borgbackup/borg/.tox/py35/bin/python3.5
self = <borg.testsuite.xattr.XattrTestCase testMethod=test_getxattr_buffer_growth>

    def setUp(self):
        self.tmpfile = tempfile.NamedTemporaryFile()
        self.symlink = os.path.join(os.path.dirname(self.tmpfile.name), 'symlink')
>       os.symlink(self.tmpfile.name, self.symlink)
E       FileExistsError: [Errno 17] File exists: '/tmp/tmpmnhbvsv9' -> '/tmp/symlink'
```

-> Create tempdir instead, put file in tempdir, make symlink in tempdir as well.


Discussion:
or just do:
```
self.symlink = self.tmpfile.name + '.symlink'
```

Resolving patch:
diff --git a/borg/testsuite/xattr.py b/borg/testsuite/xattr.py
--- a/borg/testsuite/xattr.py
+++ b/borg/testsuite/xattr.py
@@ -11,7 +11,7 @@ class XattrTestCase(BaseTestCase):
 
     def setUp(self):
         self.tmpfile = tempfile.NamedTemporaryFile()
-        self.symlink = os.path.join(os.path.dirname(self.tmpfile.name), 'symlink')
+        self.symlink = self.tmpfile.name + '.symlink'
         os.symlink(self.tmpfile.name, self.symlink)
 
     def tearDown(self):


Test patch:
Repository: borgbackup/borg

Issue:
borg mount -f should umount when SIGINT (ctrl-c)
Currently, the mount remains beyond the command execution.
This behaviour is inconvenient for those who use it as a means of assuring you don't forget to umount it (and thus remove the lock).

Thanks!


Discussion:
https://github.com/borgbackup/borg/blob/1.0.9/borg/fuse.py#L93 see there.
Umm, I do agree that if there is a crash, we should not umount. 
But IMHO receiving a SIGINT signal is something you do on purpose, and you know what you are doing, and in that particular signal at least you should umount.
SIGINT should be umount, at the very least when running in the --foreground (like the title says).

Resolving patch:
diff --git a/borg/archiver.py b/borg/archiver.py
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -1546,6 +1546,13 @@ def build_parser(self, args=None, prog=None):
         - allow_damaged_files: by default damaged files (where missing chunks were
           replaced with runs of zeros by borg check --repair) are not readable and
           return EIO (I/O error). Set this option to read such files.
+
+        When the daemonized process receives a signal or crashes, it does not unmount.
+        Unmounting in these cases could cause an active rsync or similar process
+        to unintentionally delete data.
+
+        When running in the foreground ^C/SIGINT unmounts cleanly, but other
+        signals or crashes do not.
         """)
         subparser = subparsers.add_parser('mount', parents=[common_parser],
                                           description=self.do_mount.__doc__,
diff --git a/borg/fuse.py b/borg/fuse.py
--- a/borg/fuse.py
+++ b/borg/fuse.py
@@ -6,6 +6,7 @@
 import stat
 import tempfile
 import time
+from signal import SIGINT
 from distutils.version import LooseVersion
 
 import msgpack
@@ -98,7 +99,8 @@ def mount(self, mountpoint, mount_options, foreground=False):
         umount = False
         try:
             signal = fuse_main()
-            umount = (signal is None)  # no crash and no signal -> umount request
+            # no crash and no signal (or it's ^C and we're in the foreground) -> umount request
+            umount = (signal is None or (signal == SIGINT and foreground))
         finally:
             llfuse.close(umount)
 


Test patch:
Repository: borgbackup/borg

Issue:
paperkey.html - test failure on bsd
```
==> openbsd64:         with open(export_file, 'r') as fd:
==> openbsd64: >           export_contents = fd.read()
==> openbsd64: 
==> openbsd64: py34/lib/python3.4/site-packages/borg/testsuite/archiver.py:1284: 
==> openbsd64: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
==> openbsd64: 
==> openbsd64: self = <encodings.ascii.IncrementalDecoder object at 0x1f152dd20f0>
==> openbsd64: input = b'<!doctype html>\n<html moznomarginboxes mozdisallowselectionprint>\n<!--\n    Notes:\n        This may never cause e...false);\n    }\n\n    update();\n}\n\nwindow.addEventListener("load", loaded, false);\n\n</script>\n\n</body>\n</html>'
==> openbsd64: final = True
==> openbsd64: 
==> openbsd64:     def decode(self, input, final=False):
==> openbsd64: >       return codecs.ascii_decode(input, self.errors)[0]
==> openbsd64: E       UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 544: ordinal not in range(128)
```


Discussion:

Resolving patch:
diff --git a/borg/testsuite/archiver.py b/borg/testsuite/archiver.py
--- a/borg/testsuite/archiver.py
+++ b/borg/testsuite/archiver.py
@@ -1274,6 +1274,19 @@ def test_key_export_repokey(self):
 
         assert repo_key2.enc_key == repo_key2.enc_key
 
+    def test_key_export_qr(self):
+        export_file = self.output_path + '/exported.html'
+        self.cmd('init', self.repository_location, '--encryption', 'repokey')
+        repo_id = self._extract_repository_id(self.repository_path)
+        self.cmd('key', 'export', '--qr-html', self.repository_location, export_file)
+
+        with open(export_file, 'r', encoding='utf-8') as fd:
+            export_contents = fd.read()
+
+        assert bin_to_hex(repo_id) in export_contents
+        assert export_contents.startswith('<!doctype html>')
+        assert export_contents.endswith('</html>')
+
     def test_key_import_errors(self):
         export_file = self.output_path + '/exported'
         self.cmd('init', self.repository_location, '--encryption', 'keyfile')
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -230,10 +230,13 @@ def run(self):
         print("auto-generating API documentation")
         with open("docs/api.rst", "w") as doc:
             doc.write("""
+.. IMPORTANT: this file is auto-generated by "setup.py build_api", do not edit!
+
+
 API Documentation
 =================
 """)
-            for mod in glob('borg/*.py') + glob('borg/*.pyx'):
+            for mod in sorted(glob('borg/*.py') + glob('borg/*.pyx')):
                 print("examining module %s" % mod)
                 mod = mod.replace('.pyx', '').replace('.py', '').replace('/', '.')
                 if "._" not in mod:


Test patch: