internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: borgbackup/borg

Issue:
Borg crashes after canceling of deletion
After a delete got cancelled by accident borg now throws an exception like this https://gist.github.com/BlackLotus/5b435559c95d7ff819eec1b5fe66ecca

I'm using borg from git revision e4e678790610a67ef866d569143157d80c7f6c26 (note: master branch, changeset not in 1.0-maint).



Discussion:

Resolving patch:
diff --git a/borg/repository.py b/borg/repository.py
--- a/borg/repository.py
+++ b/borg/repository.py
@@ -705,6 +705,13 @@ def iter_objects(self, segment, include_data=False):
             else:
                 yield tag, key, offset
             offset += size
+            # we must get the fd via get_fd() here again as we yielded to our caller and it might
+            # have triggered closing of the fd we had before (e.g. by calling io.read() for
+            # different segment(s)).
+            # by calling get_fd() here again we also make our fd "recently used" so it likely
+            # does not get kicked out of self.fds LRUcache.
+            fd = self.get_fd(segment)
+            fd.seek(offset)
             header = fd.read(self.header_fmt.size)
 
     def recover_segment(self, segment, filename):


Test patch:
Repository: borgbackup/borg

Issue:
show repository ID with other repository information in info
borg-info should show the repository ID and maybe where the local cache is for that repo.



Discussion:
1.0-maint?

info for repo is only in master, but could be backported as well. I'd target this on master for now and backport when this is done.


Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -920,7 +920,7 @@ def do_info(self, args, repository, manifest, key, cache):
         if any((args.location.archive, args.first, args.last, args.prefix)):
             return self._info_archives(args, repository, manifest, key, cache)
         else:
-            return self._info_repository(cache)
+            return self._info_repository(repository, key, cache)
 
     def _info_archives(self, args, repository, manifest, key, cache):
         def format_cmdline(cmdline):
@@ -957,7 +957,17 @@ def format_cmdline(cmdline):
                 print()
         return self.exit_code
 
-    def _info_repository(self, cache):
+    def _info_repository(self, repository, key, cache):
+        print('Repository ID: %s' % bin_to_hex(repository.id))
+        if key.NAME == 'plaintext':
+            encrypted = 'No'
+        else:
+            encrypted = 'Yes (%s)' % key.NAME
+        print('Encrypted: %s' % encrypted)
+        if key.NAME == 'key file':
+            print('Key file: %s' % key.find_key())
+        print('Cache: %s' % cache.path)
+        print(DASHES)
         print(STATS_HEADER)
         print(str(cache))
         return self.exit_code
diff --git a/src/borg/key.py b/src/borg/key.py
--- a/src/borg/key.py
+++ b/src/borg/key.py
@@ -118,6 +118,7 @@ def assert_id(self, id, data):
 
 class PlaintextKey(KeyBase):
     TYPE = 0x02
+    NAME = 'plaintext'
 
     chunk_seed = 0
 
@@ -281,6 +282,7 @@ class PassphraseKey(AESKeyBase):
     # - --encryption=passphrase is an invalid argument now
     # This class is kept for a while to support migration from passphrase to repokey mode.
     TYPE = 0x01
+    NAME = 'passphrase'
     iterations = 100000  # must not be changed ever!
 
     @classmethod
@@ -432,6 +434,7 @@ def get_new_target(self, args):
 
 class KeyfileKey(KeyfileKeyBase):
     TYPE = 0x00
+    NAME = 'key file'
     FILE_ID = 'BORG_KEY'
 
     def sanity_check(self, filename, id):
@@ -491,6 +494,7 @@ def save(self, target, passphrase):
 
 class RepoKey(KeyfileKeyBase):
     TYPE = 0x03
+    NAME = 'repokey'
 
     def find_key(self):
         loc = self.repository._location.canonical_path()


Test patch:
Repository: borgbackup/borg

Issue:
debug get-obj / delete-obj crash if object not found and remote repo



Discussion:

Resolving patch:
diff --git a/borg/archiver.py b/borg/archiver.py
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -767,7 +767,7 @@ def do_debug_get_obj(self, args, repository):
         else:
             try:
                 data = repository.get(id)
-            except repository.ObjectNotFound:
+            except Repository.ObjectNotFound:
                 print("object %s not found." % hex_id)
             else:
                 with open(args.path, "wb") as f:
@@ -801,7 +801,7 @@ def do_debug_delete_obj(self, args, repository):
                     repository.delete(id)
                     modified = True
                     print("object %s deleted." % hex_id)
-                except repository.ObjectNotFound:
+                except Repository.ObjectNotFound:
                     print("object %s not found." % hex_id)
         if modified:
             repository.commit()


Test patch:
Repository: borgbackup/borg

Issue:
add placeholder to IntegrityError docstring
It seems all callers pass details to IntegrityError, but they are only output via the traceback not via the main exception message, because the doc string is missing a placeholder.



Discussion:
Note: also check other Error subclasses. This is problematic esp. for the Errors that do not output a traceback.


Resolving patch:
diff --git a/borg/helpers.py b/borg/helpers.py
--- a/borg/helpers.py
+++ b/borg/helpers.py
@@ -64,7 +64,7 @@ class ErrorWithTraceback(Error):
 
 
 class IntegrityError(ErrorWithTraceback):
-    """Data integrity error"""
+    """Data integrity error: {}"""
 
 
 class ExtensionModuleError(Error):


Test patch:
Repository: borgbackup/borg

Issue:
Cache’s previous_location is only updated on cache commit
When the "Warning: The repository at location {} was previously located at" prompt is answered the location should be be updated immediately instead of only on the next commit.

Rational: Automated backup scripts often have no way to answer such prompts. A natural first try is to do a quick operation from the shell to fix that problem like a borg list or borg info. But these commands don’t commit a cache transaction so previous_location is never updated.



Discussion:
enhancement -> bug because it's so annoying; plus with the recent removal of upgradeable locks this isn't really tricky to implement any more.


Resolving patch:
diff --git a/borg/cache.py b/borg/cache.py
--- a/borg/cache.py
+++ b/borg/cache.py
@@ -77,6 +77,9 @@ def __init__(self, repository, key, manifest, path=None, sync=True, do_files=Fal
                 if not yes(msg, false_msg="Aborting.", invalid_msg="Invalid answer, aborting.",
                            retry=False, env_var_override='BORG_RELOCATED_REPO_ACCESS_IS_OK'):
                     raise self.RepositoryAccessAborted()
+                # adapt on-disk config immediately if the new location was accepted
+                self.begin_txn()
+                self.commit()
 
             if sync and self.manifest.id != self.manifest_id:
                 # If repository is older than the cache something fishy is going on


Test patch:
Repository: borgbackup/borg

Issue:
"Cache newer than repo" error should be more explicit when repo is encrypted
Ie. it should say that this is either an attack _or_ there are multiple independently updated copies of the same repo, which is also unsafe for encrypted repos.

"Cache newer than repo" error should be more explicit when repo is encrypted
Ie. it should say that this is either an attack _or_ there are multiple independently updated copies of the same repo, which is also unsafe for encrypted repos.



Discussion:

Resolving patch:
diff --git a/borg/cache.py b/borg/cache.py
--- a/borg/cache.py
+++ b/borg/cache.py
@@ -20,8 +20,11 @@
 class Cache:
     """Client Side cache
     """
+    class RepositoryIDNotUnique(Error):
+        """Cache is newer than repository - do you have multiple, independently updated repos with same ID?"""
+
     class RepositoryReplay(Error):
-        """Cache is newer than repository, refusing to continue"""
+        """Cache is newer than repository - this is either an attack or unsafe (multiple repos with same ID)"""
 
     class CacheInitAbortedError(Error):
         """Cache initialization aborted"""
@@ -81,7 +84,10 @@ def __init__(self, repository, key, manifest, path=None, sync=True, do_files=Fal
             if sync and self.manifest.id != self.manifest_id:
                 # If repository is older than the cache something fishy is going on
                 if self.timestamp and self.timestamp > manifest.timestamp:
-                    raise self.RepositoryReplay()
+                    if isinstance(key, PlaintextKey):
+                        raise self.RepositoryIDNotUnique()
+                    else:
+                        raise self.RepositoryReplay()
                 # Make sure an encrypted repository has not been swapped for an unencrypted repository
                 if self.key_type is not None and self.key_type != str(key.TYPE):
                     raise self.EncryptionMethodMismatch()


Test patch:
Repository: borgbackup/borg

Issue:
Vagrantfile: fix /dev/fuse permissions
on wheezy, it is currently "permission denied".
owner/group of /dev/fuse = root/root. mode = 600

vagrant user is in "fuse" group.



Discussion:
```
sudo chgrp fuse /dev/fuse
sudo chmod g+rw /dev/fuse
```

```
==> freebsd64: fuse: failed to open fuse device: Permission denied
crw-rw----  1 root  operator  0x53 Aug 27 22:33 /dev/fuse
uid=1001(vagrant) gid=1001(vagrant) groups=1001(vagrant),0(wheel),5(operator),1002(vboxusers)
```

strange...


Resolving patch:
diff --git a/borg/testsuite/archiver.py b/borg/testsuite/archiver.py
--- a/borg/testsuite/archiver.py
+++ b/borg/testsuite/archiver.py
@@ -27,6 +27,7 @@
 from ..remote import RemoteRepository, PathNotAllowed
 from ..repository import Repository
 from . import BaseTestCase, changedir, environment_variable, no_selinux
+from .platform import fakeroot_detected
 
 try:
     import llfuse
@@ -1316,6 +1317,12 @@ def test_basic_functionality(self):
     def test_overwrite(self):
         pass
 
+    def test_fuse(self):
+        if fakeroot_detected():
+            unittest.skip('test_fuse with the binary is not compatible with fakeroot')
+        else:
+            super().test_fuse()
+
 
 class ArchiverCheckTestCase(ArchiverTestCaseBase):
 


Test patch:
Repository: borgbackup/borg

Issue:
byte range error
```
______________________ TestKey.test_assert_id[KeyfileKey] ______________________
self = <borg.testsuite.key.TestKey object at 0x1084c6128>
key = <borg.key.KeyfileKey object at 0x1084caf28>
    def test_assert_id(self, key):
        plaintext = b'123456789'
        id = key.id_hash(plaintext)
        key.assert_id(id, plaintext)
        id_changed = bytearray(id)
>       id_changed[0] += 1
E       ValueError: byte must be in range(0, 256)

.tox/py34/lib/python3.4/site-packages/borg/testsuite/key.py:189: ValueError
```



Discussion:

Resolving patch:
diff --git a/src/borg/testsuite/key.py b/src/borg/testsuite/key.py
--- a/src/borg/testsuite/key.py
+++ b/src/borg/testsuite/key.py
@@ -155,7 +155,7 @@ def test_passphrase(self, keys_dir, monkeypatch):
 
     def _corrupt_byte(self, key, data, offset):
         data = bytearray(data)
-        data[offset] += 1
+        data[offset] ^= 1
         with pytest.raises(IntegrityError):
             key.decrypt(b'', data)
 
@@ -186,7 +186,7 @@ def test_assert_id(self, key):
         id = key.id_hash(plaintext)
         key.assert_id(id, plaintext)
         id_changed = bytearray(id)
-        id_changed[0] += 1
+        id_changed[0] ^= 1
         with pytest.raises(IntegrityError):
             key.assert_id(id_changed, plaintext)
         plaintext_changed = plaintext + b'1'


Test patch:
Repository: borgbackup/borg

Issue:
check crashes if repo is valid but completely empty
This is because of

```py
    def identify_key(self, repository):
        try:
            some_chunkid, _ = next(self.chunks.iteritems())
        except StopIteration:
            # repo is completely empty, no chunks
            return None                                             <--
        cdata = repository.get(some_chunkid)
        return key_factory(repository, cdata)
```

It later tries to use `self.key` thereby crashes.

I think this case should either abort and tell to delete repository & start over, or basically do a re-initialize.


Discussion:

Resolving patch:
diff --git a/borg/archive.py b/borg/archive.py
--- a/borg/archive.py
+++ b/borg/archive.py
@@ -833,6 +833,9 @@ def check(self, repository, repair=False, archive=None, last=None, prefix=None,
         self.repair = repair
         self.repository = repository
         self.init_chunks()
+        if not self.chunks:
+            logger.error('Repository contains no apparent data at all, cannot continue check/repair.')
+            return False
         self.key = self.identify_key(repository)
         if Manifest.MANIFEST_ID not in self.chunks:
             logger.error("Repository manifest not found!")
diff --git a/borg/repository.py b/borg/repository.py
--- a/borg/repository.py
+++ b/borg/repository.py
@@ -438,6 +438,9 @@ def report_error(msg):
             transaction_id = self.get_index_transaction_id()
         if transaction_id is None:
             transaction_id = self.io.get_latest_segment()
+        if transaction_id is None:
+            report_error('This repository contains no valid data.')
+            return False
         if repair:
             self.io.cleanup(transaction_id)
         segments_transaction_id = self.io.get_segments_transaction_id()
diff --git a/borg/testsuite/archiver.py b/borg/testsuite/archiver.py
--- a/borg/testsuite/archiver.py
+++ b/borg/testsuite/archiver.py
@@ -1444,6 +1444,13 @@ def test_extra_chunks(self):
         self.cmd('check', self.repository_location, exit_code=0)
         self.cmd('extract', '--dry-run', self.repository_location + '::archive1', exit_code=0)
 
+    def test_empty_repository(self):
+        with Repository(self.repository_location, exclusive=True) as repository:
+            for id_ in repository.list():
+                repository.delete(id_)
+            repository.commit()
+        self.cmd('check', self.repository_location, exit_code=1)
+
 
 @pytest.mark.skipif(sys.platform == 'cygwin', reason='remote is broken on cygwin and hangs')
 class RemoteArchiverTestCase(ArchiverTestCase):


Test patch:
Repository: borgbackup/borg

Issue:
Possibility of BORG_NEW_PASSPHRASE environment variable
I was hoping to be able to script the changing of the password on all of my repos every so often but I notice that there is currently only BORG_PASSPHRASE for the current passphrase and BORG_DISPLAY_PASSPHRASE to suppress the question of showing your new passphrase. Would it be possible to add a variable to hold the new passphrase, for example BORG_NEW_PASSPHRASE in order to be able to script the change-password function more easily.



Discussion:
sounds like a reasonable feature to add.

until we have it, one can work around like this (untested):

```
borg change-passphrase repo <<<EOF
newpass
newpass
N
EOF
```


Resolving patch:
diff --git a/src/borg/key.py b/src/borg/key.py
--- a/src/borg/key.py
+++ b/src/borg/key.py
@@ -247,11 +247,19 @@ def init_ciphers(self, manifest_nonce=0):
 
 class Passphrase(str):
     @classmethod
-    def env_passphrase(cls, default=None):
-        passphrase = os.environ.get('BORG_PASSPHRASE', default)
+    def _env_passphrase(cls, env_var, default=None):
+        passphrase = os.environ.get(env_var, default)
         if passphrase is not None:
             return cls(passphrase)
 
+    @classmethod
+    def env_passphrase(cls, default=None):
+        return cls._env_passphrase('BORG_PASSPHRASE', default)
+
+    @classmethod
+    def env_new_passphrase(cls, default=None):
+        return cls._env_passphrase('BORG_NEW_PASSPHRASE', default)
+
     @classmethod
     def getpass(cls, prompt):
         return cls(getpass.getpass(prompt))
@@ -276,6 +284,9 @@ def verification(cls, passphrase):
 
     @classmethod
     def new(cls, allow_empty=False):
+        passphrase = cls.env_new_passphrase()
+        if passphrase is not None:
+            return passphrase
         passphrase = cls.env_passphrase()
         if passphrase is not None:
             return passphrase
diff --git a/src/borg/testsuite/archiver.py b/src/borg/testsuite/archiver.py
--- a/src/borg/testsuite/archiver.py
+++ b/src/borg/testsuite/archiver.py
@@ -1363,6 +1363,14 @@ def test_compression_lzma_uncompressible(self):
         size, csize = self._get_sizes('lzma', compressible=False)
         assert csize >= size
 
+    def test_change_passphrase(self):
+        self.cmd('init', self.repository_location)
+        os.environ['BORG_NEW_PASSPHRASE'] = 'newpassphrase'
+        # here we have both BORG_PASSPHRASE and BORG_NEW_PASSPHRASE set:
+        self.cmd('change-passphrase', self.repository_location)
+        os.environ['BORG_PASSPHRASE'] = 'newpassphrase'
+        self.cmd('list', self.repository_location)
+
     def test_break_lock(self):
         self.cmd('init', self.repository_location)
         self.cmd('break-lock', self.repository_location)


Test patch:
Repository: borgbackup/borg

Issue:
borg mount: TypeError in error handler
```
borg mount backup-borg/ /mnt/backup                   :(
Enter passphrase for key /mnt/data/gdrive/backup-borg: 
Local Exception
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 2769, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 2706, in run
    return args.func(args)
  File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 95, in wrapper
    kwargs['manifest'], kwargs['key'] = Manifest.load(repository)
  File "/usr/lib/python3.5/site-packages/borg/helpers.py", line 210, in load
    _, data = key.decrypt(None, cdata)
  File "/usr/lib/python3.5/site-packages/borg/key.py", line 217, in decrypt
    raise IntegrityError('Chunk %s: Encryption envelope checksum mismatch' % bin_to_hex(id))
  File "/usr/lib/python3.5/site-packages/borg/helpers.py", line 880, in bin_to_hex
    return hexlify(binary).decode('ascii')
TypeError: a bytes-like object is required, not 'NoneType'

Platform: Linux protagonist 4.8.10-1-ARCH #1 SMP PREEMPT Mon Nov 21 11:55:43 CET 2016 x86_64 
Linux: arch  
Borg: 1.1.0.dev35+ng701e26a  Python: CPython 3.5.2
PID: 10538  CWD: /mnt/data/gdrive
sys.argv: ['/usr/bin/borg', 'mount', 'backup-borg/', '/mnt/backup']
SSH_ORIGINAL_COMMAND: None

```

I've also tested with 1.0.8 version in arch repositories, same issue.


Discussion:
Looks like you have some IntegrityError there and the error handler falls over his own feet.

Resolving patch:
diff --git a/borg/key.py b/borg/key.py
--- a/borg/key.py
+++ b/borg/key.py
@@ -105,7 +105,8 @@ def encrypt(self, data):
 
     def decrypt(self, id, data):
         if data[0] != self.TYPE:
-            raise IntegrityError('Chunk %s: Invalid encryption envelope' % bin_to_hex(id))
+            id_str = bin_to_hex(id) if id is not None else '(unknown)'
+            raise IntegrityError('Chunk %s: Invalid encryption envelope' % id_str)
         data = self.compressor.decompress(memoryview(data)[1:])
         if id and sha256(data).digest() != id:
             raise IntegrityError('Chunk %s: id verification failed' % bin_to_hex(id))


Test patch:
Repository: borgbackup/borg

Issue:
show broken files in list
borg-list should be able to show whether a file has any broken chunks at the moment. Perhaps it can also look up whether a --repair would fix it, since this is a cheap operation.

From the mailing list thread about annoying stuff in Borg.


---

:moneybag: [there is a bounty for this](https://www.bountysource.com/issues/38553573-show-broken-files-in-list)



Discussion:

Resolving patch:
diff --git a/src/borg/helpers.py b/src/borg/helpers.py
--- a/src/borg/helpers.py
+++ b/src/borg/helpers.py
@@ -1430,6 +1430,7 @@ class ItemFormatter(BaseFormatter):
         'csize': 'compressed size',
         'num_chunks': 'number of chunks in this file',
         'unique_chunks': 'number of unique chunks in this file',
+        'health': 'either "healthy" (file ok) or "broken" (if file has all-zero replacement chunks)',
     }
     KEY_GROUPS = (
         ('type', 'mode', 'uid', 'gid', 'user', 'group', 'path', 'bpath', 'source', 'linktarget', 'flags'),
@@ -1437,6 +1438,7 @@ class ItemFormatter(BaseFormatter):
         ('mtime', 'ctime', 'atime', 'isomtime', 'isoctime', 'isoatime'),
         tuple(sorted(hashlib.algorithms_guaranteed)),
         ('archiveid', 'archivename', 'extra'),
+        ('health', )
     )
 
     @classmethod
@@ -1526,6 +1528,7 @@ def get_item_data(self, item):
         item_data['linktarget'] = source
         item_data['extra'] = extra
         item_data['flags'] = item.get('bsdflags')
+        item_data['health'] = 'broken' if 'chunks_healthy' in item else 'healthy'
         for key in self.used_call_keys:
             item_data[key] = self.call_keys[key](item)
         return item_data
diff --git a/src/borg/testsuite/archiver.py b/src/borg/testsuite/archiver.py
--- a/src/borg/testsuite/archiver.py
+++ b/src/borg/testsuite/archiver.py
@@ -2082,6 +2082,8 @@ def test_missing_file_chunk(self):
         output = self.cmd('check', '--repair', self.repository_location, exit_code=0)
         self.assert_in('New missing file chunk detected', output)
         self.cmd('check', self.repository_location, exit_code=0)
+        output = self.cmd('list', '--format={health}#{path}{LF}', self.repository_location + '::archive1', exit_code=0)
+        self.assert_in('broken#', output)
         # check that the file in the old archives has now a different chunk list without the killed chunk
         for archive_name in ('archive1', 'archive2'):
             archive, repository = self.open_archive(archive_name)
@@ -2110,6 +2112,9 @@ def test_missing_file_chunk(self):
                         break
                 else:
                     self.assert_true(False)  # should not happen
+        # list is also all-healthy again
+        output = self.cmd('list', '--format={health}#{path}{LF}', self.repository_location + '::archive1', exit_code=0)
+        self.assert_not_in('broken#', output)
 
     def test_missing_archive_item_chunk(self):
         archive, repository = self.open_archive('archive1')


Test patch:
Repository: borgbackup/borg

Issue:
py.test has lost the nice assertion failure display
when fixing #1930, I noticed that when the assertion fails, it just says "AssertionError", it does not display the values of the terms in the assertion expression.



Discussion:
E.g. like here:
```
__________________________________________________________ TestLocationWithoutEnv.test_file ___________________________________________________________

self = <borg.testsuite.helpers.TestLocationWithoutEnv object at 0x7fced6086940>

    def test_file(self):
        #monkeypatch.delenv('BORG_REPO', raising=False)
        assert repr(Location('xfile:///some/path::archive')) == \
>           "Location(proto='file', user=None, host=None, port=None, path='/some/path', archive='archive')"
E       AssertionError

borg/testsuite/helpers.py:48: AssertionError
```
The lhs value would have been interesting...
i believe with the particular layout for the helpers, you need 

http://doc.pytest.org/en/latest/writing_plugins.html#pytest.register_assert_rewrite

Resolving patch:
diff --git a/conftest.py b/conftest.py
--- a/conftest.py
+++ b/conftest.py
@@ -4,6 +4,8 @@
 
 import pytest
 
+# needed to get pretty assertion failures in unit tests:
+pytest.register_assert_rewrite('borg')
 
 # This is a hack to fix path problems because "borg" (the package) is in the source root.
 # When importing the conftest an "import borg" can incorrectly import the borg from the


Test patch: