internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: borgbackup/borg

Issue:
Converted repo does not use existing chunks
It seems that borg cannot re-use chunks created by attic.

Here a small demo:

```
REPO=$PWD/repo
dd if=/dev/urandom of=file_to_backup bs=1M count=50
attic init $REPO
attic create $REPO::backup file_to_backup
echo -n "attic repo: " && du -hs $REPO
borg upgrade $REPO
echo -n "upgraded repo: " && du -hs $REPO
borg create $REPO::backup2 file_to_backup
echo -n "final repo: " && du -hs $REPO
```

Output:
```
[...]
attic repo: 51M /tmp/backup_test/repo
[...]
upgraded repo: 51M      /tmp/backup_test/repo
final repo: 101M        /tmp/backup_test/repo
```






Discussion:
Thanks for the reproduction instructions, these always help a lot.

This ticket is a duplicate of #787, though.

Check docs, do we have a note in borg upgrade about this or in faq?

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -2842,7 +2842,20 @@ def process_epilog(epilog):
                                help='repository to prune')
 
         upgrade_epilog = process_epilog("""
-        Upgrade an existing Borg repository.
+        Upgrade an existing, local Borg repository.
+
+        When you do not need borg upgrade
+        +++++++++++++++++++++++++++++++++
+
+        Not every change requires that you run ``borg upgrade``.
+
+        You do **not** need to run it when:
+
+        - moving your repository to a different place
+        - upgrading to another point release (like 1.0.x to 1.0.y),
+          except when noted otherwise in the changelog
+        - upgrading from 1.0.x to 1.1.x,
+          except when noted otherwise in the changelog
 
         Borg 1.x.y upgrades
         +++++++++++++++++++
@@ -2874,8 +2887,13 @@ def process_epilog(epilog):
 
         Currently, only LOCAL repositories can be upgraded (issue #465).
 
-        It will change the magic strings in the repository's segments
-        to match the new Borg magic strings. The keyfiles found in
+        Please note that ``borg create`` (since 1.0.0) uses bigger chunks by
+        default than old borg or attic did, so the new chunks won't deduplicate
+        with the old chunks in the upgraded repository.
+        See ``--chunker-params`` option of ``borg create`` and ``borg recreate``.
+
+        ``borg upgrade`` will change the magic strings in the repository's
+        segments to match the new Borg magic strings. The keyfiles found in
         $ATTIC_KEYS_DIR or ~/.attic/keys/ will also be converted and
         copied to $BORG_KEYS_DIR or ~/.config/borg/keys.
 


Test patch:
Repository: borgbackup/borg

Issue:
document "borg serve --append-only" behaviour for borg init
See #2413.


Discussion:

Resolving patch:
diff --git a/borg/archiver.py b/borg/archiver.py
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -1764,7 +1764,20 @@ def build_parser(self, args=None, prog=None):
                                help='repository to prune')
 
         upgrade_epilog = textwrap.dedent("""
-        Upgrade an existing Borg repository.
+        Upgrade an existing, local Borg repository.
+
+        When you do not need borg upgrade
+        +++++++++++++++++++++++++++++++++
+
+        Not every change requires that you run ``borg upgrade``.
+
+        You do **not** need to run it when:
+
+        - moving your repository to a different place
+        - upgrading to another point release (like 1.0.x to 1.0.y),
+          except when noted otherwise in the changelog
+        - upgrading from 1.0.x to 1.1.x,
+          except when noted otherwise in the changelog
 
         Borg 1.x.y upgrades
         +++++++++++++++++++
@@ -1796,8 +1809,13 @@ def build_parser(self, args=None, prog=None):
 
         Currently, only LOCAL repositories can be upgraded (issue #465).
 
-        It will change the magic strings in the repository's segments
-        to match the new Borg magic strings. The keyfiles found in
+        Please note that ``borg create`` (since 1.0.0) uses bigger chunks by
+        default than old borg or attic did, so the new chunks won't deduplicate
+        with the old chunks in the upgraded repository.
+        See ``--chunker-params`` option of ``borg create`` and ``borg recreate``.
+
+        ``borg upgrade`` will change the magic strings in the repository's
+        segments to match the new Borg magic strings. The keyfiles found in
         $ATTIC_KEYS_DIR or ~/.attic/keys/ will also be converted and
         copied to $BORG_KEYS_DIR or ~/.config/borg/keys.
 


Test patch:
Repository: borgbackup/borg

Issue:
--exclude is broken
**Task:** --exclude and --exclude-from were inadvertently changed to recurse during the latest --patterns fixup. Restore correct behaviour.

Reported by @jdchristensen 


Discussion:

Resolving patch:
diff --git a/src/borg/patterns.py b/src/borg/patterns.py
--- a/src/borg/patterns.py
+++ b/src/borg/patterns.py
@@ -347,8 +347,8 @@ def parse_pattern(pattern, fallback=FnmatchPattern, recurse_dir=True):
 def parse_exclude_pattern(pattern_str, fallback=FnmatchPattern):
     """Read pattern from string and return an instance of the appropriate implementation class.
     """
-    epattern_obj = parse_pattern(pattern_str, fallback)
-    return CmdTuple(epattern_obj, IECommand.Exclude)
+    epattern_obj = parse_pattern(pattern_str, fallback, recurse_dir=False)
+    return CmdTuple(epattern_obj, IECommand.ExcludeNoRecurse)
 
 
 def parse_inclexcl_command(cmd_line_str, fallback=ShellPattern):


Test patch:
Repository: borgbackup/borg

Issue:
hashindex size limitations
The size of it is limited to 2**31 entries, probably not important right now (since that would need at least ~180 GB of usable memory). [hash_sizes](https://github.com/borgbackup/borg/blob/master/src/borg/_hashindex.c#L49)

We use normal C I/O currently, which will be limited to 32 bit offsets depending on platform (I think Linux 32 bits and Windows?). This would limit the repository index to ~60 million entries and the chunks cache to ~49 million entries. This is already addressed by the patches in #1688, since Python I/O always uses large offsets if available as far as I'm aware.


Discussion:
Split this into 2 or 3 tickets?

I’m pretty sure both windows and linux allow for C i/o with full 64 bit offsets. On linux the default is controlled by a #define _FILE_OFFSET_BITS 64, but there is always the option to just use the 64 postfixed functions on 32bit systems. Might need O_LARGEFILE on opening the file.


Resolving patch:
diff --git a/src/borg/repository.py b/src/borg/repository.py
--- a/src/borg/repository.py
+++ b/src/borg/repository.py
@@ -324,9 +324,8 @@ def open_index(self, transaction_id, auto_recover=True):
         index_path = os.path.join(self.path, 'index.%d' % transaction_id).encode('utf-8')
         try:
             return NSIndex.read(index_path)
-        except RuntimeError as error:
-            assert str(error) == 'hashindex_read failed'  # everything else means we're in *deep* trouble
-            logger.warning('Repository index missing or corrupted, trying to recover')
+        except (ValueError, OSError) as exc:
+            logger.warning('Repository index missing or corrupted, trying to recover from: %s', exc)
             os.unlink(index_path)
             if not auto_recover:
                 raise
@@ -357,7 +356,8 @@ def prepare_txn(self, transaction_id, do_cleanup=True):
         if not self.index or transaction_id is None:
             try:
                 self.index = self.open_index(transaction_id, False)
-            except RuntimeError:
+            except (ValueError, OSError) as exc:
+                logger.warning('Checking repository transaction due to previous error: %s', exc)
                 self.check_transaction()
                 self.index = self.open_index(transaction_id, False)
         if transaction_id is None:


Test patch:
Repository: borgbackup/borg

Issue:
test_log_json test fails for the borg binaries
```
gw2] darwin -- Python 3.4.0 /vagrant/borg/borg/.tox/py34/bin/python3.4
self = <borg.testsuite.archiver.ArchiverTestCaseBinary testMethod=test_log_json>

    def test_log_json(self):
        self.create_test_files()
        self.cmd('init', '--encryption=repokey', self.repository_location)
        log = self.cmd('create', '--log-json', self.repository_location + '::test', 'input', '--list', '--debug')
        messages = {}  # type -> message, one of each kind
        for line in log.splitlines():
            msg = json.loads(line)
            messages[msg['type']] = msg
    
        file_status = messages['file_status']
        assert 'status' in file_status
        assert file_status['path'].startswith('input')
    
        log_message = messages['log_message']
        assert isinstance(log_message['time'], float)
        assert log_message['levelname'] == 'DEBUG'  # there should only be DEBUG messages
>       assert log_message['name'].startswith('borg.')
E       AssertionError: assert False
E        +  where False = <built-in method startswith of str object at 0x109b6b078>('borg.')
E        +    where <built-in method startswith of str object at 0x109b6b078> = 'pyimod03_importers'.startswith

.tox/py34/lib/python3.4/site-packages/borg/testsuite/archiver.py:1680: AssertionError
```


Discussion:
Looks like pyInstaller modifies the module names or something

```
{"name": "borg.logger", "type": "log_message", "levelname": "DEBUG", "message": "using builtin fallback logging configuration", "time": 1494873695.385876}
{"name": "pyimod03_importers", "type": "log_message", "levelname": "DEBUG", "message": "35 self tests completed in 0.08 seconds", "time": 1494873695.4669607}
{"name": "pyimod03_importers", "type": "log_message", "levelname": "DEBUG", "message": "TAM-verified manifest", "time": 1494873695.4725752}
{"name": "pyimod03_importers", "type": "log_message", "levelname": "DEBUG", "message": "security: read previous_location '/home/mabe/Projekte/_oss/borg/testrepo'", "time": 1494873695.473697}
{"name": "pyimod03_importers", "type": "log_message", "levelname": "DEBUG", "message": "security: read manifest timestamp '2017-05-15T18:24:32.634834'", "time": 1494873695.4740126}
{"name": "pyimod03_importers", "type": "log_message", "levelname": "DEBUG", "message": "security: determined newest manifest timestamp as 2017-05-15T18:24:32.634834", "time": 1494873695.4740918}
```

Relaxing the condition in the test should suffice.

Resolving patch:
diff --git a/src/borg/testsuite/archiver.py b/src/borg/testsuite/archiver.py
--- a/src/borg/testsuite/archiver.py
+++ b/src/borg/testsuite/archiver.py
@@ -1678,7 +1678,6 @@ def test_log_json(self):
         log_message = messages['log_message']
         assert isinstance(log_message['time'], float)
         assert log_message['levelname'] == 'DEBUG'  # there should only be DEBUG messages
-        assert log_message['name'].startswith('borg.')
         assert isinstance(log_message['message'], str)
 
     def test_common_options(self):


Test patch:
Repository: borgbackup/borg

Issue:
Very confusing error message when _version.py contains misdetected version
When installing borg from a git repository without tags or when setuptools_scm can not get the correct version for some other reason, borg fails with very confusing error messages.

```
borg list: error: argument REPOSITORY_OR_ARCHIVE: invalid validator value: '/tmp/borg-repo'
```

*How to reproduce:*
easiest way without messing up the repository is to replace _version.py with
```
version = '0.1.dev1+aaaa'
```
But installing from a git repository without tags should reproduce this too.

*Cause:*
While parsing arguments `replace_placeholders` tries to use the version, but throws an exception which is somehow replaced by a confusing message in the argument parser exception handling.
```
>>> borg.helpers.Location('/tmp/borg-repo')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/martin/sw-dev/borg/src/borg/helpers.py", line 979, in __init__
    if not self.parse(self.orig):
  File "/home/martin/sw-dev/borg/src/borg/helpers.py", line 983, in parse
    text = replace_placeholders(text)
  File "/home/martin/sw-dev/borg/src/borg/helpers.py", line 696, in replace_placeholders
    'borgpatch': '%d.%d.%d' % borg_version_tuple[:3],
TypeError: %d format: a number is required, not str
```

*Possible solution:*
Do some sanity checking in `borg/__init__.py` and abort early with a clear error message?


Discussion:
i'll take a look at this one 
the problem arrises from a custom repo having bad tags - a good solution needs a bit more discussion

Resolving patch:
diff --git a/src/borg/__init__.py b/src/borg/__init__.py
--- a/src/borg/__init__.py
+++ b/src/borg/__init__.py
@@ -4,3 +4,15 @@
 
 
 __version_tuple__ = tuple(LooseVersion(__version__).version[:3])
+
+# assert that all semver components are integers
+# this is mainly to show errors when people repackage poorly
+# and setuptools_scm determines a 0.1.dev... version
+assert all(isinstance(v, int) for v in __version_tuple__), \
+    """\
+broken borgbackup version metadata: %r
+
+version metadata is obtained dynamically on installation via setuptools_scm,
+please ensure your git repo has the correct tags or you provide the version
+using SETUPTOOLS_SCM_PRETEND_VERSION in your build script.
+""" % __version__


Test patch:
Repository: borgbackup/borg

Issue:
ux: mount fails after entering the password if llfuse is missing
just noticed that while developing,
it would be nice to get an error prior to password entry


Discussion:
Somewhat related: #1931 

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -1229,15 +1229,20 @@ def _delete_repository(self, args, repository):
         logger.info("Cache deleted.")
         return self.exit_code
 
-    @with_repository()
-    def do_mount(self, args, repository, manifest, key):
+    def do_mount(self, args):
         """Mount archive or an entire repository as a FUSE filesystem"""
         try:
-            from .fuse import FuseOperations
+            import borg.fuse
         except ImportError as e:
-            self.print_error('Loading fuse support failed [ImportError: %s]' % str(e))
+            self.print_error('borg mount not available: loading fuse support failed [ImportError: %s]' % str(e))
             return self.exit_code
 
+        return self._do_mount(args)
+
+    @with_repository()
+    def _do_mount(self, args, repository, manifest, key):
+        from .fuse import FuseOperations
+
         if not os.path.isdir(args.mountpoint) or not os.access(args.mountpoint, os.R_OK | os.W_OK | os.X_OK):
             self.print_error('%s: Mountpoint must be a writable directory' % args.mountpoint)
             return self.exit_code


Test patch:
Repository: borgbackup/borg

Issue:
Archive name placeholder {now} does weird things
Hi, 

when using archive name placeholders i get some strange results. This is borg version 1.0.10 on Arch Linux. 

This is the archive name i use (exactly like provided in 'borg help placeholders' ): 
`'{hostname}-{now:%Y-%m-%d_%H:%M:%S}'`

This is the result: 
`Archive name: viki-2017-b454999f3eae497886e6183d96d436a5-21_viki:22:32`

{hostname} is ok.
%Y is ok. 
%m should be the month of the year, right? Why its "b454999f3eae497886e6183d96d436a5" ?
%d is ok.
%H should be Hour, but is the hostname again?
%M is ok. 
%S is ok.



Discussion:
Works for me. Can you share the verbatim command line you are using (and which shell)?
I just wanted to share my systemd service file then it dawned on me. systemd has its own placeholders. 
%m is the machine ID and %H is the hostname there.


Ok. After some searching around i found the answer. I just had to "escape" the parameterization by doubling up the percent signs. systemd's parameter expansion turns %% into %. Maybe you could put a hint in the documenation about that.

`'{hostname}-{now:%%Y-%%m-%%d_%%H:%%M:%%S}'`
See also: systemd/systemd#624
And: http://stackoverflow.com/q/40532499/675646

Remarkably strange choices here. Single quotes in Unix land meant "do not interpret whatever is in these quotes" for some decades.

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -2009,7 +2009,17 @@ def do_break_lock(self, args, repository):
 
             borg create /path/to/repo::{hostname}-{user}-{utcnow} ...
             borg create /path/to/repo::{hostname}-{now:%Y-%m-%d_%H:%M:%S} ...
-            borg prune --prefix '{hostname}-' ...\n\n''')
+            borg prune --prefix '{hostname}-' ...
+
+        .. note::
+            systemd uses a difficult, non-standard syntax for command lines in unit files (refer to
+            the `systemd.unit(5)` manual page).
+
+            When invoking borg from unit files, pay particular attention to escaping,
+            especially when using the now/utcnow placeholders, since systemd performs its own
+            %-based variable replacement even in quoted text. To avoid interference from systemd,
+            double all percent signs (``{hostname}-{now:%Y-%m-%d_%H:%M:%S}``
+            becomes ``{hostname}-{now:%%Y-%%m-%%d_%%H:%%M:%%S}``).\n\n''')
     helptext['compression'] = textwrap.dedent('''
         Compression is lz4 by default. If you want something else, you have to specify what you want.
 


Test patch:
Repository: borgbackup/borg

Issue:
! pf:/some/dir Does not exclude directory
I have a folder `~/borg-test` containing a file `f0` and directories `d1` and `d2` containing files `f1` and `f2` respectively. When I created an archive of it using the following patterns file:
```
R /home/me/borg-test
! pf:/home/me/borg-test/d1
```
it included all files and directories, and did not exclude `d1`.
Using `-` instead of `!` didn't exclude it either.

This is using the standalone binary borg-linux64 of version 1.1.0b5




Discussion:
Sounds like the same issue as #2470?
In some ways it's the mirror image of #2470.  In that either the `+ /home/me/borg-test/d1` wasn't including `d1` when it should or the `! /home/me/borg-test/**` was excluding it when it shouldn't.  When the patterns file contains both `+` and `-` it's not always clear which isn't working as expected.  In this case `! pf:/home/me/borg-test/d1` isn't excluding `d1` when it should.


Resolving patch:
diff --git a/src/borg/patterns.py b/src/borg/patterns.py
--- a/src/borg/patterns.py
+++ b/src/borg/patterns.py
@@ -150,9 +150,8 @@ def match(self, path):
 
         if value is not non_existent:
             # we have a full path match!
-            # TODO: get from pattern; don't hard-code
-            self.recurse_dir = True
-            return value
+            self.recurse_dir = command_recurses_dir(value)
+            return self.is_include_cmd[value]
 
         # this is the slow way, if we have many patterns in self._items:
         for (pattern, cmd) in self._items:
@@ -325,6 +324,11 @@ class IECommand(Enum):
     ExcludeNoRecurse = 5
 
 
+def command_recurses_dir(cmd):
+    # TODO?: raise error or return None if *cmd* is RootPath or PatternStyle
+    return cmd not in [IECommand.ExcludeNoRecurse]
+
+
 def get_pattern_class(prefix):
     try:
         return _PATTERN_CLASS_BY_PREFIX[prefix]
@@ -386,7 +390,7 @@ def parse_inclexcl_command(cmd_line_str, fallback=ShellPattern):
             raise argparse.ArgumentTypeError("Invalid pattern style: {}".format(remainder_str))
     else:
         # determine recurse_dir based on command type
-        recurse_dir = cmd not in [IECommand.ExcludeNoRecurse]
+        recurse_dir = command_recurses_dir(cmd)
         val = parse_pattern(remainder_str, fallback, recurse_dir)
 
     return CmdTuple(val, cmd)


Test patch:
Repository: borgbackup/borg

Issue:
Feature flags: cache does not obey READ operation
```py
@with_repository(fake='dry_run', exclusive=True, compatibility=(Manifest.Operation.WRITE,))
def do_create(self, args, repository, manifest=None, key=None):
```

But the cache may be synced, which needs Operation.READ, which is not checked.

The cache sync thus needs to perform another operation check with READ, independent of what the command opening the cache may have checked.


Discussion:

Resolving patch:
diff --git a/src/borg/cache.py b/src/borg/cache.py
--- a/src/borg/cache.py
+++ b/src/borg/cache.py
@@ -721,6 +721,11 @@ def legacy_cleanup():
             except:
                 pass
 
+        # The cache can be used by a command that e.g. only checks against Manifest.Operation.WRITE,
+        # which does not have to include all flags from Manifest.Operation.READ.
+        # Since the sync will attempt to read archives, check compatibility with Manifest.Operation.READ.
+        self.manifest.check_repository_compatibility((Manifest.Operation.READ, ))
+
         self.begin_txn()
         with cache_if_remote(self.repository, decrypted_cache=self.key) as decrypted_repository:
             legacy_cleanup()
diff --git a/src/borg/testsuite/archiver.py b/src/borg/testsuite/archiver.py
--- a/src/borg/testsuite/archiver.py
+++ b/src/borg/testsuite/archiver.py
@@ -1440,6 +1440,12 @@ def test_unknown_feature_on_create(self):
         self.add_unknown_feature(Manifest.Operation.WRITE)
         self.cmd_raises_unknown_feature(['create', self.repository_location + '::test', 'input'])
 
+    def test_unknown_feature_on_cache_sync(self):
+        self.cmd('init', '--encryption=repokey', self.repository_location)
+        self.cmd('delete', '--cache-only', self.repository_location)
+        self.add_unknown_feature(Manifest.Operation.READ)
+        self.cmd_raises_unknown_feature(['create', self.repository_location + '::test', 'input'])
+
     def test_unknown_feature_on_change_passphrase(self):
         print(self.cmd('init', '--encryption=repokey', self.repository_location))
         self.add_unknown_feature(Manifest.Operation.CHECK)


Test patch: