internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: borgbackup/borg

Issue:
Example path for --exclude-from
<!--
Thank you for reporting an issue.

*IMPORTANT* -  *before* creating a new issue please look around:
 - Borgbackup documentation: http://borgbackup.readthedocs.io/en/stable/index.html
 - FAQ: https://borgbackup.readthedocs.io/en/stable/faq.html
 and
 - open issues in Github tracker: https://github.com/borgbackup/borg/issues
  
If you cannot find a similar problem, then create a new issue.

Please fill in as much of the template as possible.
-->

## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

ISSUE

## Describe the problem you're observing.

I found it quite hard to find out how to use/allow spaces in --exclude-from file, one extra example in manual/help might help a lot:

# No need to escape spaces 
/mnt/c/System Volume Information

(not having to escape spaces is quite counterintuitive :)




Discussion:
Well, if you consider that parsing of arguments and splitting at spaces is a feature of your shell (and quoting is for working around that feature), it is quite intuitive.

The contents of --exclude-from or --patterns-from files are not processed by your shell, but directly read by borg.

OTOH, when entering `borg ... <arguments>` on the command line or in a shell script, the arguments *are* processed by the shell.

Taking this one, documenting both patterns files and exclude files.
From what I understand (and from my tests) adding a space is not a problem since everything is handled by borg which reads one entry per line.

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -2320,8 +2320,10 @@ def do_break_lock(self, args, repository):
             /home/*/junk
             *.tmp
             fm:aa:something/*
-            re:^/home/[^/]\\.tmp/
-            sh:/home/*/.thumbnails
+            re:^home/[^/]\\.tmp/
+            sh:home/*/.thumbnails
+             # Example with spaces, no need to escape as it is processed by borg
+            some file with spaces.txt
             EOF
             $ borg create --exclude-from exclude.txt backup /
 
@@ -3307,6 +3309,7 @@ def define_archive_filters_group(subparser, *, sort_by=True, first_last=True):
         exclusive because the data is not actually compressed and deduplicated during a dry run.
 
         See the output of the "borg help patterns" command for more help on exclude patterns.
+
         See the output of the "borg help placeholders" command for more help on placeholders.
 
         .. man NOTES


Test patch:
Repository: borgbackup/borg

Issue:
write a test that reproduces the hashtable corruption issue
seems like some dists would like to have that...

#4829 


Discussion:

Resolving patch:
diff --git a/src/borg/testsuite/hashindex.py b/src/borg/testsuite/hashindex.py
--- a/src/borg/testsuite/hashindex.py
+++ b/src/borg/testsuite/hashindex.py
@@ -510,3 +510,45 @@ class AllIndexTestCase(BaseTestCase):
     def test_max_load_factor(self):
         assert NSIndex.MAX_LOAD_FACTOR < 1.0
         assert ChunkIndex.MAX_LOAD_FACTOR < 1.0
+
+
+class IndexCorruptionTestCase(BaseTestCase):
+    def test_bug_4829(self):
+
+        from struct import pack
+
+        def HH(x, y):
+            # make some 32byte long thing that depends on x and y.
+            # same x will mean a collision in the hashtable as bucket index is computed from
+            # first 4 bytes. giving a specific x targets bucket index x.
+            # y is to create different keys and does not go into the bucket index calculation.
+            # so, same x + different y --> collision
+            return pack('<IIQQQ', x, y, 0, 0, 0)  # 2 * 4 + 3 * 8 == 32
+
+        idx = NSIndex()
+
+        # create lots of colliding entries
+        for y in range(700):  # stay below max load to not trigger resize
+            idx[HH(0, y)] = (0, y)
+
+        assert idx.size() == 1031 * 40 + 18  # 1031 buckets + header
+
+        # delete lots of the collisions, creating lots of tombstones
+        for y in range(400):  # stay above min load to not trigger resize
+            del idx[HH(0, y)]
+
+        # create lots of colliding entries, within the not yet used part of the hashtable
+        for y in range(330):  # stay below max load to not trigger resize
+            # at y == 259 a resize will happen due to going beyond max EFFECTIVE load
+            # if the bug is present, that element will be inserted at the wrong place.
+            # and because it will be at the wrong place, it can not be found again.
+            idx[HH(600, y)] = 600, y
+
+        # now check if hashtable contents is as expected:
+
+        assert [idx.get(HH(0, y)) for y in range(400, 700)] == [(0, y) for y in range(400, 700)]
+
+        assert [HH(0, y) in idx for y in range(400)] == [False for y in range(400)]  # deleted entries
+
+        # this will fail at HH(600, 259) if the bug is present.
+        assert [idx.get(HH(600, y)) for y in range(330)] == [(600, y) for y in range(330)]


Test patch:
Repository: borgbackup/borg

Issue:
write a test that reproduces the hashtable corruption issue
seems like some dists would like to have that...

#4829 


Discussion:

Resolving patch:
diff --git a/borg/testsuite/hashindex.py b/borg/testsuite/hashindex.py
--- a/borg/testsuite/hashindex.py
+++ b/borg/testsuite/hashindex.py
@@ -284,3 +284,45 @@ def test_nsindex_segment_limit():
 def test_max_load_factor():
     assert NSIndex.MAX_LOAD_FACTOR < 1.0
     assert ChunkIndex.MAX_LOAD_FACTOR < 1.0
+
+
+class IndexCorruptionTestCase(BaseTestCase):
+    def test_bug_4829(self):
+
+        from struct import pack
+
+        def HH(x, y):
+            # make some 32byte long thing that depends on x and y.
+            # same x will mean a collision in the hashtable as bucket index is computed from
+            # first 4 bytes. giving a specific x targets bucket index x.
+            # y is to create different keys and does not go into the bucket index calculation.
+            # so, same x + different y --> collision
+            return pack('<IIQQQ', x, y, 0, 0, 0)  # 2 * 4 + 3 * 8 == 32
+
+        idx = NSIndex()
+
+        # create lots of colliding entries
+        for y in range(700):  # stay below max load to not trigger resize
+            idx[HH(0, y)] = (0, y)
+
+        # assert idx.size() == 1031 * 40 + 18  # 1031 buckets + header
+
+        # delete lots of the collisions, creating lots of tombstones
+        for y in range(400):  # stay above min load to not trigger resize
+            del idx[HH(0, y)]
+
+        # create lots of colliding entries, within the not yet used part of the hashtable
+        for y in range(330):  # stay below max load to not trigger resize
+            # at y == 259 a resize will happen due to going beyond max EFFECTIVE load
+            # if the bug is present, that element will be inserted at the wrong place.
+            # and because it will be at the wrong place, it can not be found again.
+            idx[HH(600, y)] = 600, y
+
+        # now check if hashtable contents is as expected:
+
+        assert [idx.get(HH(0, y)) for y in range(400, 700)] == [(0, y) for y in range(400, 700)]
+
+        assert [HH(0, y) in idx for y in range(400)] == [False for y in range(400)]  # deleted entries
+
+        # this will fail at HH(600, 259) if the bug is present.
+        assert [idx.get(HH(600, y)) for y in range(330)] == [(600, y) for y in range(330)]


Test patch:
Repository: borgbackup/borg

Issue:
add fuse info to borg debug info
at least: llfuse or pyfuse3 or none

also: llfuse/pyfuse3 version number


Discussion:

Resolving patch:
diff --git a/src/borg/helpers/misc.py b/src/borg/helpers/misc.py
--- a/src/borg/helpers/misc.py
+++ b/src/borg/helpers/misc.py
@@ -86,13 +86,17 @@ def sysinfo():
         msgpack_version = '.'.join(str(v) for v in msgpack.version)
     except:
         msgpack_version = 'unknown'
+    from ..fuse_impl import llfuse, BORG_FUSE_IMPL
+    llfuse_name = llfuse.__name__ if llfuse else 'None'
+    llfuse_version = (' %s' % llfuse.__version__) if llfuse else ''
+    llfuse_info = '%s%s [%s]' % (llfuse_name, llfuse_version, BORG_FUSE_IMPL)
     info = []
     if uname is not None:
         info.append('Platform: %s' % (' '.join(uname), ))
     if linux_distribution is not None:
         info.append('Linux: %s %s %s' % linux_distribution)
-    info.append('Borg: %s  Python: %s %s msgpack: %s' % (
-                borg_version, python_implementation, python_version, msgpack_version))
+    info.append('Borg: %s  Python: %s %s msgpack: %s fuse: %s' % (
+                borg_version, python_implementation, python_version, msgpack_version, llfuse_info))
     info.append('PID: %d  CWD: %s' % (os.getpid(), os.getcwd()))
     info.append('sys.argv: %r' % sys.argv)
     info.append('SSH_ORIGINAL_COMMAND: %r' % os.environ.get('SSH_ORIGINAL_COMMAND'))


Test patch:
Repository: borgbackup/borg

Issue:
--stats output should include repo location


## Have you checked borgbackup docs, FAQ, and open Github issues?

Y

## Is this a BUG / ISSUE report or a QUESTION?

Feature request

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).

borg 1.1.14 (from apk community repos)

#### Operating system (distribution) and version.

alpine linux

#### Full borg commandline that lead to the problem (leave away excludes and passwords)

borg create **--stats**

## Describe the problem you're observing.

I'd like `--stats` output also to include the repository location. This is to make it easier to discern different --stats outputs from a logfile. I have a script that's backing up in parallel to remote & local borg repos; It'd be nice for --stats output head to look like

    ------------------------------------------------------------------------------
    Repository: 12345@ch-s011.rsync.net:/repo/path   # for remote repo
    Repository: /path/to/local/repo                                # for local repo
    Archive name: gitea-downtime-2020-11-08-054300
    Archive fingerprint: 27992ecef61844cb5ee377d447f2984bd9fc4e62134dcf5d81706914fg97pqfe
    Time (start): Sun, 2020-11-08 05:43:08





Discussion:
Would you like to make a PR for that against master branch?
Not familiar with the codebase. I'll try to get around to it, but no promises.
@laur89 If you have some difficulties, I can have a look at it and point you to the right places :) 
@laur89 Any news, do you need some help?
If you prefer, I can develop it and explain to you how the whole thing works.
@Gu1nness local dev env setup turned out to be trickier than expected. Hoping to revisit coming weekend.
Ok nice, don't hesitate to ping me if necessary.
Okay @laur89 I'll patch it if you don't mind.

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -497,6 +497,7 @@ def info(self):
 
     def __str__(self):
         return '''\
+Repository: {location}
 Archive name: {0.name}
 Archive fingerprint: {0.fpr}
 Time (start): {start}
@@ -508,7 +509,9 @@ def __str__(self):
             self,
             start=OutputTimestamp(self.start.replace(tzinfo=timezone.utc)),
             end=OutputTimestamp(self.end.replace(tzinfo=timezone.utc)),
-            csize_max=self.cache.chunks[self.id].csize / MAX_DATA_SIZE)
+            csize_max=self.cache.chunks[self.id].csize / MAX_DATA_SIZE,
+            location=self.repository._location.canonical_path()
+)
 
     def __repr__(self):
         return 'Archive(%r)' % self.name


Test patch:
Repository: borgbackup/borg

Issue:
pytest deprecation warning about yield_fixture
pytest 6.2.1:
```
/vagrant/borg/borg/.tox/py36-none/lib/python3.6/site-packages/borg/testsuite/remote.py:73:
    PytestDeprecationWarning: @pytest.yield_fixture is deprecated.                                                                        
Use @pytest.fixture instead; they are the same.
Docs: https://docs.pytest.org/en/stable/warnings.html  
```

This should be fixed globally.


Discussion:
I've seen this in master branch, but guess it also applies to 1.1-maint.

Resolving patch:
diff --git a/src/borg/testsuite/benchmark.py b/src/borg/testsuite/benchmark.py
--- a/src/borg/testsuite/benchmark.py
+++ b/src/borg/testsuite/benchmark.py
@@ -13,7 +13,7 @@
 from .archiver import changedir, cmd
 
 
-@pytest.yield_fixture
+@pytest.fixture
 def repo_url(request, tmpdir, monkeypatch):
     monkeypatch.setenv('BORG_PASSPHRASE', '123456')
     monkeypatch.setenv('BORG_CHECK_I_KNOW_WHAT_I_AM_DOING', 'YES')
@@ -31,7 +31,7 @@ def repo(request, cmd, repo_url):
     return repo_url
 
 
-@pytest.yield_fixture(scope='session', params=["zeros", "random"])
+@pytest.fixture(scope='session', params=["zeros", "random"])
 def testdata(request, tmpdir_factory):
     count, size = 10, 1000*1000
     p = tmpdir_factory.mktemp('data')
diff --git a/src/borg/testsuite/cache.py b/src/borg/testsuite/cache.py
--- a/src/borg/testsuite/cache.py
+++ b/src/borg/testsuite/cache.py
@@ -208,7 +208,7 @@ def test_refcount_one_below_max_value(self):
 
 
 class TestAdHocCache:
-    @pytest.yield_fixture
+    @pytest.fixture
     def repository(self, tmpdir):
         self.repository_location = os.path.join(str(tmpdir), 'repository')
         with Repository(self.repository_location, exclusive=True, create=True) as repository:
diff --git a/src/borg/testsuite/remote.py b/src/borg/testsuite/remote.py
--- a/src/borg/testsuite/remote.py
+++ b/src/borg/testsuite/remote.py
@@ -70,7 +70,7 @@ def test_write(self, monkeypatch):
 
 
 class TestRepositoryCache:
-    @pytest.yield_fixture
+    @pytest.fixture
     def repository(self, tmpdir):
         self.repository_location = os.path.join(str(tmpdir), 'repository')
         with Repository(self.repository_location, exclusive=True, create=True) as repository:


Test patch:
Repository: borgbackup/borg

Issue:
pytest deprecation warning about yield_fixture
pytest 6.2.1:
```
/vagrant/borg/borg/.tox/py36-none/lib/python3.6/site-packages/borg/testsuite/remote.py:73:
    PytestDeprecationWarning: @pytest.yield_fixture is deprecated.                                                                        
Use @pytest.fixture instead; they are the same.
Docs: https://docs.pytest.org/en/stable/warnings.html  
```

This should be fixed globally.


Discussion:
I've seen this in master branch, but guess it also applies to 1.1-maint.

Resolving patch:
diff --git a/src/borg/testsuite/benchmark.py b/src/borg/testsuite/benchmark.py
--- a/src/borg/testsuite/benchmark.py
+++ b/src/borg/testsuite/benchmark.py
@@ -13,7 +13,7 @@
 from .archiver import changedir, cmd
 
 
-@pytest.yield_fixture
+@pytest.fixture
 def repo_url(request, tmpdir, monkeypatch):
     monkeypatch.setenv('BORG_PASSPHRASE', '123456')
     monkeypatch.setenv('BORG_CHECK_I_KNOW_WHAT_I_AM_DOING', 'YES')
@@ -31,7 +31,7 @@ def repo(request, cmd, repo_url):
     return repo_url
 
 
-@pytest.yield_fixture(scope='session', params=["zeros", "random"])
+@pytest.fixture(scope='session', params=["zeros", "random"])
 def testdata(request, tmpdir_factory):
     count, size = 10, 1000*1000
     p = tmpdir_factory.mktemp('data')
diff --git a/src/borg/testsuite/cache.py b/src/borg/testsuite/cache.py
--- a/src/borg/testsuite/cache.py
+++ b/src/borg/testsuite/cache.py
@@ -207,7 +207,7 @@ def test_refcount_one_below_max_value(self):
 
 
 class TestAdHocCache:
-    @pytest.yield_fixture
+    @pytest.fixture
     def repository(self, tmpdir):
         self.repository_location = os.path.join(str(tmpdir), 'repository')
         with Repository(self.repository_location, exclusive=True, create=True) as repository:
diff --git a/src/borg/testsuite/remote.py b/src/borg/testsuite/remote.py
--- a/src/borg/testsuite/remote.py
+++ b/src/borg/testsuite/remote.py
@@ -70,7 +70,7 @@ def test_write(self, monkeypatch):
 
 
 class TestRepositoryCache:
-    @pytest.yield_fixture
+    @pytest.fixture
     def repository(self, tmpdir):
         self.repository_location = os.path.join(str(tmpdir), 'repository')
         with Repository(self.repository_location, exclusive=True, create=True) as repository:


Test patch:
Repository: borgbackup/borg

Issue:
cythonize crashes on macOS with py39
Found this issue on the macOS vagrant machine after upgrading python to 3.9.1:

https://github.com/cython/cython/issues/3262



Discussion:
```
diff --git a/setup.py b/setup.py
index 4152f9da..9da06cb1 100644
--- a/setup.py
+++ b/setup.py
@@ -63,7 +63,7 @@
 system_prefix_libxxhash = os.environ.get('BORG_LIBXXHASH_PREFIX')
 
 # Number of threads to use for cythonize, not used on windows
-cpu_threads = multiprocessing.cpu_count() if multiprocessing else 1
+cpu_threads = multiprocessing.cpu_count() if multiprocessing and multiprocessing.get_start_method() != 'spawn' else None
 
 # Are we building on ReadTheDocs?
 on_rtd = os.environ.get('READTHEDOCS')
@@ -236,7 +236,7 @@ def members_appended(*ds):
         )
         if not is_win32:
             # compile .pyx extensions to .c in parallel, does not work on windows
-            cython_opts['nthreads'] = cpu_threads + 1
+            cython_opts['nthreads'] = cpu_threads
 
         # generate C code from Cython for ALL supported platforms, so we have them in the sdist.
         # the sdist does not require Cython at install time, so we need all as C.
```

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -63,7 +63,7 @@
 system_prefix_libxxhash = os.environ.get('BORG_LIBXXHASH_PREFIX')
 
 # Number of threads to use for cythonize, not used on windows
-cpu_threads = multiprocessing.cpu_count() if multiprocessing else 1
+cpu_threads = multiprocessing.cpu_count() if multiprocessing and multiprocessing.get_start_method() != 'spawn' else None
 
 # Are we building on ReadTheDocs?
 on_rtd = os.environ.get('READTHEDOCS')
@@ -236,7 +236,7 @@ def members_appended(*ds):
         )
         if not is_win32:
             # compile .pyx extensions to .c in parallel, does not work on windows
-            cython_opts['nthreads'] = cpu_threads + 1
+            cython_opts['nthreads'] = cpu_threads
 
         # generate C code from Cython for ALL supported platforms, so we have them in the sdist.
         # the sdist does not require Cython at install time, so we need all as C.


Test patch:
Repository: borgbackup/borg

Issue:
better logging for borg check
even with `--debug`, it did not show the current segment file name:

```
dietmar@haubenpinguin:~$ borg-1.2 check --debug -v -p zwergpinguin-test:borg12test
Remote: Starting repository check
Traceback (most recent call last):
  File "borg/remote.py", line 239, in serve
  File "borg/repository.py", line 1008, in check
  File "borg/repository.py", line 1470, in iter_objects
  File "borg/repository.py", line 1560, in _read
OSError: [Errno 5] Input/output error
```


Discussion:

Resolving patch:
diff --git a/src/borg/repository.py b/src/borg/repository.py
--- a/src/borg/repository.py
+++ b/src/borg/repository.py
@@ -1004,6 +1004,7 @@ def report_error(msg):
                 continue
             if segment > transaction_id:
                 continue
+            logger.debug('checking segment file %s...', filename)
             try:
                 objects = list(self.io.iter_objects(segment))
             except IntegrityError as err:


Test patch:
Repository: borgbackup/borg

Issue:
consequences of borg put shadow_index bug?
see #5636 for the bugfix.

as this bug was present for a while in borg, let's think about its consequences here (and what we can / must do about it for existing repos).



Discussion:

Resolving patch:
diff --git a/src/borg/repository.py b/src/borg/repository.py
--- a/src/borg/repository.py
+++ b/src/borg/repository.py
@@ -1191,9 +1191,11 @@ def put(self, id, data, wait=True):
         except KeyError:
             pass
         else:
-            # note: doing a delete first will do some bookkeeping,
-            # like updating the shadow_index, quota, ...
-            self._delete(id, segment, offset)
+            # note: doing a delete first will do some bookkeeping.
+            # we do not want to update the shadow_index here, because
+            # we know already that we will PUT to this id, so it will
+            # be in the repo index (and we won't need it in the shadow_index).
+            self._delete(id, segment, offset, update_shadow_index=False)
         segment, offset = self.io.write_put(id, data)
         self.storage_quota_use += len(data) + self.io.put_header_fmt.size
         self.segments.setdefault(segment, 0)
@@ -1216,11 +1218,16 @@ def delete(self, id, wait=True):
             segment, offset = self.index.pop(id)
         except KeyError:
             raise self.ObjectNotFound(id, self.path) from None
-        self._delete(id, segment, offset)
+        # if we get here, there is an object with this id in the repo,
+        # we write a DEL here that shadows the respective PUT.
+        # after the delete, the object is not in the repo index any more,
+        # for the compaction code, we need to update the shadow_index in this case.
+        self._delete(id, segment, offset, update_shadow_index=True)
 
-    def _delete(self, id, segment, offset):
+    def _delete(self, id, segment, offset, *, update_shadow_index):
         # common code used by put and delete
-        self.shadow_index.setdefault(id, []).append(segment)
+        if update_shadow_index:
+            self.shadow_index.setdefault(id, []).append(segment)
         self.segments[segment] -= 1
         size = self.io.read(segment, offset, id, read_data=False)
         self.storage_quota_use -= size


Test patch:
Repository: borgbackup/borg

Issue:
`--dry-run` does not show exclude tag file being present in the list of archived files as per `--keep-exclude-tags`
## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

ISSUE

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).

```
borg 1.1.16
```

#### Operating system (distribution) and version.

Arch Linux

#### Hardware / network configuration, and filesystems used.

EXT4

#### Full borg commandline that lead to the problem (leave away excludes and passwords)

```
borg ... --dry-run ... --exclude-if-present .nobackup --keep-exclude-tags ...
```

## Describe the problem you're observing.

The `.nobackup` file is not present in the list of output files to be backed up.


Discussion:

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -651,8 +651,9 @@ def _process(self, archive, cache, matcher, exclude_caches, exclude_if_present,
                         # if we are already recursing in an excluded dir, we do not need to do anything else than
                         # returning (we do not need to archive or recurse into tagged directories), see #3991:
                         if not recurse_excluded_dir:
-                            if keep_exclude_tags and not dry_run:
-                                archive.process_dir(path, st)
+                            if keep_exclude_tags:
+                                if not dry_run:
+                                    archive.process_dir(path, st)
                                 for tag_path in tag_paths:
                                     self._process(archive, cache, matcher, exclude_caches, exclude_if_present,
                                                   keep_exclude_tags, skip_inodes, tag_path, restrict_dev,


Test patch: