internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: borgbackup/borg

Issue:
Package description page on PyPI is unstyled
## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

Issue

## System information. For client/server mode post info for both machines.

N/A

## Describe the problem you're observing.

The [package description page on PyPI](https://pypi.org/project/borgbackup/) is unstyled (showing raw markup).  Looks like this started with release 1.1.0rc3, as 1.1.0rc2 looks fine.



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -190,10 +190,12 @@ def detect_openssl(prefixes):
 
 with open('README.rst', 'r') as fd:
     long_description = fd.read()
+    # remove header, but have one \n before first headline
+    start = long_description.find('What is BorgBackup?')
+    assert start >= 0
+    long_description = '\n' + long_description[start:]
     # remove badges
     long_description = re.compile(r'^\.\. start-badges.*^\.\. end-badges', re.M | re.S).sub('', long_description)
-    # remove |substitutions|
-    long_description = re.compile(r'\|screencast\|').sub('', long_description)
     # remove unknown directives
     long_description = re.compile(r'^\.\. highlight:: \w+$', re.M).sub('', long_description)
 


Test patch:
Repository: borgbackup/borg

Issue:
check --progress processing
In 1.1-maint, I noticed when looking at borg delete and prune that there are 2 ways `--progress` is dealt with:
- it is given to Archive() constructor
- it is given to Archive.delete() call

TODO: review and fix this.


Discussion:

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -1564,8 +1564,10 @@ def do_prune(self, args, repository, manifest, key):
                 # set up counters for the progress display
                 to_delete_len = len(to_delete)
                 archives_deleted = 0
+            pi = ProgressIndicatorPercent(total=len(to_delete), msg='Pruning archives %3.0f%%', msgid='prune')
             for archive in archives_checkpoints:
                 if archive in to_delete:
+                    pi.show()
                     if args.dry_run:
                         if args.output_list:
                             list_logger.info('Would prune:     %s' % format_archive(archive))
@@ -1574,11 +1576,11 @@ def do_prune(self, args, repository, manifest, key):
                             archives_deleted += 1
                             list_logger.info('Pruning archive: %s (%d/%d)' % (format_archive(archive),
                                                                               archives_deleted, to_delete_len))
-                        Archive(repository, key, manifest, archive.name, cache,
-                                progress=args.progress).delete(stats, forced=args.forced)
+                        Archive(repository, key, manifest, archive.name, cache).delete(stats, forced=args.forced)
                 else:
                     if args.output_list:
                         list_logger.info('Keeping archive: %s' % format_archive(archive))
+            pi.finish()
             if to_delete and not args.dry_run:
                 manifest.write()
                 repository.commit(save_space=args.save_space)


Test patch:
Repository: borgbackup/borg

Issue:
go back to system libzstd / liblz4
revert this before releasing 1.1.6:

https://github.com/borgbackup/borg/pull/3768/commits/6a6ad89f5b9125d58c0b0f39ff4fd20d39d7ed92



Discussion:
same thing for the one commit from #3871 

https://github.com/borgbackup/borg/commit/bf6fa046508e1eea9c3388928ad1809bbcbf69f5

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -17,10 +17,10 @@
 import setup_b2
 
 # True: use the shared liblz4 (>= 1.7.0 / r129) from the system, False: use the bundled lz4 code
-prefer_system_liblz4 = False
+prefer_system_liblz4 = True
 
 # True: use the shared libzstd (>= 1.3.0) from the system, False: use the bundled zstd code
-prefer_system_libzstd = False
+prefer_system_libzstd = True
 
 # True: use the shared libb2 from the system, False: use the bundled blake2 code
 prefer_system_libb2 = True


Test patch:
Repository: borgbackup/borg

Issue:
go back to system libzstd / liblz4
revert this before releasing 1.1.6:

https://github.com/borgbackup/borg/pull/3768/commits/6a6ad89f5b9125d58c0b0f39ff4fd20d39d7ed92



Discussion:
same thing for the one commit from #3871 

https://github.com/borgbackup/borg/commit/bf6fa046508e1eea9c3388928ad1809bbcbf69f5

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -30,10 +30,10 @@
 import setup_b2
 
 # True: use the shared liblz4 (>= 1.7.0 / r129) from the system, False: use the bundled lz4 code
-prefer_system_liblz4 = False
+prefer_system_liblz4 = True
 
 # True: use the shared libzstd (>= 1.3.0) from the system, False: use the bundled zstd code
-prefer_system_libzstd = False
+prefer_system_libzstd = True
 
 # True: use the shared libb2 from the system, False: use the bundled blake2 code
 prefer_system_libb2 = True


Test patch:
Repository: borgbackup/borg

Issue:
switch from msgpack-python to msgpack
I didn't switch from msgpack-python (which we used like forever) within borg 1.0.x or 1.1.x series to not break working package dependencies for distribution packages.

For the 1.2 release, bigger changes are expected anyway, so it is a good time to also switch to the new package name `msgpack`.


Discussion:
> to not break working package dependencies for distribution packages.

This is an illogical reason. Failing to switch is the thing which breaks package dependencies on Arch Linux. It's also the reason multiple distribution releases completely didn't work, when our attempts to patch borg to use msgpack failed without us noticing.
then you did not test your package although you made external breaking changes. not my fault.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -46,8 +46,8 @@
 install_requires = [
     # we are rather picky about msgpack versions, because a good working msgpack is
     # very important for borg, see https://github.com/borgbackup/borg/issues/3753
-    # best versions seem to be 0.4.6, 0.4.7, 0.4.8 and 0.5.6:
-    'msgpack-python >=0.4.6, <=0.5.6, !=0.5.0, !=0.5.1, !=0.5.2, !=0.5.3, !=0.5.4, !=0.5.5',
+    # as of now, 0.5.6 is the only preferred version of msgpack:
+    'msgpack==0.5.6',
     # if you can't satisfy the above requirement, these are versions that might
     # also work ok, IF you make sure to use the COMPILED version of msgpack-python,
     # NOT the PURE PYTHON fallback implementation: ==0.5.1, ==0.5.4


Test patch:
Repository: borgbackup/borg

Issue:
make swidth better available
`borg.platform.swidth` is currently only made available on linux, freebsd, os x.

but it also works on:
- cygwin
- openindiana


Discussion:
A bit confusing: swidth and umount are imported from posix in freebsd/darwin and then imported in init from there. Perhaps change this to just import directly from the posix module; avoids superficially unused imports and makes it more clear where these functions come from. It's all platform dependent anyway.

Resolving patch:
diff --git a/src/borg/platform/__init__.py b/src/borg/platform/__init__.py
--- a/src/borg/platform/__init__.py
+++ b/src/borg/platform/__init__.py
@@ -16,18 +16,17 @@
 
 if not sys.platform.startswith(('win32', )):
     from .posix import process_alive, local_pid_alive
+    # posix swidth implementation works for: linux, freebsd, darwin, openindiana, cygwin
+    from .posix import swidth
 
 if sys.platform.startswith('linux'):  # pragma: linux only
     from .linux import API_VERSION as OS_API_VERSION
     from .linux import acl_get, acl_set
     from .linux import set_flags, get_flags
     from .linux import SyncFile
-    from .linux import swidth
 elif sys.platform.startswith('freebsd'):  # pragma: freebsd only
     from .freebsd import API_VERSION as OS_API_VERSION
     from .freebsd import acl_get, acl_set
-    from .freebsd import swidth
 elif sys.platform == 'darwin':  # pragma: darwin only
     from .darwin import API_VERSION as OS_API_VERSION
     from .darwin import acl_get, acl_set
-    from .darwin import swidth


Test patch:
Repository: borgbackup/borg

Issue:
fuse.py: remove write_bytes, use unpacker.tell()
master branch now requires msgpack 0.5.6, so we can remove the deprecated write_bytes mechanism and use unpacker.tell() (new since 0.5.0) to get the current stream position.



Discussion:

Resolving patch:
diff --git a/src/borg/fuse.py b/src/borg/fuse.py
--- a/src/borg/fuse.py
+++ b/src/borg/fuse.py
@@ -126,7 +126,7 @@ def iter_archive_items(self, archive_item_ids, filter=None, consider_part_files=
         stream_offset = 0
         # Offset of the current chunk in the metadata stream
         chunk_begin = 0
-        # Length of the chunk preciding the current chunk
+        # Length of the chunk preceding the current chunk
         last_chunk_length = 0
         msgpacked_bytes = b''
 
@@ -134,13 +134,6 @@ def iter_archive_items(self, archive_item_ids, filter=None, consider_part_files=
         meta = self.meta
         pack_indirect_into = self.indirect_entry_struct.pack_into
 
-        def write_bytes(append_msgpacked_bytes):
-            # XXX: Future versions of msgpack include an Unpacker.tell() method that provides this for free.
-            nonlocal msgpacked_bytes
-            nonlocal stream_offset
-            msgpacked_bytes += append_msgpacked_bytes
-            stream_offset += len(append_msgpacked_bytes)
-
         for key, (csize, data) in zip(archive_item_ids, self.decrypted_repository.get_many(archive_item_ids)):
             # Store the chunk ID in the meta-array
             if write_offset + 32 >= len(meta):
@@ -149,16 +142,25 @@ def write_bytes(append_msgpacked_bytes):
             current_id_offset = write_offset
             write_offset += 32
 
-            # The chunk boundaries cannot be tracked through write_bytes, because the unpack state machine
-            # *can* and *will* consume partial items, so calls to write_bytes are unrelated to chunk boundaries.
             chunk_begin += last_chunk_length
             last_chunk_length = len(data)
 
             unpacker.feed(data)
             while True:
                 try:
-                    item = unpacker.unpack(write_bytes)
+                    item = unpacker.unpack()
+                    need_more_data = False
                 except msgpack.OutOfData:
+                    need_more_data = True
+
+                start = stream_offset - chunk_begin
+                # tell() is not helpful for the need_more_data case, but we know it is the remainder
+                # of the data in that case. in the other case, tell() works as expected.
+                length = (len(data) - start) if need_more_data else (unpacker.tell() - stream_offset)
+                msgpacked_bytes += data[start:start+length]
+                stream_offset += length
+
+                if need_more_data:
                     # Need more data, feed the next chunk
                     break
 


Test patch:
Repository: borgbackup/borg

Issue:
ioctl(fd, TCGETS, ...)  = -1 ENOTTY
just noticed while stracing:

```
ioctl(5, TCGETS, 0x7ffef1435b60)  = -1 ENOTTY (Inappropriate ioctl for device)
```

this happens for about any file `borg create` reads.

any idea how to get rid of this?

i thought it might be related to default `buffering=-1` of `os.fdopen()`, but setting it to 0 or 4096 didn't help.

python 3.6.5


Discussion:
```
    fh = os.open(path, flags_noatime)
    with os.fdopen(fh, 'rb') as fd:
        (read file)
```
Looks like we do not really need the fd there as we have the fh (and the chunker uses the fh, if valid).

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -1151,13 +1151,15 @@ def process_file(self, path, st, cache):
                 else:
                     with backup_io('open'):
                         fh = Archive._open_rb(path)
-                    with os.fdopen(fh, 'rb') as fd:
-                        self.process_file_chunks(item, cache, self.stats, self.show_progress, backup_io_iter(self.chunker.chunkify(fd, fh)))
-                        md = self.metadata_collector.stat_attrs(st, path, fd=fh)
-                    if not is_special_file:
-                        # we must not memorize special files, because the contents of e.g. a
-                        # block or char device will change without its mtime/size/inode changing.
-                        cache.memorize_file(path_hash, st, [c.id for c in item.chunks])
+                        try:
+                            self.process_file_chunks(item, cache, self.stats, self.show_progress, backup_io_iter(self.chunker.chunkify(None, fh)))
+                            md = self.metadata_collector.stat_attrs(st, path, fd=fh)
+                        finally:
+                            os.close(fh)
+                        if not is_special_file:
+                            # we must not memorize special files, because the contents of e.g. a
+                            # block or char device will change without its mtime/size/inode changing.
+                            cache.memorize_file(path_hash, st, [c.id for c in item.chunks])
                 self.stats.nfiles += 1
             if md is None:
                 fh = Archive._open_rb(path)


Test patch:
Repository: borgbackup/borg

Issue:
add BORG_HOST_ID
because of the openvz issue seen in #3968, we need to add some way to override how borg computes this by default (fqdn + uuid.getnode()).

BORG_HOST_ID would trigger that borg uses just what is in there and it is the users responsibility that that ID is really unique.


Discussion:

Resolving patch:
diff --git a/src/borg/platform/base.py b/src/borg/platform/base.py
--- a/src/borg/platform/base.py
+++ b/src/borg/platform/base.py
@@ -249,7 +249,13 @@ def getfqdn(name=''):
 # XXX this sometimes requires live internet access for issuing a DNS query in the background.
 hostname = socket.gethostname()
 fqdn = getfqdn(hostname)
-hostid = '%s@%s' % (fqdn, uuid.getnode())
+
+# uuid.getnode() is problematic in some environments (e.g. OpenVZ, see #3968) where the virtual MAC address
+# is all-zero. uuid.getnode falls back to returning a random value in that case, which is not what we want.
+# thus, we offer BORG_HOST_ID where a user can set an own, unique id for each of his hosts.
+hostid = os.environ.get('BORG_HOST_ID')
+if not hostid:
+    hostid = '%s@%s' % (fqdn, uuid.getnode())
 
 
 def get_process_id():


Test patch:
Repository: borgbackup/borg

Issue:
add BORG_HOST_ID
because of the openvz issue seen in #3968, we need to add some way to override how borg computes this by default (fqdn + uuid.getnode()).

BORG_HOST_ID would trigger that borg uses just what is in there and it is the users responsibility that that ID is really unique.


Discussion:

Resolving patch:
diff --git a/src/borg/platform/base.py b/src/borg/platform/base.py
--- a/src/borg/platform/base.py
+++ b/src/borg/platform/base.py
@@ -210,7 +210,13 @@ def getfqdn(name=''):
 # XXX this sometimes requires live internet access for issuing a DNS query in the background.
 hostname = socket.gethostname()
 fqdn = getfqdn(hostname)
-hostid = '%s@%s' % (fqdn, uuid.getnode())
+
+# uuid.getnode() is problematic in some environments (e.g. OpenVZ, see #3968) where the virtual MAC address
+# is all-zero. uuid.getnode falls back to returning a random value in that case, which is not what we want.
+# thus, we offer BORG_HOST_ID where a user can set an own, unique id for each of his hosts.
+hostid = os.environ.get('BORG_HOST_ID')
+if not hostid:
+    hostid = '%s@%s' % (fqdn, uuid.getnode())
 
 
 def get_process_id():


Test patch:
Repository: borgbackup/borg

Issue:
Flags not at the start of the expression
```
.tox/py36/lib/python3.6/site-packages/borg/patterns.py:281
  /home/travis/build/borgbackup/borg/.tox/py36/lib/python3.6/site-packages/borg/patterns.py:281: DeprecationWarning: Flags not at the start of the expression 'b\\á\\/(?:[^\\/]*\\/)*[^' (truncated)
    self.regex = re.compile(shellpattern.translate(self.pattern))
```



Discussion:

Resolving patch:
diff --git a/src/borg/shellpattern.py b/src/borg/shellpattern.py
--- a/src/borg/shellpattern.py
+++ b/src/borg/shellpattern.py
@@ -62,4 +62,4 @@ def translate(pat, match_end=r"\Z"):
         else:
             res += re.escape(c)
 
-    return res + match_end + "(?ms)"
+    return "(?ms)" + res + match_end


Test patch:
Repository: borgbackup/borg

Issue:
archives named borg-test-data
the help text in the code talks about "archives named borg-test-data", but the de-facto archives names are different.

master branch fix needed + maybe backports to 1.1 and 1.0 if there is same issue.


Discussion:
if no one is working on it I would like take the issue.

@Ashmodei sure, make a PR.

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -4211,7 +4211,7 @@ def define_archive_filters_group(subparser, *, sort_by=True, first_last=True):
 
         It creates input data below the given PATH and backups this data into the given REPO.
         The REPO must already exist (it could be a fresh empty repo or an existing repo, the
-        command will create / read / update / delete some archives named borg-test-data\\* there.
+        command will create / read / update / delete some archives named borg-benchmark-crud\\* there.
 
         Make sure you have free space there, you'll need about 1GB each (+ overhead).
 


Test patch:
Repository: borgbackup/borg

Issue:
archives named borg-test-data
the help text in the code talks about "archives named borg-test-data", but the de-facto archives names are different.

master branch fix needed + maybe backports to 1.1 and 1.0 if there is same issue.


Discussion:
if no one is working on it I would like take the issue.

@Ashmodei sure, make a PR.

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -4075,7 +4075,7 @@ def define_archive_filters_group(subparser, *, sort_by=True, first_last=True):
 
         It creates input data below the given PATH and backups this data into the given REPO.
         The REPO must already exist (it could be a fresh empty repo or an existing repo, the
-        command will create / read / update / delete some archives named borg-test-data\\* there.
+        command will create / read / update / delete some archives named borg-benchmark-crud\\* there.
 
         Make sure you have free space there, you'll need about 1GB each (+ overhead).
 


Test patch:
Repository: borgbackup/borg

Issue:
attic_repo fixture call warning
```
/opt/python/3.7-dev/lib/python3.7/unittest/case.py:615: RemovedInPytest4Warning:
 Fixture "attic_repo" called directly. Fixtures are not meant to be
 called directly, are created automatically when test functions request
 them as parameters. See https://docs.pytest.org/en/latest/fixture.html
 for more information.
    testMethod()
```


Discussion:
this is erroring tests now as pytest4 was released.

temp workaround by #4174.

Resolving patch:
diff --git a/src/borg/testsuite/archiver.py b/src/borg/testsuite/archiver.py
--- a/src/borg/testsuite/archiver.py
+++ b/src/borg/testsuite/archiver.py
@@ -56,7 +56,7 @@
 from . import BaseTestCase, changedir, environment_variable, no_selinux
 from . import are_symlinks_supported, are_hardlinks_supported, are_fifos_supported, is_utime_fully_supported, is_birthtime_fully_supported
 from .platform import fakeroot_detected
-from .upgrader import attic_repo
+from .upgrader import make_attic_repo
 from . import key
 
 
@@ -2873,7 +2873,7 @@ def test_extract_hardlinks(self):
             assert os.stat('input/dir1/source2').st_nlink == 2
 
     def test_detect_attic_repo(self):
-        path = attic_repo(self.repository_path)
+        path = make_attic_repo(self.repository_path)
         cmds = [
             ['create', path + '::test', self.tmpdir],
             ['extract', path + '::test'],
diff --git a/src/borg/testsuite/upgrader.py b/src/borg/testsuite/upgrader.py
--- a/src/borg/testsuite/upgrader.py
+++ b/src/borg/testsuite/upgrader.py
@@ -58,16 +58,20 @@ def key_valid(path):
         return f.read().startswith(KeyfileKey.FILE_ID)
 
 
-@pytest.fixture()
-def attic_repo(tmpdir):
+def make_attic_repo(dir):
     """
     create an attic repo with some stuff in it
 
-    :param tmpdir: path to the repository to be created
+    :param dir: path to the repository to be created
     :returns: path to attic repository
     """
     # there is some stuff in that repo, copied from `RepositoryTestCase.test1`
-    return untar(ATTIC_TAR, str(tmpdir), 'repo')
+    return untar(ATTIC_TAR, str(dir), 'repo')
+
+
+@pytest.fixture()
+def attic_repo(tmpdir):
+    return make_attic_repo(tmpdir)
 
 
 @pytest.fixture(params=[True, False])


Test patch:
Repository: borgbackup/borg

Issue:
attic_repo fixture call warning
```
/opt/python/3.7-dev/lib/python3.7/unittest/case.py:615: RemovedInPytest4Warning:
 Fixture "attic_repo" called directly. Fixtures are not meant to be
 called directly, are created automatically when test functions request
 them as parameters. See https://docs.pytest.org/en/latest/fixture.html
 for more information.
    testMethod()
```


Discussion:
this is erroring tests now as pytest4 was released.

temp workaround by #4174.

Resolving patch:
diff --git a/src/borg/testsuite/archiver.py b/src/borg/testsuite/archiver.py
--- a/src/borg/testsuite/archiver.py
+++ b/src/borg/testsuite/archiver.py
@@ -55,7 +55,7 @@
 from . import BaseTestCase, changedir, environment_variable, no_selinux
 from . import are_symlinks_supported, are_hardlinks_supported, are_fifos_supported, is_utime_fully_supported, is_birthtime_fully_supported
 from .platform import fakeroot_detected
-from .upgrader import attic_repo
+from .upgrader import make_attic_repo
 from . import key
 
 # 3.4.3 == first version with argparse bugfix for nargs='*' and 0 arguments given
@@ -2877,7 +2877,7 @@ def test_extract_hardlinks(self):
             assert os.stat('input/dir1/source2').st_nlink == 2
 
     def test_detect_attic_repo(self):
-        path = attic_repo(self.repository_path)
+        path = make_attic_repo(self.repository_path)
         cmds = [
             ['create', path + '::test', self.tmpdir],
             ['extract', path + '::test'],
diff --git a/src/borg/testsuite/upgrader.py b/src/borg/testsuite/upgrader.py
--- a/src/borg/testsuite/upgrader.py
+++ b/src/borg/testsuite/upgrader.py
@@ -58,16 +58,20 @@ def key_valid(path):
         return f.read().startswith(KeyfileKey.FILE_ID)
 
 
-@pytest.fixture()
-def attic_repo(tmpdir):
+def make_attic_repo(dir):
     """
     create an attic repo with some stuff in it
 
-    :param tmpdir: path to the repository to be created
+    :param dir: path to the repository to be created
     :returns: path to attic repository
     """
     # there is some stuff in that repo, copied from `RepositoryTestCase.test1`
-    return untar(ATTIC_TAR, str(tmpdir), 'repo')
+    return untar(ATTIC_TAR, str(dir), 'repo')
+
+
+@pytest.fixture()
+def attic_repo(tmpdir):
+    return make_attic_repo(tmpdir)
 
 
 @pytest.fixture(params=[True, False])


Test patch: