internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: borgbackup/borg

Issue:
only do glibc checks on client?
Copying my reply to an issue posted on the mailing list to here:

```
     $ borg init --e none ./borgtest
     Remote: FATAL: this Python was compiled for a too old (g)libc and
     misses required functionality.
```
This is the code which triggers this exception / error message:

```
    required_funcs = {os.stat, os.utime, os.chown}
    if not os.supports_follow_symlinks.issuperset(required_funcs):
        raise PythonLibcTooOld
```

So, as you see, quite basic functionality like not following symlinks is not supported for all function where borg needs that.

So far for the bad news.

The good news are that these functions might be only required for using borg as a client, but not for "borg serve" (repo server part in client/server mode, if repo access is done via ssh:).

borg currently always does this check, but I guess it could be maybe omitted for the "borg serve" server part.


Discussion:
I had a look at the code and yes, as suspected, the `follow_symlinks = False` is only used clientside (for borg create and extract).
the code in #4487 needs practical testing. if you did, please post results here.
did someone test this?
With the code from the PR, this check is not executed for `borg serve`:
```
def check_python():
    if is_win32:
        required_funcs = {os.stat}
    else:
        required_funcs = {os.stat, os.utime, os.chown}
    if not os.supports_follow_symlinks.issuperset(required_funcs):
        raise PythonLibcTooOld
```

So what needs to be tested is:
- having a "borg server" system that is rather old, so running borg (master branch or 1.2.0a5) fails with an exception telling:
```
FATAL: this Python was compiled for a too old (g)libc and misses required functionality.
```
- then run master branch code with the PR code applies, the fatal error goes away, but one needs to check of borg is practically usable and runs well on such an old system (todo: run some backups, prune, repo check).

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -4243,8 +4243,10 @@ def parse_args(self, args=None):
             parser.error('Need at least one PATH argument.')
         return args
 
-    def prerun_checks(self, logger):
-        check_python()
+    def prerun_checks(self, logger, is_serve):
+        if not is_serve:
+            # this is the borg *client*, we need to check the python:
+            check_python()
         check_extension_modules()
         selftest(logger)
 
@@ -4286,7 +4288,7 @@ def run(self, args):
             return self.exit_code
         if args.show_version:
             logging.getLogger('borg.output.show-version').info('borgbackup version %s' % __version__)
-        self.prerun_checks(logger)
+        self.prerun_checks(logger, is_serve)
         if not is_supported_msgpack():
             logger.error("You do not have a supported version of the msgpack python package installed. Terminating.")
             logger.error("This should never happen as specific, supported versions are required by our setup.py.")


Test patch:
Repository: borgbackup/borg

Issue:
Repo security_dir without location file
I found that below `~/.config/borg/security/`, there is one directory that does not have a `location` file. It does contain these files: `key-type` `manifest-timestamp` `nonce` `tam_required`

I have no idea to which repository this may have belonged to, I was looking at all the `location` files wanting to clean up after doing countless tests with borg the last few weeks, creating and using many temporary repositories for that, most of which have been deleted since.

This is on Debian Buster, using its borgbackup 1.1.9-2 package.


Discussion:
The current code saves `key-type`, `location` and `manifest-timestamp` all in one go in `borg.cache.SecurityManager.save` method. So, it should be there.

But, a missing (or unreadable) `location` file does not trigger `save()` nor questions being asked about whether the location is ok. Guess this is a bug.

One can trigger rewriting all the files by removing `key-type` (just move all the files somewhere else so you can compare whether contents are same after they got re-written). 
@enkore ^^^ can you have a look please?
Hmm, should `known` method check for all the files created by `save` maybe?

Resolving patch:
diff --git a/src/borg/cache.py b/src/borg/cache.py
--- a/src/borg/cache.py
+++ b/src/borg/cache.py
@@ -73,7 +73,8 @@ def destroy(repository, path=None):
             shutil.rmtree(path)
 
     def known(self):
-        return os.path.exists(self.key_type_file)
+        return all(os.path.exists(f)
+                   for f in (self.key_type_file, self.location_file, self.manifest_ts_file))
 
     def key_matches(self, key):
         if not self.known():


Test patch:
Repository: borgbackup/borg

Issue:
Compatibility issue (minor), error message in case of invalid repo.
Compatibility issue (minor), error message in case of invalid repo.

Old version

	jk@paju:~$ borg  --version
	borg 1.1.8

Repo dir is not valid repo:

	jk@paju:~$ borg create ~/tmp::foo ~/tmp
	/home/jk/tmp is not a valid repository. Check repo config.


Repo dir does not exist:

	jk@paju:~$ borg create ~/tmp/foo::foo ~/tmp
	Repository /home/jk/tmp/foo does not exist.

Latest version

	jk@tiira:~$ borg --version
	borg 1.1.9

Repo dir is not valid repo:

	jk@tiira:~$ borg create ~/tmp::foo ~/tmp
	Exception ignored in: <bound method Repository.__del__ of <Repository /home/jk/tmp>>
	Traceback (most recent call last):
	  File "/usr/lib/python3/dist-packages/borg/repository.py", line 179, in __del__
	    assert False, "cleanup happened in Repository.__del__"
	AssertionError: cleanup happened in Repository.__del__
	Local Exception
	Traceback (most recent call last):
	  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4455, in main
	    exit_code = archiver.run(args)
	  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4387, in run
    	return set_ec(func(args))
	  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 139, in wrapper
    	with repository:
	  File "/usr/lib/python3/dist-packages/borg/repository.py", line 189, in __enter__
	    self.open(self.path, bool(self.exclusive), lock_wait=self.lock_wait, lock=self.do_lock)
	  File "/usr/lib/python3/dist-packages/borg/repository.py", line 396, in open
	    with open(os.path.join(self.path, 'config')) as fd:
	FileNotFoundError: [Errno 2] No such file or directory: '/home/jk/tmp/config'
	
	Platform: Linux tiira 4.4.0-142-generic #168-Ubuntu SMP Wed Jan 16 21:00:45 UTC 2019 x86_64
	Linux: Ubuntu 16.04 xenial
	Borg: 1.1.9  Python: CPython 3.5.2
	PID: 23536  CWD: /home/jk
	sys.argv: ['/usr/bin/borg', 'create', '/home/jk/tmp::foo', '/home/jk/tmp']
	SSH_ORIGINAL_COMMAND: None

Repo dir does not exist:
	
	jk@tiira:~$ borg create ~/tmp/foo::foo ~/tmp
	Repository /home/jk/tmp/foo does not exist.


Please restore the v1.1.8 type of error message in case of invalid repo dir.


Discussion:

Resolving patch:
diff --git a/src/borg/repository.py b/src/borg/repository.py
--- a/src/borg/repository.py
+++ b/src/borg/repository.py
@@ -404,8 +404,12 @@ def open(self, path, exclusive, lock_wait=None, lock=True):
         else:
             self.lock = None
         self.config = ConfigParser(interpolation=None)
-        with open(os.path.join(self.path, 'config')) as fd:
-            self.config.read_file(fd)
+        try:
+            with open(os.path.join(self.path, 'config')) as fd:
+                self.config.read_file(fd)
+        except FileNotFoundError:
+            self.close()
+            raise self.InvalidRepository(self.path)
         if 'repository' not in self.config.sections() or self.config.getint('repository', 'version') != 1:
             self.close()
             raise self.InvalidRepository(path)


Test patch:
Repository: borgbackup/borg

Issue:
borg check --repair not working
`borg check --repair`is not working with error: "UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb4 in position 0: invalid start byte".

How to fix it?


```
./borg1.1.10 check -v -p --repair repo
'check --repair' is an experimental feature that might result in data loss.
Type 'YES' if you understand this and want to continue: YES (from BORG_CHECK_I_KNOW_WHAT_I_AM_DOING)
Starting repository check
Checking segments 70.5%
Starting repository index check
Completed repository check, no problems found.
Starting archive consistency check...
Analyzing archive repo-2018-05-31-012111 (1/120)
...
Analyzing archive repo-2019-07-15-024527 (39/120)
Exception while unpacking item metadata [chunk: 001604_b63ed1dca930a6932e280705c80c6d068e8985b8b94797b0e2fa61b081be7fb8]
Local Exception
Traceback (most recent call last):
  File "borg/archiver.py", line 4501, in main
  File "borg/archiver.py", line 4433, in run
  File "borg/archiver.py", line 166, in wrapper
  File "borg/archiver.py", line 334, in do_check
  File "borg/archive.py", line 1225, in check
  File "borg/archive.py", line 1612, in rebuild_refcounts
  File "borg/archive.py", line 1559, in robust_iterator
  File "borg/archive.py", line 1541, in valid_item
  File "borg/archive.py", line 1529, in list_keys_safe
  File "borg/archive.py", line 1529, in <genexpr>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb4 in position 0: invalid start byte

Platform: Linux server1 5.0.0-29-generic #31~18.04.1-Ubuntu SMP Thu Sep 12 18:29:21 UTC 2019 x86_64
Linux: debian buster/sid
Borg: 1.1.10  Python: CPython 3.5.7 msgpack: 0.5.6
PID: 26224  CWD: /home/bak01/borg
sys.argv: ['./borg1.1.10', 'check', '-v', '-p', '--repair', 'repo']
SSH_ORIGINAL_COMMAND: None
```

Repo was made with borg 1.1.9 if it does matter.


Discussion:
Strange, I'll have a look ...
can you try this patch and rerun borg check?
```
diff --git a/src/borg/archive.py b/src/borg/archive.py
index bfd4ccf9..490f08d7 100644
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -1526,7 +1526,7 @@ def report(msg, chunk_id, chunk_no):
                 logger.error(msg)
 
             def list_keys_safe(keys):
-                return ', '.join((k.decode() if isinstance(k, bytes) else str(k) for k in keys))
+                return ', '.join((k.decode(errors='replace') if isinstance(k, bytes) else str(k) for k in keys))
 
             def valid_item(obj):
                 if not isinstance(obj, StableDict):
```

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -1527,7 +1527,7 @@ def report(msg, chunk_id, chunk_no):
                 logger.error(msg)
 
             def list_keys_safe(keys):
-                return ', '.join((k.decode() if isinstance(k, bytes) else str(k) for k in keys))
+                return ', '.join((k.decode(errors='replace') if isinstance(k, bytes) else str(k) for k in keys))
 
             def valid_item(obj):
                 if not isinstance(obj, StableDict):


Test patch:
Repository: borgbackup/borg

Issue:
borg extract --strip-components --list has confusing output
The list output gives the original paths, without --strip-components applied. So if you are looking to extract files, the list output actually does not represent the paths Borg is or would be (--dry-run) writing.
borg extract --strip-components --list has confusing output
The list output gives the original paths, without --strip-components applied. So if you are looking to extract files, the list output actually does not represent the paths Borg is or would be (--dry-run) writing.


Discussion:
Can I take this one?
@Gu1nness sure (all tickets are free to take as long as nobody is assigned / told that they are working on it in a comment).
Can I take this one?
@Gu1nness sure (all tickets are free to take as long as nobody is assigned / told that they are working on it in a comment).

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -797,7 +797,7 @@ def peek_and_store_hardlink_masters(item, matched):
                     except BackupOSError as e:
                         self.print_warning('%s: %s', remove_surrogates(dir_item.path), e)
             if output_list:
-                logging.getLogger('borg.output.list').info(remove_surrogates(orig_path))
+                logging.getLogger('borg.output.list').info(remove_surrogates(item.path))
             try:
                 if dry_run:
                     archive.extract_item(item, dry_run=True, pi=pi)


Test patch:
Repository: borgbackup/borg

Issue:
msgid for borg check doesn't match documentation
I'm writing a borg frontend using the JSON interface. According to [the docs](https://borgbackup.readthedocs.io/en/stable/internals/frontends.html#message-ids), I would expect borg to emit `progress_percent` messages with `msgid='repository.check_segments'`. However, it really yields `msgid='repository.check'` which is not mentioned in the docs. I don't really know which `msgid` is preferable, (maybe `check_segments` in case there are/will be other `msgid`s related to repository checks), but at least the documentation and code should  match.

Cf. https://github.com/borgbackup/borg/blob/2209f56cd53dcb47e8056c20e25968f1e402fa94/src/borg/repository.py#L962-L963

Thanks!

---

**EDIT:** `git grep` reveals a few more inconsistencies regarding message ids for progress indicators:

- no `msgid` at all for 
https://github.com/borgbackup/borg/blob/61b8234704b83ba946df20bf129d74f8ab7dd699/src/borg/archive.py#L599
https://github.com/borgbackup/borg/blob/61b8234704b83ba946df20bf129d74f8ab7dd699/src/borg/upgrader.py#L82
- further `msgid`s not mentioned in the docs: `prune` 
https://github.com/borgbackup/borg/blob/61b8234704b83ba946df20bf129d74f8ab7dd699/src/borg/archiver.py#L1448
and `check.rebuild_manifest`
https://github.com/borgbackup/borg/blob/61b8234704b83ba946df20bf129d74f8ab7dd699/src/borg/archive.py#L1556-L1557

---
**EDIT 2:** I just verified that all references to the documentation above (which I looked up in the stable docs) still apply to the 'latest' docs on readthedocs.


Discussion:
For the repository module, the msgids are like the method name. So the docs are wrong and it should be `repository.check` not `repository.check_segments`.


Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -525,7 +525,7 @@ def add(id):
         archive_index = ChunkIndex()
         sync = CacheSynchronizer(archive_index)
         add(self.id)
-        pi = ProgressIndicatorPercent(total=len(self.metadata.items), msg='Calculating statistics... %3d%%')
+        pi = ProgressIndicatorPercent(total=len(self.metadata.items), msg='Calculating statistics... %3d%%', msgid='archive.calc_stats')
         for id, chunk in zip(self.metadata.items, self.repository.get_many(self.metadata.items)):
             pi.show(increase=1)
             add(id)
diff --git a/src/borg/upgrader.py b/src/borg/upgrader.py
--- a/src/borg/upgrader.py
+++ b/src/borg/upgrader.py
@@ -79,7 +79,7 @@ def convert_segments(segments, dryrun=True, inplace=False, progress=False):
         replace the 8 first bytes of all regular files in there."""
         logger.info("converting %d segments..." % len(segments))
         segment_count = len(segments)
-        pi = ProgressIndicatorPercent(total=segment_count, msg="Converting segments %3.0f%%")
+        pi = ProgressIndicatorPercent(total=segment_count, msg="Converting segments %3.0f%%", msgid='upgrade.convert_segments')
         for i, filename in enumerate(segments):
             if progress:
                 pi.show(i)


Test patch:
Repository: borgbackup/borg

Issue:
borg extract --strip-components --list has confusing output
The list output gives the original paths, without --strip-components applied. So if you are looking to extract files, the list output actually does not represent the paths Borg is or would be (--dry-run) writing.


Discussion:
Can I take this one?
@Gu1nness sure (all tickets are free to take as long as nobody is assigned / told that they are working on it in a comment).

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -778,7 +778,7 @@ def peek_and_store_hardlink_masters(item, matched):
                     except BackupOSError as e:
                         self.print_warning('%s: %s', remove_surrogates(dir_item.path), e)
             if output_list:
-                logging.getLogger('borg.output.list').info(remove_surrogates(orig_path))
+                logging.getLogger('borg.output.list').info(remove_surrogates(item.path))
             try:
                 if dry_run:
                     archive.extract_item(item, dry_run=True, pi=pi)


Test patch:
Repository: borgbackup/borg

Issue:
borg check --repair not working
`borg check --repair`is not working with error: "UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb4 in position 0: invalid start byte".

How to fix it?


```
./borg1.1.10 check -v -p --repair repo
'check --repair' is an experimental feature that might result in data loss.
Type 'YES' if you understand this and want to continue: YES (from BORG_CHECK_I_KNOW_WHAT_I_AM_DOING)
Starting repository check
Checking segments 70.5%
Starting repository index check
Completed repository check, no problems found.
Starting archive consistency check...
Analyzing archive repo-2018-05-31-012111 (1/120)
...
Analyzing archive repo-2019-07-15-024527 (39/120)
Exception while unpacking item metadata [chunk: 001604_b63ed1dca930a6932e280705c80c6d068e8985b8b94797b0e2fa61b081be7fb8]
Local Exception
Traceback (most recent call last):
  File "borg/archiver.py", line 4501, in main
  File "borg/archiver.py", line 4433, in run
  File "borg/archiver.py", line 166, in wrapper
  File "borg/archiver.py", line 334, in do_check
  File "borg/archive.py", line 1225, in check
  File "borg/archive.py", line 1612, in rebuild_refcounts
  File "borg/archive.py", line 1559, in robust_iterator
  File "borg/archive.py", line 1541, in valid_item
  File "borg/archive.py", line 1529, in list_keys_safe
  File "borg/archive.py", line 1529, in <genexpr>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb4 in position 0: invalid start byte

Platform: Linux server1 5.0.0-29-generic #31~18.04.1-Ubuntu SMP Thu Sep 12 18:29:21 UTC 2019 x86_64
Linux: debian buster/sid
Borg: 1.1.10  Python: CPython 3.5.7 msgpack: 0.5.6
PID: 26224  CWD: /home/bak01/borg
sys.argv: ['./borg1.1.10', 'check', '-v', '-p', '--repair', 'repo']
SSH_ORIGINAL_COMMAND: None
```

Repo was made with borg 1.1.9 if it does matter.


Discussion:
Strange, I'll have a look ...
can you try this patch and rerun borg check?
```
diff --git a/src/borg/archive.py b/src/borg/archive.py
index bfd4ccf9..490f08d7 100644
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -1526,7 +1526,7 @@ def report(msg, chunk_id, chunk_no):
                 logger.error(msg)
 
             def list_keys_safe(keys):
-                return ', '.join((k.decode() if isinstance(k, bytes) else str(k) for k in keys))
+                return ', '.join((k.decode(errors='replace') if isinstance(k, bytes) else str(k) for k in keys))
 
             def valid_item(obj):
                 if not isinstance(obj, StableDict):
```

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -1726,7 +1726,7 @@ def report(msg, chunk_id, chunk_no):
                 logger.error(msg)
 
             def list_keys_safe(keys):
-                return ', '.join((k.decode() if isinstance(k, bytes) else str(k) for k in keys))
+                return ', '.join((k.decode(errors='replace') if isinstance(k, bytes) else str(k) for k in keys))
 
             def valid_item(obj):
                 if not isinstance(obj, StableDict):


Test patch:
Repository: borgbackup/borg

Issue:
`pathconf(..., _PC_NAME_MAX) == 0` in borg mount
When I use `pathconf(..., _PC_NAME_MAX)` to allocate space for `struct dirent` as described in [`man 3 readdir`](https://linux.die.net/man/3/readdir), I get back 0. [`man 3 pathconf`](https://linux.die.net/man/3/pathconf) says it should return -1 on error.

Original bindfs bug: https://github.com/mpartel/bindfs/issues/54 by @mengelmann


Discussion:
It seems FUSE expects implementations to populate `f_namemax` in the statfs reply, but [that does not exist in llfuse](http://pythonhosted.org/llfuse/data.html#llfuse.StatvfsData).

https://bitbucket.org/nikratio/python-llfuse/issues/107/statvfs-reply-statvfsdata-is-missing
Updated, currently working links:
- https://www.rath.org/llfuse-docs/data.html#llfuse.StatvfsData
- could not find the issue 107 anywhere (archive.org, google cache)

https://github.com/python-llfuse/python-llfuse/blob/master/Changes.rst#release-13-2017-09-15

So, we have `f_namemax` since llfuse 1.3.0 (which is also the current minimum requirement for master branch borg).
So, what should f_namemax be?

filename lengths in borg archives are not really limited and it might depend on the OS and filesystem where an archive was created.
```
getconf NAME_MAX /
```
Values:
- Linux / ext4: 255

Resolving patch:
diff --git a/src/borg/fuse.py b/src/borg/fuse.py
--- a/src/borg/fuse.py
+++ b/src/borg/fuse.py
@@ -540,6 +540,7 @@ def statfs(self, ctx=None):
         stat_.f_files = 0
         stat_.f_ffree = 0
         stat_.f_favail = 0
+        stat_.f_namemax = 255  # == NAME_MAX (depends on archive source OS / FS)
         return stat_
 
     def getattr(self, inode, ctx=None):


Test patch:
Repository: borgbackup/borg

Issue:
`pathconf(..., _PC_NAME_MAX) == 0` in borg mount
When I use `pathconf(..., _PC_NAME_MAX)` to allocate space for `struct dirent` as described in [`man 3 readdir`](https://linux.die.net/man/3/readdir), I get back 0. [`man 3 pathconf`](https://linux.die.net/man/3/pathconf) says it should return -1 on error.

Original bindfs bug: https://github.com/mpartel/bindfs/issues/54 by @mengelmann


Discussion:
It seems FUSE expects implementations to populate `f_namemax` in the statfs reply, but [that does not exist in llfuse](http://pythonhosted.org/llfuse/data.html#llfuse.StatvfsData).

https://bitbucket.org/nikratio/python-llfuse/issues/107/statvfs-reply-statvfsdata-is-missing
Updated, currently working links:
- https://www.rath.org/llfuse-docs/data.html#llfuse.StatvfsData
- could not find the issue 107 anywhere (archive.org, google cache)

https://github.com/python-llfuse/python-llfuse/blob/master/Changes.rst#release-13-2017-09-15

So, we have `f_namemax` since llfuse 1.3.0 (which is also the current minimum requirement for master branch borg).
So, what should f_namemax be?

filename lengths in borg archives are not really limited and it might depend on the OS and filesystem where an archive was created.
```
getconf NAME_MAX /
```
Values:
- Linux / ext4: 255

Resolving patch:
diff --git a/src/borg/fuse.py b/src/borg/fuse.py
--- a/src/borg/fuse.py
+++ b/src/borg/fuse.py
@@ -518,6 +518,8 @@ def statfs(self, ctx=None):
         stat_.f_files = 0
         stat_.f_ffree = 0
         stat_.f_favail = 0
+        if hasattr(stat_, 'f_namemax'):  # since llfuse 1.3.0
+            stat_.f_namemax = 255  # == NAME_MAX (depends on archive source OS / FS)
         return stat_
 
     def get_item(self, inode):


Test patch: