internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: borgbackup/borg

Issue:
borg extract --stdout also writes to disk when dir is restored
From the doc I would have expected that `borg extract --stdout` *only* writes to the STDOUT. However, it actually writes to stdout and – as usual without the parameter – to the file path.

IMHO this is unexpected, or is this even a bug? (borg 1.0.10)

Actually I am also not sure what would happen if you restore a RAW device like in your doc example:
```shell
$ borg extract --stdout /path/to/repo::my-sdx | dd of=/dev/sdx bs=10M
```

Maybe the file is dumped somewhere too, or in this case it works as I expected? I only tested it with a usual text file.


Discussion:
Please give the complete sequence of commands you used and a directory listing with the unexpected files.
Ahh, I think I got the issue, I accidentally extracted a directory instead of a single file.

```shell
export BORG_REPO
[…]
borg extract ::ARCHIVE_NAME home/…/dir --stdout > file.txt
```

The dir only has one file. And so the stdout output looks legit.

**Edit:** It is legit. Same file (size).
Still a strange behaviour. Maybe you rather want to show an error message? 
well, trying to write directories (or other non-file-like items) to stdout is kind of unexpected. :)
It is, and it makes it even stranger when this actually (somehow) "works"…
--stdout must not write to disk.

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -704,7 +704,7 @@ def peek_and_store_hardlink_masters(item, matched):
                 else:
                     if stat.S_ISDIR(item.mode):
                         dirs.append(item)
-                        archive.extract_item(item, restore_attrs=False)
+                        archive.extract_item(item, stdout=stdout, restore_attrs=False)
                     else:
                         archive.extract_item(item, stdout=stdout, sparse=sparse, hardlink_masters=hardlink_masters,
                                              stripped_components=strip_components, original_path=orig_path, pi=pi)
@@ -721,7 +721,7 @@ def peek_and_store_hardlink_masters(item, matched):
                 pi.show()
                 dir_item = dirs.pop(-1)
                 try:
-                    archive.extract_item(dir_item)
+                    archive.extract_item(dir_item, stdout=stdout)
                 except BackupOSError as e:
                     self.print_warning('%s: %s', remove_surrogates(dir_item.path), e)
         for pattern in matcher.get_unmatched_include_patterns():


Test patch:
Repository: borgbackup/borg

Issue:
FUSE crashes when encountering negative uid/gid
moved from #2595, reported by @smariot:

My archive contains a lot of files with UID and GID set to -1. borg mount aborts because fuse throws an exception when it encounters the negative group id.

The archive was created in Windows under Cygwin, and these 'ownerless' files came from an external drive.


Discussion:

Resolving patch:
diff --git a/src/borg/fuse.py b/src/borg/fuse.py
--- a/src/borg/fuse.py
+++ b/src/borg/fuse.py
@@ -70,7 +70,9 @@ def __init__(self, key, repository, manifest, args, cached_repo):
         self.items = {}
         self.parent = {}
         self.contents = defaultdict(dict)
-        self.default_dir = Item(mode=0o40755, mtime=int(time.time() * 1e9), uid=os.getuid(), gid=os.getgid())
+        self.default_uid = os.getuid()
+        self.default_gid = os.getgid()
+        self.default_dir = Item(mode=0o40755, mtime=int(time.time() * 1e9), uid=self.default_uid, gid=self.default_gid)
         self.pending_archives = {}
         self.cache = ItemCache()
         data_cache_capacity = int(os.environ.get('BORG_MOUNT_DATA_CACHE_ENTRIES', os.cpu_count() or 1))
@@ -263,8 +265,8 @@ def getattr(self, inode, ctx=None):
         entry.attr_timeout = 300
         entry.st_mode = item.mode
         entry.st_nlink = item.get('nlink', 1)
-        entry.st_uid = item.uid
-        entry.st_gid = item.gid
+        entry.st_uid = item.uid if item.uid >= 0 else self.default_uid
+        entry.st_gid = item.gid if item.gid >= 0 else self.default_gid
         entry.st_rdev = item.get('rdev', 0)
         entry.st_size = item.get_size()
         entry.st_blksize = 512


Test patch:
Repository: borgbackup/borg

Issue:
div by zero on freebsd
found in 1.1.0b6 release testing:
```
==> freebsd64: __________________ ArchiverTestCase.test_create_no_cache_sync __________________
...
==> freebsd64:         duration = perf_counter() - t0
==> freebsd64:         pi.finish()
==> freebsd64:         logger.debug('AdHocCache: downloaded %d chunk IDs in %.2f s (%d requests), ~%s/s',
==> freebsd64: >                    num_chunks, duration, num_requests, format_file_size(num_chunks * 34 / duration))
==> freebsd64: E       ZeroDivisionError: float division by zero
==> freebsd64: 
==> freebsd64: .tox/py36/lib/python3.6/site-packages/borg/cache.py:1066: ZeroDivisionError
```


Discussion:

Resolving patch:
diff --git a/src/borg/cache.py b/src/borg/cache.py
--- a/src/borg/cache.py
+++ b/src/borg/cache.py
@@ -1060,7 +1060,7 @@ def _begin_txn(self):
         assert len(self.chunks) == num_chunks
         # LocalCache does not contain the manifest, either.
         del self.chunks[self.manifest.MANIFEST_ID]
-        duration = perf_counter() - t0
+        duration = perf_counter() - t0 or 0.01
         pi.finish()
         logger.debug('AdHocCache: downloaded %d chunk IDs in %.2f s (%d requests), ~%s/s',
                      num_chunks, duration, num_requests, format_file_size(num_chunks * 34 / duration))


Test patch:
Repository: borgbackup/borg

Issue:
remove hurd noatime-test-skip
the test works on GNU/Hurd 2017, so guess we can remove the skip.

before removing, first check what version debian is testing against.


Discussion:
seems like the package auto-building for hurd runs against sid and experimental, so I guess that should be at least as fresh as the GNU/Hurd 2017 release.

Resolving patch:
diff --git a/src/borg/testsuite/archiver.py b/src/borg/testsuite/archiver.py
--- a/src/borg/testsuite/archiver.py
+++ b/src/borg/testsuite/archiver.py
@@ -453,9 +453,6 @@ def test_symlink_extract(self):
             self.cmd('extract', self.repository_location + '::test')
             assert os.readlink('input/link1') == 'somewhere'
 
-    # Search for O_NOATIME there: https://www.gnu.org/software/hurd/contributing.html - we just
-    # skip the test on Hurd, it is not critical anyway, just testing a performance optimization.
-    @pytest.mark.skipif(sys.platform == 'gnu0', reason="O_NOATIME is strangely broken on GNU Hurd")
     @pytest.mark.skipif(not is_utime_fully_supported(), reason='cannot properly setup and execute test without utime')
     def test_atime(self):
         def has_noatime(some_file):


Test patch:
Repository: borgbackup/borg

Issue:
remove hurd noatime-test-skip
the test works on GNU/Hurd 2017, so guess we can remove the skip.

before removing, first check what version debian is testing against.


Discussion:
seems like the package auto-building for hurd runs against sid and experimental, so I guess that should be at least as fresh as the GNU/Hurd 2017 release.

Resolving patch:
diff --git a/borg/testsuite/archiver.py b/borg/testsuite/archiver.py
--- a/borg/testsuite/archiver.py
+++ b/borg/testsuite/archiver.py
@@ -375,9 +375,6 @@ def filter(output):
         # the interesting parts of info_output2 and info_output should be same
         self.assert_equal(filter(info_output), filter(info_output2))
 
-    # Search for O_NOATIME there: https://www.gnu.org/software/hurd/contributing.html - we just
-    # skip the test on Hurd, it is not critical anyway, just testing a performance optimization.
-    @pytest.mark.skipif(sys.platform == 'gnu0', reason="O_NOATIME is strangely broken on GNU Hurd")
     def test_atime(self):
         def has_noatime(some_file):
             atime_before = os.stat(some_file).st_atime_ns


Test patch:
Repository: borgbackup/borg

Issue:
keep file extensions for FUSE versions view
```
$ borg mount -o versions /path/to/repo /tmp/mymountpoint
$ ls -l /tmp/mymountpoint/home/user/doc.txt/
-rw-rw-r-- 1 user group 12357 Aug 26 21:19 doc.txt.cda00bc9
-rw-rw-r-- 1 user group 12204 Aug 26 21:04 doc.txt.fa760f28
```

Problem: some tools might depend on the filename extension to work correctly, so if we just append the hex stuff, they get confused.

We could also make it look that way:

```
-rw-rw-r-- 1 user group 12357 Aug 26 21:19 doc.cda00bc9.txt
-rw-rw-r-- 1 user group 12204 Aug 26 21:04 doc.fa760f28.txt
```


Discussion:

Resolving patch:
diff --git a/src/borg/fuse.py b/src/borg/fuse.py
--- a/src/borg/fuse.py
+++ b/src/borg/fuse.py
@@ -351,7 +351,10 @@ def make_versioned_name(name, version, add_dir=False):
                 # add intermediate directory with same name as filename
                 path_fname = name.rsplit(b'/', 1)
                 name += b'/' + path_fname[-1]
-            return name + os.fsencode('.%08x' % version)
+            # keep original extension at end to avoid confusing tools
+            name, ext = os.path.splitext(name)
+            version_enc = os.fsencode('.%08x' % version)
+            return name + version_enc + ext
 
         if self.versions and not is_dir:
             parent = self.process_inner(name, parent)


Test patch:
Repository: borgbackup/borg

Issue:
Limited unpacker for outer key packing
We should also use a limited unpacker for the outer key packing (`KeyfileKeyBase.decrypt_key_file`), since it is unpacked before verifying the contents. This one is easy since it's a very simple, known, unchanging structure there.

Split from #2139


Discussion:

Resolving patch:
diff --git a/src/borg/crypto/key.py b/src/borg/crypto/key.py
--- a/src/borg/crypto/key.py
+++ b/src/borg/crypto/key.py
@@ -597,7 +597,9 @@ def _load(self, key_data, passphrase):
         return False
 
     def decrypt_key_file(self, data, passphrase):
-        data = msgpack.unpackb(data)
+        unpacker = get_limited_unpacker('key')
+        unpacker.feed(data)
+        data = unpacker.unpack()
         enc_key = EncryptedKey(internal_dict=data)
         assert enc_key.version == 1
         assert enc_key.algorithm == 'sha256'
diff --git a/src/borg/helpers.py b/src/borg/helpers.py
--- a/src/borg/helpers.py
+++ b/src/borg/helpers.py
@@ -169,8 +169,16 @@ def get_limited_unpacker(kind):
                          object_hook=StableDict,
                          unicode_errors='surrogateescape',
                          ))
+    elif kind == 'key':
+        args.update(dict(use_list=True,  # default value
+                         max_array_len=0,  # not used
+                         max_map_len=10,  # EncryptedKey dict
+                         max_str_len=4000,  # inner key data
+                         object_hook=StableDict,
+                         unicode_errors='surrogateescape',
+                         ))
     else:
-        raise ValueError('kind must be "server", "client" or "manifest"')
+        raise ValueError('kind must be "server", "client", "manifest" or "key"')
     return msgpack.Unpacker(**args)
 
 


Test patch:
Repository: borgbackup/borg

Issue:
Document valid status chars for --filter
The supported status chars should be listed with the help for `borg create --filter` (`--filter STATUSCHARS`).
This could just list the available choices (and validate them), and then link to some more specifc documentation/list, where each of them is explained / listed with a short desc, similar to `borg help pattern`?!


Discussion:
Good idea.
http://borgbackup.readthedocs.io/en/stable/usage.html#item-flags
http://borgbackup.readthedocs.io/en/latest/usage/create.html#item-flags

Todo: Add short list (AME...) to --filter help text. Not sure if that's actually helpful, maybe just add a reference.

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -3469,8 +3469,8 @@ def define_archive_filters_group(subparser, *, sort_by=True, first_last=True):
         subparser.set_defaults(func=self.do_recreate)
         subparser.add_argument('--list', dest='output_list', action='store_true',
                                help='output verbose list of items (files, dirs, ...)')
-        subparser.add_argument('--filter', dest='output_filter', metavar='STATUSCHARS',
-                               help='only display items with the given status characters')
+        subparser.add_argument('--filter', metavar='STATUSCHARS', dest='output_filter',
+                               help='only display items with the given status characters (listed in borg create --help)')
         subparser.add_argument('-n', '--dry-run', dest='dry_run', action='store_true',
                                help='do not change anything')
         subparser.add_argument('-s', '--stats', dest='stats', action='store_true',


Test patch:
Repository: borgbackup/borg

Issue:
cache: truncate_and_unlink on write errors in chunks.archive.d
#2557 cont'd


Discussion:

Resolving patch:
diff --git a/src/borg/cache.py b/src/borg/cache.py
--- a/src/borg/cache.py
+++ b/src/borg/cache.py
@@ -25,6 +25,7 @@
 from .helpers import remove_surrogates
 from .helpers import ProgressIndicatorPercent, ProgressIndicatorMessage
 from .helpers import set_ec, EXIT_WARNING
+from .helpers import truncate_and_unlink
 from .item import ArchiveItem, ChunkListEntry
 from .crypto.key import PlaintextKey
 from .crypto.file_integrity import IntegrityCheckedFile, DetachedIntegrityCheckedFile, FileIntegrityError
@@ -693,7 +694,7 @@ def write_archive_index(archive_id, chunk_idx):
                                                   filename=bin_to_hex(archive_id) + '.compact') as fd:
                     chunk_idx.write(fd)
             except Exception:
-                os.unlink(fn_tmp)
+                truncate_and_unlink(fn_tmp)
             else:
                 os.rename(fn_tmp, fn)
 


Test patch:
Repository: borgbackup/borg

Issue:
Invoking "borg serve" compares None with int in version 1.0.11
```
$LOG ERROR Remote: Local Exception.
Traceback (most recent call last):
  File "/usr/lib64/python3.4/site-packages/borg/archiver.py", line 2168, in main
    exit_code = archiver.run(args)
  File "/usr/lib64/python3.4/site-packages/borg/archiver.py", line 2104, in run
    return set_ec(func(args))
  File "/usr/lib64/python3.4/site-packages/borg/helpers.py", line 85, in set_ec
    exit_code = max(exit_code, ec)
TypeError: unorderable types: NoneType() > int()

Platform: REDACTED
Linux: Gentoo Base System 2.3 
Borg: 1.0.11  Python: CPython 3.4.5
PID: 11348  CWD: /var/lib/borg
sys.argv: ['/usr/lib/python-exec/python3.4/borg', 'serve', '--append-only', '--restrict-to-path', '/var/lib/borg/REDACTED']
```

It looks like **Archiver.run** is calling **Archiver.do_serve**, which is expecting a return value from **RepositoryServer.serve**, but **RepositoryServer.serve** doesn't return anything.

The actual backup is completed successfully, prior to the exception being raised.




Discussion:
Strange, I do not see that error log output when doing a remote backup (ssh:) from borg 1.0.11 (src install) to 1.0.11 (pyinstaller binary).

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -231,12 +231,13 @@ def build_matcher(inclexcl_patterns, include_paths):
 
     def do_serve(self, args):
         """Start in server mode. This command is usually not used manually."""
-        return RepositoryServer(
+        RepositoryServer(
             restrict_to_paths=args.restrict_to_paths,
             restrict_to_repositories=args.restrict_to_repositories,
             append_only=args.append_only,
             storage_quota=args.storage_quota,
         ).serve()
+        return EXIT_SUCCESS
 
     @with_repository(create=True, exclusive=True, manifest=False)
     def do_init(self, args, repository):


Test patch:
Repository: borgbackup/borg

Issue:
Invoking "borg serve" compares None with int in version 1.0.11
```
$LOG ERROR Remote: Local Exception.
Traceback (most recent call last):
  File "/usr/lib64/python3.4/site-packages/borg/archiver.py", line 2168, in main
    exit_code = archiver.run(args)
  File "/usr/lib64/python3.4/site-packages/borg/archiver.py", line 2104, in run
    return set_ec(func(args))
  File "/usr/lib64/python3.4/site-packages/borg/helpers.py", line 85, in set_ec
    exit_code = max(exit_code, ec)
TypeError: unorderable types: NoneType() > int()

Platform: REDACTED
Linux: Gentoo Base System 2.3 
Borg: 1.0.11  Python: CPython 3.4.5
PID: 11348  CWD: /var/lib/borg
sys.argv: ['/usr/lib/python-exec/python3.4/borg', 'serve', '--append-only', '--restrict-to-path', '/var/lib/borg/REDACTED']
```

It looks like **Archiver.run** is calling **Archiver.do_serve**, which is expecting a return value from **RepositoryServer.serve**, but **RepositoryServer.serve** doesn't return anything.

The actual backup is completed successfully, prior to the exception being raised.




Discussion:
Strange, I do not see that error log output when doing a remote backup (ssh:) from borg 1.0.11 (src install) to 1.0.11 (pyinstaller binary).

Resolving patch:
diff --git a/borg/archiver.py b/borg/archiver.py
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -141,7 +141,8 @@ def print_file_status(self, status, path):
     def do_serve(self, args):
         """Start in server mode. This command is usually not used manually.
         """
-        return RepositoryServer(restrict_to_paths=args.restrict_to_paths, append_only=args.append_only).serve()
+        RepositoryServer(restrict_to_paths=args.restrict_to_paths, append_only=args.append_only).serve()
+        return EXIT_SUCCESS
 
     @with_repository(create=True, exclusive=True, manifest=False)
     def do_init(self, args, repository):


Test patch:
Repository: borgbackup/borg

Issue:
--timestamp time (vs list time)
The time set with `--timestamp` is the end time, whereas `list` shows the start time, e.g.:

```
$ borg create --timestamp 2017-01-01T00:00:00 ::foo bar
$ borg list
foo                                  Sat, 2016-12-31 23:59:59
$ borg info ::foo
[...]
Time (start): Sat, 2016-12-31 23:59:59
Time (end):   Sun, 2017-01-01 00:00:00
Command line: borg create --timestamp 2017-01-01T00:00:00 ::foo bar
[...]
```

Having a start and end time is weird with only a single timestamp given. But if current design should not be changed and the start time is considered the important one, that's what `--timestamp` should set.


Discussion:
I'll have a look. Sounds like a bug, but I have to look at docs and past behaviour.
stable and also latest docs:
```
  --timestamp yyyy-mm-ddThh:mm:ss
                        manually specify the archive creation date/time (UTC).
                        alternatively, give a reference file/directory.
```
So, it is not specified there.

But somehow the end time does not make sense, also `borg list` gives the start time (and that's also what we have in the manifest), so guess it is a bug.


Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -445,15 +445,13 @@ def save(self, name=None, comment=None, timestamp=None, additional_metadata=None
         self.items_buffer.flush(flush=True)
         duration = timedelta(seconds=time.monotonic() - self.start_monotonic)
         if timestamp is None:
-            self.end = datetime.utcnow()
-            self.start = self.end - duration
-            start = self.start
-            end = self.end
+            end = datetime.utcnow()
+            start = end - duration
         else:
-            self.end = timestamp
-            self.start = timestamp - duration
-            end = timestamp
-            start = self.start
+            end = timestamp + duration
+            start = timestamp
+        self.start = start
+        self.end = end
         metadata = {
             'version': 1,
             'name': name,


Test patch:
Repository: borgbackup/borg

Issue:
--timestamp time (vs list time)
The time set with `--timestamp` is the end time, whereas `list` shows the start time, e.g.:

```
$ borg create --timestamp 2017-01-01T00:00:00 ::foo bar
$ borg list
foo                                  Sat, 2016-12-31 23:59:59
$ borg info ::foo
[...]
Time (start): Sat, 2016-12-31 23:59:59
Time (end):   Sun, 2017-01-01 00:00:00
Command line: borg create --timestamp 2017-01-01T00:00:00 ::foo bar
[...]
```

Having a start and end time is weird with only a single timestamp given. But if current design should not be changed and the start time is considered the important one, that's what `--timestamp` should set.


Discussion:
I'll have a look. Sounds like a bug, but I have to look at docs and past behaviour.
stable and also latest docs:
```
  --timestamp yyyy-mm-ddThh:mm:ss
                        manually specify the archive creation date/time (UTC).
                        alternatively, give a reference file/directory.
```
So, it is not specified there.

But somehow the end time does not make sense, also `borg list` gives the start time (and that's also what we have in the manifest), so guess it is a bug.


Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -446,15 +446,13 @@ def save(self, name=None, comment=None, timestamp=None, additional_metadata=None
         self.items_buffer.flush(flush=True)
         duration = timedelta(seconds=time.monotonic() - self.start_monotonic)
         if timestamp is None:
-            self.end = datetime.utcnow()
-            self.start = self.end - duration
-            start = self.start
-            end = self.end
+            end = datetime.utcnow()
+            start = end - duration
         else:
-            self.end = timestamp
-            self.start = timestamp - duration
-            end = timestamp
-            start = self.start
+            end = timestamp + duration
+            start = timestamp
+        self.start = start
+        self.end = end
         metadata = {
             'version': 1,
             'name': name,


Test patch: