internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: borgbackup/borg

Issue:
borg recreate: wrong size of part files
from irc:
```
ganja\_$ When borg finds broken chunks like "var/lib/mysql/backup/mysqldump.sql.borg_part_1:
         size inconsistency detected: size 7546242262, chunks size 3184051881",
         wouldn't it be reasonable to be able to list all archives that are affected by that?
```

```
ThomasWaldmann$ do you have more details about the size inconsistency?
ganja\_$  Not except that it happened after a recreate and that check --repair isn't fixing it.
ganja\_$ And that it's consistent through multiple checks with several days between them.
       It's likely to be an effect from recreate since it has happened to 3 repos I ran
       recreate on (recompress).
```


Discussion:
Wouldn't a failure in this check either imply a Borg bug or a memory error?
part files have a size member, computed in write_part_file via item.get_size(memorize=True).
should be correct as the part files chunk id list is setup first to the chunks that make up this part.
so the size is the size of that part. sum(part_sizes) == file_size.

there is no problem when a file gets archived as multiple part files first (due to in-file checkpointing).
part file chunk lists cover the part, size is like that part only, everything sums up correctly.

but if recreate processes a file and in-file checkpointing happens, **the parts get a (wrong) size** equal to the total file size. borg check finds this and complains about the inconsistency.

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -965,7 +965,9 @@ def write_part_file(self, item, from_chunk, number):
         length = len(item.chunks)
         # the item should only have the *additional* chunks we processed after the last partial item:
         item.chunks = item.chunks[from_chunk:]
-        item.get_size(memorize=True)
+        # for borg recreate, we already have a size member in the source item (giving the total file size),
+        # but we consider only a part of the file here, thus we must recompute the size from the chunks:
+        item.get_size(memorize=True, from_chunks=True)
         item.path += '.borg_part_%d' % number
         item.part = number
         number += 1


Test patch:
Repository: borgbackup/borg

Issue:
borg recreate: wrong size of part files
from irc:
```
ganja\_$ When borg finds broken chunks like "var/lib/mysql/backup/mysqldump.sql.borg_part_1:
         size inconsistency detected: size 7546242262, chunks size 3184051881",
         wouldn't it be reasonable to be able to list all archives that are affected by that?
```

```
ThomasWaldmann$ do you have more details about the size inconsistency?
ganja\_$  Not except that it happened after a recreate and that check --repair isn't fixing it.
ganja\_$ And that it's consistent through multiple checks with several days between them.
       It's likely to be an effect from recreate since it has happened to 3 repos I ran
       recreate on (recompress).
```


Discussion:
Wouldn't a failure in this check either imply a Borg bug or a memory error?
part files have a size member, computed in write_part_file via item.get_size(memorize=True).
should be correct as the part files chunk id list is setup first to the chunks that make up this part.
so the size is the size of that part. sum(part_sizes) == file_size.

there is no problem when a file gets archived as multiple part files first (due to in-file checkpointing).
part file chunk lists cover the part, size is like that part only, everything sums up correctly.

but if recreate processes a file and in-file checkpointing happens, **the parts get a (wrong) size** equal to the total file size. borg check finds this and complains about the inconsistency.

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -897,7 +897,9 @@ def write_part_file(self, item, from_chunk, number):
         length = len(item.chunks)
         # the item should only have the *additional* chunks we processed after the last partial item:
         item.chunks = item.chunks[from_chunk:]
-        item.get_size(memorize=True)
+        # for borg recreate, we already have a size member in the source item (giving the total file size),
+        # but we consider only a part of the file here, thus we must recompute the size from the chunks:
+        item.get_size(memorize=True, from_chunks=True)
         item.path += '.borg_part_%d' % number
         item.part = number
         number += 1


Test patch:
Repository: borgbackup/borg

Issue:
hardlinked symlinks warning
see there: https://github.com/borgbackup/borg-import/issues/23

maybe the warning should include the path, so we have that information independently of --list.


Discussion:

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -1063,12 +1063,11 @@ def process_dev(self, path, st, dev_type):
     def process_symlink(self, path, st):
         # note: using hardlinkable=False because we can not support hardlinked symlinks,
         #       due to the dual-use of item.source, see issue #2343:
+        # hardlinked symlinks will be archived [and extracted] as non-hardlinked symlinks.
         with self.create_helper(path, st, 's', hardlinkable=False) as (item, status, hardlinked, hardlink_master):
             with backup_io('readlink'):
                 source = os.readlink(path)
             item.source = source
-            if st.st_nlink > 1:
-                logger.warning('hardlinked symlinks will be archived as non-hardlinked symlinks!')
             item.update(self.metadata_collector.stat_attrs(st, path))
             return status
 


Test patch:
Repository: borgbackup/borg

Issue:
hardlinked symlinks warning
see there: https://github.com/borgbackup/borg-import/issues/23

maybe the warning should include the path, so we have that information independently of --list.


Discussion:

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -883,12 +883,11 @@ def process_dev(self, path, st, dev_type):
     def process_symlink(self, path, st):
         # note: using hardlinkable=False because we can not support hardlinked symlinks,
         #       due to the dual-use of item.source, see issue #2343:
+        # hardlinked symlinks will be archived [and extracted] as non-hardlinked symlinks.
         with self.create_helper(path, st, 's', hardlinkable=False) as (item, status, hardlinked, hardlink_master):
             with backup_io('readlink'):
                 source = os.readlink(path)
             item.source = source
-            if st.st_nlink > 1:
-                logger.warning('hardlinked symlinks will be archived as non-hardlinked symlinks!')
             item.update(self.stat_attrs(st, path))
             return status
 


Test patch:
Repository: borgbackup/borg

Issue:
Exclusions from --exclude-if-present should appear in --list
Currently, when `--exclude-if-present` is used, the excluded folders do not appear in the exclusion log produced by `--list --filter x`. This is inconsistent with the behaviour of `--exclude`, and makes it harder to debug exclusion patterns based on this technique.


Discussion:
Yeah, sounds reasonable. We'll have a look at that.
```
$ borg create --list --exclude-if-present CACHE --keep-exclude-tags repo::test inp
A inp/foo
A inp/cache/CACHE   # this is the tag we wanted to keep (archive)
x inp/cache         # this is a little lie (for the --keep-exclude-tags case):
                    # we archived the directory and the tags, but no other of its contents.
                    # full truth for the other case.
A inp/baz/baz/xxx
d inp/baz/baz
d inp/baz
d inp
```

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -565,6 +565,7 @@ def _process(self, fso, cache, matcher, exclude_caches, exclude_if_present,
                                 self._process(fso, cache, matcher, exclude_caches, exclude_if_present,
                                               keep_exclude_tags, skip_inodes, tag_path, restrict_dev,
                                               read_special=read_special, dry_run=dry_run)
+                        self.print_file_status('x', path)
                         return
                 if not dry_run:
                     if not recurse_excluded_dir:


Test patch:
Repository: borgbackup/borg

Issue:
Exclusions from --exclude-if-present should appear in --list
Currently, when `--exclude-if-present` is used, the excluded folders do not appear in the exclusion log produced by `--list --filter x`. This is inconsistent with the behaviour of `--exclude`, and makes it harder to debug exclusion patterns based on this technique.


Discussion:
Yeah, sounds reasonable. We'll have a look at that.
```
$ borg create --list --exclude-if-present CACHE --keep-exclude-tags repo::test inp
A inp/foo
A inp/cache/CACHE   # this is the tag we wanted to keep (archive)
x inp/cache         # this is a little lie (for the --keep-exclude-tags case):
                    # we archived the directory and the tags, but no other of its contents.
                    # full truth for the other case.
A inp/baz/baz/xxx
d inp/baz/baz
d inp/baz
d inp
```

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -583,6 +583,7 @@ def _process(self, archive, cache, matcher, exclude_caches, exclude_if_present,
                                 self._process(archive, cache, matcher, exclude_caches, exclude_if_present,
                                               keep_exclude_tags, skip_inodes, tag_path, restrict_dev,
                                               read_special=read_special, dry_run=dry_run)
+                        self.print_file_status('x', path)
                         return
                 if not dry_run:
                     if not recurse_excluded_dir:


Test patch:
Repository: borgbackup/borg

Issue:
Exclusions from --exclude-if-present should appear in --list
Currently, when `--exclude-if-present` is used, the excluded folders do not appear in the exclusion log produced by `--list --filter x`. This is inconsistent with the behaviour of `--exclude`, and makes it harder to debug exclusion patterns based on this technique.


Discussion:
Yeah, sounds reasonable. We'll have a look at that.
```
$ borg create --list --exclude-if-present CACHE --keep-exclude-tags repo::test inp
A inp/foo
A inp/cache/CACHE   # this is the tag we wanted to keep (archive)
x inp/cache         # this is a little lie (for the --keep-exclude-tags case):
                    # we archived the directory and the tags, but no other of its contents.
                    # full truth for the other case.
A inp/baz/baz/xxx
d inp/baz/baz
d inp/baz
d inp
```

Resolving patch:
diff --git a/borg/archiver.py b/borg/archiver.py
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -369,6 +369,7 @@ def _process(self, archive, cache, matcher, exclude_caches, exclude_if_present,
                             self._process(archive, cache, matcher, exclude_caches, exclude_if_present,
                                           keep_tag_files, skip_inodes, tag_path, restrict_dev,
                                           read_special=read_special, dry_run=dry_run)
+                    self.print_file_status('x', path)
                     return
                 if not dry_run:
                     status = archive.process_dir(path, st)


Test patch:
Repository: borgbackup/borg

Issue:
--keep-tag-files for borg list?
From SanskritFritz (IRC):
```
in the docs there is --keep-tag-files for borg list.
A result of copy-paste or is there a point?

same with borg diff.
```

Note: maybe a global option? check whether it is better to make it a local option. check which branches are affected.


Discussion:

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -3040,7 +3040,7 @@ def define_archive_filters_group(subparser, *, sort_by=True, first_last=True):
         subparser.add_argument('paths', metavar='PATH', nargs='*', type=str,
                                help='paths to list; patterns are supported')
         define_archive_filters_group(subparser)
-        define_exclusion_group(subparser, tag_files=True)
+        define_exclusion_group(subparser)
 
         mount_epilog = process_epilog("""
         This command mounts an archive as a FUSE filesystem. This can be useful for


Test patch:
Repository: borgbackup/borg

Issue:
--keep-tag-files for borg list?
From SanskritFritz (IRC):
```
in the docs there is --keep-tag-files for borg list.
A result of copy-paste or is there a point?

same with borg diff.
```

Note: maybe a global option? check whether it is better to make it a local option. check which branches are affected.


Discussion:

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -3233,7 +3233,7 @@ def define_archive_filters_group(subparser, *, sort_by=True, first_last=True):
         subparser.add_argument('paths', metavar='PATH', nargs='*', type=str,
                                help='paths to list; patterns are supported')
         define_archive_filters_group(subparser)
-        define_exclusion_group(subparser, tag_files=True)
+        define_exclusion_group(subparser)
 
         mount_epilog = process_epilog("""
         This command mounts an archive as a FUSE filesystem. This can be useful for


Test patch:
Repository: borgbackup/borg

Issue:
--keep-tag-files for borg list?
From SanskritFritz (IRC):
```
in the docs there is --keep-tag-files for borg list.
A result of copy-paste or is there a point?

same with borg diff.
```

Note: maybe a global option? check whether it is better to make it a local option. check which branches are affected.


Discussion:
How about borg diff? Is the option used there?
ah, forgot that. fixing...

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -2945,7 +2945,7 @@ def define_archive_filters_group(subparser, *, sort_by=True, first_last=True):
                                help='ARCHIVE2 name (no repository location allowed)')
         subparser.add_argument('paths', metavar='PATH', nargs='*', type=str,
                                help='paths of items inside the archives to compare; patterns are supported')
-        define_exclusion_group(subparser, tag_files=True)
+        define_exclusion_group(subparser)
 
         rename_epilog = process_epilog("""
         This command renames an archive in the repository.


Test patch:
Repository: borgbackup/borg

Issue:
--keep-tag-files for borg list?
From SanskritFritz (IRC):
```
in the docs there is --keep-tag-files for borg list.
A result of copy-paste or is there a point?

same with borg diff.
```

Note: maybe a global option? check whether it is better to make it a local option. check which branches are affected.


Discussion:
How about borg diff? Is the option used there?
ah, forgot that. fixing...

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -3137,7 +3137,7 @@ def define_archive_filters_group(subparser, *, sort_by=True, first_last=True):
                                help='ARCHIVE2 name (no repository location allowed)')
         subparser.add_argument('paths', metavar='PATH', nargs='*', type=str,
                                help='paths of items inside the archives to compare; patterns are supported')
-        define_exclusion_group(subparser, tag_files=True)
+        define_exclusion_group(subparser)
 
         rename_epilog = process_epilog("""
         This command renames an archive in the repository.


Test patch:
Repository: borgbackup/borg

Issue:
wrong original size in borg create
Hello,
[borg 1.1.1  python 3.4.5 CentOs 6]
I tried 2 times with same command but different files ( oracle dump ) and both times I had
an original size reported twice bigger than the real one.
The file is 109 GB.
I tried even with smaller files but the problem did not appear.

```
# borg create -p --stats /opt/mnt/test.borg::test2 ads_orcl_daily_dmp_5
------------------------------------------------------------------------------
Archive name: test2
Archive fingerprint: ba34d3cb846d521388a14ea5cf1b9a8addb114de386a042738b99fac01e48a42
Time (start): Thu, 2017-10-26 12:33:18
Time (end):   Thu, 2017-10-26 13:33:38
Duration: 1 hours 20.12 seconds
Number of files: 1
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:              232.90 GB            205.06 GB            102.53 GB
All archives:              232.90 GB            205.06 GB            102.53 GB

                       Unique chunks         Total chunks
Chunk index:                   44512                89022
------------------------------------------------------------------------------
```
giuseppe



Discussion:
The "original size" doubles when a checkpoint is happening in the respective file.
Cosmetical bug, the chunks are referenced by the parts AND by the full file and both is going into the statistics.

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -1021,8 +1021,9 @@ def chunk_processor(data):
 
                 # if we created part files, we have referenced all chunks from the part files,
                 # but we also will reference the same chunks also from the final, complete file:
+                dummy_stats = Statistics()  # do not count this data volume twice
                 for chunk in item.chunks:
-                    cache.chunk_incref(chunk.id, stats, size=chunk.size)
+                    cache.chunk_incref(chunk.id, dummy_stats, size=chunk.size)
 
 
 class FilesystemObjectProcessors:


Test patch:
Repository: borgbackup/borg

Issue:
wrong original size in borg create
Hello,
[borg 1.1.1  python 3.4.5 CentOs 6]
I tried 2 times with same command but different files ( oracle dump ) and both times I had
an original size reported twice bigger than the real one.
The file is 109 GB.
I tried even with smaller files but the problem did not appear.

```
# borg create -p --stats /opt/mnt/test.borg::test2 ads_orcl_daily_dmp_5
------------------------------------------------------------------------------
Archive name: test2
Archive fingerprint: ba34d3cb846d521388a14ea5cf1b9a8addb114de386a042738b99fac01e48a42
Time (start): Thu, 2017-10-26 12:33:18
Time (end):   Thu, 2017-10-26 13:33:38
Duration: 1 hours 20.12 seconds
Number of files: 1
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:              232.90 GB            205.06 GB            102.53 GB
All archives:              232.90 GB            205.06 GB            102.53 GB

                       Unique chunks         Total chunks
Chunk index:                   44512                89022
------------------------------------------------------------------------------
```
giuseppe



Discussion:
The "original size" doubles when a checkpoint is happening in the respective file.
Cosmetical bug, the chunks are referenced by the parts AND by the full file and both is going into the statistics.

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -942,8 +942,9 @@ def chunk_processor(data):
 
                 # if we created part files, we have referenced all chunks from the part files,
                 # but we also will reference the same chunks also from the final, complete file:
+                dummy_stats = Statistics()  # do not count this data volume twice
                 for chunk in item.chunks:
-                    cache.chunk_incref(chunk.id, stats, size=chunk.size)
+                    cache.chunk_incref(chunk.id, dummy_stats, size=chunk.size)
 
     def process_stdin(self, path, cache):
         uid, gid = 0, 0


Test patch:
Repository: borgbackup/borg

Issue:
check docs about key export
i remember multiple people asking about whether the data exported by `borg key export` contains the key in encrypted or unencrypted form.

this should be in the docs, also some related statement about the passphrase.
check docs about key export
i remember multiple people asking about whether the data exported by `borg key export` contains the key in encrypted or unencrypted form.

this should be in the docs, also some related statement about the passphrase.


Discussion:

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -2497,11 +2497,15 @@ def define_archive_filters_group(subparser, *, sort_by=True, first_last=True):
         key_export_epilog = process_epilog("""
         If repository encryption is used, the repository is inaccessible
         without the key. This command allows to backup this essential key.
+        Note that the backup produced does not include the passphrase itself
+        (i.e. the exported key stays encrypted). In order to regain access to a
+        repository, one needs both the exported key and the original passphrase.
 
-        There are two backup formats. The normal backup format is suitable for
+        There are three backup formats. The normal backup format is suitable for
         digital storage as a file. The ``--paper`` backup format is optimized
         for printing and typing in while importing, with per line checks to
-        reduce problems with manual input.
+        reduce problems with manual input. The ``--qr-html`` creates a printable
+        HTML template with a QR code and a copy of the ``--paper``-formatted key.
 
         For repositories using keyfile encryption the key is saved locally
         on the system that is capable of doing backups. To guard against loss


Test patch:
Repository: borgbackup/borg

Issue:
check docs about key export
i remember multiple people asking about whether the data exported by `borg key export` contains the key in encrypted or unencrypted form.

this should be in the docs, also some related statement about the passphrase.


Discussion:
> some related statement about the passphrase

What do you mean by this? Just a caveat about how the passphrase isn't backed up/doesn't exist in `keyfile` mode?

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -2688,6 +2688,9 @@ def define_archive_filters_group(subparser, *, sort_by=True, first_last=True):
         key_export_epilog = process_epilog("""
         If repository encryption is used, the repository is inaccessible
         without the key. This command allows to backup this essential key.
+        Note that the backup produced does not include the passphrase itself
+        (i.e. the exported key stays encrypted). In order to regain access to a
+        repository, one needs both the exported key and the original passphrase.
 
         There are two backup formats. The normal backup format is suitable for
         digital storage as a file. The ``--paper`` backup format is optimized


Test patch: