internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: borgbackup/borg

Issue:
Running `borg check` causes `local variable 'segment' referenced before assignment`
## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes.

## Is this a BUG / ISSUE report or a QUESTION?

Bug report.

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).

1.2.0

#### Operating system (distribution) and version.

macOS 12.5.1

#### Hardware / network configuration, and filesystems used.

APFS

#### How much data is handled by borg?

Repository is empty.

#### Full borg commandline that lead to the problem (leave away excludes and passwords)

    BORG_PASSCOMMAND=test borg check koffer

## Describe the problem you're observing.

When a repository is corrupt, and `borg check` is run, the following error is written to stderr:

    UnboundLocalError: local variable 'segment' referenced before assignment

#### Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

- Create repository
- Corrupt it, e.g. by moving away files in `data/`
- Run `borg check`

#### Include any warning/errors/backtraces from the system logs

```
Local Exception
Traceback (most recent call last):
  File "/usr/local/Cellar/borgbackup/1.2.0/libexec/lib/python3.10/site-packages/borg/archiver.py", line 5089, in main
    exit_code = archiver.run(args)
  File "/usr/local/Cellar/borgbackup/1.2.0/libexec/lib/python3.10/site-packages/borg/archiver.py", line 5020, in run
    return set_ec(func(args))
  File "/usr/local/Cellar/borgbackup/1.2.0/libexec/lib/python3.10/site-packages/borg/archiver.py", line 183, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/local/Cellar/borgbackup/1.2.0/libexec/lib/python3.10/site-packages/borg/archiver.py", line 342, in do_check
    if not repository.check(repair=args.repair, save_space=args.save_space, max_duration=args.max_duration):
  File "/usr/local/Cellar/borgbackup/1.2.0/libexec/lib/python3.10/site-packages/borg/repository.py", line 1040, in check
    logger.info('finished segment check at segment %d', segment)
UnboundLocalError: local variable 'segment' referenced before assignment
```


Discussion:
Good catch, you found a bug! I could reproduce it with 1.2-maint code. :-)

Resolving patch:
diff --git a/src/borg/repository.py b/src/borg/repository.py
--- a/src/borg/repository.py
+++ b/src/borg/repository.py
@@ -1081,6 +1081,7 @@ def report_error(msg):
         pi = ProgressIndicatorPercent(
             total=segment_count, msg="Checking segments %3.1f%%", step=0.1, msgid="repository.check"
         )
+        segment = -1  # avoid uninitialized variable if there are no segment files at all
         for i, (segment, filename) in enumerate(self.io.segment_iterator()):
             pi.show(i)
             if segment <= last_segment_checked:


Test patch:
Repository: borgbackup/borg

Issue:
Add BORG_LOCK_WAIT environment variable
To compliment the `--lock-wait` option?


Discussion:
could be done, you would need to give the option rather often without the env var?
Exactly.  The way my code is written, we are already using environment variables for each command to set the passphrase and remote path.  It would be cleanest to be able to include this setting in the environment vars.

Perhaps, the options system could be augmented to always look for an environment variable that matches one of the common options.  So, `lock-wait` option automatically looks for a `BORG_LOCK_WAIT` environment variable for it's default value.  Priority of option values would come from

1. cli value
2. environment value
3. Borg default

That would solve the need for this specific option but give more flexibility for other options too.
Don't overengineer it. I see what you mean and it is basically ok, but no quick change to borg (and should not be home-grown either, but rather come from a library based on argparse doing that - it also relates to configurability and could also process a config file).

So for now, a simple PR to add that env var to code and docs would be welcome. master branch first, later backport to 1.1-maint.

Resolving patch:
diff --git a/src/borg/archiver/_common.py b/src/borg/archiver/_common.py
--- a/src/borg/archiver/_common.py
+++ b/src/borg/archiver/_common.py
@@ -468,7 +468,7 @@ def define_common_options(add_common_option):
         metavar="SECONDS",
         dest="lock_wait",
         type=int,
-        default=1,
+        default=int(os.environ.get("BORG_LOCK_WAIT", 1)),
         help="wait at most SECONDS for acquiring a repository/cache lock (default: %(default)d).",
     )
     add_common_option(


Test patch:
Repository: borgbackup/borg

Issue:
borg diff not recognizing identical chunk params when diffing v1.1 with v1.2 archive
## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

ISSUE

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).

```
$ borg -V
borg 1.2.2
```

#### Full borg commandline that lead to the problem (leave away excludes and passwords)

```
$ borg diff ::main-2022-09-30T13:41:46Z main-2022-10-03T07:50:40Z
--chunker-params might be different between archives, diff will be slow.
If you know for certain that they are the same, pass --same-chunker-params to override this check.
…
```

## Describe the problem you're observing.

The former archive in the above command was created by Borg v1.1:

```
$ borg debug dump-archive ::main-2022-09-30T13:41:46Z -
{
    …
    "_meta":
    {
        "chunker_params": [
            19,
            23,
            21,
            4095
        ],
…
```

And the latter by Borg v1.2:

```
$ borg debug dump-archive ::main-2022-10-03T07:50:40Z -
{
    …
    "_meta":
    {
        "chunker_params": [
            "buzhash",
            19,
            23,
            21,
            4095
        ],

```

As can be seen, in Borg v1.2 the name for the chunker was added to the params. I suspect this leads to this issue.

I'd consider the following changes to improve the situation:

- [x] Correct comparison between v1.1 and v1.2 chunk params in v1.2.x.
- [x] Convert the chunk params to the newer format during the `borg transfer` required to upgrade to v2.0. As of Borg 2.0.0b3 the chunk params are kept as is but this would probably be a good time to normalize them and drop the code to handle the old format.


Discussion:
Very good analysis / suggestions, thanks!

Resolving patch:
diff --git a/src/borg/upgrade.py b/src/borg/upgrade.py
--- a/src/borg/upgrade.py
+++ b/src/borg/upgrade.py
@@ -1,6 +1,6 @@
 from struct import Struct
 
-from .constants import REQUIRED_ITEM_KEYS
+from .constants import REQUIRED_ITEM_KEYS, CH_BUZHASH
 from .compress import ZLIB, ZLIB_legacy, ObfuscateSize
 from .helpers import HardLinkManager
 from .item import Item
@@ -143,6 +143,10 @@ def upgrade_archive_metadata(self, *, metadata):
         for attr in ("cmdline", "hostname", "username", "comment", "chunker_params", "recreate_cmdline"):
             if hasattr(metadata, attr):
                 new_metadata[attr] = getattr(metadata, attr)
+        if chunker_params := new_metadata.get("chunker_params"):
+            if len(chunker_params) == 4 and isinstance(chunker_params[0], int):
+                # this is a borg < 1.2 chunker_params tuple, no chunker algo specified, but we only had buzhash:
+                new_metadata["chunker_params"] = (CH_BUZHASH,) + chunker_params
         # old borg used UTC timestamps, but did not have the explicit tz offset in them.
         for attr in ("time", "time_end"):
             if hasattr(metadata, attr):


Test patch:
Repository: borgbackup/borg

Issue:
master branch / 2.0.0b3: crash in borg transfer
reported by @mbo77 there: https://github.com/borgbackup/borg/discussions/7070#discussioncomment-4234898

```
Local Exception
Traceback (most recent call last):
  File "/usr/local/lib64/python3.10/site-packages/borg/archiver/__init__.py", line 594, in main
    exit_code = archiver.run(args)
  File "/usr/local/lib64/python3.10/site-packages/borg/archiver/__init__.py", line 522, in run
    return set_ec(func(args))
  File "/usr/local/lib64/python3.10/site-packages/borg/archiver/_common.py", line 223, in wrapper
    return method(self, args, **kwargs)
  File "/usr/local/lib64/python3.10/site-packages/borg/archiver/_common.py", line 155, in wrapper
    return method(self, args, repository=repository, cache=cache_, **kwargs)
  File "/usr/local/lib64/python3.10/site-packages/borg/archiver/transfer_cmd.py", line 66, in do_transfer
    for item in other_archive.iter_items():
  File "/usr/local/lib64/python3.10/site-packages/borg/archive.py", line 610, in iter_items
    for item in self.pipeline.unpack_many(
  File "/usr/local/lib64/python3.10/site-packages/borg/archive.py", line 282, in unpack_many
    for data in self.fetch_many(ids):
  File "/usr/local/lib64/python3.10/site-packages/borg/archive.py", line 313, in fetch_many
    _, data = self.repo_objs.parse(id_, cdata)
  File "/usr/local/lib64/python3.10/site-packages/borg/repoobj.py", line 157, in parse
    meta["csize"] = len(data_compressed)
TypeError: 'NoneType' object does not support item assignment

Platform: Linux ds9 6.0.8-200.fc36.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Nov 11 15:03:58 UTC 2022 x86_64
Linux: Unknown Linux
Borg: 2.0.0b3  Python: CPython 3.10.8 msgpack: 1.0.3 fuse: llfuse 1.4.2 [pyfuse3,llfuse]
PID: 3759256  CWD: /datapool
sys.argv: ['/usr/local/bin/borg', 'transfer', '--other-repo', '/mnt/backup/bDISK/borg', '--repo', '/datapool/borg2', '--upgrader=From12To20', '--dry-run']
SSH_ORIGINAL_COMMAND: None
```



Discussion:

Resolving patch:
diff --git a/src/borg/repoobj.py b/src/borg/repoobj.py
--- a/src/borg/repoobj.py
+++ b/src/borg/repoobj.py
@@ -153,6 +153,6 @@ def parse(self, id: bytes, cdata: bytes, decompress: bool = True) -> tuple[dict,
             meta = {}
             meta["ctype"] = compressor.ID
             meta["clevel"] = compressor.level
+            meta["csize"] = len(data_compressed)
             data = data_compressed
-        meta["csize"] = len(data_compressed)
         return meta, data


Test patch:
Repository: borgbackup/borg

Issue:
create --dry-run --list does not list directory items
See #7206 for the description.

The problem here is that in `--dry-run` mode, the status for directories was just not set to `-` and thus no `--list` output was generated for directories.

status `-` means that the item was not backed up due to dry-run mode.

status `x` would be displayed for excluded items.

Without dry-run, it works like expected and also generates output (status `d`).


Discussion:

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -810,8 +810,11 @@ def _rec_walk(self, *, path, parent_fd, name, fso, cache, matcher,
                                                 restrict_dev=restrict_dev, read_special=read_special, dry_run=dry_run)
                                 self.print_file_status('x', path)
                             return
-                    if not recurse_excluded_dir and not dry_run:
-                        status = fso.process_dir_with_fd(path=path, fd=child_fd, st=st)
+                    if not recurse_excluded_dir:
+                        if not dry_run:
+                            status = fso.process_dir_with_fd(path=path, fd=child_fd, st=st)
+                        else:
+                            status = '-'
                     if recurse:
                         with backup_io('scandir'):
                             entries = helpers.scandir_inorder(path=path, fd=child_fd)


Test patch:
Repository: borgbackup/borg

Issue:
create --dry-run --list does not list directory items
See #7206 for the description.

The problem here is that in `--dry-run` mode, the status for directories was just not set to `-` and thus no `--list` output was generated for directories.

status `-` means that the item was not backed up due to dry-run mode.

status `x` would be displayed for excluded items.

Without dry-run, it works like expected and also generates output (status `d`).


Discussion:

Resolving patch:
diff --git a/src/borg/archiver/create_cmd.py b/src/borg/archiver/create_cmd.py
--- a/src/borg/archiver/create_cmd.py
+++ b/src/borg/archiver/create_cmd.py
@@ -118,7 +118,8 @@ def create_inner(archive, cache, fso):
                     if status == "C":
                         self.print_warning("%s: file changed while we backed it up", path)
                     self.print_file_status(status, path)
-                    fso.stats.files_stats[status] += 1
+                    if not dry_run and status is not None:
+                        fso.stats.files_stats[status] += 1
                 if args.paths_from_command:
                     rc = proc.wait()
                     if rc != 0:
@@ -142,7 +143,8 @@ def create_inner(archive, cache, fso):
                         else:
                             status = "-"
                         self.print_file_status(status, path)
-                        fso.stats.files_stats[status] += 1
+                        if not dry_run and status is not None:
+                            fso.stats.files_stats[status] += 1
                         continue
                     path = os.path.normpath(path)
                     parent_dir = os.path.dirname(path) or "."
@@ -442,8 +444,11 @@ def _rec_walk(
                                         )
                                 self.print_file_status("x", path)
                             return
-                    if not recurse_excluded_dir and not dry_run:
-                        status = fso.process_dir_with_fd(path=path, fd=child_fd, st=st)
+                    if not recurse_excluded_dir:
+                        if not dry_run:
+                            status = fso.process_dir_with_fd(path=path, fd=child_fd, st=st)
+                        else:
+                            status = "-"
                     if recurse:
                         with backup_io("scandir"):
                             entries = helpers.scandir_inorder(path=path, fd=child_fd)
@@ -472,7 +477,7 @@ def _rec_walk(
             self.print_warning("%s: file changed while we backed it up", path)
         if not recurse_excluded_dir:
             self.print_file_status(status, path)
-            if status is not None:
+            if not dry_run and status is not None:
                 fso.stats.files_stats[status] += 1
 
     def build_parser_create(self, subparsers, common_parser, mid_common_parser):


Test patch:
Repository: borgbackup/borg

Issue:
disallow --list with --progress?
The way that `--progress` works can't work together prettily with `--list` (or other output generating options).

borg did not disallow using them together yet, but we get issue reports by users again and again wondering/complaining about that it looks weird, so maybe we should just disallow it.



Discussion:

Resolving patch:
diff --git a/src/borg/archiver/__init__.py b/src/borg/archiver/__init__.py
--- a/src/borg/archiver/__init__.py
+++ b/src/borg/archiver/__init__.py
@@ -383,7 +383,9 @@ def parse_args(self, args=None):
         parser.common_options.resolve(args)
         func = get_func(args)
         if func == self.do_create and args.paths and args.paths_from_stdin:
-            parser.error("Must not pass PATH with ``--paths-from-stdin``.")
+            parser.error("Must not pass PATH with --paths-from-stdin.")
+        if args.progress and getattr(args, "output_list", False):
+            parser.error("Options --progress and --list do not play nicely together.")
         if func == self.do_create and not args.paths:
             if args.content_from_command or args.paths_from_command:
                 parser.error("No command given.")


Test patch:
Repository: borgbackup/borg

Issue:
hostname, user and group names encoding / binary requirements
Related to #7232 and #6151.

Guess we can rely on hostname being pure ascii, right?

But how about user and group names (for now, but also for past borg1 archives)?
- pure ascii? (guess ideally/usually yes, but we can not rely on it)
- pure unicode without surrogate-escape?
- unicode with surrogate-escapes to represent arbitrary bytestrings?


Discussion:

Resolving patch:
diff --git a/src/borg/helpers/parseformat.py b/src/borg/helpers/parseformat.py
--- a/src/borg/helpers/parseformat.py
+++ b/src/borg/helpers/parseformat.py
@@ -75,8 +75,7 @@ def text_to_json(key, value):
         value.encode(coding, errors="strict")  # check if pure unicode
     except UnicodeEncodeError:
         # value has surrogate escape sequences
-        value_replace_encoded = value.encode(coding, errors="replace")
-        data[key] = value_replace_encoded.decode(coding, errors="strict")
+        data[key] = remove_surrogates(value)
         value_bytes = value.encode(coding, errors="surrogateescape")
         data.update(binary_to_json(key, value_bytes))
     else:
@@ -715,9 +714,9 @@ def __init__(self, format, repository, manifest, key, *, json=False, iec=False):
         self.format = partial_format(format, static_keys)
         self.format_keys = {f[1] for f in Formatter().parse(format)}
         self.call_keys = {
-            "hostname": partial(self.get_meta, "hostname", rs=True),
-            "username": partial(self.get_meta, "username", rs=True),
-            "comment": partial(self.get_meta, "comment", rs=False),
+            "hostname": partial(self.get_meta, "hostname"),
+            "username": partial(self.get_meta, "username"),
+            "comment": partial(self.get_meta, "comment"),
             "end": self.get_ts_end,
             "command_line": self.get_cmdline,
         }
@@ -747,6 +746,12 @@ def get_item_data(self, archive_info):
         )
         for key in self.used_call_keys:
             item_data[key] = self.call_keys[key]()
+
+        # Note: name and comment are validated, should never contain surrogate escapes.
+        # But unsure whether hostname, username could contain surrogate escapes, play safe:
+        for key in "hostname", "username":
+            if key in item_data:
+                item_data.update(text_to_json(key, item_data[key]))
         return item_data
 
     @property
@@ -758,9 +763,8 @@ def archive(self):
             self._archive = Archive(self.manifest, self.name, iec=self.iec)
         return self._archive
 
-    def get_meta(self, key, rs):
-        value = self.archive.metadata.get(key, "")
-        return remove_surrogates(value) if rs else value
+    def get_meta(self, key):
+        return self.archive.metadata.get(key, "")
 
     def get_cmdline(self):
         cmdline = map(remove_surrogates, self.archive.metadata.get("cmdline", []))


Test patch:
Repository: borgbackup/borg

Issue:
birthtime test sporadically failing
```
    @pytest.mark.skipif(not is_darwin, reason="only for macOS")
    def test_extract_xattrs_resourcefork(self):
        self.create_regular_file("file")
        self.cmd(f"--repo={self.repository_location}", "rcreate", "-e" "none")
        input_path = os.path.abspath("input/file")
        xa_key, xa_value = b"com.apple.ResourceFork", b"whatshouldbehere"  # issue #7234
        xattr.setxattr(input_path.encode(), xa_key, xa_value)
        birthtime_expected = os.stat(input_path).st_birthtime
        mtime_expected = os.stat(input_path).st_mtime_ns
        # atime_expected = os.stat(input_path).st_atime_ns
        self.cmd(f"--repo={self.repository_location}", "create", "test", "input")
        with changedir("output"):
            self.cmd(f"--repo={self.repository_location}", "extract", "test")
            extracted_path = os.path.abspath("input/file")
            birthtime_extracted = os.stat(extracted_path).st_birthtime
            mtime_extracted = os.stat(extracted_path).st_mtime_ns
            # atime_extracted = os.stat(extracted_path).st_atime_ns
            xa_value_extracted = xattr.getxattr(extracted_path.encode(), xa_key)
        assert xa_value_extracted == xa_value
>       assert birthtime_extracted == birthtime_expected
E       assert 1675177426.389071 == 1675177426.3890712
```

birthtime extracted is slightly different than the birthtime seen in `os.stat` of input file. difference here is 200ns.




Discussion:
Guess an easy fix would be to just compute the difference and check whether it is less than 1ms.

Resolving patch:
diff --git a/src/borg/testsuite/archiver/extract_cmd.py b/src/borg/testsuite/archiver/extract_cmd.py
--- a/src/borg/testsuite/archiver/extract_cmd.py
+++ b/src/borg/testsuite/archiver/extract_cmd.py
@@ -507,7 +507,8 @@ def test_extract_xattrs_resourcefork(self):
             # atime_extracted = os.stat(extracted_path).st_atime_ns
             xa_value_extracted = xattr.getxattr(extracted_path.encode(), xa_key)
         assert xa_value_extracted == xa_value
-        assert birthtime_extracted == birthtime_expected
+        # cope with small birthtime deviations of less than 1000ns:
+        assert -1000 <= (birthtime_extracted - birthtime_expected) * 1e9 <= 1000
         assert mtime_extracted == mtime_expected
         # assert atime_extracted == atime_expected  # still broken, but not really important.
 


Test patch: