internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: borgbackup/borg

Issue:
--checkpoint-interval default too small
borgbackup writes local chunk cache every 5 minutes, causing burst writes of more than 1 GiB each time.

I just saw this again, vmstat 1 output:

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 1  3 2438352 130788    132 10609920  136   20 10316 126072 6832 28580 13 24 50 13  0
 1  3 2438400 154100    132 10558816    0   48   428 472396 3306 18648  4 14 35 47  0
 0  3 2438400 168484    132 10560680    0    0   452 452384 2268 2732  2  3 39 56  0
 0  2 2438400 183004    132 10561820    0    0   280 403788 2639 4671  5  5 38 53  0
 0  2 2438400 196108    132 10564468    0    0  1448 193200 2537 4000  6  7 43 45  0
 2  1 2438400 187444    132 10574308    0    0 12156 11252 4628 12536 14  7 65 15  0

the cache is:

merkaba:~> du -sh .cache/borg 
2,6G    .cache/borg

If I add the values of vmstat block out I get already 1,6 GiB of data written. Thats more than half of the cache. If it does that every 5 minutes its quite an amount of writes. Does borgbackup fully rewrite the cache every 5 minutes or does it just write changes to the cache?

(That said good SSDs can take a lot of writes, but still, this seems excessive to me.)



Discussion:
Okay, to be fair, it is still an BTRFS RAID 1, so divide that 1,6 GiB by two, as BTRFS wrote the same data to each of the SSDs. Still if this would be harddisks writing about 800 MiB would cause a considerable delay. Does borgbackup need to write local caches on every checkpoint on the remote repo? Can´t it rebuild the local cache in case it is not current?

(I know I can disable local cache completely and consider doing just this as I have a fast network connection here.)

You can change checkpoint-interval `--checkpoint-interval SECONDS`

Thanks, I adapted my script. I briefly looked for something like this, but overlooked it in the manpage. Its documented. So… closing.

reopening it so we change the default to a bigger value, e.g. 1800 (30 minutes).


Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -1487,8 +1487,8 @@ def build_parser(self, prog=None):
                                    help='manually specify the archive creation date/time (UTC). '
                                         'alternatively, give a reference file/directory.')
         archive_group.add_argument('-c', '--checkpoint-interval', dest='checkpoint_interval',
-                                   type=int, default=300, metavar='SECONDS',
-                                   help='write checkpoint every SECONDS seconds (Default: 300)')
+                                   type=int, default=1800, metavar='SECONDS',
+                                   help='write checkpoint every SECONDS seconds (Default: 1800)')
         archive_group.add_argument('--chunker-params', dest='chunker_params',
                                    type=ChunkerParams, default=CHUNKER_PARAMS,
                                    metavar='CHUNK_MIN_EXP,CHUNK_MAX_EXP,HASH_MASK_BITS,HASH_WINDOW_SIZE',


Test patch:
Repository: borgbackup/borg

Issue:
1.0.x LoggedIO.write_commit should do a double-fsync commit
We do this already in master, and 1.0.x should do it as well.

``` py
    def write_commit(self):
        fd = self.get_write_fd(no_new=True)
        header = self.header_no_crc_fmt.pack(self.header_fmt.size, TAG_COMMIT)
        crc = self.crc_fmt.pack(crc32(header) & 0xffffffff)
        # first fsync(fd) here (to ensure data before the commit tag supposedly hit the disk)
        fd.write(b''.join((crc, header)))
        self.close_segment()  # after-commit fsync()
```

i.e.

``` py
            crc = self.crc_fmt.pack(crc32(header) & 0xffffffff)
            fd.flush()
            os.fsync(fd.fileno())
            fd.write(b''.join((crc, header)))
```



Discussion:

Resolving patch:
diff --git a/borg/repository.py b/borg/repository.py
--- a/borg/repository.py
+++ b/borg/repository.py
@@ -727,8 +727,11 @@ def write_commit(self):
         fd = self.get_write_fd(no_new=True)
         header = self.header_no_crc_fmt.pack(self.header_fmt.size, TAG_COMMIT)
         crc = self.crc_fmt.pack(crc32(header) & 0xffffffff)
+        # first fsync(fd) here (to ensure data supposedly hits the disk before the commit tag)
+        fd.flush()
+        os.fsync(fd.fileno())
         fd.write(b''.join((crc, header)))
-        self.close_segment()
+        self.close_segment()  # after-commit fsync()
 
     def close_segment(self):
         if self._write_fd:


Test patch:
Repository: borgbackup/borg

Issue:
Exception on the client when server pipe is closed
```
Remote: packet_write_wait: Connection to 192.168.1.20 port 22: Broken pipe
Exception ignored in: <bound method RemoteRepository.__del__ of <RemoteRepository ssh://backup@backup.localnet/backup1/mordor>>
Traceback (most recent call last):
  File "/usr/lib64/python3.4/site-packages/borg/remote.py", line 181, in __del__
    assert False, "cleanup happened in Repository.__del__"
AssertionError: cleanup happened in Repository.__del__
Connection closed by remote host
```

(the connection was closed because the power was lost at the other end).

This is borg 1.0.3 from Fedora23 repo.



Discussion:
can you check again with 1.0-maint code? maybe pulling the plug is enough, no need for power switch.

Is this maybe because the context manager's `__exit__` is called on the ConnectionClosed exception and then tries to do a rollback, which requires a working connection - and if it has none, maybe triggers a ConnectionClosed (or other) exception again (which terminates borg) and does not reach the self.close() it wanted also to do? Instead, on termination the repo object gets deleted and thus the AssertionError is triggered.

See `remote.RemoteRepository.__exit__`.


Resolving patch:
diff --git a/borg/remote.py b/borg/remote.py
--- a/borg/remote.py
+++ b/borg/remote.py
@@ -189,9 +189,14 @@ def __enter__(self):
         return self
 
     def __exit__(self, exc_type, exc_val, exc_tb):
-        if exc_type is not None:
-            self.rollback()
-        self.close()
+        try:
+            if exc_type is not None:
+                self.rollback()
+        finally:
+            # in any case, we want to cleanly close the repo, even if the
+            # rollback can not succeed (e.g. because the connection was
+            # already closed) and raised another exception:
+            self.close()
 
     def borg_cmd(self, args, testing):
         """return a borg serve command line"""


Test patch:
Repository: borgbackup/borg

Issue:
1.0.x should be able to read v2 hints files
Pretty simple.

The difference between v1 and v2 is that 'compact' is a mapping of `segment -> count of deleted bytes in segment [free space]` in v2, while in v1 it's just a list of segments. So for 1.0-maint just need to check for version==2 and then take only the keys into `self.compact`.

This makes changing versions back-and-forth, if so desired, easier.



Discussion:

Resolving patch:
diff --git a/borg/repository.py b/borg/repository.py
--- a/borg/repository.py
+++ b/borg/repository.py
@@ -219,10 +219,14 @@ def prepare_txn(self, transaction_id, do_cleanup=True):
                 self.io.cleanup(transaction_id)
             with open(os.path.join(self.path, 'hints.%d' % transaction_id), 'rb') as fd:
                 hints = msgpack.unpack(fd)
-            if hints[b'version'] != 1:
-                raise ValueError('Unknown hints file version: %d' % hints['version'])
+            hints_version = hints[b'version']
+            if hints_version not in (1, 2):
+                raise ValueError('Unknown hints file version: %d' % hints_version)
             self.segments = hints[b'segments']
-            self.compact = set(hints[b'compact'])
+            if hints_version == 1:
+                self.compact = set(hints[b'compact'])
+            elif hints_version == 2:
+                self.compact = set(hints[b'compact'].keys())
 
     def write_index(self):
         hints = {b'version': 1,


Test patch:
Repository: borgbackup/borg

Issue:
add missing env var for --remote-path
`borg` supports configuring everything related to “how do I access the repository” using environment variables, in particular `BORG_REPO`, `BORG_PASSPHRASE`, `BORG_KEY_FILE` and `BORG_RSH`. The only one missing is `BORG_REMOTE_PATH`.

If that were added, it would be possible to set up an environment (either in `.profile`, or via a wrapper) where no repository-access-specific parameters need to be passed as arguments.

(#1163 is somewhat related)



Discussion:
Good idea.

Seems like this was somehow forgotten and caused the annoyance of always having to give `--remote-path X` if the executable on the remote is not "borg" (and in PATH).

rsync.net and their "borg1" come to mind...


Resolving patch:
diff --git a/borg/archiver.py b/borg/archiver.py
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -912,8 +912,8 @@ def build_parser(self, args=None, prog=None):
                                    help='do not load/update the file metadata cache used to detect unchanged files')
         common_parser.add_argument('--umask', dest='umask', type=lambda s: int(s, 8), default=UMASK_DEFAULT, metavar='M',
                                    help='set umask to M (local and remote, default: %(default)04o)')
-        common_parser.add_argument('--remote-path', dest='remote_path', default='borg', metavar='PATH',
-                                   help='set remote path to executable (default: "%(default)s")')
+        common_parser.add_argument('--remote-path', dest='remote_path', metavar='PATH',
+                                   help='set remote path to executable (default: "borg")')
 
         parser = argparse.ArgumentParser(prog=prog, description='Borg - Deduplicated Backups')
         parser.add_argument('-V', '--version', action='version', version='%(prog)s ' + __version__,
diff --git a/borg/remote.py b/borg/remote.py
--- a/borg/remote.py
+++ b/borg/remote.py
@@ -220,7 +220,8 @@ def borg_cmd(self, args, testing):
         if testing:
             return [sys.executable, '-m', 'borg.archiver', 'serve'] + opts + self.extra_test_args
         else:  # pragma: no cover
-            return [args.remote_path, 'serve'] + opts
+            remote_path = args.remote_path or os.environ.get('BORG_REMOTE_PATH', 'borg')
+            return [remote_path, 'serve'] + opts
 
     def ssh_cmd(self, location):
         """return a ssh command line that can be prefixed to a borg command line"""


Test patch:
Repository: borgbackup/borg

Issue:
Document recovery from full disk
I filled my backup partition with borg and now even borg list is crashing.
The documentation should descript how to recover from a full disk. 
[This comment](https://github.com/borgbackup/borg/issues/1114#issuecomment-222656759) suggests:

```
you can try deleting the segment files of the last incomplete backup and then run borg check --repair. After that, use borg delete or prune with the --save-space flag.
```

But i don't know how to safely do this and am afraid of destroying my whole backup.



Discussion:

Resolving patch:
diff --git a/borg/repository.py b/borg/repository.py
--- a/borg/repository.py
+++ b/borg/repository.py
@@ -84,6 +84,12 @@ def __enter__(self):
 
     def __exit__(self, exc_type, exc_val, exc_tb):
         if exc_type is not None:
+            no_space_left_on_device = exc_type is OSError and exc_val.errno == errno.ENOSPC
+            # The ENOSPC could have originated somewhere else besides the Repository. The cleanup is always safe, unless
+            # EIO or FS corruption ensues, which is why we specifically check for ENOSPC.
+            if self._active_txn and no_space_left_on_device:
+                logger.warning('No space left on device, cleaning up partial transaction to free space.')
+                self.io.cleanup(self.io.get_segments_transaction_id())
             self.rollback()
         self.close()
 
@@ -543,7 +549,7 @@ def get_latest_segment(self):
         return None
 
     def get_segments_transaction_id(self):
-        """Verify that the transaction id is consistent with the index transaction id
+        """Return last committed segment
         """
         for segment, filename in self.segment_iterator(reverse=True):
             if self.is_committed_segment(segment):


Test patch:
Repository: borgbackup/borg

Issue:
cygwin: sparse test failure
the sparse test fails in its preparation phase (1.0-maint):

the "is input sparse" st.st_blocks \* 512 < total_len / 9 assertion fails.

blocks are ~42M size, total_len is ~84M.

cygwin / ntfs / win10



Discussion:

Resolving patch:
diff --git a/borg/testsuite/archiver.py b/borg/testsuite/archiver.py
--- a/borg/testsuite/archiver.py
+++ b/borg/testsuite/archiver.py
@@ -398,7 +398,7 @@ def test_sparse_file(self):
         st = os.stat(filename)
         self.assert_equal(st.st_size, total_len)
         if sparse_support and hasattr(st, 'st_blocks'):
-            self.assert_true(st.st_blocks * 512 < total_len / 9)  # is input sparse?
+            self.assert_true(st.st_blocks * 512 < total_len)  # is input sparse?
         self.cmd('init', self.repository_location)
         self.cmd('create', self.repository_location + '::test', 'input')
         with changedir('output'):
@@ -415,7 +415,7 @@ def test_sparse_file(self):
         if sparse_support:
             if hasattr(st, 'st_blocks'):
                 # do only check if it is less, do NOT check if it is much less
-                # as that causes troubles on xfs and zfs:
+                # as that causes troubles on xfs, zfs, ntfs:
                 self.assert_true(st.st_blocks * 512 < total_len)
             if hasattr(os, 'SEEK_HOLE') and hasattr(os, 'SEEK_DATA'):
                 with open(filename, 'rb') as fd:


Test patch:
Repository: borgbackup/borg

Issue:
cygwin test issue: chown throws errno 22 invalid argument
In testsuite/archiver.py line 191:

```
os.chown('input/file1', 100, 200)
OSError: [Errno 22] Invalid argument: 'input/file1'
```

The filename argument is valid, but it does not like the made up uid/gid.



Discussion:

Resolving patch:
diff --git a/borg/testsuite/archiver.py b/borg/testsuite/archiver.py
--- a/borg/testsuite/archiver.py
+++ b/borg/testsuite/archiver.py
@@ -288,10 +288,14 @@ def create_test_files(self):
             # File mode
             os.chmod('input/dir2', 0o555)  # if we take away write perms, we need root to remove contents
             # File owner
-            os.chown('input/file1', 100, 200)
+            os.chown('input/file1', 100, 200)  # raises OSError invalid argument on cygwin
             have_root = True  # we have (fake)root
         except PermissionError:
             have_root = False
+        except OSError as e:
+            if e.errno != errno.EINVAL:
+                raise
+            have_root = False
         return have_root
 
     def test_basic_functionality(self):


Test patch:
Repository: borgbackup/borg

Issue:
add a comment header to usage/*.rst.inc
these files are automatically generated from borg's builtin help and should not be edited.

this should be pointed out so noone wastes time editing them directly.



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -163,6 +163,7 @@ def run(self):
         for command, parser in choices.items():
             print('generating help for %s' % command)
             with open('docs/usage/%s.rst.inc' % command, 'w') as doc:
+                doc.write(".. IMPORTANT: this file is auto-generated from borg's built-in help, do not edit!\n\n")
                 if command == 'help':
                     for topic in Archiver.helptext:
                         params = {"topic": topic,


Test patch:
Repository: borgbackup/borg

Issue:
FUSE xattr code broken on Linux
borg mount crashes on Linux.

```
# ls -l /mnt
ls: /mnt: Input/output error
ls: cannot open directory /mnt: Transport endpoint is not connected
```

When running borg mount -f, the real problem gets visible:

```
$ ls -ld /mnt
File "/home/askalski/private/src/borg/borg/fuse.py", line 197, in getxattr
    raise llfuse.FUSEError(errno.ENOATTR) from None
AttributeError: module 'errno' has no attribute 'ENOATTR'
```

Reason:

After the initial fix for FUSE xattr code on OS X, it is now broken on Linux (in borg 1.0.4), because OS X Python has errno.ENOATTR, but Linux Python does not. :-(

The llfuse documentation just said that ENOATTR needs to be returned, but did not point out that it means llfuse.ENOATTR and _not_ errno.ENOATTR.



Discussion:

Resolving patch:
diff --git a/borg/fuse.py b/borg/fuse.py
--- a/borg/fuse.py
+++ b/borg/fuse.py
@@ -194,7 +194,7 @@ def getxattr(self, inode, name, ctx=None):
         try:
             return item.get(b'xattrs', {})[name]
         except KeyError:
-            raise llfuse.FUSEError(errno.ENOATTR) from None
+            raise llfuse.FUSEError(llfuse.ENOATTR) from None
 
     def _load_pending_archive(self, inode):
         # Check if this is an archive we need to load


Test patch:
Repository: borgbackup/borg

Issue:
add input_io wrapper around get_flags
in master branch ("1.1") get_flags() is also working on the input file on some OSes, so it needs wrapping / error handling. fix all the places.

See also #1211.



Discussion:

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -597,7 +597,7 @@ def chunk_decref(id, stats):
             try:
                 self.cache.chunk_decref(id, stats)
             except KeyError:
-                cid = hexlify(id).decode('ascii')
+                cid = bin_to_hex(id)
                 raise ChunksIndexError(cid)
             except Repository.ObjectNotFound as e:
                 # object not in repo - strange, but we wanted to delete it anyway.
@@ -657,13 +657,12 @@ def stat_attrs(self, st, path):
             attrs['user'] = attrs['group'] = None
         with backup_io():
             xattrs = xattr.get_all(path, follow_symlinks=False)
+            bsdflags = get_flags(path, st)
+            acl_get(path, attrs, st, self.numeric_owner)
         if xattrs:
             attrs['xattrs'] = StableDict(xattrs)
-        bsdflags = get_flags(path, st)
         if bsdflags:
             attrs['bsdflags'] = bsdflags
-        with backup_io():
-            acl_get(path, attrs, st, self.numeric_owner)
         return attrs
 
     def process_dir(self, path, st):
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -324,8 +324,12 @@ def _process(self, archive, cache, matcher, exclude_caches, exclude_if_present,
             return
         status = None
         # Ignore if nodump flag is set
-        if get_flags(path, st) & stat.UF_NODUMP:
-            self.print_file_status('x', path)
+        try:
+            if get_flags(path, st) & stat.UF_NODUMP:
+                self.print_file_status('x', path)
+                return
+        except OSError as e:
+            self.print_warning('%s: %s', path, e)
             return
         if stat.S_ISREG(st.st_mode):
             if not dry_run:


Test patch:
Repository: borgbackup/borg

Issue:
somehow warn when extracting a damaged file
borg check --repair replaces file chunks that are missing in the repo by all-zero chunks of same length.

but there currently is no indication when extracting such a file, it looks normally, no warning.

so, emit a warning? rename the file in the metadata, e.g. ".damaged"?

when a file is healed later, it should extract normally again.



Discussion:
also maybe the archive as a whole could maybe gain a flag "contains damaged files". This could then be displayed in list.


Resolving patch:
diff --git a/borg/archive.py b/borg/archive.py
--- a/borg/archive.py
+++ b/borg/archive.py
@@ -18,7 +18,7 @@
 from io import BytesIO
 from . import xattr
 from .helpers import Error, uid2user, user2uid, gid2group, group2gid, \
-    parse_timestamp, to_localtime, format_time, format_timedelta, \
+    parse_timestamp, to_localtime, format_time, format_timedelta, remove_surrogates, \
     Manifest, Statistics, decode_dict, make_path_safe, StableDict, int_to_bigint, bigint_to_int, \
     ProgressIndicatorPercent
 from .platform import acl_get, acl_set
@@ -344,6 +344,8 @@ def add_file_chunks(chunks):
         return stats
 
     def extract_item(self, item, restore_attrs=True, dry_run=False, stdout=False, sparse=False):
+        has_damaged_chunks = b'chunks_healthy' in item
+
         if dry_run or stdout:
             if b'chunks' in item:
                 for data in self.pipeline.fetch_many([c[0] for c in item[b'chunks']], is_preloaded=True):
@@ -351,6 +353,9 @@ def extract_item(self, item, restore_attrs=True, dry_run=False, stdout=False, sp
                         sys.stdout.buffer.write(data)
                 if stdout:
                     sys.stdout.buffer.flush()
+            if has_damaged_chunks:
+                logger.warning('File %s has damaged (all-zero) chunks. Try running borg check --repair.' %
+                               remove_surrogates(item[b'path']))
             return
 
         dest = self.cwd
@@ -397,6 +402,9 @@ def extract_item(self, item, restore_attrs=True, dry_run=False, stdout=False, sp
                         fd.truncate(pos)
                         fd.flush()
                         self.restore_attrs(path, item, fd=fd.fileno())
+                if has_damaged_chunks:
+                    logger.warning('File %s has damaged (all-zero) chunks. Try running borg check --repair.' %
+                                   remove_surrogates(item[b'path']))
             return
         with backup_io():
             # No repository access beyond this point.


Test patch:
Repository: borgbackup/borg

Issue:
borg break-lock doesn't use BORG_REPO env variable
Most other commands do.
Using 1.0.6.



Discussion:

Resolving patch:
diff --git a/borg/archiver.py b/borg/archiver.py
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -1316,7 +1316,7 @@ def build_parser(self, args=None, prog=None):
                                           formatter_class=argparse.RawDescriptionHelpFormatter,
                                           help='break repository and cache locks')
         subparser.set_defaults(func=self.do_break_lock)
-        subparser.add_argument('location', metavar='REPOSITORY',
+        subparser.add_argument('location', metavar='REPOSITORY', nargs='?', default='',
                                type=location_validator(archive=False),
                                help='repository for which to break the locks')
 


Test patch: