internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: borgbackup/borg

Issue:
borg version to log
I want suggest add ability to show borg version in logs when using `-v` key.



Discussion:
could be --show-version (we already have --show-rc).


Resolving patch:
diff --git a/borg/archiver.py b/borg/archiver.py
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -814,6 +814,8 @@ def build_parser(self, args=None, prog=None):
                                    help='enable debug output, work on log level DEBUG')
         common_parser.add_argument('--lock-wait', dest='lock_wait', type=int, metavar='N', default=1,
                                    help='wait for the lock, but max. N seconds (default: %(default)d).')
+        common_parser.add_argument('--show-version', dest='show_version', action='store_true', default=False,
+                                   help='show/log the borg version')
         common_parser.add_argument('--show-rc', dest='show_rc', action='store_true', default=False,
                                    help='show/log the return code (rc)')
         common_parser.add_argument('--no-files-cache', dest='cache_files', action='store_false',
@@ -1414,6 +1416,8 @@ def run(self, args):
         os.umask(args.umask)  # early, before opening files
         self.lock_wait = args.lock_wait
         setup_logging(level=args.log_level, is_serve=args.func == self.do_serve)  # do not use loggers before this!
+        if args.show_version:
+            logger.info('borgbackup version %s' % __version__)
         check_extension_modules()
         if is_slow_msgpack():
             logger.warning("Using a pure-python msgpack! This will result in lower performance.")


Test patch:
Repository: borgbackup/borg

Issue:
add more info to prune log
Please add flowing info to prune log
1. Archive fingerprint (In case of archive rename we can't automatically handle pruned archive)
2. Repository name



Discussion:
So, what do you want to do with the archive fingerprint from the log?

"repository name" -> "repository URL"

We are store info from borg logs to mysql, after prune we check prune log and look for pruned archive and set what archive was pruned. But in case of archive rename we can't automatically handle pruned archive. 

We currently develop (php+mysql) webapp to manage\analyze borg logs and we need a bit more info from logs for this. After app will finished i push it to github.

I have changed format_archive() so it outputs:

```
<name (26 chars)> <datetime> [<id>]
```

it's bit less pretty now due to the long hex id, but of course the id is more stable than the name.

It is used by prune, list and delete.


Resolving patch:
diff --git a/borg/helpers.py b/borg/helpers.py
--- a/borg/helpers.py
+++ b/borg/helpers.py
@@ -1,4 +1,5 @@
 import argparse
+from binascii import hexlify
 from collections import namedtuple
 from functools import wraps
 import grp
@@ -610,7 +611,11 @@ def sizeof_fmt_decimal(num, suffix='B', sep='', precision=2):
 
 
 def format_archive(archive):
-    return '%-36s %s' % (archive.name, format_time(to_localtime(archive.ts)))
+    return '%-36s %s [%s]' % (
+        archive.name,
+        format_time(to_localtime(archive.ts)),
+        hexlify(archive.id).decode('ascii'),
+    )
 
 
 def memoize(function):


Test patch:
Repository: borgbackup/borg

Issue:
borg client waiting forever to acquire the lock
I have been experimenting with borg 1.0.0 and I am wondering how locking is supposed to work.

I have 2 computers being backed up to the same shared repository over SSH (eg. 192.168.0.1:/var/tmp/mybackup').

Each client is run like this: `borg create --lock-wait=10000 [lots of --excludes] 192.168.0.1:/var/tmp/mybackup::$(hostname)-$(date '+%Y-%m-%d_%H:%M:%S' ) /`

When I start backup at the same time on all clients, only 1 client can start the backup and the others are waiting on the lock. So far so good. But when the first client finishes its backup then I expect the next client to acquire the lock yet all stay stuck waiting on the lock!

Here is what `ps aux | grep borg` displays on the backup server:

``` sh
root@backupserver:/etc# ps aux | grep borg
root     19833  0.0  0.0  93012 27412 ?        Ss   15:46   0:00 /usr/bin/python3 /usr/local/bin/borg serve --umask=077 --info
```

I have to CTRL-C the waiting clients then restart them manually; then again only the first to acquire the lock can complete, all other clients stay stuck.

Worse, the 'borg serve' processes on the server keep running, even after I CTRL-C out of the clients.

Am I doing anything wrong?



Discussion:

Resolving patch:
diff --git a/borg/locking.py b/borg/locking.py
--- a/borg/locking.py
+++ b/borg/locking.py
@@ -33,7 +33,8 @@ def __init__(self, timeout=None, sleep=None):
 
         :param timeout: time out interval [s] or None (no timeout)
         :param sleep: sleep interval [s] (>= 0: do sleep call, <0: don't call sleep)
-                      or None (autocompute: use 10% of timeout, or 1s for no timeout)
+                      or None (autocompute: use 10% of timeout [but not more than 60s],
+                      or 1s for no timeout)
         """
         if timeout is not None and timeout < 0:
             raise ValueError("timeout must be >= 0")
@@ -42,7 +43,7 @@ def __init__(self, timeout=None, sleep=None):
             if timeout is None:
                 sleep = 1.0
             else:
-                sleep = timeout / 10.0
+                sleep = min(60.0, timeout / 10.0)
         self.sleep_interval = sleep
         self.start_time = None
         self.end_time = None


Test patch:
Repository: borgbackup/borg

Issue:
Capitalization is not consistent
This is just a nitpicking issue, but borg does not have a consistent usage of capitalized words in log messages. Also ellipsis (...) are not used consistently; sometimes there are some, sometimes there are none.

```
Archives: 1, w/ cached Idx: 0, w/ outdated Idx: 0, w/o cached Idx: 1.
Fetching and building archive index for tatooine-2016-03-25-150454-CET ...
Merging into master chunks index ...
Done.
reading files cache
processing files
```



Discussion:
Yeah, you're right.

These need to get capitalized and ...ed.

Also, I guess it makes sense to emit them at DEBUG level, not at INFO level.


Resolving patch:
diff --git a/borg/archive.py b/borg/archive.py
--- a/borg/archive.py
+++ b/borg/archive.py
@@ -535,7 +535,7 @@ def process_file(self, path, st, cache):
         first_run = not cache.files
         ids = cache.file_known_and_unchanged(path_hash, st)
         if first_run:
-            logger.info('processing files')
+            logger.debug('Processing files ...')
         chunks = None
         if ids is not None:
             # Make sure all ids are available
diff --git a/borg/cache.py b/borg/cache.py
--- a/borg/cache.py
+++ b/borg/cache.py
@@ -161,7 +161,7 @@ def close(self):
     def _read_files(self):
         self.files = {}
         self._newest_mtime = 0
-        logger.info('reading files cache')
+        logger.debug('Reading files cache ...')
         with open(os.path.join(self.path, 'files'), 'rb') as fd:
             u = msgpack.Unpacker(use_list=True)
             while True:


Test patch:
Repository: borgbackup/borg

Issue:
failing hashindex tests on netbsd
I've bisected the issue and this is the changeset breaking the tests: https://github.com/borgbackup/borg/commit/f63be633470128a84a2640cd3202d28c62223e21

@pyhalov can you fix it in a way so it does not break netbsd (64bit)?

`py.test -k hashindex` shows the failing tests.

I think the issue is that you check for _BIG_ENDIAN or _LITTLE_ENDIAN being defined. They are both defined on netbsd (and have a 4321, 1234 value).

http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/sys/sys/endian.h



Discussion:
I can do something like this https://github.com/pyhalov/borg/commit/6965f7d4383b7e2fa263afff3d8f74f4e63dbc0d 

Yes, that does not fail on netbsd, please commit/PR and append ", fixes #804" to your commit comment.


Resolving patch:
diff --git a/borg/fuse.py b/borg/fuse.py
--- a/borg/fuse.py
+++ b/borg/fuse.py
@@ -174,15 +174,15 @@ def getattr(self, inode, ctx=None):
             else:
                 entry.st_ctime_ns = bigint_to_int(item[b'mtime'])
         else:
-            entry.st_mtime_ns = bigint_to_int(item[b'mtime']) / 1e9
+            entry.st_mtime = bigint_to_int(item[b'mtime']) / 1e9
             if b'atime' in item:
-                entry.st_atime_ns = bigint_to_int(item[b'atime']) / 1e9
+                entry.st_atime = bigint_to_int(item[b'atime']) / 1e9
             else:
-                entry.st_atime_ns = bigint_to_int(item[b'mtime']) / 1e9
+                entry.st_atime = bigint_to_int(item[b'mtime']) / 1e9
             if b'ctime' in item:
-                entry.st_ctime_ns = bigint_to_int(item[b'ctime']) / 1e9
+                entry.st_ctime = bigint_to_int(item[b'ctime']) / 1e9
             else:
-                entry.st_ctime_ns = bigint_to_int(item[b'mtime']) / 1e9
+                entry.st_ctime = bigint_to_int(item[b'mtime']) / 1e9
         return entry
 
     def listxattr(self, inode, ctx=None):
diff --git a/borg/testsuite/chunker.py b/borg/testsuite/chunker.py
--- a/borg/testsuite/chunker.py
+++ b/borg/testsuite/chunker.py
@@ -1,7 +1,7 @@
 from io import BytesIO
 
 from ..chunker import Chunker, buzhash, buzhash_update
-from ..archive import CHUNK_MAX_EXP
+from ..archive import CHUNK_MAX_EXP, CHUNKER_PARAMS
 from . import BaseTestCase
 
 
@@ -29,3 +29,14 @@ def test_buzhash(self):
         self.assert_equal(buzhash(b'abcdefghijklmnop', 1), buzhash_update(buzhash(b'Xabcdefghijklmno', 1), ord('X'), ord('p'), 16, 1))
         # Test with more than 31 bytes to make sure our barrel_shift macro works correctly
         self.assert_equal(buzhash(b'abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz', 0), 566521248)
+
+    def test_small_reads(self):
+        class SmallReadFile:
+            input = b'a' * (20 + 1)
+
+            def read(self, nbytes):
+                self.input = self.input[:-1]
+                return self.input[:1]
+
+        reconstructed = b''.join(Chunker(0, *CHUNKER_PARAMS).chunkify(SmallReadFile()))
+        assert reconstructed == b'a' * 20


Test patch:
Repository: borgbackup/borg

Issue:
Add --error to only output log messages of error or higher
Back in 2015, the following was added, but refactored out prior to the 1.0.0 release:

https://github.com/borgbackup/borg/commit/25140e8c8218fe19ef9ffc8029e3f53516b2d546#diff-dfaa777fd33eac4e20768ac811e52696R649

That gave the ability to specify the log level for all levels (debug, info, warning, error, critical).  The current default seems to be to only output "warning" or higher, but there is no way to tell borg that you only want to see "error" or higher.

Adding "--error" would allow me to suppress warning messages in the output.



Discussion:
There's the BORG_LOGGING_CONF environment variable, which uses these: https://docs.python.org/3/library/logging.config.html

Usually applications have `-q`, `--quiet` for this.

To suppress warnings? Borg doesn't generate many of those, and the few around ought to be noticed/fixed.

quiet is the default, but it does not include warnings and errors.

and I am rather tending to remove -v / --verbose than to add -q / --quiet (because it seems to trigger expectations about the way it works that are not like it really works).

that said, adding --error might be an option if somebody prefers to not have some specific (and expected) warnings show up. @tgharold had the specific issue that a `borg check` cron job always emails him that "orphan chunks check skipped" warning because he only checked the last 5 archives, not all.

this could be fixed in 3 ways (to stop the spamming):
- add --error so one can decide to not see warnings (which might have bad side effects in case some important warning passes unnoticed)
- change log level of that message to info (orphan chunks are no big problem, the just waste space, but do not cause malfunction)
- kill that message completely

```
logger.warning('Orphaned objects check skipped (needs all archives checked).')
```

This should be on INFO level imho, because it's expected from the command given to Borg.

@enkore agreed, that's a good reason.


Resolving patch:
diff --git a/borg/archive.py b/borg/archive.py
--- a/borg/archive.py
+++ b/borg/archive.py
@@ -879,7 +879,7 @@ def orphan_chunks_check(self):
                 for id_ in unused:
                     self.repository.delete(id_)
         else:
-            logger.warning('Orphaned objects check skipped (needs all archives checked).')
+            logger.info('Orphaned objects check skipped (needs all archives checked).')
 
     def finish(self, save_space=False):
         if self.repair:


Test patch:
Repository: borgbackup/borg

Issue:
How to check correctness of passphrase in a script?
I'm working on a shellscript that is run interactively which asks for the repo passphrase and then invokes borg multiple times. Is there a simple way to check in a script that the passphrase is correct?

My idea at first was to set `BORG_PASSPHRASE` and then run a simple command (`borg list`) and check the exit code, but unfortunately borg queries the tty for a passphrase if the one in the environment does not match. Right now I'm using _expect_ to get around this, but this is quite hackish -- is there a simpler method?



Discussion:
A command "borg ask-passphrase" might be useful.

It would ask for the passphrase, try it, repeat until valid and then ... echo it on stdout?

If borg list doesn't prompt for the passphrase and just fails (as in 94b61ea apparently, haven't run it yet) that is sufficient for my use case, thanks.

Maybe a quick check command that only checks the repo exists and the passphrase works (maybe also: the temp dir and the cache dir exist and are writeable, other assumptions borg makes...), would be generally useful in different kinds of scripts. I'm not sure having borg ask for the passphrase itself and then echo it would save the people who write scripts enough work to justify the additional complexity.

Thanks for the quick reaction btw, I love the project!


Resolving patch:
diff --git a/borg/key.py b/borg/key.py
--- a/borg/key.py
+++ b/borg/key.py
@@ -18,6 +18,10 @@
 PREFIX = b'\0' * 8
 
 
+class PassphraseWrong(Error):
+    """passphrase supplied in BORG_PASSPHRASE is incorrect"""
+
+
 class PasswordRetriesExceeded(Error):
     """exceeded the maximum password retries"""
 
@@ -284,13 +288,19 @@ def detect(cls, repository, manifest_data):
         key = cls(repository)
         target = key.find_key()
         prompt = 'Enter passphrase for key %s: ' % target
-        passphrase = Passphrase.env_passphrase(default='')
-        for retry in range(1, 4):
-            if key.load(target, passphrase):
-                break
-            passphrase = Passphrase.getpass(prompt)
+        passphrase = Passphrase.env_passphrase()
+        if passphrase is None:
+            passphrase = Passphrase()
+            if not key.load(target, passphrase):
+                for retry in range(0, 3):
+                    passphrase = Passphrase.getpass(prompt)
+                    if key.load(target, passphrase):
+                        break
+                else:
+                    raise PasswordRetriesExceeded
         else:
-            raise PasswordRetriesExceeded
+            if not key.load(target, passphrase):
+                raise PassphraseWrong
         num_blocks = num_aes_blocks(len(manifest_data) - 41)
         key.init_ciphers(PREFIX + long_to_bytes(key.extract_nonce(manifest_data) + num_blocks))
         return key


Test patch:
Repository: borgbackup/borg

Issue:
test crash on freebsd 
Testing the borg binary runs into an issue on freebsd:

```
==> freebsd64: ___________________ ArchiverTestCaseBinary.test_delete_repo ____________________
==> freebsd64: self = <borg.testsuite.archiver.ArchiverTestCaseBinary testMethod=test_delete_repo>
==> freebsd64:     def test_delete_repo(self):
==> freebsd64:         self.create_regular_file('file1', size=1024 * 80)
==> freebsd64:         self.create_regular_file('dir2/file2', size=1024 * 80)
==> freebsd64:         self.cmd('init', self.repository_location)
==> freebsd64:         self.cmd('create', self.repository_location + '::test', 'input')
==> freebsd64:         self.cmd('create', self.repository_location + '::test.2', 'input')
==> freebsd64:         os.environ['BORG_DELETE_I_KNOW_WHAT_I_AM_DOING'] = 'no'
==> freebsd64:         self.cmd('delete', self.repository_location, exit_code=2)
==> freebsd64: >       self.archiver.exit_code = 0
==> freebsd64: E       AttributeError: 'NoneType' object has no attribute 'exit_code'
==> freebsd64: py34/lib/python3.4/site-packages/borg/testsuite/archiver.py:736: AttributeError
==> freebsd64: ======= 1 failed, 441 passed, 112 skipped, 3 xpassed in 2917.08 seconds ========
```



Discussion:
> Testing the borg binary runs into an issue on freebsd:

I downloaded the binary and tested this on freebsd 10.2 and there was no problem. 

```
./borg-freebsd64
usage: borg-freebsd64 [-h] [-V] <command> ...

Borg - Deduplicated Backups

optional arguments:
  -h, --help            show this help message and exit
  -V, --version         show version number and exit

```

How did you have this issue?


Resolving patch:
diff --git a/borg/testsuite/archiver.py b/borg/testsuite/archiver.py
--- a/borg/testsuite/archiver.py
+++ b/borg/testsuite/archiver.py
@@ -59,6 +59,7 @@ def exec_cmd(*args, archiver=None, fork=False, exe=None, **kw):
             sys.stdout = sys.stderr = output = StringIO()
             if archiver is None:
                 archiver = Archiver()
+            archiver.exit_code = EXIT_SUCCESS
             args = archiver.parse_args(list(args))
             ret = archiver.run(args)
             return ret, output.getvalue()
@@ -733,7 +734,6 @@ def test_delete_repo(self):
         self.cmd('create', self.repository_location + '::test.2', 'input')
         os.environ['BORG_DELETE_I_KNOW_WHAT_I_AM_DOING'] = 'no'
         self.cmd('delete', self.repository_location, exit_code=2)
-        self.archiver.exit_code = 0
         assert os.path.exists(self.repository_path)
         os.environ['BORG_DELETE_I_KNOW_WHAT_I_AM_DOING'] = 'YES'
         self.cmd('delete', self.repository_location)


Test patch:
Repository: borgbackup/borg

Issue:
mix of neg. & pos. --keep-xxx may trigger validation error
It is valid to give positive or negative numbers X for `--keep-TIMEUNIT X`.

A validator was implemented badly so that if you gave e.g. -1 once and +1 once (or in general: if the sum of them all was zero), it triggered a validation error telling you that you have to give at least one of these options.

Workaround: as any negative value means the same, just give another negative value so that the sum is not 0.



Discussion:

Resolving patch:
diff --git a/borg/archiver.py b/borg/archiver.py
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -565,11 +565,13 @@ def do_info(self, args, repository, manifest, key, archive, cache):
     @with_repository()
     def do_prune(self, args, repository, manifest, key):
         """Prune repository archives according to specified rules"""
-        archives = manifest.list_archive_infos(sort_by='ts', reverse=True)  # just a ArchiveInfo list
-        if args.hourly + args.daily + args.weekly + args.monthly + args.yearly == 0 and args.within is None:
-            self.print_error('At least one of the "keep-within", "keep-hourly", "keep-daily", "keep-weekly", '
-                             '"keep-monthly" or "keep-yearly" settings must be specified')
+        if not any((args.hourly, args.daily,
+                    args.weekly, args.monthly, args.yearly, args.within)):
+            self.print_error('At least one of the "keep-within", "keep-last", '
+                             '"keep-hourly", "keep-daily", '
+                             '"keep-weekly", "keep-monthly" or "keep-yearly" settings must be specified.')
             return self.exit_code
+        archives = manifest.list_archive_infos(sort_by='ts', reverse=True)  # just a ArchiveInfo list
         if args.prefix:
             archives = [archive for archive in archives if archive.name.startswith(args.prefix)]
         keep = []


Test patch:
Repository: borgbackup/borg

Issue:
Borg fails when creating an archive
When I create an archive with commit 4f507c0 borg crashes with the following error. Commit 77966c7 works as expected.

```
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 1961, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 1911, in run
    return args.func(args)
  File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 80, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 290, in do_create
    create_inner(archive, cache)
  File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 262, in create_inner
    read_special=args.read_special, dry_run=dry_run, st=st)
  File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 345, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 345, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 345, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 345, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 345, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 345, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 345, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 319, in _process
    status = archive.process_file(path, st, cache, self.ignore_inode)
  File "/usr/lib/python3.5/site-packages/borg/archive.py", line 602, in process_file
    self.add_item(item)
  File "/usr/lib/python3.5/site-packages/borg/archive.py", line 228, in add_item
    self.items_buffer.add(item)
  File "/usr/lib/python3.5/site-packages/borg/archive.py", line 81, in add
    self.flush()
  File "/usr/lib/python3.5/site-packages/borg/archive.py", line 98, in flush
    self.buffer.write(chunks[-1])
TypeError: a bytes-like object is required, not '_Chunk'
```



Discussion:

Resolving patch:
diff --git a/borg/archive.py b/borg/archive.py
--- a/borg/archive.py
+++ b/borg/archive.py
@@ -95,7 +95,7 @@ def flush(self, flush=False):
         for chunk in chunks[:end]:
             self.chunks.append(self.write_chunk(chunk))
         if end == -1:
-            self.buffer.write(chunks[-1])
+            self.buffer.write(chunks[-1].data)
 
     def is_full(self):
         return self.buffer.tell() > self.BUFFER_SIZE


Test patch: