internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: borgbackup/borg

Issue:
stat() errors in scandir_generic() cause the whole directory to be skipped
When backing up a directory, ``scandir_inorder()`` is used to read the directory entries ordered by inode number:
https://github.com/borgbackup/borg/blob/1.1.4/src/borg/archiver.py#L613-L614
https://github.com/borgbackup/borg/blob/1.1.4/src/borg/helpers.py#L2104-L2105

 If the ``scandir_generic()`` implementation is used (python3.4 without the scandir package installed), the implicit `stat()` that is done on ``dirent.inode()`` will raise `Permission denied` e.g. on fuse mountpoints without `allow_other`. As a result, the whole directory will be skipped just because of one failing stat.
https://github.com/borgbackup/borg/blob/1.1.4/src/borg/helpers.py#L2046-L2090

1.) A failing ``stat()`` should only skip the corresponding directory entry.
2.) If the ``stat()`` fails on a directory entry that is going to be skipped anyway, the error message should be suppressed.



Discussion:
Good analysis. Would you like to do a PR to fix that issue?
BTW: ``sorted()`` in the ``scandir_generic()`` implementation is useless and causes unnecessary CPU load:
https://github.com/borgbackup/borg/blob/1.1.4/src/borg/helpers.py#L2087

Borg only uses ``scandir_inorder()`` which sorts the directory entries a second time anyways.
And even ``os.scandir()`` yields entries in arbitrary order:
https://docs.python.org/3/library/os.html#os.scandir
@leo-b sounds good, just do a PR.
fixed by #3558.

Resolving patch:
diff --git a/src/borg/helpers/fs.py b/src/borg/helpers/fs.py
--- a/src/borg/helpers/fs.py
+++ b/src/borg/helpers/fs.py
@@ -11,6 +11,9 @@
 
 from ..constants import *  # NOQA
 
+from ..logger import create_logger
+logger = create_logger()
+
 
 def get_base_dir():
     """Get home directory / base directory for borg:
@@ -128,8 +131,21 @@ def hardlinkable(mode):
     return stat.S_ISREG(mode) or stat.S_ISBLK(mode) or stat.S_ISCHR(mode) or stat.S_ISFIFO(mode)
 
 
+def scandir_keyfunc(dirent):
+    try:
+        return (0, dirent.inode())
+    except OSError as e:
+        # maybe a permission denied error while doing a stat() on the dirent
+        logger.debug('scandir_inorder: Unable to stat %s: %s', dirent.path, e)
+        # order this dirent after all the others lexically by file name
+        # we may not break the whole scandir just because of an exception in one dirent
+        # ignore the exception for now, since another stat will be done later anyways
+        # (or the entry will be skipped by an exclude pattern)
+        return (1, dirent.name)
+
+
 def scandir_inorder(path='.'):
-    return sorted(os.scandir(path), key=lambda dirent: dirent.inode())
+    return sorted(os.scandir(path), key=scandir_keyfunc)
 
 
 def secure_erase(path):


Test patch:
Repository: borgbackup/borg

Issue:
zstd tests: are erroneously skipped
in master and 1.1.x, the zstd tests were skipped because a not anymore used zstd import.

borg initially used the "zstd" python binding, but later replaced this by an own Cython-based binding, so we do not have the zstd module installed any more - thus the tests checking for that were always skipped.

note: no impact on productive use of borg.


Discussion:

Resolving patch:
diff --git a/src/borg/testsuite/compress.py b/src/borg/testsuite/compress.py
--- a/src/borg/testsuite/compress.py
+++ b/src/borg/testsuite/compress.py
@@ -5,11 +5,6 @@
 except ImportError:
     lzma = None
 
-try:
-    import zstd
-except ImportError:
-    zstd = None
-
 import pytest
 
 from ..compress import get_compressor, Compressor, CompressionSpec, CNONE, ZLIB, LZ4, LZMA, ZSTD, Auto
@@ -75,8 +70,6 @@ def test_lzma():
 
 
 def test_zstd():
-    if zstd is None:
-        pytest.skip("No zstd support found.")
     c = get_compressor(name='zstd')
     cdata = c.compress(data)
     assert len(cdata) < len(data)
@@ -109,6 +102,9 @@ def test_compressor():
     params_list = [
         dict(name='none'),
         dict(name='lz4'),
+        dict(name='zstd', level=1),
+        dict(name='zstd', level=3),
+        # avoiding high zstd levels, memory needs unclear
         dict(name='zlib', level=0),
         dict(name='zlib', level=6),
         dict(name='zlib', level=9),
@@ -119,12 +115,6 @@ def test_compressor():
             dict(name='lzma', level=6),
             # we do not test lzma on level 9 because of the huge memory needs
         ]
-    if zstd:
-        params_list += [
-            dict(name='zstd', level=1),
-            dict(name='zstd', level=3),
-            # also avoiding high zstd levels, memory needs unclear
-        ]
     for params in params_list:
         c = Compressor(**params)
         assert data == c.decompress(c.compress(data))


Test patch:
Repository: borgbackup/borg

Issue:
ascii decoding of header files can lead to UnicodeDecodeError
https://github.com/borgbackup/borg/blob/1.1.4/setup_lz4.py#L27 e.g. there, but also some other places for other header files.

https://github.com/openbsd/ports/commit/1232bc2b8d672eb5b4fa1873e0402f2d94da3101


Discussion:
2 potential ways to fix:
- decode using utf-8 (will still crash on other encodings)
- use 'rb' and b'...' to detect specific content

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -101,8 +101,8 @@ def detect_openssl(prefixes):
     for prefix in prefixes:
         filename = os.path.join(prefix, 'include', 'openssl', 'evp.h')
         if os.path.exists(filename):
-            with open(filename, 'r') as fd:
-                if 'PKCS5_PBKDF2_HMAC(' in fd.read():
+            with open(filename, 'rb') as fd:
+                if b'PKCS5_PBKDF2_HMAC(' in fd.read():
                     return prefix
 
 
@@ -110,8 +110,8 @@ def detect_lz4(prefixes):
     for prefix in prefixes:
         filename = os.path.join(prefix, 'include', 'lz4.h')
         if os.path.exists(filename):
-            with open(filename, 'r') as fd:
-                if 'LZ4_decompress_safe' in fd.read():
+            with open(filename, 'rb') as fd:
+                if b'LZ4_decompress_safe' in fd.read():
                     return prefix
 
 


Test patch:
Repository: borgbackup/borg

Issue:
stdin and filename
Hi,

i have problem with specific filename in stdin backup.

example:

mysqldump ... | borg create -v --stats ssh://backup_host:/....::{now:%Y-%m-%d} -

i expected param --stdin-filename (for example in rectis mysqldump | restic backup --stdin --stdin-filename production.sql)





Can i change filename from stdin to custom name?


Thanks,
Kolar



Discussion:
There is no such option yet, the name 'stdin' is currently hardcoded.

As there is usually only one such file in such an archive and you can name the archive as you like, isn't that sufficient?
Many people have the problem of rename the ending to, for example, .sql ... because windows 10 hiding extension of files... 
Not sure I understand the problem.

BTW, isn't showing the file extension part of security best practices on windows? People should really see whether it is foo.pdf or foo.pdf.exe.
Most people are stupid if they find a set of stdin in the archive, so they will not know what to do. For example, webhost customers. It would be great to change the stdin file name.
Hi ThomasWaldmann,

any news? Can i help you?


Thanks,
Kolar Michal
@michalkolar if you like, implement the feature and do a PR.

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -455,7 +455,7 @@ def create_inner(archive, cache, fso):
                     pass
             for path in args.paths:
                 if path == '-':  # stdin
-                    path = 'stdin'
+                    path = args.stdin_name
                     if not dry_run:
                         try:
                             status = fso.process_stdin(path, cache)
@@ -2940,6 +2940,8 @@ def define_archive_filters_group(subparser, *, sort_by=True, first_last=True):
                                help='experimental: do not synchronize the cache. Implies not using the files cache.')
         subparser.add_argument('--no-files-cache', dest='cache_files', action='store_false',
                                help='do not load/update the file metadata cache used to detect unchanged files')
+        subparser.add_argument('--stdin-name', metavar='NAME', dest='stdin_name', default='stdin',
+                               help='use NAME in archive for stdin data (default: "stdin")')
 
         exclude_group = define_exclusion_group(subparser, tag_files=True)
         exclude_group.add_argument('--exclude-nodump', dest='exclude_nodump', action='store_true',


Test patch:
Repository: borgbackup/borg

Issue:
stdin and filename
Hi,

i have problem with specific filename in stdin backup.

example:

mysqldump ... | borg create -v --stats ssh://backup_host:/....::{now:%Y-%m-%d} -

i expected param --stdin-filename (for example in rectis mysqldump | restic backup --stdin --stdin-filename production.sql)





Can i change filename from stdin to custom name?


Thanks,
Kolar



Discussion:
There is no such option yet, the name 'stdin' is currently hardcoded.

As there is usually only one such file in such an archive and you can name the archive as you like, isn't that sufficient?
Many people have the problem of rename the ending to, for example, .sql ... because windows 10 hiding extension of files... 
Not sure I understand the problem.

BTW, isn't showing the file extension part of security best practices on windows? People should really see whether it is foo.pdf or foo.pdf.exe.
Most people are stupid if they find a set of stdin in the archive, so they will not know what to do. For example, webhost customers. It would be great to change the stdin file name.
Hi ThomasWaldmann,

any news? Can i help you?


Thanks,
Kolar Michal
@michalkolar if you like, implement the feature and do a PR.

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -485,7 +485,7 @@ def create_inner(archive, cache):
             logger.debug('Processing files ...')
             for path in args.paths:
                 if path == '-':  # stdin
-                    path = 'stdin'
+                    path = args.stdin_name
                     if not dry_run:
                         try:
                             status = archive.process_stdin(path, cache)
@@ -3138,6 +3138,8 @@ def define_archive_filters_group(subparser, *, sort_by=True, first_last=True):
                                help='experimental: do not synchronize the cache. Implies not using the files cache.')
         subparser.add_argument('--no-files-cache', dest='cache_files', action='store_false',
                                help='do not load/update the file metadata cache used to detect unchanged files')
+        subparser.add_argument('--stdin-name', metavar='NAME', dest='stdin_name', default='stdin',
+                               help='use NAME in archive for stdin data (default: "stdin")')
 
         exclude_group = define_exclusion_group(subparser, tag_files=True)
         exclude_group.add_argument('--exclude-nodump', dest='exclude_nodump', action='store_true',


Test patch:
Repository: borgbackup/borg

Issue:
Placeholders {now} and {utcnow} are replaced with timestamps microseconds apart
Both `{now}` and `{utcnow}` should point to exact same moment, save for timezone adjustment. They are not, because current time is queried twice when replacing placeholders:

Borg version: 1.0.11, but master is also affected.

https://github.com/borgbackup/borg/blob/26f1cd148603e2c2762c5f27ef56bf5b9ca17d18/src/borg/helpers/parseformat.py#L189

```
$ borg create \
  --info \
  --stats \
  ':: utcnow: {utcnow:%H:%M:%S.%f}, now: {now:%H:%M:%S.%f}' \
  /dev/null
------------------------------------------------------------------------------
Archive name:  utcnow: 21:16:01.582465, now: 22:16:01.582458
Archive fingerprint: ...
Time (start): Mon, 2018-01-15 22:16:03
Time (end):   Mon, 2018-01-15 22:16:03
```




Discussion:

Resolving patch:
diff --git a/src/borg/helpers/parseformat.py b/src/borg/helpers/parseformat.py
--- a/src/borg/helpers/parseformat.py
+++ b/src/borg/helpers/parseformat.py
@@ -180,14 +180,14 @@ def format_line(format, data):
 def replace_placeholders(text):
     """Replace placeholders in text with their values."""
     from ..platform import fqdn
-    current_time = datetime.now()
+    current_time = datetime.now(timezone.utc)
     data = {
         'pid': os.getpid(),
         'fqdn': fqdn,
         'reverse-fqdn': '.'.join(reversed(fqdn.split('.'))),
         'hostname': socket.gethostname(),
-        'now': DatetimeWrapper(current_time.now()),
-        'utcnow': DatetimeWrapper(current_time.utcnow()),
+        'now': DatetimeWrapper(current_time.astimezone(None)),
+        'utcnow': DatetimeWrapper(current_time),
         'user': uid2user(os.getuid(), os.getuid()),
         'uuid4': str(uuid.uuid4()),
         'borgversion': borg_version,


Test patch:
Repository: borgbackup/borg

Issue:
Placeholders {now} and {utcnow} are replaced with timestamps microseconds apart
Both `{now}` and `{utcnow}` should point to exact same moment, save for timezone adjustment. They are not, because current time is queried twice when replacing placeholders:

Borg version: 1.0.11, but master is also affected.

https://github.com/borgbackup/borg/blob/26f1cd148603e2c2762c5f27ef56bf5b9ca17d18/src/borg/helpers/parseformat.py#L189

```
$ borg create \
  --info \
  --stats \
  ':: utcnow: {utcnow:%H:%M:%S.%f}, now: {now:%H:%M:%S.%f}' \
  /dev/null
------------------------------------------------------------------------------
Archive name:  utcnow: 21:16:01.582465, now: 22:16:01.582458
Archive fingerprint: ...
Time (start): Mon, 2018-01-15 22:16:03
Time (end):   Mon, 2018-01-15 22:16:03
```




Discussion:

Resolving patch:
diff --git a/src/borg/helpers.py b/src/borg/helpers.py
--- a/src/borg/helpers.py
+++ b/src/borg/helpers.py
@@ -665,14 +665,14 @@ def format_line(format, data):
 def replace_placeholders(text):
     """Replace placeholders in text with their values."""
     from .platform import fqdn
-    current_time = datetime.now()
+    current_time = datetime.now(timezone.utc)
     data = {
         'pid': os.getpid(),
         'fqdn': fqdn,
         'reverse-fqdn': '.'.join(reversed(fqdn.split('.'))),
         'hostname': socket.gethostname(),
-        'now': DatetimeWrapper(current_time.now()),
-        'utcnow': DatetimeWrapper(current_time.utcnow()),
+        'now': DatetimeWrapper(current_time.astimezone(None)),
+        'utcnow': DatetimeWrapper(current_time),
         'user': uid2user(os.getuid(), os.getuid()),
         'uuid4': str(uuid.uuid4()),
         'borgversion': borg_version,


Test patch:
Repository: borgbackup/borg

Issue:
Manifest rebuilding does not report progress (even with -p option on).
<!--
Thank you for reporting an issue.

*IMPORTANT* -  *before* creating a new issue please look around:
 - Borgbackup documentation: http://borgbackup.readthedocs.io/en/stable/index.html
 - FAQ: https://borgbackup.readthedocs.io/en/stable/faq.html
 and
 - open issues in Github tracker: https://github.com/borgbackup/borg/issues
  
If you cannot find a similar problem, then create a new issue.

Please fill in as much of the template as possible.
-->

## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

BUG

## Describe the problem you're observing.

I have quite a large backup (400GB) where the manifest is missing. I am waiting on:
```
$ borg -p -v check --repair backup
...
Rebuilding missing manifest, this might take some time...
```
for more than 100 hours now and no progress is being reported. Seems that borg does not report progress for manifest rebuilding even though the -p option has been turned on.


Discussion:
There is a bit of progress reporting: `logger.info('Found archive %s', name)`

But that only happens IF/WHEN it finds some archive and can be rare (if you only have few / large archives).

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -1318,7 +1318,10 @@ def valid_archive(obj):
         # within this repository (assuming that newer borg versions support more item keys).
         manifest = Manifest(self.key, self.repository)
         archive_keys_serialized = [msgpack.packb(name.encode()) for name in ARCHIVE_KEYS]
+        pi = ProgressIndicatorPercent(total=len(self.chunks), msg="Rebuilding manifest %6.2f%%", step=0.01,
+                                      msgid='check.rebuild_manifest')
         for chunk_id, _ in self.chunks.iteritems():
+            pi.show()
             cdata = self.repository.get(chunk_id)
             try:
                 data = self.key.decrypt(chunk_id, cdata)
@@ -1350,6 +1353,7 @@ def valid_archive(obj):
                     logger.warning('Duplicate archive name %s, storing as %s', name, new_name)
                     name = new_name
                 manifest.archives[name] = (chunk_id, archive.time)
+        pi.finish()
         logger.info('Manifest rebuild complete.')
         return manifest
 


Test patch:
Repository: borgbackup/borg

Issue:
improve borg check output
https://github.com/borgbackup/borg/pull/3792/files

This needs more work, it now always says 'Deleting orphaned objects...'.


Discussion:

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -1585,11 +1585,12 @@ def orphan_chunks_check(self):
             if orphaned:
                 logger.error('{} orphaned objects found!'.format(len(orphaned)))
                 self.error_found = True
-            if self.repair:
-                logger.info('Deleting orphaned objects...')
+            if self.repair and unused:
+                logger.info('Deleting %d orphaned and %d superseded objects...' % (
+                    len(orphaned), len(self.possibly_superseded)))
                 for id_ in unused:
                     self.repository.delete(id_)
-                logger.info('Finished deleting orphaned objects.')
+                logger.info('Finished deleting orphaned/superseded objects.')
         else:
             logger.info('Orphaned objects check skipped (needs all archives checked).')
 


Test patch:
Repository: borgbackup/borg

Issue:
add a hint to setup.py that py37 requires llfuse >= 1.3.4



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -69,6 +69,9 @@
     # llfuse 0.41.1, 1.1 are ok
     extras_require['fuse'] = ['llfuse <2.0, !=0.42.*, !=0.43, !=1.0', ]
 
+if my_python >= (3, 7):
+    extras_require['fuse'][0] += ', >=1.3.4'
+
 from setuptools import setup, find_packages, Extension
 from setuptools.command.sdist import sdist
 from distutils.command.clean import clean


Test patch:
Repository: borgbackup/borg

Issue:
improve borg check output
https://github.com/borgbackup/borg/pull/3792/files

This needs more work, it now always says 'Deleting orphaned objects...'.


Discussion:

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -1721,16 +1721,22 @@ def orphan_chunks_check(self):
             if orphaned:
                 logger.error('{} orphaned objects found!'.format(len(orphaned)))
                 self.error_found = True
-            if self.repair:
+            if self.repair and unused:
+                logger.info('Deleting %d orphaned and %d superseded objects...' % (
+                    len(orphaned), len(self.possibly_superseded)))
                 for id_ in unused:
                     self.repository.delete(id_)
+                logger.info('Finished deleting orphaned/superseded objects.')
         else:
             logger.info('Orphaned objects check skipped (needs all archives checked).')
 
     def finish(self, save_space=False):
         if self.repair:
+            logger.info('Writing Manifest.')
             self.manifest.write()
+            logger.info('Committing repo (may take a while, due to compact_segments)...')
             self.repository.commit(save_space=save_space)
+            logger.info('Finished committing repo.')
 
 
 class ArchiveRecreater:


Test patch:
Repository: borgbackup/borg

Issue:
Manifest rebuilding does not report progress (even with -p option on).
<!--
Thank you for reporting an issue.

*IMPORTANT* -  *before* creating a new issue please look around:
 - Borgbackup documentation: http://borgbackup.readthedocs.io/en/stable/index.html
 - FAQ: https://borgbackup.readthedocs.io/en/stable/faq.html
 and
 - open issues in Github tracker: https://github.com/borgbackup/borg/issues
  
If you cannot find a similar problem, then create a new issue.

Please fill in as much of the template as possible.
-->

## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

BUG

## Describe the problem you're observing.

I have quite a large backup (400GB) where the manifest is missing. I am waiting on:
```
$ borg -p -v check --repair backup
...
Rebuilding missing manifest, this might take some time...
```
for more than 100 hours now and no progress is being reported. Seems that borg does not report progress for manifest rebuilding even though the -p option has been turned on.


Discussion:
There is a bit of progress reporting: `logger.info('Found archive %s', name)`

But that only happens IF/WHEN it finds some archive and can be rare (if you only have few / large archives).

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -1458,7 +1458,10 @@ def valid_archive(obj):
         # within this repository (assuming that newer borg versions support more item keys).
         manifest = Manifest(self.key, self.repository)
         archive_keys_serialized = [msgpack.packb(name.encode()) for name in ARCHIVE_KEYS]
+        pi = ProgressIndicatorPercent(total=len(self.chunks), msg="Rebuilding manifest %6.2f%%", step=0.01,
+                                      msgid='check.rebuild_manifest')
         for chunk_id, _ in self.chunks.iteritems():
+            pi.show()
             cdata = self.repository.get(chunk_id)
             try:
                 data = self.key.decrypt(chunk_id, cdata)
@@ -1490,6 +1493,7 @@ def valid_archive(obj):
                     logger.warning('Duplicate archive name %s, storing as %s', name, new_name)
                     name = new_name
                 manifest.archives[name] = (chunk_id, archive.time)
+        pi.finish()
         logger.info('Manifest rebuild complete.')
         return manifest
 


Test patch:
Repository: borgbackup/borg

Issue:
Package description page on PyPI is unstyled
## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

Issue

## System information. For client/server mode post info for both machines.

N/A

## Describe the problem you're observing.

The [package description page on PyPI](https://pypi.org/project/borgbackup/) is unstyled (showing raw markup).  Looks like this started with release 1.1.0rc3, as 1.1.0rc2 looks fine.



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -215,10 +215,12 @@ def detect_openssl(prefixes):
 
 with open('README.rst', 'r') as fd:
     long_description = fd.read()
+    # remove header, but have one \n before first headline
+    start = long_description.find('What is BorgBackup?')
+    assert start >= 0
+    long_description = '\n' + long_description[start:]
     # remove badges
     long_description = re.compile(r'^\.\. start-badges.*^\.\. end-badges', re.M | re.S).sub('', long_description)
-    # remove |substitutions|
-    long_description = re.compile(r'\|screencast\|').sub('', long_description)
     # remove unknown directives
     long_description = re.compile(r'^\.\. highlight:: \w+$', re.M).sub('', long_description)
 


Test patch: