Repository: borgbackup/borg

Issue:
non-readable files in source directory massively stall backup ("borg create")
## Have you checked borgbackup docs, FAQ, and open GitHub issues?

yes

## Is this a BUG / ISSUE report or a QUESTION?

Issue / Feature Request / bug?

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).

borg-2.0.0b5

#### Operating system (distribution) and version.

Ubuntu Server 22.04
#### Hardware / network configuration, and filesystems used.

Intel(R) Core(TM) i3-8145U CPU @ 2.10GHz
16 GB RAM
ext4

#### How much data is handled by borg?

Not much, just test data. Maybe 5 MB.

#### Full borg commandline that lead to the problem (leave away excludes and passwords)

`/path/to/borg-2.0.0b5-linuxnew64 create --stats --list --checkpoint-interval=60 --repo /path/to/testrepo1 --verbose archive-{utcnow} /etc`

## Describe the problem you're observing.

I am getting my feet wet with the new syntax and related repositories coming in borg 2.0. For that purpose I created some test repos and backed up my /etc directory (as a dummy source directory - a directory with many text files which do not change while running borg). Some files under this directory are not readable for the user borg runs under. This causes borg to log a warning and retry for a total of about 15 seconds - for each file. Example:

```
/etc/sudoers.d/README: open: [Errno 13] Permission denied: 'README', slept 0.001s, next: retry: 1 of 9...
/etc/sudoers.d/README: open: [Errno 13] Permission denied: 'README', slept 0.003s, next: retry: 2 of 9...
/etc/sudoers.d/README: open: [Errno 13] Permission denied: 'README', slept 0.010s, next: retry: 3 of 9...
/etc/sudoers.d/README: open: [Errno 13] Permission denied: 'README', slept 0.032s, next: retry: 4 of 9...
/etc/sudoers.d/README: open: [Errno 13] Permission denied: 'README', slept 0.100s, next: retry: 5 of 9...
/etc/sudoers.d/README: open: [Errno 13] Permission denied: 'README', slept 0.316s, next: retry: 6 of 9...
/etc/sudoers.d/README: open: [Errno 13] Permission denied: 'README', slept 1.000s, next: retry: 7 of 9...
/etc/sudoers.d/README: open: [Errno 13] Permission denied: 'README', slept 3.162s, next: retry: 8 of 9...
/etc/sudoers.d/README: open: [Errno 13] Permission denied: 'README', slept 10.000s, next: retry: 9 of 9...
/etc/sudoers.d/README: open: [Errno 13] Permission denied: 'README'
E /etc/sudoers.d/README
```
While waiting for the file(s) to become accessible (which never happens, of course) the backup stalls. This causes the backup to run for about 14 minutes for a measly 4.6 MB of source files. For comparison: Running with the correct permissions (i.e. all files are readable by borg) the backup take WAY less than one second.

Suggestion / feature request:

Either make waiting for "Permission denied" errors non-blocking by using asyncio or threads (--> backup can continue while borg waits) and/or provide an option to skip non-accessible files. 

#### Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

yes - does not appear to be a bug but rather "works as intended". But maybe not very smart... ;-)

#### Include any warning/errors/backtraces from the system logs

None



Discussion:
Oh. Yeah, guess we should not retry on "Permission denied", because that won't go away by retrying.

For system backups one would usually run borg as root (or use other ways to give borg read permissions on everything), but good that you tried without. :-)

Resolving patch:
diff --git a/src/borg/archiver/create_cmd.py b/src/borg/archiver/create_cmd.py
--- a/src/borg/archiver/create_cmd.py
+++ b/src/borg/archiver/create_cmd.py
@@ -1,3 +1,4 @@
+import errno
 import sys
 import argparse
 import logging
@@ -370,6 +371,10 @@ def _process_any(self, *, path, parent_fd, name, st, fso, cache, read_special, d
                     self.print_warning("Unknown file type: %s", path)
                     return
             except (BackupError, BackupOSError) as err:
+                if isinstance(err, BackupOSError):
+                    if err.errno in (errno.EPERM, errno.EACCES):
+                        # Do not try again, such errors can not be fixed by retrying.
+                        raise
                 # sleep a bit, so temporary problems might go away...
                 sleep_s = 1000.0 / 1e6 * 10 ** (retry / 2)  # retry 0: 1ms, retry 6: 1s, ...
                 time.sleep(sleep_s)
diff --git a/src/borg/testsuite/archiver/create_cmd.py b/src/borg/testsuite/archiver/create_cmd.py
--- a/src/borg/testsuite/archiver/create_cmd.py
+++ b/src/borg/testsuite/archiver/create_cmd.py
@@ -5,6 +5,7 @@
 import shutil
 import socket
 import stat
+import subprocess
 import time
 import unittest
 
@@ -208,6 +209,7 @@ def test_create_erroneous_file(self):
             input=flist.encode(),
             exit_code=0,
         )
+        assert "retry: 3 of " in out
         assert "E input/file2" not in out  # we managed to read it in the 3rd retry (after 3 failed reads)
         # repo looking good overall? checks for rc == 0.
         self.cmd(f"--repo={self.repository_location}", "check", "--debug")
@@ -217,6 +219,37 @@ def test_create_erroneous_file(self):
         assert "input/file2" in out
         assert "input/file3" in out
 
+    def test_create_no_permission_file(self):
+        file_path = os.path.join(self.input_path, "file")
+        self.create_regular_file(file_path + "1", size=1000)
+        self.create_regular_file(file_path + "2", size=1000)
+        self.create_regular_file(file_path + "3", size=1000)
+        # revoke read permissions on file2 for everybody, including us:
+        if is_win32:
+            subprocess.run(["icacls.exe", file_path + "2", "/deny", "everyone:(R)"])
+        else:
+            os.chmod(file_path + "2", 0o000)
+        self.cmd(f"--repo={self.repository_location}", "rcreate", RK_ENCRYPTION)
+        flist = "".join(f"input/file{n}\n" for n in range(1, 4))
+        out = self.cmd(
+            f"--repo={self.repository_location}",
+            "create",
+            "--paths-from-stdin",
+            "--list",
+            "test",
+            input=flist.encode(),
+            exit_code=1,  # WARNING status: could not back up file2.
+        )
+        assert "retry: 1 of " not in out  # retries were NOT attempted!
+        assert "E input/file2" in out  # no permissions!
+        # repo looking good overall? checks for rc == 0.
+        self.cmd(f"--repo={self.repository_location}", "check", "--debug")
+        # check files in created archive
+        out = self.cmd(f"--repo={self.repository_location}", "list", "test")
+        assert "input/file1" in out
+        assert "input/file2" not in out  # it skipped file2
+        assert "input/file3" in out
+
     def test_create_content_from_command(self):
         self.cmd(f"--repo={self.repository_location}", "rcreate", RK_ENCRYPTION)
         input_data = "some test content"


Test patch: