Repository: borgbackup/borg

Issue:
persistent crash in free space check after ctrl-c-ing borg operation
From the borgbackup mailing list:

I decided to start from scratch for various unrelated reasons.
However, I made a different mistake this time, and the repo won't allow
creations of any more archives as a result. Here are the details:

```
$ borg init --encryption none borgrepo

$ borg create --exclude-if-present '.nobackup' --stats borgrepo::first /home/me

$ borg create --exclude-if-present '.nobackup' --stats borgrepo::second /home/me
```

all worked, then the mistake (the actual user name wasn't 'me', so the
actual typo wasn't 'notme' - it was a single letter omission):

```
$ borg create --exclude-if-present '.nobackup' --stats borgrepo::third /home/notme
/home/notme: [Errno 2] No such file or directory: '/home/notme'
```

---> ^C'ed it here - immediately after the error message, and
retried without the mistake:

```
$ borg create --exclude-if-present '.nobackup' --stats borgrepo::third /home/me
Archive third already exists
```

it made the archive, or part of it, even though it had no source:

```
$ borg list borgrepo
first                                Sat, 2019-10-19 17:43:39
[32e1e882ef4948ba8a59abcfbfb655df71704f88366506ba307b8b8d12bc52a1]
second                               Sat, 2019-10-19 17:45:12
[b0459333ddece1ce236aecf55f666c5ab639c596f088d86c2fed09596b448a07]
third                                Sun, 2019-10-20 08:54:22
[e6277e63fe56ddbff1cd410dfe6826efd2e703a91a306d113e870a4df80c1bf4]
```

but won't let go:

```
$ borg delete borgrepo::third
Local Exception
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 4501,
in main exit_code = archiver.run(args)
  File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 4433,
in run return set_ec(func(args))
  File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 166,
in wrapper return method(self, args, repository=repository, **kwargs)
  File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 1257,
in do_delete return self._delete_archives(args, repository)
  File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 1310,
in _delete_archives repository.commit(save_space=args.save_space)
  File "/usr/lib/python3.7/site-packages/borg/repository.py", line 438,
in commit self.check_free_space()
  File "/usr/lib/python3.7/site-packages/borg/repository.py", line 644,
in check_free_space compact_working_space =
sum(self.io.segment_size(segment) - free for segment, free in
self.compact.items()) File
"/usr/lib/python3.7/site-packages/borg/repository.py", line 644, in
<genexpr> compact_working_space = sum(self.io.segment_size(segment) -
free for segment, free in self.compact.items()) File
"/usr/lib/python3.7/site-packages/borg/repository.py", line 1349, in
segment_size return os.path.getsize(self.segment_filename(segment))
File "/usr/lib/python3.7/genericpath.py", line 50, in getsize return
os.stat(filename).st_size FileNotFoundError: [Errno 2] No such file or
directory: '/<...>/borgrepo/data/0/7'

Platform: Linux <...> 5.3.6-1-MANJARO #1 SMP PREEMPT Sat Oct 12
09:30:05 UTC 2019 x86_64 Linux: arch Manjaro Linux
Borg: 1.1.10  Python: CPython 3.7.4 msgpack: 0.5.6
PID: 1650  CWD: /home/me/Desktop
sys.argv: ['/usr/bin/borg', 'delete',
'/<...>/borgrepo::third'] SSH_ORIGINAL_COMMAND: None
```

```
$ borg info borgrepo::third
Archive name: third
Archive fingerprint:
e6277e63fe56ddbff1cd410dfe6826efd2e703a91a306d113e870a4df80c1bf4
Comment: Hostname: <...>
Username: me
Time (start): Sun, 2019-10-20 08:54:22
Time (end): Sun, 2019-10-20 08:54:22
Duration: 0.07 seconds
Number of files: 0
Command line: /usr/bin/borg create --exclude-if-present .nobackup --stats borgrepo::third /home/notme
Utilization of maximum supported archive size: 0%
------------------------------------------------------------------------------
Original size      Compressed size    Deduplicated size This archive:
                 0 B                  0 B                429 B All
archives:              480.81 MB            260.03 MB            129.74
MB

                       Unique chunks         Total chunks
Chunk index:                    2244                 4823
```


Worst of all, further creates fail:

```
$ borg create --exclude-if-present '.nobackup' --stats borgrepo::fourth /home/me
Local Exception
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 4501,
in main exit_code = archiver.run(args)
  File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 4433,
in run return set_ec(func(args))
  File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 166,
in wrapper return method(self, args, repository=repository, **kwargs)
  File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 573,
in do_create create_inner(archive, cache)
  File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 538,
in create_inner archive.save(comment=args.comment,
timestamp=args.timestamp) File
"/usr/lib/python3.7/site-packages/borg/archive.py", line 515, in save
self.repository.commit() File
"/usr/lib/python3.7/site-packages/borg/repository.py", line 438, in
commit self.check_free_space() File
"/usr/lib/python3.7/site-packages/borg/repository.py", line 644, in
check_free_space compact_working_space =
sum(self.io.segment_size(segment) - free for segment, free in
self.compact.items()) File
"/usr/lib/python3.7/site-packages/borg/repository.py", line 644, in
<genexpr> compact_working_space = sum(self.io.segment_size(segment) -
free for segment, free in self.compact.items()) File
"/usr/lib/python3.7/site-packages/borg/repository.py", line 1349, in
segment_size return os.path.getsize(self.segment_filename(segment))
File "/usr/lib/python3.7/genericpath.py", line 50, in getsize return
os.stat(filename).st_size FileNotFoundError: [Errno 2] No such file or
directory: '/.../borgrepo/data/0/7'

Platform: Linux <...> 5.3.6-1-MANJARO #1 SMP PREEMPT Sat Oct 12
09:30:05 UTC 2019 x86_64 Linux: arch Manjaro Linux
Borg: 1.1.10  Python: CPython 3.7.4 msgpack: 0.5.6
PID: 1669  CWD: /home/me/Desktop
sys.argv: ['/usr/bin/borg', 'create', '--exclude-if-present',
'.nobackup', '--stats', '/<...>/borgrepo::fourth',
'/home/me'] SSH_ORIGINAL_COMMAND: None
```

```
$ borg list borgrepo
first                                Sat, 2019-10-19 17:43:39
[32e1e882ef4948ba8a59abcfbfb655df71704f88366506ba307b8b8d12bc52a1]
second                               Sat, 2019-10-19 17:45:12
[b0459333ddece1ce236aecf55f666c5ab639c596f088d86c2fed09596b448a07]
third                                Sun, 2019-10-20 08:54:22
[e6277e63fe56ddbff1cd410dfe6826efd2e703a91a306d113e870a4df80c1bf4]
```


```
$ borg check --verify-data borgrepo
$ echo $?
0
```

Sure, once I do adequate testing and have this all scripted, there won't be anymore such pilot errors.  Still...



Discussion:
When using Borg 1.1.10 on WSL (windows system linux; running ubuntu) and when interrupting operations (generally a 'create') I would (still do) often get errors on subsequent commands.  These would be long/intricate and with big tracebacks, etc. (I read through them once or twice but nothing stood out that I understood. No, I don't have a specific example to paste here, but maybe I can dig one up shortly.)

Anyway...after a bit of thought and troubleshooting I came to realize all i had to do was delete/clear-away the cache for the repository and everything would work again.

"borg  delete  --cache-only  myRepository"

I guess I could/should have filed a report, but everything otherwise seemed to work fine. I just had to remember that interrupting an operation I needed to start over without cache (and if there was a partial archive hanging free in the repository I would just delete that as well).

It isn't a perfect solution and I'm not sure it applies to what you have above, but it is how I have avoided any hanging mess from an interrupt.
By the way (side note), whenever I attempt that command, I am SCARED TO DEATH that I forget or MISTYPE the command flag.  I've been terrified that one day I might type:

"borg   delete  --cach-only  myRepository"

Or something else that will not match the cache flag, but somehow be accepted (or ignored) and my entire repository will delete.  (No, I'm not sure if I'm given a "are you sure y/n" prompt on a delete).   Anyway, just a side comment and thus asking whether that delete command is smart/safe enough not to blow away a repository when a person is simply trying to gently clear a cache.    ???

And an additional note to my first post above, you have referenced that you will do/fix something in 1.1.11.  Will this be a fix like "if a command is interrupted, THEN clear cache"? in order to help in these situations?
I spent 15 minutes or so creating repositories and interrupting them and such and am unable (currently) to induce the same behavior.  But the use of control-c on multiple occasions did then prevent subsequent commands from working in my use over a few days.

Can you (or whoever has the problem you posted) clear cache and see if it fixed things?

(I will now keep a lookout on my side and when ctrl-c causes subsequent errors again make a point to copy info here.  Maybe it can help.)

Resolving patch:
diff --git a/src/borg/repository.py b/src/borg/repository.py
--- a/src/borg/repository.py
+++ b/src/borg/repository.py
@@ -669,7 +669,13 @@ def check_free_space(self):
             if len(self.compact) < 10:
                 # This is mostly for the test suite to avoid overestimated free space needs. This can be annoying
                 # if TMP is a small-ish tmpfs.
-                compact_working_space = sum(self.io.segment_size(segment) - free for segment, free in self.compact.items())
+                compact_working_space = 0
+                for segment, free in self.compact.items():
+                    try:
+                        compact_working_space += self.io.segment_size(segment) - free
+                    except FileNotFoundError:
+                        # looks like self.compact is referring to a non-existent segment file, ignore it.
+                        pass
                 logger.debug('check_free_space: few segments, not requiring a full free segment')
                 compact_working_space = min(compact_working_space, full_segment_size)
                 logger.debug('check_free_space: calculated working space for compact as %d bytes', compact_working_space)


Test patch: