Repository: borgbackup/borg

Issue:
inaccessible hard links are stored
similar to: https://github.com/jborg/attic/issues/393
```
$ touch hl1
$ chmod 000 hl1
$ ln hl1 hl2
$ borg create repo::hl-check .
hl1: [Errno 13] Permission denied: 'hl1'
$ mkdir extract
$ cd extract
$ borg extract ../repo::hl-check
hl2: [Errno 2] No such file or directory: '/home/tw/w/borg/extract/hl1' -> '/home/tw/w/borg/extract/hl2'
$ ls -l hl*
ls: cannot access 'hl*': No such file or directory
$ borg list ../repo::hl-check | grep hl
h--------- tw     tw            0 Sat, 2017-01-21 20:08:04 hl2 link to hl1
```
Race between storing hardlink master and later hardlink references
https://github.com/borgbackup/borg/blob/1.0.9/borg/archive.py#L664
- inodes are recycled, so an (dev, inode) pair doesn't really identify a file during a backup run
- If there's an error later on, then `self.hard_links[st.st_ino, st.st_dev] = safe_path` would have created a reference to an item that didn't make it into the archive
- I'm not sure, it may be possible to sneak in a reference to file contents with a stat result that doesn't belong to them
  1. have file with inode x, contents y and permission z
  2. have a hardlink there
  3. get rid of the inode (to delete an inode one may not have the necessary permissions to read the contents)
  4. create new file with permissions you like, lodge it so that they have the same inode, create another dirent for the file so that you get nlink > 1
  5. assuming you manage the proper timing:
     - the file (x, y, z) will end up in the archive as usual
     - the new file (x, y, z') has the same contents (due to being recognized as a hard link), the ones of file (x, y, z), but permissions you can control z'
     - This is kinda low-impact though. Not only requires that a very peculiar permissions layout, but also precise timing and the ability to control/predict inodes (which may be not hard or may be very hard)



Discussion:
Also:
```
(borg-env)tw@tux:~/w/borg/extract$ borg mount ../repo::hl-check mnt
Enter passphrase for key /home/tw/w/borg/repo: 
Local Exception.
Traceback (most recent call last):
  File "/home/tw/w/borg/borg/archiver.py", line 2076, in main
    exit_code = archiver.run(args)
  File "/home/tw/w/borg/borg/archiver.py", line 2012, in run
    return func(args)
  File "/home/tw/w/borg/borg/archiver.py", line 91, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/home/tw/w/borg/borg/archiver.py", line 560, in do_mount
    operations = FuseOperations(key, repository, manifest, archive, cached_repo)
  File "/home/tw/w/borg/borg/fuse.py", line 71, in __init__
    self.process_archive(archive)
  File "/home/tw/w/borg/borg/fuse.py", line 139, in process_archive
    inode = self._find_inode(item[b'source'], prefix)
  File "/home/tw/w/borg/borg/fuse.py", line 182, in _find_inode
    inode = self.contents[inode][segment]
KeyError: b'hl1'

Platform: Linux tux 4.9.4-tw2 #1 SMP Wed Jan 18 07:03:50 CET 2017 x86_64 x86_64
Linux: Ubuntu 16.04 xenial
Borg: 1.0.10.dev80+ngca0c1da  Python: CPython 3.5.2
PID: 22263  CWD: /home/tw/w/borg/extract
sys.argv: ['/home/tw/w/borg-env/bin/borg', 'mount', '../repo::hl-check', 'mnt']
SSH_ORIGINAL_COMMAND: None
```
- Correct `create` code
- Add proper error handling to `mount`
- `extract` fails on the item and so does approximately the right thing by extension.
If we skip hl1, we should also skip hl2. Hardlinks are symmetric. No problem with extract and mount then.
Since there is still #1297 it has to be handled in mount. This is also just a variation on #1297, just that the master backup does not fail due to a lost race, but due to lack of permissions.
Close wontfix?, likely impossible to fix without losing that functionality.

@textshell:

> Well, i guess it doesn‘t matter if the hardlink is not identified. That would be possible with comparing a checksum or FIEMAP (not quite as save) But then you get false negatives.
- One purpose of this feature is to save time (so re-reading isn't an option, if we did that, we can just remove it)
- Another purpose is to preserve hard links. Unless things like the FIEMAP ioctl where used an assumed as an identity that is also not possible without races, but using FIEMAP leads to races and un-preserved hardlinks w.r.t. changes of the file.

A possible fix, if this does turn out to be a relevant issue, would be to use the item metadata (esp. permission bits, ACL) from the hardlink master, either in create code or in all other code.

Yet another option would be to also take the inode generation number into account (but only for the hard_links mapping, nowhere else). On Linux the number may be retrieved for some filesystems via the ioctl(2) FS_IOC_GETVERSION (thanks @verygreen).

This seems to be implemented for btrfs, ext2/4, XFS, f2fs, ecryptfs (lxr).

i think a quick fix would be to just move the assignment down, to *after* the file has been processed - after add_item(), before the final return.
Yup, sounds good. You do it?

Resolving patch:
diff --git a/borg/archive.py b/borg/archive.py
--- a/borg/archive.py
+++ b/borg/archive.py
@@ -660,8 +660,6 @@ def process_file(self, path, st, cache, ignore_inode=False):
                 self.add_item(item)
                 status = 'h'  # regular file, hardlink (to already seen inodes)
                 return status
-            else:
-                self.hard_links[st.st_ino, st.st_dev] = safe_path
         is_special_file = is_special(st.st_mode)
         if not is_special_file:
             path_hash = self.key.id_hash(os.path.join(self.cwd, path).encode('utf-8', 'surrogateescape'))
@@ -709,6 +707,9 @@ def process_file(self, path, st, cache, ignore_inode=False):
             item[b'mode'] = stat.S_IFREG | stat.S_IMODE(item[b'mode'])
         self.stats.nfiles += 1
         self.add_item(item)
+        if st.st_nlink > 1 and source is None:
+            # Add the hard link reference *after* the file has been added to the archive.
+            self.hard_links[st.st_ino, st.st_dev] = safe_path
         return status
 
     @staticmethod
diff --git a/borg/fuse.py b/borg/fuse.py
--- a/borg/fuse.py
+++ b/borg/fuse.py
@@ -128,15 +128,21 @@ def process_archive(self, archive, prefix=[]):
                 else:
                     self.items[inode] = item
                     continue
-                segments = prefix + os.fsencode(os.path.normpath(item[b'path'])).split(b'/')
-                del item[b'path']
+                path = item.pop(b'path')
+                segments = prefix + os.fsencode(os.path.normpath(path)).split(b'/')
                 num_segments = len(segments)
                 parent = 1
                 for i, segment in enumerate(segments, 1):
                     # Leaf segment?
                     if i == num_segments:
                         if b'source' in item and stat.S_ISREG(item[b'mode']):
-                            inode = self._find_inode(item[b'source'], prefix)
+                            try:
+                                inode = self._find_inode(item[b'source'], prefix)
+                            except KeyError:
+                                file = path.decode(errors='surrogateescape')
+                                source = item[b'source'].decode(errors='surrogateescape')
+                                logger.warning('Skipping broken hard link: %s -> %s', file, source)
+                                continue
                             item = self.cache.get(inode)
                             item[b'nlink'] = item.get(b'nlink', 1) + 1
                             self.items[inode] = item


Test patch: