Repository: rdiff-backup/rdiff-backup (language: python)

Issue:
[ENH] add feature to set access rights in testing/fileset.py
## Is your feature request related to a problem? Please describe.

In order to test the issue described at #738 I'd need a file set with only readable files, but the feature doesn't exist yet.

## Describe the solution you'd like

Add a "mode/chmod" parameter to the fileset description

## Describe alternatives you've considered

Don't write such tests (not an option on the long run, especially considering #76 we need more features for fileset

## Additional context

Need to consider making it work under Linux _and_ Windows.

[BUG] Removing Old Increments Fail with Permission denied for 2.1.2b1
## Bug summary

Executing the beta version of rdiff-backup to remove old increments fail if the related directory is write protected.

This worked in the previous version.

## Version, Python, Operating System

Call `rdiff-backup info` (version >= 2.1) or `rdiff-backup -v9` and replace the following line with the output, repeat for each environment impacted:

```yaml
exec:
  api_version:
    actual: 0
    default: 200
    max: 201
    min: 200
  argv:
  - /usr/bin/rdiff-backup
  - info
  parsed: null
  version: 2.1.2b1
python:
  executable: /usr/bin/python3
  name: cpython
  version: 3.10.6
system:
  fs_encoding: utf-8
  platform: Linux-5.19.7-200.fc36.x86_64-x86_64-with-glibc2.35
```

## rdiff-backup call

How did you call rdiff-backup, with which parameters:

```
su -m safekeep -c "rdiff-backup --force remove increments --older-than 9D /srv/archive/safekeep-client/backup"
```

## What happened and what did you expect?

The run failed, with the following traceback

```
Traceback (most recent call last):
  File "/usr/bin/rdiff-backup", line 32, in <module>
    sys.exit(rdiffbackup.run.main_run(sys.argv[1:]))
  File "/usr/lib64/python3.10/site-packages/rdiffbackup/run.py", line 86, in main_run
    ret_val |= conn_act.setup()
  File "/usr/lib64/python3.10/site-packages/rdiffbackup/actions/remove.py", line 93, in setup
    self.repo.base_dir.conn.fs_abilities.single_set_globals(
  File "/usr/lib64/python3.10/site-packages/rdiff_backup/fs_abilities.py", line 1130, in single_set_globals
    fsa = FSAbilities(rp.path, rp)
  File "/usr/lib64/python3.10/site-packages/rdiff_backup/fs_abilities.py", line 72, in __init__
    self._init_readwrite()
  File "/usr/lib64/python3.10/site-packages/rdiff_backup/fs_abilities.py", line 160, in _init_readwrite
    subdir.mkdir()
  File "/usr/lib64/python3.10/site-packages/rdiff_backup/rpath.py", line 793, in mkdir
    self.conn.os.mkdir(self.path)
PermissionError: [Errno 13] Permission denied: b'/srv/archive/safekeep-client/backup/rdiff-backup.tmp.0'
```

## More information

Permissions on the top level directory (which is copied from the root of the system being backed up) does not allow writing.  See the directory listing:

```
dr-xr-xr-x.  11 safekeep safekeep  4096 Sep 16 22:10 .
drwxr-xr-x.   3 root     root      4096 Sep 21  2019 ..
dr-xr-xr-x.   2 safekeep safekeep  4096 Jan 20  2022 afs
lrwxrwxrwx.   1 safekeep safekeep     7 May 23 21:53 bin -> usr/bin
dr-xr-xr-x.   6 safekeep safekeep  4096 May 25 06:56 boot
drwxr-xr-x. 120 safekeep safekeep 12288 Jun 26 15:21 etc
drwxr-xr-x.   4 safekeep safekeep  4096 Jan 20  2022 home
lrwxrwxrwx.   1 safekeep safekeep     7 May 23 21:53 lib -> usr/lib
lrwxrwxrwx.   1 safekeep safekeep     9 May 23 21:53 lib64 -> usr/lib64
drwxr-xr-x.   2 safekeep safekeep  4096 Jan 20  2022 opt
drwx------.   4 safekeep safekeep 12288 Sep 16 03:34 rdiff-backup-data
dr-xr-x---.   3 safekeep safekeep  4096 Jun 25 21:14 root
lrwxrwxrwx.   1 safekeep safekeep     8 May 23 21:53 sbin -> usr/sbin
drwxr-xr-x.  12 safekeep safekeep  4096 May 23 21:04 usr
drwxr-xr-x.  19 safekeep safekeep  4096 May 23 21:17 var
```

Also, if you allow writing on the top level it appears to fail futher, when it finds other write-protected directories.



Discussion / hints:

Fix patch:
diff --git a/tox.ini b/tox.ini
index 271a6c58b..45305f1bb 100644
--- a/tox.ini
+++ b/tox.ini
@@ -42,6 +42,7 @@ commands =
 	coverage run testing/action_test_test.py --verbose --buffer
 	coverage run testing/action_verify_test.py --verbose --buffer
 	sh ./testing/verbosity_actions_test.sh 9 --verbose --buffer
+	coverage run testing/readonly_actions_test.py --verbose --buffer
 	coverage run testing/api_test.py --verbose --buffer
 	coverage run testing/location_map_filenames_test.py --verbose --buffer
 	coverage run testing/location_map_hardlinks_test.py --verbose --buffer
diff --git a/tox_win.ini b/tox_win.ini
index 37f4f4118..f05c6a98e 100644
--- a/tox_win.ini
+++ b/tox_win.ini
@@ -31,6 +31,7 @@ commands =
     python testing/action_remove_test.py --verbose --buffer
     python testing/action_test_test.py --verbose --buffer
     python testing/action_verify_test.py --verbose --buffer
+    python testing/readonly_actions_test.py --verbose --buffer
     python testing/api_test.py --verbose --buffer
     python testing/location_map_filenames_test.py --verbose --buffer
 #    python testing/location_map_hardlinks_test.py --verbose --buffer


Test patch:
diff --git a/testing/action_backuprestore_test.py b/testing/action_backuprestore_test.py
index f776defb3..0486e62d3 100644
--- a/testing/action_backuprestore_test.py
+++ b/testing/action_backuprestore_test.py
@@ -20,37 +20,37 @@ def setUp(self):
         # Windows can't handle too long filenames
         long_multi = 10 if os.name == "nt" else 25
         self.from1_struct = {
-            "from1": {"subs": {
-                "fileA": {"content": "initial"},
+            "from1": {"contents": {
+                "fileA": {"content": "initial", "inode": "fileA"},
                 "fileB": {},
                 "dirOld": {"type": "dir"},
                 "itemX": {"type": "dir"},
                 "itemY": {"type": "file"},
                 "longdirnam" * long_multi: {"type": "dir"},
                 "longfilnam" * long_multi: {"content": "not so long content"},
-                # "somehardlink": {"link": "fileA"},
+                # "somehardlink": {"inode": "fileA"},
             }}
         }
         self.from1_path = os.path.join(self.base_dir, b"from1")
         self.from2_struct = {
-            "from2": {"subs": {
-                "fileA": {"content": "modified"},
+            "from2": {"contents": {
+                "fileA": {"content": "modified", "inode": "fileA"},
                 "fileC": {},
                 "dirNew": {"type": "dir"},
                 "itemX": {"type": "file"},
                 "itemY": {"type": "dir"},
                 "longdirnam" * long_multi: {"type": "dir"},
                 "longfilnam" * long_multi: {"content": "differently long"},
-                # "somehardlink": {"link": "fileA"},
+                # "somehardlink": {"inode": "fileA"},
             }}
         }
         self.from2_path = os.path.join(self.base_dir, b"from2")
         if os.name != "nt":
             # rdiff-backup can't handle (yet) hardlinks under Windows
-            self.from1_struct["from1"]["subs"]["somehardlink"] = {
-                "link": "fileA"}
-            self.from2_struct["from2"]["subs"]["somehardlink"] = {
-                "link": "fileA"}
+            self.from1_struct["from1"]["contents"]["somehardlink"] = {
+                "inode": "fileA"}
+            self.from2_struct["from2"]["contents"]["somehardlink"] = {
+                "inode": "fileA"}
         fileset.create_fileset(self.base_dir, self.from1_struct)
         fileset.create_fileset(self.base_dir, self.from2_struct)
         fileset.remove_fileset(self.base_dir, {"bak": {"type": "dir"}})
@@ -218,13 +218,13 @@ def setUp(self):
         self.base_dir = os.path.join(comtst.abs_test_dir,
                                      b"action_backuprestore")
         self.from1_struct = {
-            "from1": {"subs": {  # CODE.txt pre-quoted
+            "from1": {"contents": {  # CODE.txt pre-quoted
                 ";067;079;068;069.txt": {"content": "initial"},
             }}
         }
         self.from1_path = os.path.join(self.base_dir, b"from1")
         self.from2_struct = {
-            "from2": {"subs": {  # CODE.txt pre-quoted and non-quoted
+            "from2": {"contents": {  # CODE.txt pre-quoted and non-quoted
                 ";067;079;068;069.txt": {"content": "modified"},
                 "CODE.txt": {"content": "whatever"},
             }}
diff --git a/testing/action_calculate_test.py b/testing/action_calculate_test.py
index 32a046124..0bad41def 100644
--- a/testing/action_calculate_test.py
+++ b/testing/action_calculate_test.py
@@ -22,37 +22,37 @@ def setUp(self):
         # Windows can't handle too long filenames
         long_multi = 10 if os.name == "nt" else 25
         self.from1_struct = {
-            "from1": {"subs": {
-                "fileA": {"content": "initial"},
+            "from1": {"contents": {
+                "fileA": {"content": "initial", "inode": "fileA"},
                 "fileB": {},
                 "dirOld": {"type": "dir"},
                 "itemX": {"type": "dir"},
                 "itemY": {"type": "file"},
                 "longdirnam" * long_multi: {"type": "dir"},
                 "longfilnam" * long_multi: {"content": "not so long content"},
-                # "somehardlink": {"link": "fileA"},
+                # "somehardlink": {"inode": "fileA"},
             }}
         }
         self.from1_path = os.path.join(self.base_dir, b"from1")
         self.from2_struct = {
-            "from2": {"subs": {
-                "fileA": {"content": "modified"},
+            "from2": {"contents": {
+                "fileA": {"content": "modified", "inode": "fileA"},
                 "fileC": {},
                 "dirNew": {"type": "dir"},
                 "itemX": {"type": "file"},
                 "itemY": {"type": "dir"},
                 "longdirnam" * long_multi: {"type": "dir"},
                 "longfilnam" * long_multi: {"content": "differently long"},
-                # "somehardlink": {"link": "fileA"},
+                # "somehardlink": {"inode": "fileA"},
             }}
         }
         self.from2_path = os.path.join(self.base_dir, b"from2")
         if os.name != "nt":
             # rdiff-backup can't handle (yet) hardlinks under Windows
-            self.from1_struct["from1"]["subs"]["somehardlink"] = {
-                "link": "fileA"}
-            self.from2_struct["from2"]["subs"]["somehardlink"] = {
-                "link": "fileA"}
+            self.from1_struct["from1"]["contents"]["somehardlink"] = {
+                "inode": "fileA"}
+            self.from2_struct["from2"]["contents"]["somehardlink"] = {
+                "inode": "fileA"}
         fileset.create_fileset(self.base_dir, self.from1_struct)
         fileset.create_fileset(self.base_dir, self.from2_struct)
         fileset.remove_fileset(self.base_dir, {"bak": {"type": "dir"}})
diff --git a/testing/action_compare_test.py b/testing/action_compare_test.py
index cb4813409..5b93c3ba0 100644
--- a/testing/action_compare_test.py
+++ b/testing/action_compare_test.py
@@ -18,7 +18,7 @@ class ActionCompareTest(unittest.TestCase):
     def setUp(self):
         self.base_dir = os.path.join(comtst.abs_test_dir, b"action_compare")
         self.from1_struct = {
-            "from1": {"subs": {
+            "from1": {"contents": {
                 "fileChanged": {"content": "initial"},
                 "fileOld": {},
                 "fileUnchanged": {"content": "unchanged"},
@@ -26,7 +26,7 @@ def setUp(self):
         }
         self.from1_path = os.path.join(self.base_dir, b"from1")
         self.from2_struct = {
-            "from2": {"subs": {
+            "from2": {"contents": {
                 "fileChanged": {"content": "modified"},
                 "fileNew": {},
                 "fileUnchanged": {"content": "unchanged"},
@@ -34,7 +34,7 @@ def setUp(self):
         }
         self.from2_path = os.path.join(self.base_dir, b"from2")
         self.from3_struct = {
-            "from3": {"subs": {
+            "from3": {"contents": {
                 "fileChanged": {"content": "samesize"},
                 "fileNew": {},
                 "fileUnchanged": {"content": "unchanged"},
diff --git a/testing/action_list_test.py b/testing/action_list_test.py
index 930c541c9..1af30ae0f 100644
--- a/testing/action_list_test.py
+++ b/testing/action_list_test.py
@@ -16,7 +16,7 @@ class ActionListTest(unittest.TestCase):
     def setUp(self):
         self.base_dir = os.path.join(comtst.abs_test_dir, b"action_list")
         self.from1_struct = {
-            "from1": {"subs": {
+            "from1": {"contents": {
                 "fileChanged": {"content": "initial"},
                 "fileOld": {},
                 "fileUnchanged": {"content": "unchanged"},
@@ -24,7 +24,7 @@ def setUp(self):
         }
         self.from1_path = os.path.join(self.base_dir, b"from1")
         self.from2_struct = {
-            "from2": {"subs": {
+            "from2": {"contents": {
                 "fileChanged": {"content": "modified"},
                 "fileNew": {},
                 "fileUnchanged": {"content": "unchanged"},
diff --git a/testing/action_regress_test.py b/testing/action_regress_test.py
index 9b0cd4d53..c65aa507c 100644
--- a/testing/action_regress_test.py
+++ b/testing/action_regress_test.py
@@ -19,7 +19,7 @@ class ActionRegressTest(unittest.TestCase):
     def setUp(self):
         self.base_dir = os.path.join(comtst.abs_test_dir, b"action_regress")
         self.from1_struct = {
-            "from1": {"subs": {
+            "from1": {"contents": {
                 "fileChanged": {"content": "initial"},
                 "fileOld": {},
                 "fileUnchanged": {"content": "unchanged"},
@@ -29,7 +29,7 @@ def setUp(self):
         }
         self.from1_path = os.path.join(self.base_dir, b"from1")
         self.from2_struct = {
-            "from2": {"subs": {
+            "from2": {"contents": {
                 "fileChanged": {"content": "modified"},
                 "fileNew": {},
                 "fileUnchanged": {"content": "unchanged"},
@@ -39,7 +39,7 @@ def setUp(self):
         }
         self.from2_path = os.path.join(self.base_dir, b"from2")
         self.from3_struct = {
-            "from3": {"subs": {
+            "from3": {"contents": {
                 "fileChanged": {"content": "modified again"},
                 "fileNew": {},
                 "fileUnchanged": {"content": "unchanged"},
@@ -49,7 +49,7 @@ def setUp(self):
         }
         self.from3_path = os.path.join(self.base_dir, b"from3")
         self.from4_struct = {
-            "from4": {"subs": {
+            "from4": {"contents": {
                 "fileChanged": {"content": "modified again"},
                 "fileEvenNewer": {},
                 "fileUnchanged": {"content": "unchanged"},
diff --git a/testing/action_remove_test.py b/testing/action_remove_test.py
index c656299a8..bdf068f5a 100644
--- a/testing/action_remove_test.py
+++ b/testing/action_remove_test.py
@@ -17,7 +17,7 @@ class ActionRemoveTest(unittest.TestCase):
     def setUp(self):
         self.base_dir = os.path.join(comtst.abs_test_dir, b"action_remove")
         self.from1_struct = {
-            "from1": {"subs": {
+            "from1": {"contents": {
                 "fileChanged": {"content": "initial"},
                 "fileOld": {},
                 "fileUnchanged": {"content": "unchanged"},
@@ -25,7 +25,7 @@ def setUp(self):
         }
         self.from1_path = os.path.join(self.base_dir, b"from1")
         self.from2_struct = {
-            "from2": {"subs": {
+            "from2": {"contents": {
                 "fileChanged": {"content": "modified"},
                 "fileNew": {},
                 "fileUnchanged": {"content": "unchanged"},
@@ -33,7 +33,7 @@ def setUp(self):
         }
         self.from2_path = os.path.join(self.base_dir, b"from2")
         self.from3_struct = {
-            "from3": {"subs": {
+            "from3": {"contents": {
                 "fileChanged": {"content": "modified again"},
                 "fileNew": {},
                 "fileUnchanged": {"content": "unchanged"},
@@ -41,7 +41,7 @@ def setUp(self):
         }
         self.from3_path = os.path.join(self.base_dir, b"from3")
         self.from4_struct = {
-            "from4": {"subs": {
+            "from4": {"contents": {
                 "fileChanged": {"content": "modified again"},
                 "fileEvenNewer": {},
                 "fileUnchanged": {"content": "unchanged"},
diff --git a/testing/action_verify_test.py b/testing/action_verify_test.py
index d1a576b57..8f6cff398 100644
--- a/testing/action_verify_test.py
+++ b/testing/action_verify_test.py
@@ -16,7 +16,7 @@ class ActionVerifyTest(unittest.TestCase):
     def setUp(self):
         self.base_dir = os.path.join(comtst.abs_test_dir, b"action_verify")
         self.from1_struct = {
-            "from1": {"subs": {
+            "from1": {"contents": {
                 "fileChanged": {"content": "initial"},
                 "fileOld": {},
                 "fileUnchanged": {"content": "unchanged"},
@@ -24,7 +24,7 @@ def setUp(self):
         }
         self.from1_path = os.path.join(self.base_dir, b"from1")
         self.from2_struct = {
-            "from2": {"subs": {
+            "from2": {"contents": {
                 "fileChanged": {"content": "modified"},
                 "fileNew": {},
                 "fileUnchanged": {"content": "unchanged"},
diff --git a/testing/fileset.py b/testing/fileset.py
index b4315422b..e552bb5f1 100644
--- a/testing/fileset.py
+++ b/testing/fileset.py
@@ -16,45 +16,53 @@
 The name of the files and directories in the base directory are used as keys
 in a dictionary of dictionaries.
 
-A directory has the type "dir" or a "subs" key to contain sub-directories or
-files.
+A directory has the type "dir/directory" or a "contents" key to contain sub-directories or files.
+A directory can also have the "rec" key, which is a dictionary of keys to apply by default to itself and all its children.
 
 Both directories and files can have the following keys:
-* a "mode" to define the chmod to apply
+* a "mode" to define the chmod to apply, differentiated as "dmode" resp. "fmode"
 
 Files can also have the following keys:
 * "content" which is written to the file.
 * "open" flag, "t" or "b" for the write mode to the file
+* "inode" to hardlink files together, the value is not important, it must just be unique within the fileset (it might not even be an integer).
+
+Then there are symlinks with type "link", and the "target" parameter, applied as-is.
 
 Example:
 {
-    "a_dir": {"subs": {"fileA": {"content": "initial"}, "fileB": {}}},
-    "empty_dir": {"type": "dir", "mode": 0o777},
+    "a_dir": {
+        "rec": {"fmode": 0o444, "content": "default content"},
+        "contents": {"fileA": {"content": "initial"}, "fileB": {}}
+    },
+    "empty_dir": {"type": "dir", "dmode": 0o777},
     "a_bin_file": {"content": b"some_binary_content", "open": "b"},
 }
+
+NOTE: the format is meant to be as similar as possible to `tree -J`, so that its output could be eventually used to re-create a file structure.
+
+TODO: fully support tree format with a list of dictionaries, instead of a dictionary of dictionaries, where the key is the filename.
 """
 
 import os
 import shutil
 import stat
 
-DEFAULT_DIR_MODE = 0o755
-DEFAULT_FILE_MODE = 0o644
-DEFAULT_FILE_OPEN = "t"  # or "b"
-DEFAULT_FILE_CONTENT = ""
 
-
-def create_fileset(base_dir, structure):
+def create_fileset(base_dir, structure, recurse={}):
     """
     Create the file set as represented by the structure in the base directory
 
     base_dir can be path-like, str or bytes,
-    structure contains names as str
+    structure contains names as str pointing to further structures
+    recurse can be used to set settings in the whole hierarchy
     """
-    _create_directory(base_dir)
+    # we create a new inodes dictionary for each fileset
+    recurse["inodes"] = {}
+    _create_directory(SetPath(base_dir, {"type": "directory"}, recurse))
     for name in structure:
         _create_fileset(os.path.join(os.fsdecode(base_dir), name),
-                        structure[name])
+                        structure[name], recurse)
 
 
 def remove_fileset(base_dir, structure):
@@ -66,15 +74,16 @@ def remove_fileset(base_dir, structure):
     for name in structure:
         fullname = os.path.join(os.fsdecode(base_dir), name)
         struct = structure[name]
+        set_path = SetPath(fullname, struct)
         try:
-            if struct.get("type") == "dir" or "subs" in struct:
-                shutil.rmtree(fullname)
+            if set_path.get_type() == "directory":
+                _rmtree(fullname)
             else:
                 os.remove(fullname)
         except FileNotFoundError:
             pass  # if the file doesn't exist, we don't need to remove it
         except IsADirectoryError:
-            shutil.rmtree(fullname)
+            _rmtree(fullname)
         except NotADirectoryError:
             os.remove(fullname)
 
@@ -135,68 +144,191 @@ def compare_paths(path1, path2):
     return differences
 
 
-def _create_fileset(fullname, struct):
+class SetPath():
+    """
+    Holds a path's own settings and the recursive ones, so that they can be
+    transparently combined
+    """
+    defaults = {
+        "dmode": 0o755,
+        "fmode": 0o644,
+        "open": "t",
+        "content": "",
+    }
+    type_synonyms = {
+        "dir": "directory",
+        "symlink": "link",
+        "hardlink": "file",
+    }
+    path = ""
+    path_type = ""
+    values = {}
+    recurse = {}
+
+    @classmethod
+    def get_canonic_type(cls, path_type):
+        return cls.type_synonyms.get(path_type, path_type)
+
+    def __init__(self, path, values={}, recurse={}, new_rec={}):
+        """
+        Initiate the settings based on own values, recursive ones, and
+        additional new ones, to be combined into one recursive set of settings
+        """
+        self.path = path
+        path_type = values.get("type")
+        if path_type:
+            self.path_type = self.get_canonic_type(path_type)
+        elif "contents" in values:
+            self.path_type = "directory"
+        elif "target" in values:
+            self.path_type = "link"
+        else:
+            self.path_type = "file"
+        self.values = values
+        self.recurse = recurse.copy()
+        self.recurse.update(new_rec)
+        # the copy of recurse is shallow, so that the "inodes" dictionary is
+        # always the same throughout one fileset
+        if "inode" in values:
+            inode = values["inode"]
+            if inode in self.recurse["inodes"]:
+                self.values["target"] = self.recurse["inodes"][inode]
+            else:
+                self.recurse["inodes"][inode] = self
+
+    def __fspath__(self):
+        return self.path
+
+    def __repr__(self):
+        return str([self.path, self.path_type, self.values, self.recurse])
+
+    def get_type(self):
+        return self.path_type
+
+    def get_mode(self):
+        """
+        Get the file access rights according to file type and current settings
+        """
+        assert self.path_type in ["file", "directory"], (
+            "Type {pt} can't get a mode".format(pt=self.path_type))
+        generic = "mode"
+        if self.path_type == "file":
+            specific = "fmode"
+        elif self.path_type == "directory":
+            specific = "dmode"
+        default = self.defaults.get(specific)
+
+        mode = self.values.get(
+            specific, self.values.get(
+                generic, self.recurse.get(
+                    specific, self.recurse.get(
+                        generic, default))))
+
+        if isinstance(mode, int):
+            return mode
+        else:
+            return int(mode, base=8)
+
+    def get(self, param):
+        """
+        Get the value of the given parameters across own values, recursive ones
+        and potential default value.
+        Returns None as last resort.
+        """
+        default = self.defaults.get(param)
+        return self.values.get(param, self.recurse.get(param, default))
+
+    def is_hardlinked(self):
+        return self.get_type() != "link" and "target" in self.values
+
+
+# --- INTERNAL FUNCTIONS ---
+
+
+def _create_fileset(fullname, struct, recurse={}):
     """
     Recursive part of the fileset creation
     """
-    if struct.get("type") == "dir" or "subs" in struct:
-        _create_directory(fullname, settings=struct, always_delete=True)
-        for name in struct.get("subs", {}):
-            _create_fileset(os.path.join(fullname, name), struct["subs"][name])
-    elif (struct.get("type") == "hardlink"
-            or ("link" in struct and "type" not in struct)):
-        _create_link(fullname, settings=struct)
-    elif struct.get("type") == "symlink":  # a link is hard by default
-        _create_link(fullname, settings=struct, linker=os.symlink)
+    set_path = SetPath(fullname, struct, recurse, struct.get("rec", {}))
+    if set_path.get_type() == "directory":
+        _create_directory(set_path, always_delete=True)
+        for name in struct.get("contents", {}):
+            _create_fileset(os.path.join(fullname, name),
+                            struct["contents"][name], set_path.recurse)
+        _finish_directory(set_path)
     else:
-        _create_file(fullname, settings=struct)
-    # other types of items are ignored for now
+        if set_path.is_hardlinked():
+            _create_hardlink(set_path)
+        elif set_path.get_type() == "link":  # symlink
+            _create_symlink(set_path)
+        else:  # this must be a file
+            _create_file(set_path)
+        # other types of items are ignored for now
 
 
-def _create_directory(dir_name, settings={}, always_delete=False):
+def _create_directory(set_path, always_delete=False):
     """
     Create a directory according to settings.
 
-    The directory is created according to "mode". It is first destroyed if
-    requested. It is currently the recommended approach to make sure the
+    It is first destroyed if requested.
+    It is currently the recommended approach to make sure the
     structure is exactly as expected (a delta mechanism could be added).
     """
-    if os.path.exists(dir_name):
-        if always_delete or not os.path.isdir(dir_name):
-            shutil.rmtree(dir_name)
+    if os.path.exists(set_path):
+        if always_delete or not os.path.isdir(set_path):
+            _rmtree(set_path)
         else:
             return
-    os.makedirs(dir_name, mode=settings.get("mode", DEFAULT_DIR_MODE))
+    os.makedirs(set_path)
 
 
-def _create_file(file_name, settings, always_delete=False):
+def _finish_directory(set_path):
     """
-    Creates a file according to settings
+    The directory is chmod according to "mode", _after_ the contained elements
+    have been created.
+    """
+    os.chmod(set_path, set_path.get_mode())
+
+
+def _create_file(set_path, always_delete=False):
+    """
+    Creates a file according to set_path
 
     The file will have the access rights according to "mode", and the "content"
     from the corresponding key, written in binary mode if "open" is set to "b",
     else "t".
     """
-    if os.path.exists(file_name):
-        if always_delete or not os.path.isfile(file_name):
-            shutil.rmtree(file_name)
-    with open(file_name, "w" + settings.get("open", DEFAULT_FILE_OPEN)) as fd:
-        fd.write(settings.get("content", DEFAULT_FILE_CONTENT))
-    os.chmod(file_name, mode=settings.get("mode", DEFAULT_FILE_MODE))
+    if os.path.exists(set_path):
+        if always_delete or not os.path.isfile(set_path):
+            _rmtree(set_path)
+    with open(set_path, "w" + set_path.get("open")) as fd:
+        fd.write(set_path.get("content"))
+    os.chmod(set_path, set_path.get_mode())
 
 
-def _create_link(link_name, settings, linker=os.link):
+def _create_hardlink(set_path):
     """
-    Creates a link according to settings, hard or soft depending on function
+    Creates a hardlink according to set_path
     """
-    link = settings["link"]
-    if os.path.isabs(link):
-        linker(link, link_name)
-    else:
-        currdir = os.getcwd()
-        os.chdir(os.path.dirname(link_name))
-        linker(link, link_name)
-        os.chdir(currdir)
+    os.sync()
+    os.link(set_path.get("target"), set_path)
+
+
+def _create_symlink(set_path):
+    """
+    Creates a symlink according to set_path
+    """
+    os.symlink(set_path.get("target"), set_path)
+
+
+def _rmtree(set_path):
+    """
+    Remove a complete tree making sure that access rights don't get in the way
+    """
+    for dir_name, dirs, files in os.walk(set_path):  # topdown
+        mode = os.stat(dir_name).st_mode
+        os.chmod(dir_name, mode | 0o222)
+    shutil.rmtree(set_path)
 
 
 def _compare_files(file1, stat1, file2, stat2):
@@ -276,3 +408,48 @@ def _compare_files(file1, stat1, file2, stat2):
                 pa1=file1, pa2=file2, go1=stat1.st_uid, go2=stat2.st_uid))
 
     return differences
+
+
+if __name__ == "__main__":
+    # just doing some minimal test when called as script
+    # requires `tree` to work!
+    import subprocess
+    import tempfile
+
+    base_temp_dir = tempfile.mkdtemp(".d", "fileset_")
+    structure = {
+        "a_dir": {
+            "rec": {"fmode": 0o444, "content": "default content"},
+            "contents": {
+                "fileA": {"content": "initial", "inode": 0},
+                "fileB": {"mode": "0544", "inode": "B"},
+                "fileC": {"target": "../a_bin_file"},
+                "fileD": {"inode": "B"},
+            }
+        },
+        "empty_dir": {"type": "dir", "dmode": 0o777},
+        "a_bin_file": {"content": b"some_binary_content", "open": "b"},
+    }
+
+    print("base directory: {bd}".format(bd=base_temp_dir))
+    create_fileset(base_temp_dir, structure)
+    subprocess.call(["tree", "-aJps", "--inodes", base_temp_dir])
+    remove_fileset(base_temp_dir, structure)
+
+"""
+$ tree -aJps --inodes /tmp/fileset_eh24xnxy.d
+[
+  {"type":"directory","name":"/tmp/fileset_eh24xnxy.d","inode":0,"mode":"0700","prot":"drwx------","size":100,"contents":[
+    {"type":"file","name":"a_bin_file","inode":190,"mode":"0644","prot":"-rw-r--r--","size":19},
+    {"type":"directory","name":"a_dir","inode":185,"mode":"0755","prot":"drwxr-xr-x","size":120,"contents":[
+      {"type":"file","name":"fileA","inode":186,"mode":"0444","prot":"-r--r--r--","size":7},
+      {"type":"file","name":"fileB","inode":187,"mode":"0544","prot":"-r-xr--r--","size":15},
+      {"type":"link","name":"fileC","target":"../a_bin_file","inode":190,"mode":"0777","prot":"lrwxrwxrwx","size":13},
+      {"type":"file","name":"fileD","inode":187,"mode":"0544","prot":"-r-xr--r--","size":15}
+  ]},
+    {"type":"directory","name":"empty_dir","inode":189,"mode":"0777","prot":"drwxrwxrwx","size":40}
+  ]}
+,
+  {"type":"report","directories":2,"files":5}
+]
+"""
diff --git a/testing/location_map_filenames_test.py b/testing/location_map_filenames_test.py
index 2674428dc..8ed203e55 100644
--- a/testing/location_map_filenames_test.py
+++ b/testing/location_map_filenames_test.py
@@ -20,7 +20,7 @@ def setUp(self):
         # Windows can't handle too long filenames
         long_multi = 5 if os.name == "nt" else 25
         self.from1_struct = {
-            "from1": {"subs": {
+            "from1": {"contents": {
                 "fileABC": {"content": "initial"},
                 "fileXYZ": {},
                 "dirAbCXyZ": {"type": "dir"},
@@ -36,7 +36,7 @@ def setUp(self):
         }
         self.from1_path = os.path.join(self.base_dir, b"from1")
         self.from2_struct = {
-            "from2": {"subs": {
+            "from2": {"contents": {
                 "fileABC": {"content": "modified"},
                 "fileXYZ": {},
                 "diraBcXyZ": {"type": "dir"},
diff --git a/testing/location_map_hardlinks_test.py b/testing/location_map_hardlinks_test.py
index 47c594a7b..6bd13edba 100644
--- a/testing/location_map_hardlinks_test.py
+++ b/testing/location_map_hardlinks_test.py
@@ -17,10 +17,10 @@ def setUp(self):
         self.base_dir = os.path.join(comtst.abs_test_dir,
                                      b"location_map_hardlinks")
         self.from1_struct = {
-            "from1": {"subs": {
-                "hardlink1": {"content": "initial"},
-                "hardlink2": {"link": "hardlink1"},
-                "hardlink3": {"link": "hardlink1"},
+            "from1": {"contents": {
+                "hardlink1": {"content": "initial", "inode": "hardlink1"},
+                "hardlink2": {"inode": "hardlink1"},
+                "hardlink3": {"inode": "hardlink1"},
             }}
         }
         self.from1_path = os.path.join(self.base_dir, b"from1")
diff --git a/testing/readonly_actions_test.py b/testing/readonly_actions_test.py
new file mode 100644
index 000000000..cfc13bb61
--- /dev/null
+++ b/testing/readonly_actions_test.py
@@ -0,0 +1,81 @@
+"""
+Test the ability to regress/remove read-only paths with api version >= 201
+"""
+import os
+import unittest
+
+import commontest as comtst
+import fileset
+
+
+class ActionReadOnlyTest(unittest.TestCase):
+    """
+    Test that rdiff-backup can properly handle read-only paths
+    """
+
+    def setUp(self):
+        self.base_dir = os.path.join(comtst.abs_test_dir,
+                                     b"readonly_actions")
+        self.from1_struct = {
+            "from1": {"contents": {
+                "dirA": {"contents": {"fileA": {"content": "initial"}}},
+                "fileB": {"content": "something"}
+            }}}
+        self.from1_path = os.path.join(self.base_dir, b"from1")
+        self.from2_struct = {
+            "from2": {"contents": {
+                "dirA": {"contents": {"fileA": {"content": "afterwards"}}},
+                "fileB": {"content": "now else"}
+            }}}
+        self.from2_path = os.path.join(self.base_dir, b"from2")
+        fileset.create_fileset(self.base_dir, self.from1_struct)
+        fileset.create_fileset(self.base_dir, self.from2_struct)
+        fileset.remove_fileset(self.base_dir, {"bak": {"type": "dir"}})
+        self.bak_path = os.path.join(self.base_dir, b"bak")
+
+        # we backup twice to the same backup repository at different times
+        self.assertEqual(comtst.rdiff_backup_action(
+            False, False, self.from1_path, self.bak_path,
+            ("--api-version", "201", "--current-time", "10000"),
+            b"backup", ()), 0)
+        self.assertEqual(comtst.rdiff_backup_action(
+            False, True, self.from2_path, self.bak_path,
+            ("--api-version", "201", "--current-time", "20000"),
+            b"backup", ()), 0)
+
+        self.success = False
+
+    def test_readonly_regress(self):
+        """test the "regress" action on a read-only repository"""
+
+        # then we regress forcefully
+        self.assertEqual(comtst.rdiff_backup_action(
+            False, None, self.bak_path, None,
+            ("--api-version", "201", "--force"),
+            b"regress", ()), 0)
+
+        # all tests were successful
+        self.success = True
+
+    def test_readonly_remove(self):
+        """test the "remove" action on a read-only repository"""
+
+        # then we regress forcefully
+        self.assertEqual(comtst.rdiff_backup_action(
+            True, None, self.bak_path, None,
+            ("--api-version", "201", "--force"),
+            b"remove", ("increments", "--older-than", "0B")), 0)
+
+        # all tests were successful
+        self.success = True
+
+    def tearDown(self):
+        # we clean-up only if the test was successful
+        if self.success:
+            fileset.remove_fileset(self.base_dir, self.from1_struct)
+            fileset.remove_fileset(self.base_dir, self.from2_struct)
+            fileset.remove_fileset(self.base_dir, {"bak": {"type": "dir"}})
+
+
+if __name__ == "__main__":
+    unittest.main()
diff --git a/testing/selectiontest.py b/testing/selectiontest.py
index eb513e249..6c95ac708 100644
--- a/testing/selectiontest.py
+++ b/testing/selectiontest.py
@@ -504,19 +504,19 @@ def setUp(self):
         self.base_dir = os.path.join(comtst.abs_test_dir,
                                      b"select_if_present")
         self.from1_struct = {
-            "from1": {"subs": {
+            "from1": {"contents": {
                 "dirWith": {
-                    "subs": {
+                    "contents": {
                         "check_for_me": {"content": "whatever"},
                         "fileWithin": {"content": "initial"},
                         "emptyDir": {"type": "dir"},
                     },
                 },
                 "dirWithout": {
-                    "subs": {
+                    "contents": {
                         "fileWithout": {"content": "initial"},
                         "nonEmptyDir": {
-                            "subs": {
+                            "contents": {
                                 "check_for_me": {"content": "whatever"},
                                 "fileWithin": {"content": "initial"},
                                 "anotherEmptyDir": {"type": "dir"},