Repository: matrix-org/synapse

Issue:
The DELETE method for the `unstable/org.matrix.msc3814.v1/dehydrated_device` endpoint doesn't delete the device
Seems that the patch I provided in https://github.com/matrix-org/synapse/pull/15929 doesn't do what it should be doing, at least not for the `DELETE` method.

After trying to delete the dehydrated device a subsequent `/keys/query` will still return the dehydrated device.

Uploading a new dehydrated device using the `PUT` method might also not delete the old dehydrated device.

Steps to reproduce:

- Create a new dehydrated device using the PUT method on the `unstable/org.matrix.msc3814.v1/dehydrated_device` endpoint.
- Observe the device over the `/keys/query` endpoint.
- Delete the dehydrated device using the DELETE method on the endpoint from step 1.
- Continue observing that the device exists using the `/keys/query` endpoint, calling DELETE from the previous step claims that the device is deleted.

@H-Shay any chance you could take a look at this?


Discussion:
> @H-Shay any chance you could take a look at this?

Absolutely
I think the problem is somewhere in here: 
- The current `DELETE` endpoint calls `rehydrate_device` which sets the access token to the current device (current device being the device we most recently uploaded via `PUT` and presumably the one we are trying to delete), and then deletes everything for the _previous_ device, as determined by whatever device the access token was set for previously:
https://github.com/matrix-org/synapse/blob/68b2611783ab00fdad567654a95492442722c106/synapse/handlers/device.py#L705-L715
-  If this is the first device added, this deletes everything for the current device, otherwise it deletes everything for the previous device. 
- Thus if you `PUT` a new device and then try to delete it via the endpoint it will to fail (unless this is the very first dehydrated device for the account) to deleted the current device

Could this be solved by separating out the rehydration and deletion steps? This also raised a question for me as to what the difference between rehydrating and deleting the device is - this wasn't very clear in either MSC and it appears that in MSC3814 it is stated that "Rather than changing its device ID when it rehydrates the
device, it will keep its device ID and upload its own device keys" - which leads me to believe rehydrating and deleting are separate actions? 
Oh I see. We (or rather I) reused the code the previous MSC added and that has incompatible semantics.

> Could this be solved by separating out the rehydration and deletion steps? This also raised a question for me as to what the difference between rehydrating and deleting the device is

I think part of the problem is that "rehydration" isn't the same thing between the two MSCs. Yes we should separate those concerns. There are three steps where deletion of an existing dehydrated device is OK to happen:

1. When we consume all the to-device events from the `/events` endpoint (as opposed to when we start consuming, like the MSC suggested).
2. When the user PUTs a new dehydrated device.
3. When the user explicitly DELETEs the dehydrated device.

I think no. 1 is too implicit and magical. If we only support no. 2 and 3, then the mechanism becomes obvious and simple. The server has a single slot for a dehydrated device and you can insert, delete, and get a value from this slot, the getter should not remove the value.

In pseudo-rust code it becomes a simple optional: `Option<DehydratedDevice>`.

Resolving patch:
diff --git a/synapse/handlers/device.py b/synapse/handlers/device.py
--- a/synapse/handlers/device.py
+++ b/synapse/handlers/device.py
@@ -722,6 +722,22 @@ async def rehydrate_device(
 
         return {"success": True}
 
+    async def delete_dehydrated_device(self, user_id: str, device_id: str) -> None:
+        """
+        Delete a stored dehydrated device.
+
+        Args:
+            user_id: the user_id to delete the device from
+            device_id: id of the dehydrated device to delete
+        """
+        success = await self.store.remove_dehydrated_device(user_id, device_id)
+
+        if not success:
+            raise errors.NotFoundError()
+
+        await self.delete_devices(user_id, [device_id])
+        await self.store.delete_e2e_keys_by_device(user_id=user_id, device_id=device_id)
+
     @wrap_as_background_process("_handle_new_device_update_async")
     async def _handle_new_device_update_async(self) -> None:
         """Called when we have a new local device list update that we need to
diff --git a/synapse/rest/client/devices.py b/synapse/rest/client/devices.py
--- a/synapse/rest/client/devices.py
+++ b/synapse/rest/client/devices.py
@@ -513,10 +513,8 @@ async def on_DELETE(self, request: SynapseRequest) -> Tuple[int, JsonDict]:
         if dehydrated_device is not None:
             (device_id, device_data) = dehydrated_device
 
-            result = await self.device_handler.rehydrate_device(
-                requester.user.to_string(),
-                self.auth.get_access_token_from_request(request),
-                device_id,
+            await self.device_handler.delete_dehydrated_device(
+                requester.user.to_string(), device_id
             )
 
             result = {"device_id": device_id}
@@ -538,6 +536,14 @@ async def on_PUT(self, request: SynapseRequest) -> Tuple[int, JsonDict]:
         requester = await self.auth.get_user_by_req(request)
         user_id = requester.user.to_string()
 
+        old_dehydrated_device = await self.device_handler.get_dehydrated_device(user_id)
+
+        # if an old device exists, delete it before creating a new one
+        if old_dehydrated_device:
+            await self.device_handler.delete_dehydrated_device(
+                user_id, old_dehydrated_device[0]
+            )
+
         device_info = submission.dict()
         if "device_keys" not in device_info.keys():
             raise SynapseError(


Test patch:
diff --git a/tests/rest/client/test_devices.py b/tests/rest/client/test_devices.py
--- a/tests/rest/client/test_devices.py
+++ b/tests/rest/client/test_devices.py
@@ -379,4 +379,141 @@ def test_dehydrate_msc3814(self) -> None:
             access_token=token,
             shorthand=False,
         )
-        self.assertEqual(channel.code, 404)
+        self.assertEqual(channel.code, 401)
+
+    @unittest.override_config(
+        {"experimental_features": {"msc2697_enabled": False, "msc3814_enabled": True}}
+    )
+    def test_msc3814_dehydrated_device_delete_works(self) -> None:
+        user = self.register_user("mikey", "pass")
+        token = self.login(user, "pass", device_id="device1")
+        content: JsonDict = {
+            "device_data": {
+                "algorithm": "m.dehydration.v1.olm",
+            },
+            "device_id": "device2",
+            "initial_device_display_name": "foo bar",
+            "device_keys": {
+                "user_id": "@mikey:test",
+                "device_id": "device2",
+                "valid_until_ts": "80",
+                "algorithms": [
+                    "m.olm.curve25519-aes-sha2",
+                ],
+                "keys": {
+                    "<algorithm>:<device_id>": "<key_base64>",
+                },
+                "signatures": {
+                    "<user_id>": {"<algorithm>:<device_id>": "<signature_base64>"}
+                },
+            },
+        }
+        channel = self.make_request(
+            "PUT",
+            "_matrix/client/unstable/org.matrix.msc3814.v1/dehydrated_device",
+            content=content,
+            access_token=token,
+            shorthand=False,
+        )
+        self.assertEqual(channel.code, 200)
+        device_id = channel.json_body.get("device_id")
+        assert device_id is not None
+        self.assertIsInstance(device_id, str)
+        self.assertEqual("device2", device_id)
+
+        # ensure that keys were uploaded and available
+        channel = self.make_request(
+            "POST",
+            "/_matrix/client/r0/keys/query",
+            {
+                "device_keys": {
+                    user: ["device2"],
+                },
+            },
+            token,
+        )
+        self.assertEqual(
+            channel.json_body["device_keys"][user]["device2"]["keys"],
+            {
+                "<algorithm>:<device_id>": "<key_base64>",
+            },
+        )
+
+        # delete the dehydrated device
+        channel = self.make_request(
+            "DELETE",
+            "_matrix/client/unstable/org.matrix.msc3814.v1/dehydrated_device",
+            access_token=token,
+            shorthand=False,
+        )
+        self.assertEqual(channel.code, 200)
+
+        # ensure that keys are no longer available for deleted device
+        channel = self.make_request(
+            "POST",
+            "/_matrix/client/r0/keys/query",
+            {
+                "device_keys": {
+                    user: ["device2"],
+                },
+            },
+            token,
+        )
+        self.assertEqual(channel.json_body["device_keys"], {"@mikey:test": {}})
+
+        # check that an old device is deleted when user PUTs a new device
+        # First, create a device
+        content["device_id"] = "device3"
+        content["device_keys"]["device_id"] = "device3"
+        channel = self.make_request(
+            "PUT",
+            "_matrix/client/unstable/org.matrix.msc3814.v1/dehydrated_device",
+            content=content,
+            access_token=token,
+            shorthand=False,
+        )
+        self.assertEqual(channel.code, 200)
+        device_id = channel.json_body.get("device_id")
+        assert device_id is not None
+        self.assertIsInstance(device_id, str)
+        self.assertEqual("device3", device_id)
+
+        # create a second device without deleting first device
+        content["device_id"] = "device4"
+        content["device_keys"]["device_id"] = "device4"
+        channel = self.make_request(
+            "PUT",
+            "_matrix/client/unstable/org.matrix.msc3814.v1/dehydrated_device",
+            content=content,
+            access_token=token,
+            shorthand=False,
+        )
+        self.assertEqual(channel.code, 200)
+        device_id = channel.json_body.get("device_id")
+        assert device_id is not None
+        self.assertIsInstance(device_id, str)
+        self.assertEqual("device4", device_id)
+
+        # check that the second device that was created is what is returned when we GET
+        channel = self.make_request(
+            "GET",
+            "_matrix/client/unstable/org.matrix.msc3814.v1/dehydrated_device",
+            access_token=token,
+            shorthand=False,
+        )
+        self.assertEqual(channel.code, 200)
+        returned_device_id = channel.json_body["device_id"]
+        self.assertEqual(returned_device_id, "device4")
+
+        # and that if we query the keys for the first device they are not there
+        channel = self.make_request(
+            "POST",
+            "/_matrix/client/r0/keys/query",
+            {
+                "device_keys": {
+                    user: ["device3"],
+                },
+            },
+            token,
+        )
+        self.assertEqual(channel.json_body["device_keys"], {"@mikey:test": {}})