Repository: matrix-org/synapse

Issue:
/admin/quarantine_media destroys stickerpacks
we need a way to exclude stickerpacks from being nuked by quarantine_media somehow. suggestions welcome


Discussion:
Probably the easiest way to do this is to maintain a list of users (or a community?) in the config of users who are immune to quarantining, on the assumption that they are trustworthy and not going to start uploading virii or whatever.
And we can't just prevent removal of "m.sticker" events?
I don't think we even create events, it looks like the media is just uploaded and a manifest bundled with the referring mxc as url. So maybe first the script should exclude certain users while waiting for stickerpacks to be done some other way.
This has popped up again on matrix.org. Seems like you're proposing that we add a config option featuring a list of trusted users whom we won't quarantine media of. This would work for our local stickerpacks.

However, this would not work for remote users (as it stands) as we don't know who uploaded a piece of media without asking the homeserver that's hosting it.

Thinking in terms of community sticker packs, *and custom emoji*, this may become a problem in the future.

However, for now, this should solve the issue of a homeserver operator's own stickerpacks from being quarantined accidentally.
the config option could also just be a list of mxc uris to never destroy, which makes it agnostic to stickers and agnostic to users.
> the config option could also just be a list of mxc uris to never destroy, which makes it agnostic to stickers and agnostic to users.

Yes, and I think this would be useful in addition to users. Having a user which we can upload all of our stickerpacks through and knowing they won't be quarantined is quite practical.
> the config option could also just be a list of mxc uris to never destroy, which makes it agnostic to stickers and agnostic to users.

How about a new column on the `local_media` and `remote_media_cache` tables, rather than having a big list in the config (which would presumably require us to restart the server each time we wanted to add a new item?). Initially this could be updated manually: if it became useful, we could add an admin API later.

One problem with that approach would be that it would be difficult to quarantine-lock media from remote servers before we had cached a copy of it. Is that likely to be a problem in practice?



You could take the other ideas and put them in a database table instead, forgoing the need for a restart.

For those idea, doing it by user would still be quite practical for the stickerpack use-case. If we had a good relationship with another homeserver operator who uploads stickerpacks, we could whitelist a user from their server which uploads them at any time - rather than servicing their requests to do so each time they uploaded a new piece of media.

However, this breaks down when you want to prevent quarantine of a legitimate stickerpack someone uploads, but not other images they generally upload in their spare time.

Perhaps we could have a hybrid approach of being able to mark specific media via @richvdh's proposed method, but additionally have a table of users that marks any media they upload at time of upload. We could then add and remove users from this list at will.
> For those idea, doing it by user would still be quite practical for the stickerpack use-case. If we had a good relationship with another homeserver operator who uploads stickerpacks, we could whitelist a user from their server which uploads them at any time - rather than servicing their requests to do so each time they uploaded a new piece of media.

I don't think we know the uploader for remote media, so I don't think that works.

I suggest that for now we either a) add a column in the tables to say media shouldn't get quarantined and/or b) add a table for users whose images should not be quarantined.

I don't think we can do any better for now while the mxc's are so divorced from rooms, so we should just do something sub optimal to unblock the specific case of sticker packs on matrix.org. Given this won't be a public API we don't really need to worry about backwards compat and can change our minds down the line 
> add a column in the tables to say media shouldn't get quarantined

I seem to recall a conversation where it was decided this ^ is the way forward. Am I misremembering?

Otherwise, any hints on how to decide between the two approaches? Adding the column seems like it would have less impact overall, but not sure it would meet the requirements.
@clokep I *think* all involved parties are happy with the database column solution. I was rooting for the config option for remote server, but considering we don't know who uploaded a piece of media from a remote server, it loses its advantage.
I think adding a database column for media will solve this problem for now. We can always do something else later.



Resolving patch:
diff --git a/synapse/storage/data_stores/main/media_repository.py b/synapse/storage/data_stores/main/media_repository.py
--- a/synapse/storage/data_stores/main/media_repository.py
+++ b/synapse/storage/data_stores/main/media_repository.py
@@ -81,6 +81,15 @@ def store_local_media(
             desc="store_local_media",
         )
 
+    def mark_local_media_as_safe(self, media_id: str):
+        """Mark a local media as safe from quarantining."""
+        return self.db.simple_update_one(
+            table="local_media_repository",
+            keyvalues={"media_id": media_id},
+            updatevalues={"safe_from_quarantine": True},
+            desc="mark_local_media_as_safe",
+        )
+
     def get_url_cache(self, url, ts):
         """Get the media_id and ts for a cached URL as of the given timestamp
         Returns:
diff --git a/synapse/storage/data_stores/main/room.py b/synapse/storage/data_stores/main/room.py
--- a/synapse/storage/data_stores/main/room.py
+++ b/synapse/storage/data_stores/main/room.py
@@ -626,36 +626,10 @@ def quarantine_media_ids_in_room(self, room_id, quarantined_by):
 
         def _quarantine_media_in_room_txn(txn):
             local_mxcs, remote_mxcs = self._get_media_mxcs_in_room_txn(txn, room_id)
-            total_media_quarantined = 0
-
-            # Now update all the tables to set the quarantined_by flag
-
-            txn.executemany(
-                """
-                UPDATE local_media_repository
-                SET quarantined_by = ?
-                WHERE media_id = ?
-            """,
-                ((quarantined_by, media_id) for media_id in local_mxcs),
-            )
-
-            txn.executemany(
-                """
-                    UPDATE remote_media_cache
-                    SET quarantined_by = ?
-                    WHERE media_origin = ? AND media_id = ?
-                """,
-                (
-                    (quarantined_by, origin, media_id)
-                    for origin, media_id in remote_mxcs
-                ),
+            return self._quarantine_media_txn(
+                txn, local_mxcs, remote_mxcs, quarantined_by
             )
 
-            total_media_quarantined += len(local_mxcs)
-            total_media_quarantined += len(remote_mxcs)
-
-            return total_media_quarantined
-
         return self.db.runInteraction(
             "quarantine_media_in_room", _quarantine_media_in_room_txn
         )
@@ -805,17 +779,17 @@ def _quarantine_media_txn(
         Returns:
             The total number of media items quarantined
         """
-        total_media_quarantined = 0
-
         # Update all the tables to set the quarantined_by flag
         txn.executemany(
             """
             UPDATE local_media_repository
             SET quarantined_by = ?
-            WHERE media_id = ?
+            WHERE media_id = ? AND safe_from_quarantine = ?
         """,
-            ((quarantined_by, media_id) for media_id in local_mxcs),
+            ((quarantined_by, media_id, False) for media_id in local_mxcs),
         )
+        # Note that a rowcount of -1 can be used to indicate no rows were affected.
+        total_media_quarantined = txn.rowcount if txn.rowcount > 0 else 0
 
         txn.executemany(
             """
@@ -825,9 +799,7 @@ def _quarantine_media_txn(
             """,
             ((quarantined_by, origin, media_id) for origin, media_id in remote_mxcs),
         )
-
-        total_media_quarantined += len(local_mxcs)
-        total_media_quarantined += len(remote_mxcs)
+        total_media_quarantined += txn.rowcount if txn.rowcount > 0 else 0
 
         return total_media_quarantined
 


Test patch:
diff --git a/tests/rest/admin/test_admin.py b/tests/rest/admin/test_admin.py
--- a/tests/rest/admin/test_admin.py
+++ b/tests/rest/admin/test_admin.py
@@ -220,6 +220,24 @@ def write_to(r):
 
         return hs
 
+    def _ensure_quarantined(self, admin_user_tok, server_and_media_id):
+        """Ensure a piece of media is quarantined when trying to access it."""
+        request, channel = self.make_request(
+            "GET", server_and_media_id, shorthand=False, access_token=admin_user_tok,
+        )
+        request.render(self.download_resource)
+        self.pump(1.0)
+
+        # Should be quarantined
+        self.assertEqual(
+            404,
+            int(channel.code),
+            msg=(
+                "Expected to receive a 404 on accessing quarantined media: %s"
+                % server_and_media_id
+            ),
+        )
+
     def test_quarantine_media_requires_admin(self):
         self.register_user("nonadmin", "pass", admin=False)
         non_admin_user_tok = self.login("nonadmin", "pass")
@@ -292,24 +310,7 @@ def test_quarantine_media_by_id(self):
         self.assertEqual(200, int(channel.code), msg=channel.result["body"])
 
         # Attempt to access the media
-        request, channel = self.make_request(
-            "GET",
-            server_name_and_media_id,
-            shorthand=False,
-            access_token=admin_user_tok,
-        )
-        request.render(self.download_resource)
-        self.pump(1.0)
-
-        # Should be quarantined
-        self.assertEqual(
-            404,
-            int(channel.code),
-            msg=(
-                "Expected to receive a 404 on accessing quarantined media: %s"
-                % server_name_and_media_id
-            ),
-        )
+        self._ensure_quarantined(admin_user_tok, server_name_and_media_id)
 
     def test_quarantine_all_media_in_room(self, override_url_template=None):
         self.register_user("room_admin", "pass", admin=True)
@@ -371,45 +372,10 @@ def test_quarantine_all_media_in_room(self, override_url_template=None):
         server_and_media_id_2 = mxc_2[6:]
 
         # Test that we cannot download any of the media anymore
-        request, channel = self.make_request(
-            "GET",
-            server_and_media_id_1,
-            shorthand=False,
-            access_token=non_admin_user_tok,
-        )
-        request.render(self.download_resource)
-        self.pump(1.0)
-
-        # Should be quarantined
-        self.assertEqual(
-            404,
-            int(channel.code),
-            msg=(
-                "Expected to receive a 404 on accessing quarantined media: %s"
-                % server_and_media_id_1
-            ),
-        )
-
-        request, channel = self.make_request(
-            "GET",
-            server_and_media_id_2,
-            shorthand=False,
-            access_token=non_admin_user_tok,
-        )
-        request.render(self.download_resource)
-        self.pump(1.0)
-
-        # Should be quarantined
-        self.assertEqual(
-            404,
-            int(channel.code),
-            msg=(
-                "Expected to receive a 404 on accessing quarantined media: %s"
-                % server_and_media_id_2
-            ),
-        )
+        self._ensure_quarantined(admin_user_tok, server_and_media_id_1)
+        self._ensure_quarantined(admin_user_tok, server_and_media_id_2)
 
-    def test_quaraantine_all_media_in_room_deprecated_api_path(self):
+    def test_quarantine_all_media_in_room_deprecated_api_path(self):
         # Perform the above test with the deprecated API path
         self.test_quarantine_all_media_in_room("/_synapse/admin/v1/quarantine_media/%s")
 
@@ -449,25 +415,52 @@ def test_quarantine_all_media_by_user(self):
         )
 
         # Attempt to access each piece of media
+        self._ensure_quarantined(admin_user_tok, server_and_media_id_1)
+        self._ensure_quarantined(admin_user_tok, server_and_media_id_2)
+
+    def test_cannot_quarantine_safe_media(self):
+        self.register_user("user_admin", "pass", admin=True)
+        admin_user_tok = self.login("user_admin", "pass")
+
+        non_admin_user = self.register_user("user_nonadmin", "pass", admin=False)
+        non_admin_user_tok = self.login("user_nonadmin", "pass")
+
+        # Upload some media
+        response_1 = self.helper.upload_media(
+            self.upload_resource, self.image_data, tok=non_admin_user_tok
+        )
+        response_2 = self.helper.upload_media(
+            self.upload_resource, self.image_data, tok=non_admin_user_tok
+        )
+
+        # Extract media IDs
+        server_and_media_id_1 = response_1["content_uri"][6:]
+        server_and_media_id_2 = response_2["content_uri"][6:]
+
+        # Mark the second item as safe from quarantine.
+        _, media_id_2 = server_and_media_id_2.split("/")
+        self.get_success(self.store.mark_local_media_as_safe(media_id_2))
+
+        # Quarantine all media by this user
+        url = "/_synapse/admin/v1/user/%s/media/quarantine" % urllib.parse.quote(
+            non_admin_user
+        )
         request, channel = self.make_request(
-            "GET",
-            server_and_media_id_1,
-            shorthand=False,
-            access_token=non_admin_user_tok,
+            "POST", url.encode("ascii"), access_token=admin_user_tok,
         )
-        request.render(self.download_resource)
+        self.render(request)
         self.pump(1.0)
-
-        # Should be quarantined
+        self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
         self.assertEqual(
-            404,
-            int(channel.code),
-            msg=(
-                "Expected to receive a 404 on accessing quarantined media: %s"
-                % server_and_media_id_1,
-            ),
+            json.loads(channel.result["body"].decode("utf-8")),
+            {"num_quarantined": 1},
+            "Expected 1 quarantined item",
         )
 
+        # Attempt to access each piece of media, the first should fail, the
+        # second should succeed.
+        self._ensure_quarantined(admin_user_tok, server_and_media_id_1)
+
         # Attempt to access each piece of media
         request, channel = self.make_request(
             "GET",
@@ -478,12 +471,12 @@ def test_quarantine_all_media_by_user(self):
         request.render(self.download_resource)
         self.pump(1.0)
 
-        # Should be quarantined
+        # Shouldn't be quarantined
         self.assertEqual(
-            404,
+            200,
             int(channel.code),
             msg=(
-                "Expected to receive a 404 on accessing quarantined media: %s"
+                "Expected to receive a 200 on accessing not-quarantined media: %s"
                 % server_and_media_id_2
             ),
         )