Repository: matrix-org/synapse

Issue:
We need to actually censor redactions from the DB (SYN-284)
We don't seem to remove redacted event contents from the DB when they are redacted.  This kinda defeats the point; we should.

(Imported from https://matrix.org/jira/browse/SYN-284)

(Reported by @ara4n)


Discussion:
Jira watchers: @erikjohnston @ara4n

Links exported from Jira:

is duplicated by [SYN-493](https://matrix.org/jira/browse/SYN-493)

My first question is: what is the point of the current redactions system? The current implementation of it seems to be trying to do two separate things.

Originally, the intent was that redactions should be used to remove content that _server admins_ wouldn't want on their server, e.g. illegal content. In this scenario it's important that admins can cause redacted content to be deleted. The idea was to have an admin API that allowed you to look at the redacted event before deciding whether or not to delete it.

However, people can now redact their own events, which is used as a crude "retraction" mechanism. In my opinion these shouldn't be auto deleted (and also shouldn't be flagged to server admins lest they look at the retracted content). For example, if someone sends a bunch of abusive messages into Matrix HQ and later retracts them, moderators still need to be able to see what those messages were when people complain so that they can take the appropriate action.

On a more practical level, whether we delete content or not is not something we can enforce on a server. Certainly the first thing I would do would be to disable it on my server, and I'm fairly sure I would not be in the minority in doing that. Why? Because its infuriating to have messages disappear while I'm reading/responding simply because the author decided to or accidentally redacted their event, which has happened several times now. Given this, I'm not entirely sure what the purpose or need of having redacted events auto deleted from any server really is.

My opinion on this mess is: we should have separate redact and retract mechanisms and implement the admin API for dealing with redactions, but never delete retractions from the DB (we could possibly have a separate flag for asking server admins to redact the event).

-- @erikjohnston

Is this the issue behind https://github.com/matrix-org/matrix-appservice-irc/issues/473 or should I open a new one?
Are there any news on this or how does GDPR affect this and referenced issues?

I started thinking about this/them more, because of https://github.com/tulir/mautrix-telegram/issues/248 and https://github.com/t2bot/t2bot.io/issues/15.
From the duplicate https://github.com/matrix-org/synapse/issues/3211#issuecomment-388311845:

> @richvdh commented on May 11:
> (we'll be fixing this for GDPR)

GDPR has been in force for months. Is anyone from Matrix.org following this issue? I seem to have asked previously asked about this nine days ago.
This is kind of an important issue AFAICT because redactions are the only way to signal non-consent to processing of data. Imagine the following:

1. Person joins #matrix and sends a message
2. They decide that message was not what they want to send
  * Consent can be withdrawn
  * In a most obvious case, lets imagine a 12 year old sent their name and address and then a parent noticed what was happening and intervened. When the data subject is a minor, no justification needs to be given
3. Knowing that the "remove" button doesn't actually delete the message, the person is completely within their rights to send a GDPR request to the operators of each and every synapse server that federates #matrix 

* If this issue is fixed, then the person is nolonger within their rights to send a GDPR request to each server operator because there is an obvious technical solution
* This has a high probability of being attacked to grief the network in one of two ways:
  a. Create an undelete bot which is able to repeat the content of a deleted message, thus illustrating the elusiveness of the remove button
  b. Create a service which sends GDPR email to every known server operator
* To add to the mess, the griefter might also begin recommending people upgrade to their "gdpr-compliant" fork of synapse, giving them a user base and thus a seat at the table in protocol design discussions.
> within their rights to send a GDPR request to the operators of each and every synapse server that federates #matrix

I'm not so sure this is actually true, assume for a second that my homeserver serves only users outside of the EU, but is in the room, I would have no reason to comply with GDPR.
I mean similarly, if I GDPR gmail, do I really have a stance to GDPR the mail server of every person I have ever emailed?
the right soln here imo is to provide a server admin specific setting for how long to keep redactions before GCing them. it could probably default to 0 other than for server admins who need an audit trail of the abuse on their platform.
> it could probably default to 0

To be clear, 0 means immediate deletion correct?
FYI there's a temporary fix which people can implement on their own homeserver in order to:

1. Clear redacted messages
2. Drop rooms which nolonger have any members on your HS

https://github.com/xwiki-labs/synapse_scripts/blob/master/synapse_janitor.sql

On a synapse which I used to administer, this is setup in a crontab which brings the server down once per week, runs these, and then re-launches it.
Is this going to be covered by Brendan's PR? https://github.com/matrix-org/synapse/pull/5815
I just checked the code and it looks like a no.

Resolving patch:
diff --git a/synapse/config/server.py b/synapse/config/server.py
--- a/synapse/config/server.py
+++ b/synapse/config/server.py
@@ -162,6 +162,16 @@ def read_config(self, config, **kwargs):
 
         self.mau_trial_days = config.get("mau_trial_days", 0)
 
+        # How long to keep redacted events in the database in unredacted form
+        # before redacting them.
+        redaction_retention_period = config.get("redaction_retention_period", "7d")
+        if redaction_retention_period is not None:
+            self.redaction_retention_period = self.parse_duration(
+                redaction_retention_period
+            )
+        else:
+            self.redaction_retention_period = None
+
         # Options to disable HS
         self.hs_disabled = config.get("hs_disabled", False)
         self.hs_disabled_message = config.get("hs_disabled_message", "")
@@ -718,6 +728,13 @@ def generate_config_section(
         # Defaults to 'true'.
         #
         #allow_per_room_profiles: false
+
+        # How long to keep redacted events in unredacted form in the database. After
+        # this period redacted events get replaced with their redacted form in the DB.
+        #
+        # Defaults to `7d`. Set to `null` to disable.
+        #
+        redaction_retention_period: 7d
         """
             % locals()
         )
diff --git a/synapse/storage/events.py b/synapse/storage/events.py
--- a/synapse/storage/events.py
+++ b/synapse/storage/events.py
@@ -23,7 +23,7 @@
 from six import iteritems, text_type
 from six.moves import range
 
-from canonicaljson import json
+from canonicaljson import encode_canonical_json, json
 from prometheus_client import Counter, Histogram
 
 from twisted.internet import defer
@@ -33,6 +33,7 @@
 from synapse.api.errors import SynapseError
 from synapse.events import EventBase  # noqa: F401
 from synapse.events.snapshot import EventContext  # noqa: F401
+from synapse.events.utils import prune_event_dict
 from synapse.logging.context import PreserveLoggingContext, make_deferred_yieldable
 from synapse.logging.utils import log_function
 from synapse.metrics import BucketCollector
@@ -262,6 +263,14 @@ def read_forward_extremities():
 
         hs.get_clock().looping_call(read_forward_extremities, 60 * 60 * 1000)
 
+        def _censor_redactions():
+            return run_as_background_process(
+                "_censor_redactions", self._censor_redactions
+            )
+
+        if self.hs.config.redaction_retention_period is not None:
+            hs.get_clock().looping_call(_censor_redactions, 5 * 60 * 1000)
+
     @defer.inlineCallbacks
     def _read_forward_extremities(self):
         def fetch(txn):
@@ -1548,6 +1557,98 @@ def _store_redaction(self, txn, event):
             (event.event_id, event.redacts),
         )
 
+    @defer.inlineCallbacks
+    def _censor_redactions(self):
+        """Censors all redactions older than the configured period that haven't
+        been censored yet.
+
+        By censor we mean update the event_json table with the redacted event.
+
+        Returns:
+            Deferred
+        """
+
+        if self.hs.config.redaction_retention_period is None:
+            return
+
+        max_pos = yield self.find_first_stream_ordering_after_ts(
+            self._clock.time_msec() - self.hs.config.redaction_retention_period
+        )
+
+        # We fetch all redactions that:
+        #   1. point to an event we have,
+        #   2. has a stream ordering from before the cut off, and
+        #   3. we haven't yet censored.
+        #
+        # This is limited to 100 events to ensure that we don't try and do too
+        # much at once. We'll get called again so this should eventually catch
+        # up.
+        #
+        # We use the range [-max_pos, max_pos] to handle backfilled events,
+        # which are given negative stream ordering.
+        sql = """
+            SELECT redact_event.event_id, redacts FROM redactions
+            INNER JOIN events AS redact_event USING (event_id)
+            INNER JOIN events AS original_event ON (
+                redact_event.room_id = original_event.room_id
+                AND redacts = original_event.event_id
+            )
+            WHERE NOT have_censored
+            AND ? <= redact_event.stream_ordering AND redact_event.stream_ordering <= ?
+            ORDER BY redact_event.stream_ordering ASC
+            LIMIT ?
+        """
+
+        rows = yield self._execute(
+            "_censor_redactions_fetch", None, sql, -max_pos, max_pos, 100
+        )
+
+        updates = []
+
+        for redaction_id, event_id in rows:
+            redaction_event = yield self.get_event(redaction_id, allow_none=True)
+            original_event = yield self.get_event(
+                event_id, allow_rejected=True, allow_none=True
+            )
+
+            # The SQL above ensures that we have both the redaction and
+            # original event, so if the `get_event` calls return None it
+            # means that the redaction wasn't allowed. Either way we know that
+            # the result won't change so we mark the fact that we've checked.
+            if (
+                redaction_event
+                and original_event
+                and original_event.internal_metadata.is_redacted()
+            ):
+                # Redaction was allowed
+                pruned_json = encode_canonical_json(
+                    prune_event_dict(original_event.get_dict())
+                )
+            else:
+                # Redaction wasn't allowed
+                pruned_json = None
+
+            updates.append((redaction_id, event_id, pruned_json))
+
+        def _update_censor_txn(txn):
+            for redaction_id, event_id, pruned_json in updates:
+                if pruned_json:
+                    self._simple_update_one_txn(
+                        txn,
+                        table="event_json",
+                        keyvalues={"event_id": event_id},
+                        updatevalues={"json": pruned_json},
+                    )
+
+                self._simple_update_one_txn(
+                    txn,
+                    table="redactions",
+                    keyvalues={"event_id": redaction_id},
+                    updatevalues={"have_censored": True},
+                )
+
+        yield self.runInteraction("_update_censor_txn", _update_censor_txn)
+
     @defer.inlineCallbacks
     def count_daily_messages(self):
         """


Test patch:
diff --git a/tests/storage/test_redaction.py b/tests/storage/test_redaction.py
--- a/tests/storage/test_redaction.py
+++ b/tests/storage/test_redaction.py
@@ -17,6 +17,8 @@
 
 from mock import Mock
 
+from canonicaljson import json
+
 from twisted.internet import defer
 
 from synapse.api.constants import EventTypes, Membership
@@ -29,8 +31,10 @@
 
 class RedactionTestCase(unittest.HomeserverTestCase):
     def make_homeserver(self, reactor, clock):
+        config = self.default_config()
+        config["redaction_retention_period"] = "30d"
         return self.setup_test_homeserver(
-            resource_for_federation=Mock(), http_client=None
+            resource_for_federation=Mock(), http_client=None, config=config
         )
 
     def prepare(self, reactor, clock, hs):
@@ -286,3 +290,74 @@ def room_id(self):
         self.assertEqual(
             fetched.unsigned["redacted_because"].event_id, redaction_event_id2
         )
+
+    def test_redact_censor(self):
+        """Test that a redacted event gets censored in the DB after a month
+        """
+
+        self.get_success(
+            self.inject_room_member(self.room1, self.u_alice, Membership.JOIN)
+        )
+
+        msg_event = self.get_success(self.inject_message(self.room1, self.u_alice, "t"))
+
+        # Check event has not been redacted:
+        event = self.get_success(self.store.get_event(msg_event.event_id))
+
+        self.assertObjectHasAttributes(
+            {
+                "type": EventTypes.Message,
+                "user_id": self.u_alice.to_string(),
+                "content": {"body": "t", "msgtype": "message"},
+            },
+            event,
+        )
+
+        self.assertFalse("redacted_because" in event.unsigned)
+
+        # Redact event
+        reason = "Because I said so"
+        self.get_success(
+            self.inject_redaction(self.room1, msg_event.event_id, self.u_alice, reason)
+        )
+
+        event = self.get_success(self.store.get_event(msg_event.event_id))
+
+        self.assertTrue("redacted_because" in event.unsigned)
+
+        self.assertObjectHasAttributes(
+            {
+                "type": EventTypes.Message,
+                "user_id": self.u_alice.to_string(),
+                "content": {},
+            },
+            event,
+        )
+
+        event_json = self.get_success(
+            self.store._simple_select_one_onecol(
+                table="event_json",
+                keyvalues={"event_id": msg_event.event_id},
+                retcol="json",
+            )
+        )
+
+        self.assert_dict(
+            {"content": {"body": "t", "msgtype": "message"}}, json.loads(event_json)
+        )
+
+        # Advance by 30 days, then advance again to ensure that the looping call
+        # for updating the stream position gets called and then the looping call
+        # for the censoring gets called.
+        self.reactor.advance(60 * 60 * 24 * 31)
+        self.reactor.advance(60 * 60 * 2)
+
+        event_json = self.get_success(
+            self.store._simple_select_one_onecol(
+                table="event_json",
+                keyvalues={"event_id": msg_event.event_id},
+                retcol="json",
+            )
+        )
+
+        self.assert_dict({"content": {}}, json.loads(event_json))