Repository: matrix-org/synapse

Issue:
Synapse does not apply edits to events other than `m.room.message`
MSC2676 [says](https://github.com/uhoreg/matrix-doc/blob/aggregations-edits/proposals/2676-message-editing.md#server-behaviour):

> Whenever a homeserver would return an event via the Client-Server API, it should check for any valid, applicable `m.replace` event, and if one is found, it should first modify the content of the original event according to the `m.new_content` of the most recent edit.

It also [gives](https://github.com/uhoreg/matrix-doc/blob/aggregations-edits/proposals/2676-message-editing.md#validity-of-message-edit-events) a comprehensive list of what comprises a "valid" edit event.

Synapse does this, but only where the events are of type `m.room.message`. This is incorrect: it should also do so for other event types. Examples include polls ([`org.matrix.msc3381.poll.start`](https://github.com/matrix-org/matrix-spec-proposals/pull/3381)), which can be edited via the Element UI, and stickers ([`m.sticker`](https://spec.matrix.org/v1.2/client-server-api/#sticker-messages)), which cannot be edited via the UI but can via the API.

This is closely related to #12503, but that concerns aggregations, whereas this issue is about updating the `content` of the original event.
Synapse does not bundle `m.replace` aggregations for events other than `m.room.message`
Per MSC2675, when the server returns an event, it is supposed to bundle the aggregation of any related events into the response.

This includes `m.replace` relations: for an `m.room.message` event, `/context/$original_event_id` might return:

```json5
{
  "event": {
    ...
    "event_id": "$original_event_id",
    "type": "m.room.message",
    "unsigned": {
      "m.relations": {
        "m.replace": {
          "event_id": "$replacement_event_id",
          "origin_server_ts": 1650390324643,
          "sender": "@richvdh:matrix.org"
        }
      }
    },
    "user_id": "@richvdh:matrix.org"
  }
}
```

However, for other event types, no `m.replace` aggregation is returned.

It's not as if `m.room.message` is the only event type you can edit (consider, for example, encrypted events), so this seems odd.


Discussion:
I came across this issue while doing tests on matrix.org. As far as I can tell, Synapse doesn't even seem to "modify the content of the original event according to the m.new_content of the most recent edit.", even for `m.room.message` events. The original message content does not get modified after one or even several edits. Below are the message objects from a test I just did on matrix.org. The message was created and edited in a fresh non-encrypted private room using Element Web. After editing the message, I logged out, refreshed the browser, and logged back in to make sure the events were fetched again from the server.

Original message, before and after editing:

```
content:
  body: "hi"
  msgtype: "m.text"
  org.matrix.msc1767.text: "hi"
event_id: "$PbrLqfPm7MBDHLZd5gVREn3VA8UyFc8qb9220xiKYlw"
origin_server_ts: 1665067355108
sender: "@xyz:matrix.org"
type: "m.room.message"
```

Message created by editing:

```
content: 
  body: " * hi hi"
  msgtype: "m.text"
  org.matrix.msc1767.text: " * hi hi"
  m.new_content: 
    body: "hi hi"
    msgtype: "m.text"
    org.matrix.msc1767.text: "hi hi"
  m.relates_to: 
    event_id: "$PbrLqfPm7MBDHLZd5gVREn3VA8UyFc8qb9220xiKYlw"
    rel_type: "m.replace"
event_id: "$rOEXMZKBNp8iYoBcSvdOFIE7bAqVVpJGLhc68ENPLw8"
origin_server_ts: 1665067377760
sender: "@xyz:matrix.org"
type: "m.room.message"
```
I guess this would not get picked up by users as Element will anyway display the edited content of messages, but I'm wondering if something has changed in the protocol or implementation since this issue was opened. Or am I missing something completely obvious here?
As of today, the correct behavior should be applied, but only for `m.room.message` events. How are you requesting the event?
Via the sync API.
> Via the sync API.

Bundled aggregations are only returned if the result is `limited`, see [here](https://spec.matrix.org/latest/client-server-api/#server-side-replacement-of-content):

> Whenever an `m.replace` is to be bundled with an event as above, the server should also modify the content of the original event according to the `m.new_content` of the most recent replacement event (determined as above).

You can then see [here](https://spec.matrix.org/v1.4/client-server-api/#aggregations) when aggregations are bundled:

> `[GET /sync](https://spec.matrix.org/v1.4/client-server-api/#get_matrixclientv3sync)` when the relevant section has a `limited` value of `true`.
Oh right, I had missed the point with the "limited". Not quite sure about the logic of it, but in any case, thanks for the info!
See #14252.
Presumably this means that if you forward-paginate through a room with `/messages` and get as far as the updated event, you'll see the old content rather than the edited content.
> Presumably this means that if you forward-paginate through a room with `/messages` and get as far as the updated event, you'll see the old content rather than the edited content.

Another situation this can impact clients is when viewing a thread (which uses the `/relations` API). @t3chguy mentioned to me that clients essentially assume all encrypted events have an edit and attempt to re-fetch them from the server to see if this is accurate. (matrix-js-sdk is using `/relations` to see if there are edits, see https://github.com/matrix-org/matrix-js-sdk/blob/v17.1.0-rc.1/src/models/thread.ts#L240.)

See also vector-im/element-web#21447 and vector-im/element-web#21445
> @t3chguy mentioned to me that clients essentially assume all encrypted events have an edit and attempt to re-fetch them from the server to see if this is accurate.

so to be clear: that is a workaround for this bug, and if we fixed this bug, clients could remove that workaround?
>> @t3chguy mentioned to me that clients essentially assume all encrypted events have an edit and attempt to re-fetch them from the server to see if this is accurate.
>
> so to be clear: that is a workaround for this bug, and if we fixed this bug, clients could remove that workaround?

This is my understanding, yes. 👍 
Yes, once enough server uptake EW will gladly remove the check which asks the server for /relations on each and every thread response
See #14252.
@richvdh I guess we might be able to re-land this once MSC3925 becomes generally available? Otherwise, the support for that might be lacking as-is.
> @richvdh I guess we might be able to re-land this once [MSC3925](https://github.com/matrix-org/matrix-spec-proposals/pull/3925) becomes generally available?

That is my hope, yes!
> > @richvdh I guess we might be able to re-land this once [MSC3925](https://github.com/matrix-org/matrix-spec-proposals/pull/3925) becomes generally available?
> 
> That is my hope, yes!

We've now merged #15193, so we can probably try this again soon.
> > > @richvdh I guess we might be able to re-land this once [MSC3925](https://github.com/matrix-org/matrix-spec-proposals/pull/3925) becomes generally available?
> > 
> > 
> > That is my hope, yes!
> 
> We've now merged #15193, so we can probably try this again soon.

The changes in #15193 just shipped this week (in v1.79.0). I haven't seen any reports of regression due to it, but to be safe I think we should likely wait until v1.81 before making this change.

Resolving patch:
diff --git a/synapse/storage/databases/main/relations.py b/synapse/storage/databases/main/relations.py
--- a/synapse/storage/databases/main/relations.py
+++ b/synapse/storage/databases/main/relations.py
@@ -472,12 +472,11 @@ async def get_applicable_edits(
             the event will map to None.
         """
 
-        # We only allow edits for `m.room.message` events that have the same sender
-        # and event type. We can't assert these things during regular event auth so
-        # we have to do the checks post hoc.
+        # We only allow edits for events that have the same sender and event type.
+        # We can't assert these things during regular event auth so we have to do
+        # the checks post hoc.
 
-        # Fetches latest edit that has the same type and sender as the
-        # original, and is an `m.room.message`.
+        # Fetches latest edit that has the same type and sender as the original.
         if isinstance(self.database_engine, PostgresEngine):
             # The `DISTINCT ON` clause will pick the *first* row it encounters,
             # so ordering by origin server ts + event ID desc will ensure we get
@@ -493,7 +492,6 @@ async def get_applicable_edits(
                 WHERE
                     %s
                     AND relation_type = ?
-                    AND edit.type = 'm.room.message'
                 ORDER by original.event_id DESC, edit.origin_server_ts DESC, edit.event_id DESC
             """
         else:
@@ -512,7 +510,6 @@ async def get_applicable_edits(
                 WHERE
                     %s
                     AND relation_type = ?
-                    AND edit.type = 'm.room.message'
                 ORDER by edit.origin_server_ts, edit.event_id
             """
 


Test patch: