Repository: matrix-org/synapse

Issue:
`/messages` endpoint returns `end` token identical to `from` token
<!--

**THIS IS NOT A SUPPORT CHANNEL!**
**IF YOU HAVE SUPPORT QUESTIONS ABOUT RUNNING OR CONFIGURING YOUR OWN HOME SERVER**,
please ask in **#synapse:matrix.org** (using a matrix.org account if necessary)

If you want to report a security issue, please see https://matrix.org/security-disclosure-policy/

This is a bug report template. By following the instructions below and
filling out the sections with your information, you will help the us to get all
the necessary data to fix your issue.

You can also preview your report before submitting it. You may remove sections
that aren't relevant to your particular case.

Text between <!-- and --​> marks will be invisible in the report.

-->

### Description

<!-- Describe here the problem that you are experiencing -->

Goal is to fetch messages in a room backwards.

### Steps to reproduce

- My client first does a `/sync` to get the latest `next_batch` token
- I'm then using that token to pass it into the [messages endpoint](https://spec.matrix.org/v1.2/client-server-api/#get_matrixclientv3roomsroomidmessages) as `{ from: $next_batch, direction: "b", .. }`.
- The response from the server has an `end` field that's the same as the `next_batch` token that was used in the initial request, additionally to all room messages.
 
So it seems there's an issue with synapse not leaving the `end` token empty if it reaches the end of the timeline.

For reference, here's the raw response from synapse, where `start` and `end` are identical.

> matrix_sdk::http_client: Got response: Response { status: 200, version: HTTP/1.1, headers: {"transfer-encoding": "chunked", "server": "Synapse/1.53.0", "date": "Mon, 28 Feb 2022 14:45:37 GMT", "content-type": "application/json", "cache-control": "no-cache, no-store, must-revalidate", "access-control-allow-origin": "*", "access-control-allow-methods": "GET, HEAD, POST, PUT, DELETE, OPTIONS", "access-control-allow-headers": "X-Requested-With, Content-Type, Authorization, Date"}, body: b"{\"chunk\":[],\"start\":\"t1-915_1917_28_2_75_1_1_77_1\",\"end\":\"t1-915_1917_28_2_75_1_1_77_1\"}" }

<!--
Describe how what happens differs from what you expected.

If you can identify any relevant log snippets from _homeserver.log_, please include
those (please be careful to remove any personal or private data). Please surround them with
``` (three backticks, on a line on their own), so that they are formatted legibly.
-->

### Version information

<!-- IMPORTANT: please answer the following questions, to help us narrow down the problem -->

<!-- Was this issue identified on matrix.org or another homeserver? -->
- **Homeserver**: Local dev instance

If not matrix.org:

<!--
 What version of Synapse is running?

You can find the Synapse version with this command:

$ curl http://localhost:8008/_synapse/admin/v1/server_version

(You may need to replace `localhost:8008` if Synapse is not configured to
listen on that port.)
-->
- **Version**: 1.53.0

- **Install method**: docker hub
<!-- examples: package manager/git clone/pip  -->

- **Platform**: x86 64bit, docker container
<!--
Tell us about the environment in which your homeserver is operating
distro, hardware, if it's running in a vm/container, etc.
-->



Discussion:
I find it amazing that we don't have an issue for this already, but I don't think we do.

It was discussed at https://github.com/matrix-org/matrix-doc/issues/2251#issuecomment-932821615, but there doesn't appear to have been an issue filed.
I'd be happy to see a PR that fixes this.

Resolving patch:
diff --git a/synapse/handlers/pagination.py b/synapse/handlers/pagination.py
--- a/synapse/handlers/pagination.py
+++ b/synapse/handlers/pagination.py
@@ -515,14 +515,25 @@ async def get_messages(
 
             next_token = from_token.copy_and_replace(StreamKeyType.ROOM, next_key)
 
-        if events:
-            if event_filter:
-                events = await event_filter.filter(events)
+        # if no events are returned from pagination, that implies
+        # we have reached the end of the available events.
+        # In that case we do not return end, to tell the client
+        # there is no need for further queries.
+        if not events:
+            return {
+                "chunk": [],
+                "start": await from_token.to_string(self.store),
+            }
 
-            events = await filter_events_for_client(
-                self.storage, user_id, events, is_peeking=(member_event_id is None)
-            )
+        if event_filter:
+            events = await event_filter.filter(events)
+
+        events = await filter_events_for_client(
+            self.storage, user_id, events, is_peeking=(member_event_id is None)
+        )
 
+        # if after the filter applied there are no more events
+        # return immediately - but there might be more in next_token batch
         if not events:
             return {
                 "chunk": [],


Test patch: