Repository: matrix-org/synapse

Issue:
`/messages` can't paginate from the beginning of the room forwards - MSC3567
<!--

**THIS IS NOT A SUPPORT CHANNEL!**
**IF YOU HAVE SUPPORT QUESTIONS ABOUT RUNNING OR CONFIGURING YOUR OWN HOME SERVER**,
please ask in **#synapse:matrix.org** (using a matrix.org account if necessary)

If you want to report a security issue, please see https://matrix.org/security-disclosure-policy/

This is a bug report template. By following the instructions below and
filling out the sections with your information, you will help the us to get all
the necessary data to fix your issue.

You can also preview your report before submitting it. You may remove sections
that aren't relevant to your particular case.

Text between <!-- and --​> marks will be invisible in the report.

-->

As discovered in https://github.com/matrix-org/matrix-spec/issues/601#issuecomment-1081269883

### Description

In [MSC3567](https://github.com/matrix-org/matrix-spec-proposals/pull/3567),

> The `from` field on the [`/_matrix/client/v3/rooms/{roomId}/messages`](https://spec.matrix.org/v1.1/client-server-api/#get_matrixclientv3roomsroomidmessages)
becomes optional. If it is not provided, the homeserver shall return a list of
messages from the first or last (per the value of the `dir` parameter) visible
event in the room history for the requesting user.

But Synapse doesn't seem to behave this way:

 - ✅ `GET /_matrix/client/v3/rooms/{roomId}/messages?dir=b`: Gets the most recent events in the room
 - ❌ `GET /_matrix/client/v3/rooms/{roomId}/messages?dir=f`: Gets no messages
    - Expected to get the first messages in the room (oldest)

### Steps to reproduce

 1. `GET /_matrix/client/v3/rooms/{roomId}/messages?dir=f`
 1. Notice that the response doesn't include any messages and appears to just be the most recent position in the timeline looking forward
    ```
    {
        "chunk": [],
        "start": "s962_0_0_0_0_0_0_0_0",
        "end": "s962_0_0_0_0_0_0_0_0"
    }
    ```

<!--
Describe how what happens differs from what you expected.

If you can identify any relevant log snippets from _homeserver.log_, please include
those (please be careful to remove any personal or private data). Please surround them with
``` (three backticks, on a line on their own), so that they are formatted legibly.
-->

### Version information

<!-- IMPORTANT: please answer the following questions, to help us narrow down the problem -->

<!-- Was this issue identified on matrix.org or another homeserver? -->
- **Homeserver**: local dev homeserver

If not matrix.org: 

<!--
 What version of Synapse is running?

You can find the Synapse version with this command:

$ curl http://localhost:8008/_synapse/admin/v1/server_version

(You may need to replace `localhost:8008` if Synapse is not configured to
listen on that port.)
-->
- **Version**: Most recent `develop` (163fd686b5db10eab2c70855dedef23204c6f18c)

- **Install method**: [Local development setup](https://github.com/matrix-org/synapse/tree/fd1e7d0fc2c65e2cebc7c7b8d1d4d49b05d99168#synapse-development)
<!-- examples: package manager/git clone/pip  -->

- **Platform**: macOS 12.3
<!--
Tell us about the environment in which your homeserver is operating
distro, hardware, if it's running in a vm/container, etc.
-->

`/messages` can't paginate from the beginning of the room forwards - MSC3567
<!--

**THIS IS NOT A SUPPORT CHANNEL!**
**IF YOU HAVE SUPPORT QUESTIONS ABOUT RUNNING OR CONFIGURING YOUR OWN HOME SERVER**,
please ask in **#synapse:matrix.org** (using a matrix.org account if necessary)

If you want to report a security issue, please see https://matrix.org/security-disclosure-policy/

This is a bug report template. By following the instructions below and
filling out the sections with your information, you will help the us to get all
the necessary data to fix your issue.

You can also preview your report before submitting it. You may remove sections
that aren't relevant to your particular case.

Text between <!-- and --​> marks will be invisible in the report.

-->

As discovered in https://github.com/matrix-org/matrix-spec/issues/601#issuecomment-1081269883

### Description

In [MSC3567](https://github.com/matrix-org/matrix-spec-proposals/pull/3567),

> The `from` field on the [`/_matrix/client/v3/rooms/{roomId}/messages`](https://spec.matrix.org/v1.1/client-server-api/#get_matrixclientv3roomsroomidmessages)
becomes optional. If it is not provided, the homeserver shall return a list of
messages from the first or last (per the value of the `dir` parameter) visible
event in the room history for the requesting user.

But Synapse doesn't seem to behave this way:

 - ✅ `GET /_matrix/client/v3/rooms/{roomId}/messages?dir=b`: Gets the most recent events in the room
 - ❌ `GET /_matrix/client/v3/rooms/{roomId}/messages?dir=f`: Gets no messages
    - Expected to get the first messages in the room (oldest)

### Steps to reproduce

 1. `GET /_matrix/client/v3/rooms/{roomId}/messages?dir=f`
 1. Notice that the response doesn't include any messages and appears to just be the most recent position in the timeline looking forward
    ```
    {
        "chunk": [],
        "start": "s962_0_0_0_0_0_0_0_0",
        "end": "s962_0_0_0_0_0_0_0_0"
    }
    ```

<!--
Describe how what happens differs from what you expected.

If you can identify any relevant log snippets from _homeserver.log_, please include
those (please be careful to remove any personal or private data). Please surround them with
``` (three backticks, on a line on their own), so that they are formatted legibly.
-->

### Version information

<!-- IMPORTANT: please answer the following questions, to help us narrow down the problem -->

<!-- Was this issue identified on matrix.org or another homeserver? -->
- **Homeserver**: local dev homeserver

If not matrix.org: 

<!--
 What version of Synapse is running?

You can find the Synapse version with this command:

$ curl http://localhost:8008/_synapse/admin/v1/server_version

(You may need to replace `localhost:8008` if Synapse is not configured to
listen on that port.)
-->
- **Version**: Most recent `develop` (163fd686b5db10eab2c70855dedef23204c6f18c)

- **Install method**: [Local development setup](https://github.com/matrix-org/synapse/tree/fd1e7d0fc2c65e2cebc7c7b8d1d4d49b05d99168#synapse-development)
<!-- examples: package manager/git clone/pip  -->

- **Platform**: macOS 12.3
<!--
Tell us about the environment in which your homeserver is operating
distro, hardware, if it's running in a vm/container, etc.
-->



Discussion:

Resolving patch:
diff --git a/synapse/handlers/pagination.py b/synapse/handlers/pagination.py
--- a/synapse/handlers/pagination.py
+++ b/synapse/handlers/pagination.py
@@ -448,6 +448,12 @@ async def get_messages(
 
         if pagin_config.from_token:
             from_token = pagin_config.from_token
+        elif pagin_config.direction == "f":
+            from_token = (
+                await self.hs.get_event_sources().get_start_token_for_pagination(
+                    room_id
+                )
+            )
         else:
             from_token = (
                 await self.hs.get_event_sources().get_current_token_for_pagination(
diff --git a/synapse/streams/events.py b/synapse/streams/events.py
--- a/synapse/streams/events.py
+++ b/synapse/streams/events.py
@@ -73,6 +73,19 @@ def get_current_token(self) -> StreamToken:
         )
         return token
 
+    @trace
+    async def get_start_token_for_pagination(self, room_id: str) -> StreamToken:
+        """Get the start token for a given room to be used to paginate
+        events.
+
+        The returned token does not have the current values for fields other
+        than `room`, since they are not used during pagination.
+
+        Returns:
+            The start token for pagination.
+        """
+        return StreamToken.START
+
     @trace
     async def get_current_token_for_pagination(self, room_id: str) -> StreamToken:
         """Get the current token for a given room to be used to paginate


Test patch:
diff --git a/tests/rest/admin/test_room.py b/tests/rest/admin/test_room.py
--- a/tests/rest/admin/test_room.py
+++ b/tests/rest/admin/test_room.py
@@ -1857,6 +1857,46 @@ def test_stream_token_is_accepted_for_fwd_pagianation(self) -> None:
         self.assertIn("chunk", channel.json_body)
         self.assertIn("end", channel.json_body)
 
+    def test_room_messages_backward(self) -> None:
+        """Test room messages can be retrieved by an admin that isn't in the room."""
+        latest_event_id = self.helper.send(
+            self.room_id, body="message 1", tok=self.user_tok
+        )["event_id"]
+
+        # Check that we get the first and second message when querying /messages.
+        channel = self.make_request(
+            "GET",
+            "/_synapse/admin/v1/rooms/%s/messages?dir=b" % (self.room_id,),
+            access_token=self.admin_user_tok,
+        )
+        self.assertEqual(channel.code, 200, channel.json_body)
+
+        chunk = channel.json_body["chunk"]
+        self.assertEqual(len(chunk), 6, [event["content"] for event in chunk])
+
+        # in backwards, this is the first event
+        self.assertEqual(chunk[0]["event_id"], latest_event_id)
+
+    def test_room_messages_forward(self) -> None:
+        """Test room messages can be retrieved by an admin that isn't in the room."""
+        latest_event_id = self.helper.send(
+            self.room_id, body="message 1", tok=self.user_tok
+        )["event_id"]
+
+        # Check that we get the first and second message when querying /messages.
+        channel = self.make_request(
+            "GET",
+            "/_synapse/admin/v1/rooms/%s/messages?dir=f" % (self.room_id,),
+            access_token=self.admin_user_tok,
+        )
+        self.assertEqual(channel.code, 200, channel.json_body)
+
+        chunk = channel.json_body["chunk"]
+        self.assertEqual(len(chunk), 6, [event["content"] for event in chunk])
+
+        # in forward, this is the last event
+        self.assertEqual(chunk[5]["event_id"], latest_event_id)
+
     def test_room_messages_purge(self) -> None:
         """Test room messages can be retrieved by an admin that isn't in the room."""
         store = self.hs.get_datastores().main