Repository: matrix-org/synapse

Issue:
Hierarchy pagination is limited to 50 items per space
`_matrix/client/unstable/org.matrix.msc2946/rooms/{roomId}/hierarchy` returns a next_batch string for spaces with more than 50 children as expected.
Pparsing the pagination token then only returns 11 rooms in my case, even though there should be much more. The Space in question has 135 space children.
Different spaces return different seemingly random numbers after the second call, but never the actual number.

**To Reproduce**
Steps to reproduce the behavior:

- call `_matrix/client/unstable/org.matrix.msc2946/rooms({roomId}/hierarchy` for a space with more than 50 children, ideally more than 100

- repeat the call and parse the next_batch string

- wrong number of spaces are returned

**Expected behavior**

- return actual number of spaces and if >50 another next_batch string

tested with v1.43.0



Discussion:
Are you always seeing the same incorrect number of rooms returned, or does it differ between pagination attempts? And is it the same subset of rooms returned every time?

What's the structure of the space with 135 children? ie. how many rooms and subspaces are there in each subspace?
Is this a public space? If so, can you provide the room ID?
> Are you always seeing the same incorrect number of rooms returned, or does it differ between pagination attempts? 

Yes, I always get the same incorrect number of rooms returned for a specific space (but the number varies from one space to another). 
> And is it the same subset of rooms returned every time?

Yes, always the same rooms.
> What's the structure of the space with 135 children? ie. how many rooms and subspaces are there in each subspace?

The room structure is as follows:
```
parent space
  | children space 1
  | children space 2
  | children space 3
  ...
  | children space 134
  | children space 135
```

no further sub-spaces

> Is this a public space? If so, can you provide the room ID?

I've created a public dummy space with 120 space children here: `!iUCIUjePntDkDGSnjb:matrix.org`
In this case the second API call with the pagination token only returns one room.

edit: just to verify, `GET https://matrix.org/_matrix/client/r0/rooms/!iUCIUjePntDkDGSnjb:matrix.org/state`
has 120 events of the type`m.space.child`

I suspect what's happening here is that only the first 50 children of each subspace are being returned.
@marcel-klasse does this line up with what you're seeing?
that's possible. I have just added 5 children to the first subspace like this:
```
parent space
  | children space 1
     | child 1
     | child 2
     | child 3
     | child 4
     | child 5
  | children space 2
  | children space 3
  ...
  | children space 119
  | children space 120
```
which is now returning 4 rooms after the second call.


I'm seeing this exact issue when trying to migrate a list of rooms from another service over to a matrix server space.

The issue appears to be that the 50 room limit is hard coded [on line 52 in `handlers/room_summary.py`](https://github.com/matrix-org/synapse/blob/1035663833a76196c3e3ba425fd6500c5420bbe2/synapse/handlers/room_summary.py#L52) with the request cap limiting logic appearing [on line 653 in `handlers/room_summary.py`](https://github.com/matrix-org/synapse/blob/1035663833a76196c3e3ba425fd6500c5420bbe2/synapse/handlers/room_summary.py#L653).

The limiting logic appears to come in from PR `Federation API for Space summary` (#9652) where the `max_children` check against `MAX_ROOMS_PER_SPACE` is first introduced into `_summarize_local_room`.

I've tested this on a local test VM. When I set this variable (`MAX_ROOMS_PER_SPACE`) to 100, it works as expected (I have 90 rooms and got 90 rooms).

The easiest solution would seem to be expose this as a configuration variable so individual servers can override the limit as needed, but I figured I'd drop my research in for the synapse team to consider.

Thanks!
I've reworded the issue since it isn't really broken, but working as designed. Part of the problem with this code right now is that it is trying to support both the legacy `/spaces` API and the newer `/hierarchy` API so it gets a bit messy. We could likely up the limit of 50 higher, but it likely makes more sense to ensure we can handle arbitrary limits. I would need to look deeper into the code to see how easy that is.

Resolving patch:
diff --git a/synapse/handlers/room_summary.py b/synapse/handlers/room_summary.py
--- a/synapse/handlers/room_summary.py
+++ b/synapse/handlers/room_summary.py
@@ -153,6 +153,9 @@ async def get_space_summary(
         rooms_result: List[JsonDict] = []
         events_result: List[JsonDict] = []
 
+        if max_rooms_per_space is None or max_rooms_per_space > MAX_ROOMS_PER_SPACE:
+            max_rooms_per_space = MAX_ROOMS_PER_SPACE
+
         while room_queue and len(rooms_result) < MAX_ROOMS:
             queue_entry = room_queue.popleft()
             room_id = queue_entry.room_id
@@ -167,7 +170,7 @@ async def get_space_summary(
             # The client-specified max_rooms_per_space limit doesn't apply to the
             # room_id specified in the request, so we ignore it if this is the
             # first room we are processing.
-            max_children = max_rooms_per_space if processed_rooms else None
+            max_children = max_rooms_per_space if processed_rooms else MAX_ROOMS
 
             if is_in_room:
                 room_entry = await self._summarize_local_room(
@@ -395,7 +398,7 @@ async def _get_room_hierarchy(
                     None,
                     room_id,
                     suggested_only,
-                    # TODO Handle max children.
+                    # Do not limit the maximum children.
                     max_children=None,
                 )
 
@@ -525,6 +528,10 @@ async def federation_space_summary(
         rooms_result: List[JsonDict] = []
         events_result: List[JsonDict] = []
 
+        # Set a limit on the number of rooms to return.
+        if max_rooms_per_space is None or max_rooms_per_space > MAX_ROOMS_PER_SPACE:
+            max_rooms_per_space = MAX_ROOMS_PER_SPACE
+
         while room_queue and len(rooms_result) < MAX_ROOMS:
             room_id = room_queue.popleft()
             if room_id in processed_rooms:
@@ -583,7 +590,9 @@ async def get_federation_hierarchy(
 
         # Iterate through each child and potentially add it, but not its children,
         # to the response.
-        for child_room in root_room_entry.children_state_events:
+        for child_room in itertools.islice(
+            root_room_entry.children_state_events, MAX_ROOMS_PER_SPACE
+        ):
             room_id = child_room.get("state_key")
             assert isinstance(room_id, str)
             # If the room is unknown, skip it.
@@ -633,8 +642,8 @@ async def _summarize_local_room(
             suggested_only: True if only suggested children should be returned.
                 Otherwise, all children are returned.
             max_children:
-                The maximum number of children rooms to include. This is capped
-                to a server-set limit.
+                The maximum number of children rooms to include. A value of None
+                means no limit.
 
         Returns:
             A room entry if the room should be returned. None, otherwise.
@@ -656,8 +665,13 @@ async def _summarize_local_room(
             # we only care about suggested children
             child_events = filter(_is_suggested_child_event, child_events)
 
-        if max_children is None or max_children > MAX_ROOMS_PER_SPACE:
-            max_children = MAX_ROOMS_PER_SPACE
+        # TODO max_children is legacy code for the /spaces endpoint.
+        if max_children is not None:
+            child_iter: Iterable[EventBase] = itertools.islice(
+                child_events, max_children
+            )
+        else:
+            child_iter = child_events
 
         stripped_events: List[JsonDict] = [
             {
@@ -668,7 +682,7 @@ async def _summarize_local_room(
                 "sender": e.sender,
                 "origin_server_ts": e.origin_server_ts,
             }
-            for e in itertools.islice(child_events, max_children)
+            for e in child_iter
         ]
         return _RoomEntry(room_id, room_entry, stripped_events)
 


Test patch:
diff --git a/tests/handlers/test_room_summary.py b/tests/handlers/test_room_summary.py
--- a/tests/handlers/test_room_summary.py
+++ b/tests/handlers/test_room_summary.py
@@ -253,6 +253,38 @@ def test_simple_space(self):
         )
         self._assert_hierarchy(result, expected)
 
+    def test_large_space(self):
+        """Test a space with a large number of rooms."""
+        rooms = [self.room]
+        # Make at least 51 rooms that are part of the space.
+        for _ in range(55):
+            room = self.helper.create_room_as(self.user, tok=self.token)
+            self._add_child(self.space, room, self.token)
+            rooms.append(room)
+
+        result = self.get_success(self.handler.get_space_summary(self.user, self.space))
+        # The spaces result should have the space and the first 50 rooms in it,
+        # along with the links from space -> room for those 50 rooms.
+        expected = [(self.space, rooms[:50])] + [(room, []) for room in rooms[:49]]
+        self._assert_rooms(result, expected)
+
+        # The result should have the space and the rooms in it, along with the links
+        # from space -> room.
+        expected = [(self.space, rooms)] + [(room, []) for room in rooms]
+
+        # Make two requests to fully paginate the results.
+        result = self.get_success(
+            self.handler.get_room_hierarchy(create_requester(self.user), self.space)
+        )
+        result2 = self.get_success(
+            self.handler.get_room_hierarchy(
+                create_requester(self.user), self.space, from_token=result["next_batch"]
+            )
+        )
+        # Combine the results.
+        result["rooms"] += result2["rooms"]
+        self._assert_hierarchy(result, expected)
+
     def test_visibility(self):
         """A user not in a space cannot inspect it."""
         user2 = self.register_user("user2", "pass")