Repository: matrix-org/synapse

Issue:
Faster joins: potential infinite loop in resync 
Resyncing the state for a room follows the following algorithm:
 * Find a batch of events with partial state. We choose the first 100 events ordered by arrival time. For each event in the batch:
   * Skip if we have `prev_events` with partial state
   * Request the state at any completely missing `prev_events`
   * Resolve the state across the `prev_events` and persist

It's possible for that to get into a loop, where we pull out the first 100 events, fail to make progress on any of them, and repeat.

https://github.com/matrix-org/synapse/blob/7c6b2204d143550d81e5bf9612c4e69fe0866b4c/synapse/handlers/federation_event.py#L527-L540

part of https://github.com/matrix-org/synapse/issues/12646


Discussion:
I don't think that this is actually a situation that can happen, even for one event, never mind 100.

To hit this "still has partial state" case, note that we must be processing an event `E` where:
 * we have one or more `prev_events` as non-outlier events in the database, and
 * one or more of those `prev_events` (say `E'`) have partial state.
 
... which implies that `E'` was received *after* `E` (otherwise we would already have de-partial-stated it). Which in turn implies that, at the time we received `E`, we didn't have `E'` - ie, we were persisting an event without having previously persisted its `prev_events`. (Note that we pick events to de-partial-state in order of `stream_ordering` - ie, order that events were *persisted* - rather than received timestamp.)

But when we persist an event (as a non-outlier) without having its `prev_events`, then we always request the state at that point in the DAG - ie, `E` will not have partial state.


... except, if we receive an event as an outlier, we don't give it a new stream_ordering when we de-outlier it. So, if we receive 100 outliers, *then* an event early in the room history, *then* de-outlier the outliers, then we may hit this. Will test.

Resolving patch:
diff --git a/synapse/handlers/federation_event.py b/synapse/handlers/federation_event.py
--- a/synapse/handlers/federation_event.py
+++ b/synapse/handlers/federation_event.py
@@ -543,15 +543,15 @@ async def update_state_for_partial_state_event(
             )
             if context.partial_state:
                 # this can happen if some or all of the event's prev_events still have
-                # partial state - ie, an event has an earlier stream_ordering than one
-                # or more of its prev_events, so we de-partial-state it before its
-                # prev_events.
+                # partial state. We were careful to only pick events from the db without
+                # partial-state prev events, so that implies that a prev event has
+                # been persisted (with partial state) since we did the query.
                 #
-                # TODO(faster_joins): we probably need to be more intelligent, and
-                #    exclude partial-state prev_events from consideration
-                #    https://github.com/matrix-org/synapse/issues/13001
+                # So, let's just ignore `event` for now; when we re-run the db query
+                # we should instead get its partial-state prev event, which we will
+                # de-partial-state, and then come back to event.
                 logger.warning(
-                    "%s still has partial state: can't de-partial-state it yet",
+                    "%s still has prev_events with partial state: can't de-partial-state it yet",
                     event.event_id,
                 )
                 return
diff --git a/synapse/storage/databases/main/events_worker.py b/synapse/storage/databases/main/events_worker.py
--- a/synapse/storage/databases/main/events_worker.py
+++ b/synapse/storage/databases/main/events_worker.py
@@ -2110,11 +2110,29 @@ async def get_partial_state_events_batch(self, room_id: str) -> List[str]:
     def _get_partial_state_events_batch_txn(
         txn: LoggingTransaction, room_id: str
     ) -> List[str]:
+        # we want to work through the events from oldest to newest, so
+        # we only want events whose prev_events do *not* have partial state - hence
+        # the 'NOT EXISTS' clause in the below.
+        #
+        # This is necessary because ordering by stream ordering isn't quite enough
+        # to ensure that we work from oldest to newest event (in particular,
+        # if an event is initially persisted as an outlier and later de-outliered,
+        # it can end up with a lower stream_ordering than its prev_events).
+        #
+        # Typically this means we'll only return one event per batch, but that's
+        # hard to do much about.
+        #
+        # See also: https://github.com/matrix-org/synapse/issues/13001
         txn.execute(
             """
             SELECT event_id FROM partial_state_events AS pse
                 JOIN events USING (event_id)
-            WHERE pse.room_id = ?
+            WHERE pse.room_id = ? AND
+               NOT EXISTS(
+                  SELECT 1 FROM event_edges AS ee
+                     JOIN partial_state_events AS prev_pse ON (prev_pse.event_id=ee.prev_event_id)
+                     WHERE ee.event_id=pse.event_id
+               )
             ORDER BY events.stream_ordering
             LIMIT 100
             """,


Test patch: