Repository: matrix-org/synapse

Issue:
Accepting invites on matrix.org is no longer reliable.
At least on my account and Amandine's, it seems that it's very easy for something to go wrong with Synapse since ~November which means the client believes the accept has failed; either a timeout or an exception. You are then stuck in limbo unless you flush the cache, as per:

https://github.com/vector-im/riot-ios/issues/2153

Someone needs to hunt whatever happened with the logs on those rageshakes.


Discussion:
(i've also seen lots of other anecdotal evidence of this happening over the last ~month, and have to keep telling people to flush their cache).
I wouldn't be *entirely* surprised to discover this is related to some of the unfinished lazy-loading work, fwiw.
An instance of this from matthew on 1 Feb:

```
2019-02-01 23:36:13,013 - synapse.handlers.sync - 1402 - INFO - GET-3120426 - Membership changes in <room_id>: [$1549064172149644GTXRt:matrix.org (join)]
2019-02-01 23:36:13,333 - synapse.storage.roommember - 94 - INFO - GET-3120426 - Fetched room membership for ! <room_id> (2 users) in 9 ms
2019-02-01 23:36:13,341 - synapse.access.http.8098 - 302 - INFO - GET-3120426 - 194.78.14.74 - 8098 - {@matthew:matrix.org} Processed request: 2.950sec/0.001sec (0.076sec, 0.004sec) (0.009sec/0.266sec/11) 432B 200 "GET /_matri/client/r0/sync?etc"
```

Another on 14 Feb:

```
2019-02-14 11:34:48,750 - synapse.access.http.8098 - 233 - INFO - GET-6199846 - 80.250.101.234 - 8098 - Received request: GET /_matrix/client/r0/sync?filter=139&set_presence=offline&since=s840348390_754815119_586165_237988328_113322494_596735_18432257_11836808_31828&timeout=0
2019-02-14 11:34:48,751 - synapse.util.caches.response_cache - 151 - INFO - GET-6199846 - [sync]: no cached result for [(@matthew:matrix.org, 0, 's840348390_754815119_586165_237988328_113322494_596735_18432257_11836808_31828', '139', False, 'NBHCUILQXV')], calculating new one
2019-02-14 11:34:48,751 - synapse.handlers.sync - 909 - INFO - GET-6199846 - Calculating sync response for @matthew:matrix.org between StreamToken(room_key='s840348390', presence_key='754815119', typing_key='586165', receipt_key='237988328', account_data_key='113322494', push_rules_key='596735', to_device_key='18432257', device_list_key='11836808', groups_key='31828') and StreamToken(room_key='s840349675', presence_key=754817859, typing_key=586790, receipt_key=237988945, account_data_key=113322868, push_rules_key=596735, to_device_key=18432300, device_list_key=11836831, groups_key=31828)
2019-02-14 11:34:48,772 - synapse.handlers.sync - 1402 - INFO - GET-6199846 - Membership changes in !<room_id>: [$1550143986277856SDgkf:matrix.org (join)]
2019-02-14 11:34:49,919 - synapse.access.http.8098 - 302 - INFO - GET-6199846 - 80.250.101.234 - 8098 - {@matthew:matrix.org} Processed request: 1.167sec/0.002sec (0.100sec, 0.000sec) (0.057sec/0.247sec/21) 17582B 200 "GET /_matrix/client/r0/sync?filter=139&set_presence=offline&since=s840348390_754815119_586165_237988328_113322494_596735_18432257_11836808_31828&timeout=0 HTTP/1.1" "Riot/0.7.11 (iPhone; iOS 12.1.2; Scale/3.00)" [6 dbevts]

```
I've added some more debugging for this to matrix.org, so please let me know when you see it again
so I managed to reproduce an instance of this, and got some debug out of it:

```
2019-03-12 12:23:40,449 - synapse.handlers.sync - 912 - INFO - GET-15459791 - Calculating sync response for @richvdh:matrix.org between StreamToken(room_key='s872000229', presence_key='757242052', typing_key='832786', receipt_key='254199105', account_data_key='125873516', push_rules_key='643156', to_device_key='20082070', device_list_key='12533569', groups_key=
2019-03-12 12:23:40,480 - synapse.handlers.sync - 1414 - INFO - GET-15459791 - Membership changes in !dSodagOCcZCEPOhgZW:sw1v.org: [$1552393420415287uTeoF:matrix.org (join)]
2019-03-12 12:23:40,493 - synapse.handler.sync.4422_debug - 1451 - DEBUG - GET-15459791 - Previous membership for room !dSodagOCcZCEPOhgZW:sw1v.org with join: invite (event $1552393414489avAHM:sw1v.org)
2019-03-12 12:23:40,521 - synapse.storage.roommember - 94 - INFO - GET-15459791 - Fetched room membership for !dSodagOCcZCEPOhgZW:sw1v.org (3 users) in 2 ms
2019-03-12 12:23:40,530 - synapse.access.http.8088 - 302 - INFO - GET-15459791 - 2001:8b0:ba7a:e98f:3252:cbff:fee8:6489 - 8088 - {@richvdh:matrix.org} Processed request: 2.563sec/0.000sec (0.012sec, 0.004sec) (0.015sec/0.043sec/12) 453B 200 "GET /_matrix/client/r0/sync?filter=148&timeout=30000&since=s872000229_757242052_832786_254199105_125873516_643156_2008207
```

The "previous membership..." line comes from [here](https://github.com/matrix-org/synapse/blob/18b1a92162edbbab9c85666d48fb1734b31850b8/synapse/handlers/sync.py#L1449) and the fact that it is not then followed by a bunch of debug about newly joined rooms from [here](https://github.com/matrix-org/synapse/blob/18b1a92162edbbab9c85666d48fb1734b31850b8/synapse/handlers/sync.py#L1576) implies that the new room has not made it into `sync_result_builder.joined_room_ids` despite the join having landed with stream id 872000269.

`joined_room_ids` is built [here](https://github.com/matrix-org/synapse/blob/18b1a92162edbbab9c85666d48fb1734b31850b8/synapse/handlers/sync.py#L1823). So presumably `get_rooms_for_user_with_stream_ordering` (which has a cache) returned a stale result, though I'm unsure how this happened.

get_rooms_for_user_with_stream_ordering's cache should have been invalidated by this entry in the cache invalidation stream:

```
matrix=# select * from cache_invalidation_stream where stream_id=558594235;
 stream_id |  cache_func   |                        keys                        | invalidation_ts 
-----------+---------------+----------------------------------------------------+-----------------
 558594235 | cs_cache_fake | {!dSodagOCcZCEPOhgZW:sw1v.org,@richvdh:matrix.org} |   1552393420347
```

(invalidation_ts is 12:23:40.347).

However, this wasn't sent out by the master until 12:23:40.450:

```
2019-03-12 12:23:40,450 - synapse.replication.tcp.resource - 181 - INFO - replication_notifier-2492793 - Streaming: caches -> 558594235
```

The join event (stream id 872000269) was sent out at 12:23:40.426:

```
2019-03-12 12:23:40,426 - synapse.replication.tcp.resource - 181 - INFO - replication_notifier-2492793 - Streaming: events -> 872000271
```

It's therefore clear that there is a window where the worker can see the new join event (because its event stream id has been updated to include it), but has not yet received the cache invalidation.

The replication notifier makes no particular effort to preserve the ordering of `events` and `caches` notifications, so this isn't entirely surprising. I'm not really sure how best to fix it, though.

Resolving patch:
diff --git a/synapse/replication/slave/storage/events.py b/synapse/replication/slave/storage/events.py
--- a/synapse/replication/slave/storage/events.py
+++ b/synapse/replication/slave/storage/events.py
@@ -16,7 +16,10 @@
 import logging
 
 from synapse.api.constants import EventTypes
-from synapse.replication.tcp.streams.events import EventsStreamEventRow
+from synapse.replication.tcp.streams.events import (
+    EventsStreamCurrentStateRow,
+    EventsStreamEventRow,
+)
 from synapse.storage.event_federation import EventFederationWorkerStore
 from synapse.storage.event_push_actions import EventPushActionsWorkerStore
 from synapse.storage.events_worker import EventsWorkerStore
@@ -80,14 +83,7 @@ def process_replication_rows(self, stream_name, token, rows):
         if stream_name == "events":
             self._stream_id_gen.advance(token)
             for row in rows:
-                if row.type != EventsStreamEventRow.TypeId:
-                    continue
-                data = row.data
-                self.invalidate_caches_for_event(
-                    token, data.event_id, data.room_id, data.type, data.state_key,
-                    data.redacts,
-                    backfilled=False,
-                )
+                self._process_event_stream_row(token, row)
         elif stream_name == "backfill":
             self._backfill_id_gen.advance(-token)
             for row in rows:
@@ -100,6 +96,23 @@ def process_replication_rows(self, stream_name, token, rows):
             stream_name, token, rows
         )
 
+    def _process_event_stream_row(self, token, row):
+        data = row.data
+
+        if row.type == EventsStreamEventRow.TypeId:
+            self.invalidate_caches_for_event(
+                token, data.event_id, data.room_id, data.type, data.state_key,
+                data.redacts,
+                backfilled=False,
+            )
+        elif row.type == EventsStreamCurrentStateRow.TypeId:
+            if data.type == EventTypes.Member:
+                self.get_rooms_for_user_with_stream_ordering.invalidate(
+                    (data.state_key, ),
+                )
+        else:
+            raise Exception("Unknown events stream row type %s" % (row.type, ))
+
     def invalidate_caches_for_event(self, stream_ordering, event_id, room_id,
                                     etype, state_key, redacts, backfilled):
         self._invalidate_get_event_cache(event_id)
diff --git a/synapse/storage/_base.py b/synapse/storage/_base.py
--- a/synapse/storage/_base.py
+++ b/synapse/storage/_base.py
@@ -1355,11 +1355,6 @@ def _invalidate_state_caches(self, room_id, members_changed):
             members_changed (iterable[str]): The user_ids of members that have
                 changed
         """
-        for member in members_changed:
-            self._attempt_to_invalidate_cache(
-                "get_rooms_for_user_with_stream_ordering", (member,),
-            )
-
         for host in set(get_domain_from_id(u) for u in members_changed):
             self._attempt_to_invalidate_cache(
                 "is_host_joined", (room_id, host,),
diff --git a/synapse/storage/events.py b/synapse/storage/events.py
--- a/synapse/storage/events.py
+++ b/synapse/storage/events.py
@@ -79,7 +79,7 @@ def encode_json(json_object):
     """
     out = frozendict_json_encoder.encode(json_object)
     if isinstance(out, bytes):
-        out = out.decode('utf8')
+        out = out.decode("utf8")
     return out
 
 
@@ -813,9 +813,10 @@ def _persist_events_txn(
         """
         all_events_and_contexts = events_and_contexts
 
+        min_stream_order = events_and_contexts[0][0].internal_metadata.stream_ordering
         max_stream_order = events_and_contexts[-1][0].internal_metadata.stream_ordering
 
-        self._update_current_state_txn(txn, state_delta_for_room, max_stream_order)
+        self._update_current_state_txn(txn, state_delta_for_room, min_stream_order)
 
         self._update_forward_extremities_txn(
             txn,
@@ -890,7 +891,7 @@ def _persist_events_txn(
             backfilled=backfilled,
         )
 
-    def _update_current_state_txn(self, txn, state_delta_by_room, max_stream_order):
+    def _update_current_state_txn(self, txn, state_delta_by_room, stream_id):
         for room_id, current_state_tuple in iteritems(state_delta_by_room):
             to_delete, to_insert = current_state_tuple
 
@@ -899,6 +900,12 @@ def _update_current_state_txn(self, txn, state_delta_by_room, max_stream_order):
             # that we can use it to calculate the `prev_event_id`. (This
             # allows us to not have to pull out the existing state
             # unnecessarily).
+            #
+            # The stream_id for the update is chosen to be the minimum of the stream_ids
+            # for the batch of the events that we are persisting; that means we do not
+            # end up in a situation where workers see events before the
+            # current_state_delta updates.
+            #
             sql = """
                 INSERT INTO current_state_delta_stream
                 (stream_id, room_id, type, state_key, event_id, prev_event_id)
@@ -911,7 +918,7 @@ def _update_current_state_txn(self, txn, state_delta_by_room, max_stream_order):
                 sql,
                 (
                     (
-                        max_stream_order,
+                        stream_id,
                         room_id,
                         etype,
                         state_key,
@@ -929,7 +936,7 @@ def _update_current_state_txn(self, txn, state_delta_by_room, max_stream_order):
                 sql,
                 (
                     (
-                        max_stream_order,
+                        stream_id,
                         room_id,
                         etype,
                         state_key,
@@ -970,7 +977,7 @@ def _update_current_state_txn(self, txn, state_delta_by_room, max_stream_order):
             txn.call_after(
                 self._curr_state_delta_stream_cache.entity_has_changed,
                 room_id,
-                max_stream_order,
+                stream_id,
             )
 
             # Invalidate the various caches
@@ -986,6 +993,11 @@ def _update_current_state_txn(self, txn, state_delta_by_room, max_stream_order):
                 if ev_type == EventTypes.Member
             )
 
+            for member in members_changed:
+                txn.call_after(
+                    self.get_rooms_for_user_with_stream_ordering.invalidate, (member,)
+                )
+
             self._invalidate_state_caches_and_stream(txn, room_id, members_changed)
 
     def _update_forward_extremities_txn(


Test patch:
diff --git a/tests/replication/slave/storage/_base.py b/tests/replication/slave/storage/_base.py
--- a/tests/replication/slave/storage/_base.py
+++ b/tests/replication/slave/storage/_base.py
@@ -56,7 +56,9 @@ def prepare(self, reactor, clock, hs):
         client = client_factory.buildProtocol(None)
 
         client.makeConnection(FakeTransport(server, reactor))
-        server.makeConnection(FakeTransport(client, reactor))
+
+        self.server_to_client_transport = FakeTransport(client, reactor)
+        server.makeConnection(self.server_to_client_transport)
 
     def replicate(self):
         """Tell the master side of replication that something has happened, and then
@@ -69,6 +71,24 @@ def check(self, method, args, expected_result=None):
         master_result = self.get_success(getattr(self.master_store, method)(*args))
         slaved_result = self.get_success(getattr(self.slaved_store, method)(*args))
         if expected_result is not None:
-            self.assertEqual(master_result, expected_result)
-            self.assertEqual(slaved_result, expected_result)
-        self.assertEqual(master_result, slaved_result)
+            self.assertEqual(
+                master_result,
+                expected_result,
+                "Expected master result to be %r but was %r" % (
+                    expected_result, master_result
+                ),
+            )
+            self.assertEqual(
+                slaved_result,
+                expected_result,
+                "Expected slave result to be %r but was %r" % (
+                    expected_result, slaved_result
+                ),
+            )
+        self.assertEqual(
+            master_result,
+            slaved_result,
+            "Slave result %r does not match master result %r" % (
+                slaved_result, master_result
+            ),
+        )
diff --git a/tests/replication/slave/storage/test_events.py b/tests/replication/slave/storage/test_events.py
--- a/tests/replication/slave/storage/test_events.py
+++ b/tests/replication/slave/storage/test_events.py
@@ -11,11 +11,13 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+import logging
 
 from canonicaljson import encode_canonical_json
 
 from synapse.events import FrozenEvent, _EventInternalMetadata
 from synapse.events.snapshot import EventContext
+from synapse.handlers.room import RoomEventSource
 from synapse.replication.slave.storage.events import SlavedEventStore
 from synapse.storage.roommember import RoomsForUser
 
@@ -26,6 +28,8 @@
 OUTLIER = {"outlier": True}
 ROOM_ID = "!room:blue"
 
+logger = logging.getLogger(__name__)
+
 
 def dict_equals(self, other):
     me = encode_canonical_json(self.get_pdu_json())
@@ -172,18 +176,142 @@ def test_push_actions_for_user(self):
             {"highlight_count": 1, "notify_count": 2},
         )
 
+    def test_get_rooms_for_user_with_stream_ordering(self):
+        """Check that the cache on get_rooms_for_user_with_stream_ordering is invalidated
+        by rows in the events stream
+        """
+        self.persist(type="m.room.create", key="", creator=USER_ID)
+        self.persist(type="m.room.member", key=USER_ID, membership="join")
+        self.replicate()
+        self.check("get_rooms_for_user_with_stream_ordering", (USER_ID_2,), set())
+
+        j2 = self.persist(
+            type="m.room.member", sender=USER_ID_2, key=USER_ID_2, membership="join"
+        )
+        self.replicate()
+        self.check(
+            "get_rooms_for_user_with_stream_ordering",
+            (USER_ID_2,),
+            {(ROOM_ID, j2.internal_metadata.stream_ordering)},
+        )
+
+    def test_get_rooms_for_user_with_stream_ordering_with_multi_event_persist(self):
+        """Check that current_state invalidation happens correctly with multiple events
+        in the persistence batch.
+
+        This test attempts to reproduce a race condition between the event persistence
+        loop and a worker-based Sync handler.
+
+        The problem occurred when the master persisted several events in one batch. It
+        only updates the current_state at the end of each batch, so the obvious thing
+        to do is then to issue a current_state_delta stream update corresponding to the
+        last stream_id in the batch.
+
+        However, that raises the possibility that a worker will see the replication
+        notification for a join event before the current_state caches are invalidated.
+
+        The test involves:
+         * creating a join and a message event for a user, and persisting them in the
+           same batch
+
+         * controlling the replication stream so that updates are sent gradually
+
+         * between each bunch of replication updates, check that we see a consistent
+           snapshot of the state.
+        """
+        self.persist(type="m.room.create", key="", creator=USER_ID)
+        self.persist(type="m.room.member", key=USER_ID, membership="join")
+        self.replicate()
+        self.check("get_rooms_for_user_with_stream_ordering", (USER_ID_2,), set())
+
+        # limit the replication rate
+        repl_transport = self.server_to_client_transport
+        repl_transport.autoflush = False
+
+        # build the join and message events and persist them in the same batch.
+        logger.info("----- build test events ------")
+        j2, j2ctx = self.build_event(
+            type="m.room.member", sender=USER_ID_2, key=USER_ID_2, membership="join"
+        )
+        msg, msgctx = self.build_event()
+        self.get_success(self.master_store.persist_events([
+            (j2, j2ctx),
+            (msg, msgctx),
+        ]))
+        self.replicate()
+
+        event_source = RoomEventSource(self.hs)
+        event_source.store = self.slaved_store
+        current_token = self.get_success(event_source.get_current_key())
+
+        # gradually stream out the replication
+        while repl_transport.buffer:
+            logger.info("------ flush ------")
+            repl_transport.flush(30)
+            self.pump(0)
+
+            prev_token = current_token
+            current_token = self.get_success(event_source.get_current_key())
+
+            # attempt to replicate the behaviour of the sync handler.
+            #
+            # First, we get a list of the rooms we are joined to
+            joined_rooms = self.get_success(
+                self.slaved_store.get_rooms_for_user_with_stream_ordering(
+                    USER_ID_2,
+                ),
+            )
+
+            # Then, we get a list of the events since the last sync
+            membership_changes = self.get_success(
+                self.slaved_store.get_membership_changes_for_user(
+                    USER_ID_2, prev_token, current_token,
+                )
+            )
+
+            logger.info(
+                "%s->%s: joined_rooms=%r membership_changes=%r",
+                prev_token,
+                current_token,
+                joined_rooms,
+                membership_changes,
+            )
+
+            # the membership change is only any use to us if the room is in the
+            # joined_rooms list.
+            if membership_changes:
+                self.assertEqual(
+                    joined_rooms, {(ROOM_ID, j2.internal_metadata.stream_ordering)}
+                )
+
     event_id = 0
 
-    def persist(
+    def persist(self, backfill=False, **kwargs):
+        """
+        Returns:
+            synapse.events.FrozenEvent: The event that was persisted.
+        """
+        event, context = self.build_event(**kwargs)
+
+        if backfill:
+            self.get_success(
+                self.master_store.persist_events([(event, context)], backfilled=True)
+            )
+        else:
+            self.get_success(
+                self.master_store.persist_event(event, context)
+            )
+
+        return event
+
+    def build_event(
         self,
         sender=USER_ID,
         room_id=ROOM_ID,
-        type={},
+        type="m.room.message",
         key=None,
         internal={},
         state=None,
-        reset_state=False,
-        backfill=False,
         depth=None,
         prev_events=[],
         auth_events=[],
@@ -192,10 +320,7 @@ def persist(
         push_actions=[],
         **content
     ):
-        """
-        Returns:
-            synapse.events.FrozenEvent: The event that was persisted.
-        """
+
         if depth is None:
             depth = self.event_id
 
@@ -234,23 +359,11 @@ def persist(
             )
         else:
             state_handler = self.hs.get_state_handler()
-            context = self.get_success(state_handler.compute_event_context(event))
+            context = self.get_success(state_handler.compute_event_context(
+                event
+            ))
 
         self.master_store.add_push_actions_to_staging(
             event.event_id, {user_id: actions for user_id, actions in push_actions}
         )
-
-        ordering = None
-        if backfill:
-            self.get_success(
-                self.master_store.persist_events([(event, context)], backfilled=True)
-            )
-        else:
-            ordering, _ = self.get_success(
-                self.master_store.persist_event(event, context)
-            )
-
-        if ordering:
-            event.internal_metadata.stream_ordering = ordering
-
-        return event
+        return event, context
diff --git a/tests/server.py b/tests/server.py
--- a/tests/server.py
+++ b/tests/server.py
@@ -365,6 +365,7 @@ class FakeTransport(object):
     disconnected = False
     buffer = attr.ib(default=b'')
     producer = attr.ib(default=None)
+    autoflush = attr.ib(default=True)
 
     def getPeer(self):
         return None
@@ -415,31 +416,44 @@ def _produce():
     def write(self, byt):
         self.buffer = self.buffer + byt
 
-        def _write():
-            if not self.buffer:
-                # nothing to do. Don't write empty buffers: it upsets the
-                # TLSMemoryBIOProtocol
-                return
-
-            if self.disconnected:
-                return
-            logger.info("%s->%s: %s", self._protocol, self.other, self.buffer)
-
-            if getattr(self.other, "transport") is not None:
-                try:
-                    self.other.dataReceived(self.buffer)
-                    self.buffer = b""
-                except Exception as e:
-                    logger.warning("Exception writing to protocol: %s", e)
-                return
-
-            self._reactor.callLater(0.0, _write)
-
         # always actually do the write asynchronously. Some protocols (notably the
         # TLSMemoryBIOProtocol) get very confused if a read comes back while they are
         # still doing a write. Doing a callLater here breaks the cycle.
-        self._reactor.callLater(0.0, _write)
+        if self.autoflush:
+            self._reactor.callLater(0.0, self.flush)
 
     def writeSequence(self, seq):
         for x in seq:
             self.write(x)
+
+    def flush(self, maxbytes=None):
+        if not self.buffer:
+            # nothing to do. Don't write empty buffers: it upsets the
+            # TLSMemoryBIOProtocol
+            return
+
+        if self.disconnected:
+            return
+
+        if getattr(self.other, "transport") is None:
+            # the other has no transport yet; reschedule
+            if self.autoflush:
+                self._reactor.callLater(0.0, self.flush)
+            return
+
+        if maxbytes is not None:
+            to_write = self.buffer[:maxbytes]
+        else:
+            to_write = self.buffer
+
+        logger.info("%s->%s: %s", self._protocol, self.other, to_write)
+
+        try:
+            self.other.dataReceived(to_write)
+        except Exception as e:
+            logger.warning("Exception writing to protocol: %s", e)
+            return
+
+        self.buffer = self.buffer[len(to_write):]
+        if self.buffer and self.autoflush:
+            self._reactor.callLater(0.0, self.flush)