Repository: matrix-org/synapse

Issue:
A room lost all its forward extremities, leading to "no create event in auth events"
this actually goes back to March; sadly this is the first time I've had a chance to write it up in a github issue.

Somehow a room (on the `mozilla.org` instance) ended up with no `event_forward_extremities` in the database at all, despite still having a number of local members.

The room was manually "fixed" by inserting a new row into `event_forward_extremities` and restarting synapse.

The only thing that looks odd is this entry in the postgres error log:

```
2020-03-04 15:14:33 UTC:172.20.37.63(58809):8e71e000-3607-11ea-8fb7-3d4a5344b740-live@8e71e000-3607-11ea-8fb7-3d4a5344b740-live:[14889]:ERROR: duplicate key value violates unique constraint "events_pkey"
2020-03-04 15:14:33 UTC:172.20.37.63(58809):8e71e000-3607-11ea-8fb7-3d4a5344b740-live@8e71e000-3607-11ea-8fb7-3d4a5344b740-live:[14889]:DETAIL: Key (stream_ordering)=(1691477) already exists.
2020-03-04 15:14:33 UTC:172.20.37.63(58809):8e71e000-3607-11ea-8fb7-3d4a5344b740-live@8e71e000-3607-11ea-8fb7-3d4a5344b740-live:[14889]:STATEMENT: INSERT INTO events (contains_url, depth, event_id, origin_server_ts, outlier, processed, received_ts, room_id, sender, stream_ordering, topological_ordering, type) VALUES(false, 5557, '$iVFN6H4LReXpOG_N0ygHoatqCO61LmyzNj5B9D6x-8M', 1583334868334, false, true, 1583334873818, '!<room>:mozilla.org', '@<sender>:mozilla.org', 1691477, 5557, 'm.room.message')
```



Discussion:
After the above error, synapse will have removed all traces of the failed event from the database, and retried.

The eventual event, as persisted, was:

```
 stream_ordering | topological_ordering |                   event_id                   |      type      |       room_id       | content | unrecognized_keys | processed | outlier | depth | origin_server_ts |  received_ts  |        sender        | contains_url 
-----------------+----------------------+----------------------------------------------+----------------+---------------------+---------+-------------------+-----------+---------+-------+------------------+---------------+----------------------+--------------
         1691479 |                 5557 | $iVFN6H4LReXpOG_N0ygHoatqCO61LmyzNj5B9D6x-8M | m.room.message | !<room>:mozilla.org |         |                   | t         | f       |  5557 |    1583334868334 | 1583334875837 | @<sender>:mozilla.org | f
```

The slightly later `received_ts` and `stream_ordering` are consistent with this being the result of a retry. The original `stream_ordering` (1691477) is assigned to the previous event in the same room (which happens to be a redaction, not that that should matter).

So, questions:
 * why was 1691477 used for two events?
 * why did we end up with an empty extremity set?




> why did we end up with an empty extremity set?

well, this part is easily answered. When we get an integrity error and retry, `_persist_events_txn` is called with `delete_existing=True`, which makes it delete all previous traces of the previous attempt at https://github.com/matrix-org/synapse/blob/v1.17.0/synapse/storage/data_stores/main/events.py#L396. However, this happens *after* updating the forward extremities at https://github.com/matrix-org/synapse/blob/v1.17.0/synapse/storage/data_stores/main/events.py#L372. Net effect is that we end up with no forward extremities.
ok, I *think* this was caused by running two master instances at once. Don't do that, people.

Resolving patch:
diff --git a/synapse/storage/data_stores/main/events.py b/synapse/storage/data_stores/main/events.py
--- a/synapse/storage/data_stores/main/events.py
+++ b/synapse/storage/data_stores/main/events.py
@@ -17,7 +17,6 @@
 import itertools
 import logging
 from collections import OrderedDict, namedtuple
-from functools import wraps
 from typing import TYPE_CHECKING, Dict, Iterable, List, Tuple
 
 import attr
@@ -69,27 +68,6 @@ def encode_json(json_object):
 _EventCacheEntry = namedtuple("_EventCacheEntry", ("event", "redacted_event"))
 
 
-def _retry_on_integrity_error(func):
-    """Wraps a database function so that it gets retried on IntegrityError,
-    with `delete_existing=True` passed in.
-
-    Args:
-        func: function that returns a Deferred and accepts a `delete_existing` arg
-    """
-
-    @wraps(func)
-    @defer.inlineCallbacks
-    def f(self, *args, **kwargs):
-        try:
-            res = yield func(self, *args, delete_existing=False, **kwargs)
-        except self.database_engine.module.IntegrityError:
-            logger.exception("IntegrityError, retrying.")
-            res = yield func(self, *args, delete_existing=True, **kwargs)
-        return res
-
-    return f
-
-
 @attr.s(slots=True)
 class DeltaState:
     """Deltas to use to update the `current_state_events` table.
@@ -134,7 +112,6 @@ def __init__(self, hs: "HomeServer", db: Database, main_data_store: "DataStore")
             hs.config.worker.writers.events == hs.get_instance_name()
         ), "Can only instantiate EventsStore on master"
 
-    @_retry_on_integrity_error
     @defer.inlineCallbacks
     def _persist_events_and_state_updates(
         self,
@@ -143,7 +120,6 @@ def _persist_events_and_state_updates(
         state_delta_for_room: Dict[str, DeltaState],
         new_forward_extremeties: Dict[str, List[str]],
         backfilled: bool = False,
-        delete_existing: bool = False,
     ):
         """Persist a set of events alongside updates to the current state and
         forward extremities tables.
@@ -157,7 +133,6 @@ def _persist_events_and_state_updates(
             new_forward_extremities: Map from room_id to list of event IDs
                 that are the new forward extremities of the room.
             backfilled
-            delete_existing
 
         Returns:
             Deferred: resolves when the events have been persisted
@@ -197,7 +172,6 @@ def _persist_events_and_state_updates(
                 self._persist_events_txn,
                 events_and_contexts=events_and_contexts,
                 backfilled=backfilled,
-                delete_existing=delete_existing,
                 state_delta_for_room=state_delta_for_room,
                 new_forward_extremeties=new_forward_extremeties,
             )
@@ -341,7 +315,6 @@ def _persist_events_txn(
         txn: LoggingTransaction,
         events_and_contexts: List[Tuple[EventBase, EventContext]],
         backfilled: bool,
-        delete_existing: bool = False,
         state_delta_for_room: Dict[str, DeltaState] = {},
         new_forward_extremeties: Dict[str, List[str]] = {},
     ):
@@ -393,13 +366,6 @@ def _persist_events_txn(
         # From this point onwards the events are only events that we haven't
         # seen before.
 
-        if delete_existing:
-            # For paranoia reasons, we go and delete all the existing entries
-            # for these events so we can reinsert them.
-            # This gets around any problems with some tables already having
-            # entries.
-            self._delete_existing_rows_txn(txn, events_and_contexts=events_and_contexts)
-
         self._store_event_txn(txn, events_and_contexts=events_and_contexts)
 
         # Insert into event_to_state_groups.
@@ -797,39 +763,6 @@ def _update_outliers_txn(self, txn, events_and_contexts):
 
         return [ec for ec in events_and_contexts if ec[0] not in to_remove]
 
-    @classmethod
-    def _delete_existing_rows_txn(cls, txn, events_and_contexts):
-        if not events_and_contexts:
-            # nothing to do here
-            return
-
-        logger.info("Deleting existing")
-
-        for table in (
-            "events",
-            "event_auth",
-            "event_json",
-            "event_edges",
-            "event_forward_extremities",
-            "event_reference_hashes",
-            "event_search",
-            "event_to_state_groups",
-            "state_events",
-            "rejections",
-            "redactions",
-            "room_memberships",
-        ):
-            txn.executemany(
-                "DELETE FROM %s WHERE event_id = ?" % (table,),
-                [(ev.event_id,) for ev, _ in events_and_contexts],
-            )
-
-        for table in ("event_push_actions",):
-            txn.executemany(
-                "DELETE FROM %s WHERE room_id = ? AND event_id = ?" % (table,),
-                [(ev.room_id, ev.event_id) for ev, _ in events_and_contexts],
-            )
-
     def _store_event_txn(self, txn, events_and_contexts):
         """Insert new events into the event and event_json tables
 


Test patch: