Repository: matrix-org/synapse

Issue:
Duplicate `txn.call_after` calls when transactions are retried
Synapse provides the ability for database transactions to defer some processing until after the transaction succeeds or fails, via the [`Cursor.call_after` and `Cursor.call_on_exception` methods](https://github.com/matrix-org/synapse/blob/release-v1.53/synapse/storage/database.py#L241-L259). These methods append the given callback to an internal list which [gets run once the transaction is done](https://github.com/matrix-org/synapse/blob/release-v1.53/synapse/storage/database.py#L755-L759).

When a transaction fails with a SerializationFailure, Synapse will retry it up to 5 times (inside [`new_transaction`](https://github.com/matrix-org/synapse/blob/release-v1.53/synapse/storage/database.py#L514)). Although a new Cursor is created for each attempt, the same `after_callbacks` list is reused. This means that if a transaction fails, then succeeds on the next attempt, the `after_callbacks` from the previous failed attempt will also run.

These callbacks are mostly used for cache invalidation, so this is probably okay?

-------------------

Note that [MultiWriterIdGenerator needs either the `call_after` or `call_on_exception` callbacks from failed transaction attempts to run](https://github.com/matrix-org/synapse/blob/release-v1.53/synapse/storage/util/id_generators.py#L542-L543), otherwise it wouldn't know when the ids handed out to failed transactions are done. If we were to clear out `after_callbacks` in between attempts, MultiWriterIdGenerator would break.


Discussion:
Does this need discussion of what the proper behavior should be?
Probably? Let's stick it in the needs-discussion bucket.
Conclusion from meeting: let's note this behavior in in the `call_*` function docstrings, but leave the behavior unchanged.

There *are* legitimate uses for both modalities, so if we wanted to address it we'd likely want to extend the API surface to indicate whether duplicate callbacks should be enqueued or not. But we'll hold off on that until we have a concrete motivation to make the change.

Resolving patch:
diff --git a/synapse/storage/database.py b/synapse/storage/database.py
--- a/synapse/storage/database.py
+++ b/synapse/storage/database.py
@@ -241,9 +241,17 @@ def __init__(
         self.exception_callbacks = exception_callbacks
 
     def call_after(self, callback: Callable[..., object], *args: Any, **kwargs: Any):
-        """Call the given callback on the main twisted thread after the
-        transaction has finished. Used to invalidate the caches on the
-        correct thread.
+        """Call the given callback on the main twisted thread after the transaction has
+        finished.
+
+        Mostly used to invalidate the caches on the correct thread.
+
+        Note that transactions may be retried a few times if they encounter database
+        errors such as serialization failures. Callbacks given to `call_after`
+        will accumulate across transaction attempts and will _all_ be called once a
+        transaction attempt succeeds, regardless of whether previous transaction
+        attempts failed. Otherwise, if all transaction attempts fail, all
+        `call_on_exception` callbacks will be run instead.
         """
         # if self.after_callbacks is None, that means that whatever constructed the
         # LoggingTransaction isn't expecting there to be any callbacks; assert that
@@ -254,6 +262,15 @@ def call_after(self, callback: Callable[..., object], *args: Any, **kwargs: Any)
     def call_on_exception(
         self, callback: Callable[..., object], *args: Any, **kwargs: Any
     ):
+        """Call the given callback on the main twisted thread after the transaction has
+        failed.
+
+        Note that transactions may be retried a few times if they encounter database
+        errors such as serialization failures. Callbacks given to `call_on_exception`
+        will accumulate across transaction attempts and will _all_ be called once the
+        final transaction attempt fails. No `call_on_exception` callbacks will be run
+        if any transaction attempt succeeds.
+        """
         # if self.exception_callbacks is None, that means that whatever constructed the
         # LoggingTransaction isn't expecting there to be any callbacks; assert that
         # is not the case.


Test patch: