Repository: matrix-org/synapse

Issue:
Exception in pusher on_new_receipts:  Cursor needed to be reset
occasionally `/_matrix/client/r0/rooms/{roomId}/read_markers` spits out this exception:

```
2019-11-22 13:44:21,146 - synapse.push.pusherpool - 204 - ERROR - POST-162567 - Exception in pusher on_new_receipts
Traceback (most recent call last):
  File "/home/path/to/server/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: <generator object ReceiptsWorkerStore.get_all_updated_receipts.<locals>.get_all_updated_receipts_txn.<locals>.<genexpr> at 0x7faca013cca8>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/path/to/server/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: <generator object ReceiptsWorkerStore.get_all_updated_receipts.<locals>.get_all_updated_receipts_txn.<locals>.<genexpr> at 0x7faca013cca8>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/path/to/server/lib/python3.6/site-packages/synapse/push/pusherpool.py", line 196, in on_new_receipts
    users_affected = set([r[3] for r in updated_receipts])
  File "/home/path/to/server/lib/python3.6/site-packages/synapse/push/pusherpool.py", line 196, in <listcomp>
    users_affected = set([r[3] for r in updated_receipts])
  File "/home/path/to/server/lib/python3.6/site-packages/synapse/storage/data_stores/main/receipts.py", line 283, in <genexpr>
    return (r[0:5] + (json.loads(r[5]),) for r in txn)
sqlite3.InterfaceError: Cursor needed to be reset because of commit/rollback and can no longer be fetched from.
```

The symptoms are presumably that people get pushes for messages they have already read.

I think it's because `get_all_updated_receipts` returns a generator expression which refers to a database cursor, which refers to a database connection, which might get re-used before the results are read.


Discussion:

Resolving patch:
diff --git a/synapse/storage/_base.py b/synapse/storage/_base.py
--- a/synapse/storage/_base.py
+++ b/synapse/storage/_base.py
@@ -409,16 +409,15 @@ def _new_transaction(
             i = 0
             N = 5
             while True:
+                cursor = LoggingTransaction(
+                    conn.cursor(),
+                    name,
+                    self.database_engine,
+                    after_callbacks,
+                    exception_callbacks,
+                )
                 try:
-                    txn = conn.cursor()
-                    txn = LoggingTransaction(
-                        txn,
-                        name,
-                        self.database_engine,
-                        after_callbacks,
-                        exception_callbacks,
-                    )
-                    r = func(txn, *args, **kwargs)
+                    r = func(cursor, *args, **kwargs)
                     conn.commit()
                     return r
                 except self.database_engine.module.OperationalError as e:
@@ -456,6 +455,40 @@ def _new_transaction(
                                 )
                             continue
                     raise
+                finally:
+                    # we're either about to retry with a new cursor, or we're about to
+                    # release the connection. Once we release the connection, it could
+                    # get used for another query, which might do a conn.rollback().
+                    #
+                    # In the latter case, even though that probably wouldn't affect the
+                    # results of this transaction, python's sqlite will reset all
+                    # statements on the connection [1], which will make our cursor
+                    # invalid [2].
+                    #
+                    # In any case, continuing to read rows after commit()ing seems
+                    # dubious from the PoV of ACID transactional semantics
+                    # (sqlite explicitly says that once you commit, you may see rows
+                    # from subsequent updates.)
+                    #
+                    # In psycopg2, cursors are essentially a client-side fabrication -
+                    # all the data is transferred to the client side when the statement
+                    # finishes executing - so in theory we could go on streaming results
+                    # from the cursor, but attempting to do so would make us
+                    # incompatible with sqlite, so let's make sure we're not doing that
+                    # by closing the cursor.
+                    #
+                    # (*named* cursors in psycopg2 are different and are proper server-
+                    # side things, but (a) we don't use them and (b) they are implicitly
+                    # closed by ending the transaction anyway.)
+                    #
+                    # In short, if we haven't finished with the cursor yet, that's a
+                    # problem waiting to bite us.
+                    #
+                    # TL;DR: we're done with the cursor, so we can close it.
+                    #
+                    # [1]: https://github.com/python/cpython/blob/v3.8.0/Modules/_sqlite/connection.c#L465
+                    # [2]: https://github.com/python/cpython/blob/v3.8.0/Modules/_sqlite/cursor.c#L236
+                    cursor.close()
         except Exception as e:
             logger.debug("[TXN FAIL] {%s} %s", name, e)
             raise
diff --git a/synapse/storage/data_stores/main/receipts.py b/synapse/storage/data_stores/main/receipts.py
--- a/synapse/storage/data_stores/main/receipts.py
+++ b/synapse/storage/data_stores/main/receipts.py
@@ -280,7 +280,7 @@ def get_all_updated_receipts_txn(txn):
                 args.append(limit)
             txn.execute(sql, args)
 
-            return (r[0:5] + (json.loads(r[5]),) for r in txn)
+            return list(r[0:5] + (json.loads(r[5]),) for r in txn)
 
         return self.runInteraction(
             "get_all_updated_receipts", get_all_updated_receipts_txn


Test patch: