Repository: matrix-org/synapse

Issue:
MAU transaction errors
There's hundreds of these in my logs. I do have #3830 and #3831 included on my server, but I don't think either would cause this kind of failure?

```
homeserver_1 - 2018-09-12 22:21:44,478 - synapse.storage._base - 254 - WARNING - [TXN OPERROR] {upsert_monthly_active_user-2ed258} could not serialize access due to concurrent update
 4/5
homeserver_1 - 2018-09-12 22:21:44,478 - synapse.storage._base - 254 - WARNING - [TXN OPERROR] {upsert_monthly_active_user-2ed264} could not serialize access due to concurrent update
 0/5
homeserver_1 - 2018-09-12 22:21:44,479 - synapse.storage._base - 254 - WARNING - [TXN OPERROR] {upsert_monthly_active_user-2ed253} could not serialize access due to concurrent update
 5/5
homeserver_1 - 2018-09-12 22:21:44,479 - synapse.storage._base - 254 - WARNING - [TXN OPERROR] {upsert_monthly_active_user-2ed269} could not serialize access due to concurrent update
 0/5
homeserver_1 - 2018-09-12 22:21:44,479 - synapse.storage._base - 254 - WARNING - [TXN OPERROR] {upsert_monthly_active_user-2ed26e} could not serialize access due to concurrent update
 0/5
homeserver_1 - 2018-09-12 22:21:44,479 - synapse.storage._base - 254 - WARNING - [TXN OPERROR] {upsert_monthly_active_user-2ed265} could not serialize access due to concurrent update
 0/5
homeserver_1 - 2018-09-12 22:21:44,479 - synapse.storage._base - 254 - WARNING - [TXN OPERROR] {upsert_monthly_active_user-2ed268} could not serialize access due to concurrent update
 0/5
homeserver_1 - 2018-09-12 22:21:44,479 - synapse.storage._base - 254 - WARNING - [TXN OPERROR] {upsert_monthly_active_user-2ed26b} could not serialize access due to concurrent update
 0/5
homeserver_1 - 2018-09-12 22:21:44,479 - synapse.storage._base - 254 - WARNING - [TXN OPERROR] {upsert_monthly_active_user-2ed271} could not serialize access due to concurrent update
 0/5
homeserver_1 - 2018-09-12 22:21:44,480 - synapse.storage._base - 254 - WARNING - [TXN OPERROR] {upsert_monthly_active_user-2ed26d} could not serialize access due to concurrent update
 0/5
homeserver_1 - 2018-09-12 22:21:44,480 - synapse.storage._base - 254 - WARNING - [TXN OPERROR] {upsert_monthly_active_user-2ed24f} could not serialize access due to concurrent update
 5/5
homeserver_1 - 2018-09-12 22:21:44,480 - synapse.storage._base - 254 - WARNING - [TXN OPERROR] {upsert_monthly_active_user-2ed26a} could not serialize access due to concurrent update
 0/5
homeserver_1 - 2018-09-12 22:21:44,481 - synapse.storage._base - 254 - WARNING - [TXN OPERROR] {upsert_monthly_active_user-2ed26c} could not serialize access due to concurrent update
 0/5
homeserver_1 - 2018-09-12 22:21:44,481 - synapse.storage._base - 254 - WARNING - [TXN OPERROR] {upsert_monthly_active_user-2ed270} could not serialize access due to concurrent update
 0/5
homeserver_1 - 2018-09-12 22:21:44,481 - synapse.storage._base - 254 - WARNING - [TXN OPERROR] {upsert_monthly_active_user-2ed245} could not serialize access due to concurrent update
 5/5
homeserver_1 - 2018-09-12 22:21:44,482 - synapse.storage._base - 254 - WARNING - [TXN OPERROR] {upsert_monthly_active_user-2ed26f} could not serialize access due to concurrent update
 0/5
```


Discussion:
Agreed the PRs ref'd shouldn't make any difference, I've not seen this in the wild as yet.
It might be one of those bugs that shows up with massive traffic. In a matter of 5 days I've had the metric climb to 8000 users with no signs of stopping, and I'm not sure if MAU has been tested at this level of traffic yet.
FWIW, those errors are "non fatal", and basically mean we tried to edit the same row from two different transactions, and so one gets error'ed with that exception. They should get retried.

OTOH, we probably shouldn't be updating so often that this is happening....
Does it still happen? am wondering if this only a problem the first time the feature is enabled and lots of ‘new’ users are added. 
Having grepped the logs moments ago, it still happens. It seems to happen in batches though and isn't constant, but that might be a function of my load.
Its quite possible that its happening in batches because we're only updating if they're older than X minutes, and so they all become due for updating at the same time
Okay, I'll batch up the requests to prevent this mode of failure
So I tried to implement batched upsert much like we do for user_ips - the difference being that in the general case this info is used for control logic immediately after the upsert.

This means that it is necessary to check on outstanding upserts before making any reads of the table, this doesn't work where the output is dependent on whether the queued item will be an insert or update (get_monthly_active_count()) - the only way I can see to do this would be to store a version of the table in memory. (You can't use something like user_last_seen_monthly_active because it also needs to incorporate the batch queued items)

It also means that tests that previous had logic like 'create new mau, call get_monthly_active_count() to check it works', need arbitrary reactor advances to account for the batching, which is possible but seems like something that will continually confuse writers of tests in the area.

Without infra readily available to store the whole table and manage invalidation and also mindful of return of investment in doing so given the nature of the bug, reconsidered the table lock option. In almost all cases the table will be small (typically < 100 and almost certainly < 10,000), and updates are restricted to an hourly granularity (at the time of writing) it seems reasonable to allow locking.

Will resolve to implement locking and ask Travis to check on vacuum stats for the table on his HS since it will be the largest install using this behaviour (it is disabled on matrix.org).

For reference my abandoned branch lives here https://github.com/matrix-org/synapse/tree/neilj/batch-unsert-mau-users

Resolving patch:
diff --git a/synapse/storage/monthly_active_users.py b/synapse/storage/monthly_active_users.py
--- a/synapse/storage/monthly_active_users.py
+++ b/synapse/storage/monthly_active_users.py
@@ -172,6 +172,10 @@ def upsert_monthly_active_user(self, user_id):
             Deferred[bool]: True if a new entry was created, False if an
                 existing one was updated.
         """
+        # Am consciously deciding to lock the table on the basis that is ought
+        # never be a big table and alternative approaches (batching multiple
+        # upserts into a single txn) introduced a lot of extra complexity.
+        # See https://github.com/matrix-org/synapse/issues/3854 for more
         is_insert = yield self._simple_upsert(
             desc="upsert_monthly_active_user",
             table="monthly_active_users",
@@ -181,7 +185,6 @@ def upsert_monthly_active_user(self, user_id):
             values={
                 "timestamp": int(self._clock.time_msec()),
             },
-            lock=False,
         )
         if is_insert:
             self.user_last_seen_monthly_active.invalidate((user_id,))


Test patch: