Repository: matrix-org/synapse

Issue:
Synapse refreshes device lists over and over
My homesever logs this sort of thing, over and over and over:

```
2021-04-22 14:14:06,870 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745113} [matrix.org] Completed request: 200 OK in 0.08 secs - GET matrix://matrix.org/_matrix/federation/v1/user/devices/%40USER%3Amatrix.org
2021-04-22 14:14:06,891 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745114} [inso.ovh] Completed request: 200 OK in 0.02 secs - GET matrix://inso.ovh/_matrix/federation/v1/user/devices/%40USER%3Ainso.ovh
2021-04-22 14:14:06,963 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745124} [matrix.org] Completed request: 200 OK in 0.04 secs - GET matrix://matrix.org/_matrix/federation/v1/user/devices/%40USER%3Amatrix.org
2021-04-22 14:14:06,995 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745130} [madduck.net] Completed request: 200 OK in 0.03 secs - GET matrix://madduck.net/_matrix/federation/v1/user/devices/%40USER%3Amadduck.net
2021-04-22 14:14:07,290 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745132} [owdo.org] Completed request: 200 OK in 0.29 secs - GET matrix://owdo.org/_matrix/federation/v1/user/devices/%40USER%3Aowdo.org
2021-04-22 14:14:07,378 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745134} [cody.to] Completed request: 200 OK in 0.09 secs - GET matrix://cody.to/_matrix/federation/v1/user/devices/%40USER%3Acody.to
2021-04-22 14:14:07,508 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745137} [kb1rd.net] Completed request: 200 OK in 0.13 secs - GET matrix://kb1rd.net/_matrix/federation/v1/user/devices/%40USER%3Akb1rd.net
2021-04-22 14:14:07,531 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745148} [matrix.org] Completed request: 200 OK in 0.02 secs - GET matrix://matrix.org/_matrix/federation/v1/user/devices/%40USER%3Amatrix.org
2021-04-22 14:14:07,638 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745150} [ufopensource.club] Completed request: 200 OK in 0.11 secs - GET matrix://ufopensource.club/_matrix/federation/v1/user/devices/%40USER%3Aufopensource.club
2021-04-22 14:14:07,662 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745154} [armylane.com] Completed request: 200 OK in 0.02 secs - GET matrix://armylane.com/_matrix/federation/v1/user/devices/%40USER%3Aarmylane.com
2021-04-22 14:14:07,885 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745156} [rozekengineering.com] Completed request: 200 OK in 0.22 secs - GET matrix://rozekengineering.com/_matrix/federation/v1/user/devices/%40USER%3Arozekengineering.com
2021-04-22 14:14:07,936 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745159} [matrix.tele2.ee] Completed request: 200 OK in 0.05 secs - GET matrix://matrix.tele2.ee/_matrix/federation/v1/user/devices/%40USER%3Amatrix.tele2.ee
2021-04-22 14:14:07,970 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745161} [strahlungsfrei.de] Completed request: 200 OK in 0.03 secs - GET matrix://strahlungsfrei.de/_matrix/federation/v1/user/devices/%40USER%3Astrahlungsfrei.de
2021-04-22 14:14:08,009 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745162} [qtrnn.io] Completed request: 200 OK in 0.04 secs - GET matrix://qtrnn.io/_matrix/federation/v1/user/devices/%40USER%3Aqtrnn.io
2021-04-22 14:14:08,048 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745163} [matrix.org] Completed request: 200 OK in 0.04 secs - GET matrix://matrix.org/_matrix/federation/v1/user/devices/%40USER%3Amatrix.org
2021-04-22 14:14:08,159 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745164} [vangrinsven.org] Completed request: 200 OK in 0.11 secs - GET matrix://vangrinsven.org/_matrix/federation/v1/user/devices/%40USER%3Avangrinsven.org
2021-04-22 14:14:08,187 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745165} [mleuser.de] Completed request: 200 OK in 0.03 secs - GET matrix://mleuser.de/_matrix/federation/v1/user/devices/%40USER%3Amleuser.de
2021-04-22 14:14:08,229 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745166} [mrks.network] Completed request: 200 OK in 0.04 secs - GET matrix://mrks.network/_matrix/federation/v1/user/devices/%40USER%3Amrks.network
2021-04-22 14:14:08,271 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745167} [nilsson.link] Completed request: 200 OK in 0.04 secs - GET matrix://nilsson.link/_matrix/federation/v1/user/devices/%40USER%3Anilsson.link
2021-04-22 14:14:08,296 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745168} [matrix.org] Completed request: 200 OK in 0.02 secs - GET matrix://matrix.org/_matrix/federation/v1/user/devices/%40USER%3Amatrix.org
2021-04-22 14:14:08,335 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745171} [das-labor.org] Completed request: 200 OK in 0.04 secs - GET matrix://das-labor.org/_matrix/federation/v1/user/devices/%40USER%3Adas-labor.org
2021-04-22 14:14:08,362 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745172} [luschmar.ch] Completed request: 200 OK in 0.03 secs - GET matrix://luschmar.ch/_matrix/federation/v1/user/devices/%40USER%3Aluschmar.ch
2021-04-22 14:14:08,424 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745177} [quranic.network] Completed request: 200 OK in 0.04 secs - GET matrix://quranic.network/_matrix/federation/v1/user/devices/%40USER%3Aquranic.network
2021-04-22 14:14:08,462 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745181} [maralorn.de] Completed request: 200 OK in 0.04 secs - GET matrix://maralorn.de/_matrix/federation/v1/user/devices/%40USER%3Amaralorn.de
2021-04-22 14:14:08,488 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745185} [matrix.org] Completed request: 200 OK in 0.03 secs - GET matrix://matrix.org/_matrix/federation/v1/user/devices/%40USER%3Amatrix.org
2021-04-22 14:14:08,533 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745186} [matrix.org] Completed request: 200 OK in 0.04 secs - GET matrix://matrix.org/_matrix/federation/v1/user/devices/%40USER%3Amatrix.org
2021-04-22 14:14:08,705 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745190} [humble.host] Completed request: 200 OK in 0.17 secs - GET matrix://humble.host/_matrix/federation/v1/user/devices/%40USER%3Ahumble.host
2021-04-22 14:14:08,800 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745199} [matrix.org] Completed request: 200 OK in 0.05 secs - GET matrix://matrix.org/_matrix/federation/v1/user/devices/%40USER%3Amatrix.org
2021-04-22 14:14:08,821 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745203} [matrix.org] Completed request: 200 OK in 0.02 secs - GET matrix://matrix.org/_matrix/federation/v1/user/devices/%40USER%3Amatrix.org
2021-04-22 14:14:08,846 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745205} [matrix.org] Completed request: 200 OK in 0.02 secs - GET matrix://matrix.org/_matrix/federation/v1/user/devices/%40USER%3Amatrix.org
2021-04-22 14:14:08,896 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745206} [amorgan.xyz] Completed request: 200 OK in 0.05 secs - GET matrix://amorgan.xyz/_matrix/federation/v1/user/devices/%40USER%3Aamorgan.xyz
2021-04-22 14:14:08,947 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745210} [imbitbu.de] Completed request: 200 OK in 0.05 secs - GET matrix://imbitbu.de/_matrix/federation/v1/user/devices/%40USER%3Aimbitbu.de
2021-04-22 14:14:08,961 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745211} [nopl.de] Completed request: 200 OK in 0.01 secs - GET matrix://nopl.de/_matrix/federation/v1/user/devices/%40USER%3Anopl.de
2021-04-22 14:14:09,034 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745213} [matrix.itschulz.com] Completed request: 200 OK in 0.05 secs - GET matrix://matrix.itschulz.com/_matrix/federation/v1/user/devices/%40USER%3Amatrix.itschulz.com
2021-04-22 14:14:09,059 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745216} [b3web.de] Completed request: 200 OK in 0.02 secs - GET matrix://b3web.de/_matrix/federation/v1/user/devices/%40USER%3Ab3web.de
2021-04-22 14:14:09,525 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745219} [matrix.nck1236.com] Completed request: 200 OK in 0.47 secs - GET matrix://matrix.nck1236.com/_matrix/federation/v1/user/devices/%40USER%3Amatrix.nck1236.com
2021-04-22 14:14:15,789 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745220} [librem.one] Completed request: 200 OK in 6.26 secs - GET matrix://librem.one/_matrix/federation/v1/user/devices/%40USER%3Alibrem.one
2021-04-22 14:14:15,887 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745226} [iddqd.social] Completed request: 200 OK in 0.09 secs - GET matrix://iddqd.social/_matrix/federation/v1/user/devices/%40USER%3Aiddqd.social
2021-04-22 14:14:16,061 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745230} [zner0l.de] Completed request: 200 OK in 0.17 secs - GET matrix://zner0l.de/_matrix/federation/v1/user/devices/%40USER%3Azner0l.de
2021-04-22 14:14:16,091 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745240} [freiburg.social] Completed request: 200 OK in 0.03 secs - GET matrix://freiburg.social/_matrix/federation/v1/user/devices/%40USER%3Afreiburg.social
2021-04-22 14:14:16,141 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745245} [truf-kin.com] Completed request: 200 OK in 0.05 secs - GET matrix://truf-kin.com/_matrix/federation/v1/user/devices/%40USER%3Atruf-kin.com
2021-04-22 14:14:16,240 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4714 - {GET-O-745251} [kittenface.studio] Completed request: 200 OK in 0.10 secs - GET matrix://kittenface.studio/_matrix/federation/v1/user/devices/%40USER%3Akittenface.studio

2021-04-22 14:14:36,877 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4715 - {GET-O-745252} [matrix.org] Completed request: 200 OK in 0.06 secs - GET matrix://matrix.org/_matrix/federation/v1/user/devices/%40USER%3Amatrix.org
2021-04-22 14:14:36,897 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4715 - {GET-O-745253} [inso.ovh] Completed request: 200 OK in 0.02 secs - GET matrix://inso.ovh/_matrix/federation/v1/user/devices/%40USER%3Ainso.ovh
2021-04-22 14:14:36,996 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4715 - {GET-O-745263} [matrix.org] Completed request: 200 OK in 0.06 secs - GET matrix://matrix.org/_matrix/federation/v1/user/devices/%40USER%3Amatrix.org
2021-04-22 14:14:37,028 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4715 - {GET-O-745269} [madduck.net] Completed request: 200 OK in 0.03 secs - GET matrix://madduck.net/_matrix/federation/v1/user/devices/%40USER%3Amadduck.net
2021-04-22 14:14:37,328 - synapse.http.matrixfederationclient - 216 - INFO - _maybe_retry_device_resync-4715 - {GET-O-745271} [owdo.org] Completed request: 200 OK in 0.30 secs - GET matrix://owdo.org/_matrix/federation/v1/user/devices/%40USER%3Aowdo.org
```

The same users are resynced each loop.



Discussion:

Resolving patch:
diff --git a/synapse/handlers/device.py b/synapse/handlers/device.py
--- a/synapse/handlers/device.py
+++ b/synapse/handlers/device.py
@@ -926,6 +926,10 @@ async def user_device_resync(
         else:
             cached_devices = await self.store.get_cached_devices_for_user(user_id)
             if cached_devices == {d["device_id"]: d for d in devices}:
+                logging.info(
+                    "Skipping device list resync for %s, as our cache matches already",
+                    user_id,
+                )
                 devices = []
                 ignore_devices = True
 
@@ -941,6 +945,9 @@ async def user_device_resync(
             await self.store.update_remote_device_list_cache(
                 user_id, devices, stream_id
             )
+        # mark the cache as valid, whether or not we actually processed any device
+        # list updates.
+        await self.store.mark_remote_user_device_cache_as_valid(user_id)
         device_ids = [device["device_id"] for device in devices]
 
         # Handle cross-signing keys.
diff --git a/synapse/storage/databases/main/devices.py b/synapse/storage/databases/main/devices.py
--- a/synapse/storage/databases/main/devices.py
+++ b/synapse/storage/databases/main/devices.py
@@ -717,7 +717,15 @@ async def mark_remote_user_device_cache_as_stale(self, user_id: str) -> None:
             keyvalues={"user_id": user_id},
             values={},
             insertion_values={"added_ts": self._clock.time_msec()},
-            desc="make_remote_user_device_cache_as_stale",
+            desc="mark_remote_user_device_cache_as_stale",
+        )
+
+    async def mark_remote_user_device_cache_as_valid(self, user_id: str) -> None:
+        # Remove the database entry that says we need to resync devices, after a resync
+        await self.db_pool.simple_delete(
+            table="device_lists_remote_resync",
+            keyvalues={"user_id": user_id},
+            desc="mark_remote_user_device_cache_as_valid",
         )
 
     async def mark_remote_user_device_list_as_unsubscribed(self, user_id: str) -> None:
@@ -1289,15 +1297,6 @@ def _update_remote_device_list_cache_txn(
             lock=False,
         )
 
-        # If we're replacing the remote user's device list cache presumably
-        # we've done a full resync, so we remove the entry that says we need
-        # to resync
-        self.db_pool.simple_delete_txn(
-            txn,
-            table="device_lists_remote_resync",
-            keyvalues={"user_id": user_id},
-        )
-
     async def add_device_change_to_streams(
         self, user_id: str, device_ids: Collection[str], hosts: List[str]
     ):


Test patch: