Repository: matrix-org/synapse

Issue #5442: we should backoff on 40x errors
Created: 2019-06-12T19:32:35Z
Currently we do not increment the per-host backoff counter on 405 (method not allowed) responses. If we get a 405, it suggests a poorly-configured destination, so we should back off.
Indeed, the only 40x codes we do back off for are 401, 429, and (sometimes) 404. If our spec actually properly defined the valid error responses, we could do much better here.

Duplicate issue #15365: No backoff on `v1/user/devices` requests for homeservers that have been reset
Created: 2023-04-01T12:59:10Z
Description
I used run a Conduit instance on the domain matrix.example.com with a user @phesch:matrix.example.com and joined several rooms with that user. However, this was before I knew about .well-known redirects, so I eventually decided to nuke that server and set up a new one under example.com with a well-known redirect to matrix.example.com (so the homeserver's "actual" domain and DNS didn't change, but the name did).
It has been about 18 months since I reset the homeserver and everything works correctly except for the fact that it receives a large amount of request spam from the Synapse homeservers that were in the rooms my old user @phesch:matrix.example.com was in.
These servers constantly request /federation/v1/user/devices/@phesch:matrix.example.com despite the fact that they always receive a 403 response (since that user obviously doesn't exist). I guess this has something to do with the usual backoff being reset since my homeserver responds normally to other (current) requests.
This behavior has become rather annoying since it amounts to >60000 requests a day and constitutes about 70% of my server's total incoming HTTP traffic.
Note that Dendrites and other Conduits do not show up as performing these requests in my logs.
Steps to reproduce

Set up a matrix homeserver (unsure whether it must be a conduit server) on a subdomain
Make a user and join some rooms
Delete the homeserver and make a new one with the homeserver itself on the same subdomain, but with a .well-known entry so the homeserver's name is the main domain
Make a user and join some rooms
Observe that Synapse homeservers keep requesting v1/user/devices at a high rate even though it always returns an error

Homeserver
Many Synapse servers
Synapse Version
1.70.1 - 1.80.0rc2 are the versions I found in my logs
Installation Method
Other (please mention below)
Database
N/A
Workers
I don't know
Platform
N/A
Configuration
N/A
Relevant log output
Here's an excerpt of what this looks like in my nginx logs:
<IP> - - [<TIME>] "GET /_matrix/federation/v1/user/devices/%40phesch%3Amatrix.example.com HTTP/1.1" 403 86 "-" "Synapse/1.79.0" "-" 313 0.011
<IP> - - [<TIME+0s>] "GET /_matrix/federation/v1/user/devices/%40phesch%3Amatrix.example.com HTTP/1.1" 403 86 "-" "Synapse/1.79.0" "-" 317 0.010
<IP> - - [<TIME+3s>] "GET /_matrix/federation/v1/user/devices/%40phesch%3Amatrix.example.com HTTP/1.1" 403 86 "-" "Synapse/1.79.0" "-" 327 0.011
<IP> - - [<TIME+7s>] "GET /_matrix/federation/v1/user/devices/%40phesch%3Amatrix.example.com HTTP/1.1" 403 86 "-" "Synapse/1.74.0" "-" 310 0.010
...
Anything else that would be useful to know?
No response

Closing comment (2023-05-26T10:32:10Z): Duplicate of #5442, I think?