Repository: matrix-org/synapse

Issue:
CS-API `/messages` requests fail on develop when using postgres due to NumericValueOutOfRange if some events have repeatedly failed to backfill
https://sentry.tools.element.io/organizations/element/issues/33447/?project=2&query=is%3Aunresolved

Running this query: https://github.com/matrix-org/synapse/blob/ac1a31740b6d0dfda4d57a25762aaddfde981caf/synapse/storage/databases/main/event_federation.py#L753-L805

with args

```
[['!redacted_room_name:example.com', False, 1664362899049, 3600000, 604800000]]
```

from https://github.com/matrix-org/synapse/blob/ac1a31740b6d0dfda4d57a25762aaddfde981caf/synapse/storage/databases/main/event_federation.py#L815-L824

```
Failed handle request via "RoomMessageListRestServlet": "<XForwardedForRequest at 0x7fba292b3490 method='GET' uri='/_matrix/client/r0/rooms/!redacted_room_id:example.com/messages?from=s3305809885_757284974_68989_1616590947_1642068190_3598845_602714760_5320124732_0&dir=b&limit=50&filter=%7B%22lazy_load_members%22%3Atrue%7D' clientproto='HTTP/1.1' site='11106'>"
```

Running on `1.68.0 (b=matrix-org-hotfixes,3f30bdca19)`


Discussion:
I don't like the look of this:

```
matrix=> SELECT count(*), (
matrix(>     1 << event_failed_pull_attempts.num_attempts
matrix(> ) c
matrix-> FROM event_failed_pull_attempts
matrix-> GROUP BY c
matrix-> ORDER BY c ASC;
 count │      c      
═══════╪═════════════
    15 │ -2147483648
    11 │           1
   848 │           2
   921 │           4
   378 │           8
   180 │          16
   117 │          32
    70 │          64
    40 │         128
    39 │         256
    26 │         512
    53 │        1024
    26 │        2048
    33 │        4096
    15 │        8192
    25 │       16384
    18 │       32768
    13 │       65536
    20 │      131072
    18 │      262144
     6 │      524288
    18 │     1048576
     6 │     2097152
    19 │     4194304
    35 │     8388608
    18 │    16777216
    17 │    33554432
     8 │    67108864
    13 │   134217728
     3 │   268435456
     4 │   536870912
    13 │  1073741824
(32 rows)
```

I think that first row is the smallest 32 bit signed int?
Right. In the query

```sql
SELECT count(*)
FROM event_failed_pull_attempts
WHERE 1664364716150 >= last_attempt_ts + LEAST((1 << num_attempts) * 3600000, 604800000);
```

(which causes `ERROR:  integer out of range`), consider the boolean expression

```sql
1664364716150 >= last_attempt_ts + LEAST((1 << num_attempts) * 3600000, 604800000) 
```

The types of "leaf" expressions involved here are

```
matrix=> SELECT (pg_typeof(1664364716150) currentSELECT_time, pg_typeof(last_attempt_ts) last_attempt_ts, pg_typeof(num_attempts) num_attempts, pg_typeof(3600000) ms_per_hour, pg_typeof(604800000) ms_per_week
FROM event_failed_pull_attempts
LIMIT 1;
matrix(>     pg_typeof(1664364716150)   AS current_time,
matrix(>     pg_typeof(last_attempt_ts) AS last_attempt_ts,
matrix(>     pg_typeof(num_attempts)    AS num_attempts,
matrix(>     pg_typeof(3600000)         AS ms_per_hour,
matrix(>     pg_typeof(604800000)       AS ms_per_week
matrix(> FROM event_failed_pull_attempts
matrix(> LIMIT 1;
matrix(> ^C
matrix=> SELECT
matrix->     pg_typeof(1664364716150)   AS current_time,
matrix->     pg_typeof(last_attempt_ts) AS last_attempt_ts,
matrix->     pg_typeof(num_attempts)    AS num_attempts,
matrix->     pg_typeof(3600000)         AS ms_per_hour,
matrix->     pg_typeof(604800000)       AS ms_per_week
matrix-> FROM event_failed_pull_attempts
matrix-> LIMIT 1;
 current_time │ last_attempt_ts │ num_attempts │ ms_per_hour │ ms_per_week 
══════════════╪═════════════════╪══════════════╪═════════════╪═════════════
 bigint       │ bigint          │ integer      │ integer     │ integer
(1 row)
```
The left-shift expression is

```
SELECT pg_typeof(1 << num_attempts) AS shifted FROM event_failed_pull_attempts LIMIT 1;
 shifted 
═════════
 integer
```

[because `1` is an integer](https://www.postgresql.org/docs/current/functions-math.html#FUNCTIONS-MATH-OP-TABLE).

our expression is now `bigint >= bigint + LEAST(integer * integer, integer)`.

It looks like multiplication of integers ([32 bit signed ints](https://www.postgresql.org/docs/current/datatype-numeric.html#DATATYPE-NUMERIC-TABLE)) simply errors on overflow, but I couldn't see anything in the docs which explains this.

```
matrix=> select (1<<30) * (1 << 30);
ERROR:  integer out of range
```

Similarly for bigints (64 bit signed ints):

```
matrix=> select (1::bigint << 60) * (1::bigint << 60);
ERROR:  bigint out of range
```

In contrast the left shift operation will happily overflow, wrapping around to negative ints.

```
matrix=> WITH series(x) AS (SELECT generate_series(1, 40)) SELECT x, 1 << x AS shifted FROM series;
 x  │   shifted   
════╪═════════════
  1 │           2
  2 │           4
  3 │           8
  4 │          16
  5 │          32
  6 │          64
  7 │         128
  8 │         256
  9 │         512
 10 │        1024
 11 │        2048
 12 │        4096
 13 │        8192
 14 │       16384
 15 │       32768
 16 │       65536
 17 │      131072
 18 │      262144
 19 │      524288
 20 │     1048576
 21 │     2097152
 22 │     4194304
 23 │     8388608
 24 │    16777216
 25 │    33554432
 26 │    67108864
 27 │   134217728
 28 │   268435456
 29 │   536870912
 30 │  1073741824
 31 │ -2147483648
 32 │           1
 33 │           2
 34 │           4
 35 │           8
 36 │          16
 37 │          32
 38 │          64
 39 │         128
 40 │         256
(40 rows)
```
TL;DR: `(1 << num_attempts) * 3600000` is too large for a 32 bit integer in for some value of `num_attempts` that we have in our database.

As for why we sometimes `num_attempts % 32` is very large, and hence `1 << num_attempts` is large. To try to illustrate this:

```
matrix=> SELECT MAX(num_attempts), MAX(num_attempts % 32), MIN(1 << num_attempts), MAX(1 << num_attempts) FROM event_failed_pull_attempts;
  max   │ max │     min     │    max     
════════╪═════╪═════════════╪════════════
 157177 │  31 │ -2147483648 │ 1073741824
(1 row)
```

```
matrix=> select pct, percentile_disc(pct) WITHIN GROUP (ORDER BY num_attempts) FROM event_failed_pull_attempts, generate_series(0, 1, 0.05) AS pct group by pct;
 pct  │ percentile_disc 
══════╪═════════════════
    0 │               1
 0.05 │               1
 0.10 │               1
 0.15 │               1
 0.20 │               1
 0.25 │               1
 0.30 │               2
 0.35 │               2
 0.40 │               2
 0.45 │               2
 0.50 │               2
 0.55 │               2
 0.60 │               3
 0.65 │               3
 0.70 │               3
 0.75 │               4
 0.80 │               6
 0.85 │              10
 0.90 │              17
 0.95 │              38
 1.00 │          157406
```

I think we need to clamp `1 << num_attempts` somehow, e.g. by using `1 << LEAST(num_attempts, 5)` instead. If I understand correctly, that should mean a maximum backoff time of `1 << 5` = 32 hours?
Assuming a regression in #13635.
> I think we need to clamp `1 << num_attempts` somehow

or cast the left hand side to a bigint?

```
synapse=# SELECT CAST(1 AS BIGINT) << 33;
  ?column?
------------
 8589934592
(1 row)
```
> or cast the left hand side to a bigint?

Even if we do this, `(bigint << integer) * integer` will become `(bigint * integer)`. Postgres still errors on overflows for those multiplications.

```
matrix=> SELECT (1::bigint << 50) * 3600000;
ERROR:  bigint out of range
```

(Maybe there's some setting to  control overflow behaviour?)
Ah yes you're right. I missed that `num_attempts` would keep going up even though we limit the backoff interval.

Resolving patch:
diff --git a/synapse/storage/databases/main/event_federation.py b/synapse/storage/databases/main/event_federation.py
--- a/synapse/storage/databases/main/event_federation.py
+++ b/synapse/storage/databases/main/event_federation.py
@@ -73,13 +73,30 @@
 
 logger = logging.getLogger(__name__)
 
-BACKFILL_EVENT_BACKOFF_UPPER_BOUND_SECONDS: int = int(
-    datetime.timedelta(days=7).total_seconds()
-)
-BACKFILL_EVENT_EXPONENTIAL_BACKOFF_STEP_SECONDS: int = int(
-    datetime.timedelta(hours=1).total_seconds()
+# Parameters controlling exponential backoff between backfill failures.
+# After the first failure to backfill, we wait 2 hours before trying again. If the
+# second attempt fails, we wait 4 hours before trying again. If the third attempt fails,
+# we wait 8 hours before trying again, ... and so on.
+#
+# Each successive backoff period is twice as long as the last. However we cap this
+# period at a maximum of 2^8 = 256 hours: a little over 10 days. (This is the smallest
+# power of 2 which yields a maximum backoff period of at least 7 days---which was the
+# original maximum backoff period.) Even when we hit this cap, we will continue to
+# make backfill attempts once every 10 days.
+BACKFILL_EVENT_EXPONENTIAL_BACKOFF_MAXIMUM_DOUBLING_STEPS = 8
+BACKFILL_EVENT_EXPONENTIAL_BACKOFF_STEP_MILLISECONDS = int(
+    datetime.timedelta(hours=1).total_seconds() * 1000
 )
 
+# We need a cap on the power of 2 or else the backoff period
+#   2^N * (milliseconds per hour)
+# will overflow when calcuated within the database. We ensure overflow does not occur
+# by checking that the largest backoff period fits in a 32-bit signed integer.
+_LONGEST_BACKOFF_PERIOD_MILLISECONDS = (
+    2**BACKFILL_EVENT_EXPONENTIAL_BACKOFF_MAXIMUM_DOUBLING_STEPS
+) * BACKFILL_EVENT_EXPONENTIAL_BACKOFF_STEP_MILLISECONDS
+assert 0 < _LONGEST_BACKOFF_PERIOD_MILLISECONDS <= ((2**31) - 1)
+
 
 # All the info we need while iterating the DAG while backfilling
 @attr.s(frozen=True, slots=True, auto_attribs=True)
@@ -767,7 +784,15 @@ def get_backfill_points_in_room_txn(
             # persisted in our database yet (meaning we don't know their depth
             # specifically). So we need to look for the approximate depth from
             # the events connected to the current backwards extremeties.
-            sql = """
+
+            if isinstance(self.database_engine, PostgresEngine):
+                least_function = "LEAST"
+            elif isinstance(self.database_engine, Sqlite3Engine):
+                least_function = "MIN"
+            else:
+                raise RuntimeError("Unknown database engine")
+
+            sql = f"""
                 SELECT backward_extrem.event_id, event.depth FROM events AS event
                 /**
                  * Get the edge connections from the event_edges table
@@ -825,7 +850,10 @@ def get_backfill_points_in_room_txn(
                      */
                     AND (
                         failed_backfill_attempt_info.event_id IS NULL
-                        OR ? /* current_time */ >= failed_backfill_attempt_info.last_attempt_ts + /*least*/%s((1 << failed_backfill_attempt_info.num_attempts) * ? /* step */, ? /* upper bound */)
+                        OR ? /* current_time */ >= failed_backfill_attempt_info.last_attempt_ts + (
+                            (1 << {least_function}(failed_backfill_attempt_info.num_attempts, ? /* max doubling steps */))
+                            * ? /* step */
+                        )
                     )
                 /**
                  * Sort from highest (closest to the `current_depth`) to the lowest depth
@@ -837,22 +865,15 @@ def get_backfill_points_in_room_txn(
                 LIMIT ?
             """
 
-            if isinstance(self.database_engine, PostgresEngine):
-                least_function = "least"
-            elif isinstance(self.database_engine, Sqlite3Engine):
-                least_function = "min"
-            else:
-                raise RuntimeError("Unknown database engine")
-
             txn.execute(
-                sql % (least_function,),
+                sql,
                 (
                     room_id,
                     False,
                     current_depth,
                     self._clock.time_msec(),
-                    1000 * BACKFILL_EVENT_EXPONENTIAL_BACKOFF_STEP_SECONDS,
-                    1000 * BACKFILL_EVENT_BACKOFF_UPPER_BOUND_SECONDS,
+                    BACKFILL_EVENT_EXPONENTIAL_BACKOFF_MAXIMUM_DOUBLING_STEPS,
+                    BACKFILL_EVENT_EXPONENTIAL_BACKOFF_STEP_MILLISECONDS,
                     limit,
                 ),
             )
@@ -902,7 +923,14 @@ async def get_insertion_event_backward_extremities_in_room(
         def get_insertion_event_backward_extremities_in_room_txn(
             txn: LoggingTransaction, room_id: str
         ) -> List[Tuple[str, int]]:
-            sql = """
+            if isinstance(self.database_engine, PostgresEngine):
+                least_function = "LEAST"
+            elif isinstance(self.database_engine, Sqlite3Engine):
+                least_function = "MIN"
+            else:
+                raise RuntimeError("Unknown database engine")
+
+            sql = f"""
                 SELECT
                     insertion_event_extremity.event_id, event.depth
                 /* We only want insertion events that are also marked as backwards extremities */
@@ -942,7 +970,10 @@ def get_insertion_event_backward_extremities_in_room_txn(
                      */
                     AND (
                         failed_backfill_attempt_info.event_id IS NULL
-                        OR ? /* current_time */ >= failed_backfill_attempt_info.last_attempt_ts + /*least*/%s((1 << failed_backfill_attempt_info.num_attempts) * ? /* step */, ? /* upper bound */)
+                        OR ? /* current_time */ >= failed_backfill_attempt_info.last_attempt_ts + (
+                            (1 << {least_function}(failed_backfill_attempt_info.num_attempts, ? /* max doubling steps */))
+                            * ? /* step */
+                        )
                     )
                 /**
                  * Sort from highest (closest to the `current_depth`) to the lowest depth
@@ -954,21 +985,14 @@ def get_insertion_event_backward_extremities_in_room_txn(
                 LIMIT ?
             """
 
-            if isinstance(self.database_engine, PostgresEngine):
-                least_function = "least"
-            elif isinstance(self.database_engine, Sqlite3Engine):
-                least_function = "min"
-            else:
-                raise RuntimeError("Unknown database engine")
-
             txn.execute(
-                sql % (least_function,),
+                sql,
                 (
                     room_id,
                     current_depth,
                     self._clock.time_msec(),
-                    1000 * BACKFILL_EVENT_EXPONENTIAL_BACKOFF_STEP_SECONDS,
-                    1000 * BACKFILL_EVENT_BACKOFF_UPPER_BOUND_SECONDS,
+                    BACKFILL_EVENT_EXPONENTIAL_BACKOFF_MAXIMUM_DOUBLING_STEPS,
+                    BACKFILL_EVENT_EXPONENTIAL_BACKOFF_STEP_MILLISECONDS,
                     limit,
                 ),
             )


Test patch:
diff --git a/tests/storage/test_event_federation.py b/tests/storage/test_event_federation.py
--- a/tests/storage/test_event_federation.py
+++ b/tests/storage/test_event_federation.py
@@ -766,9 +766,7 @@ def test_get_backfill_points_in_room(self):
             self.store.get_backfill_points_in_room(room_id, depth_map["B"], limit=100)
         )
         backfill_event_ids = [backfill_point[0] for backfill_point in backfill_points]
-        self.assertListEqual(
-            backfill_event_ids, ["b6", "b5", "b4", "2", "b3", "b2", "b1"]
-        )
+        self.assertEqual(backfill_event_ids, ["b6", "b5", "b4", "2", "b3", "b2", "b1"])
 
         # Try at "A"
         backfill_points = self.get_success(
@@ -814,7 +812,7 @@ def test_get_backfill_points_in_room_excludes_events_we_have_attempted(
         )
         backfill_event_ids = [backfill_point[0] for backfill_point in backfill_points]
         # Only the backfill points that we didn't record earlier exist here.
-        self.assertListEqual(backfill_event_ids, ["b6", "2", "b1"])
+        self.assertEqual(backfill_event_ids, ["b6", "2", "b1"])
 
     def test_get_backfill_points_in_room_attempted_event_retry_after_backoff_duration(
         self,
@@ -860,7 +858,7 @@ def test_get_backfill_points_in_room_attempted_event_retry_after_backoff_duratio
             self.store.get_backfill_points_in_room(room_id, depth_map["A"], limit=100)
         )
         backfill_event_ids = [backfill_point[0] for backfill_point in backfill_points]
-        self.assertListEqual(backfill_event_ids, ["b3", "b2"])
+        self.assertEqual(backfill_event_ids, ["b3", "b2"])
 
         # Now advance time by 20 hours (above 2^4 because we made 4 attemps) and
         # see if we can now backfill it
@@ -871,7 +869,48 @@ def test_get_backfill_points_in_room_attempted_event_retry_after_backoff_duratio
             self.store.get_backfill_points_in_room(room_id, depth_map["A"], limit=100)
         )
         backfill_event_ids = [backfill_point[0] for backfill_point in backfill_points]
-        self.assertListEqual(backfill_event_ids, ["b3", "b2", "b1"])
+        self.assertEqual(backfill_event_ids, ["b3", "b2", "b1"])
+
+    def test_get_backfill_points_in_room_works_after_many_failed_pull_attempts_that_could_naively_overflow(
+        self,
+    ) -> None:
+        """
+        A test that reproduces #13929 (Postgres only).
+
+        Test to make sure we can still get backfill points after many failed pull
+        attempts that cause us to backoff to the limit. Even if the backoff formula
+        would tell us to wait for more seconds than can be expressed in a 32 bit
+        signed int.
+        """
+        setup_info = self._setup_room_for_backfill_tests()
+        room_id = setup_info.room_id
+        depth_map = setup_info.depth_map
+
+        # Pretend that we have tried and failed 10 times to backfill event b1.
+        for _ in range(10):
+            self.get_success(
+                self.store.record_event_failed_pull_attempt(room_id, "b1", "fake cause")
+            )
+
+        # If the backoff periods grow without limit:
+        # After the first failed attempt, we would have backed off for 1 << 1 = 2 hours.
+        # After the second failed attempt we would have backed off for 1 << 2 = 4 hours,
+        # so after the 10th failed attempt we should backoff for 1 << 10 == 1024 hours.
+        # Wait 1100 hours just so we have a nice round number.
+        self.reactor.advance(datetime.timedelta(hours=1100).total_seconds())
+
+        # 1024 hours in milliseconds is 1024 * 3600000, which exceeds the largest 32 bit
+        # signed integer. The bug we're reproducing is that this overflow causes an
+        # error in postgres preventing us from fetching a set of backwards extremities
+        # to retry fetching.
+        backfill_points = self.get_success(
+            self.store.get_backfill_points_in_room(room_id, depth_map["A"], limit=100)
+        )
+
+        # We should aim to fetch all backoff points: b1's latest backoff period has
+        # expired, and we haven't tried the rest.
+        backfill_event_ids = [backfill_point[0] for backfill_point in backfill_points]
+        self.assertEqual(backfill_event_ids, ["b3", "b2", "b1"])
 
     def _setup_room_for_insertion_backfill_tests(self) -> _BackfillSetupInfo:
         """
@@ -965,9 +1004,7 @@ def test_get_insertion_event_backward_extremities_in_room(self):
             )
         )
         backfill_event_ids = [backfill_point[0] for backfill_point in backfill_points]
-        self.assertListEqual(
-            backfill_event_ids, ["insertion_eventB", "insertion_eventA"]
-        )
+        self.assertEqual(backfill_event_ids, ["insertion_eventB", "insertion_eventA"])
 
         # Try at "insertion_eventA"
         backfill_points = self.get_success(
@@ -1011,7 +1048,7 @@ def test_get_insertion_event_backward_extremities_in_room_excludes_events_we_hav
         )
         backfill_event_ids = [backfill_point[0] for backfill_point in backfill_points]
         # Only the backfill points that we didn't record earlier exist here.
-        self.assertListEqual(backfill_event_ids, ["insertion_eventB"])
+        self.assertEqual(backfill_event_ids, ["insertion_eventB"])
 
     def test_get_insertion_event_backward_extremities_in_room_attempted_event_retry_after_backoff_duration(
         self,
@@ -1069,7 +1106,7 @@ def test_get_insertion_event_backward_extremities_in_room_attempted_event_retry_
             )
         )
         backfill_event_ids = [backfill_point[0] for backfill_point in backfill_points]
-        self.assertListEqual(backfill_event_ids, [])
+        self.assertEqual(backfill_event_ids, [])
 
         # Now advance time by 20 hours (above 2^4 because we made 4 attemps) and
         # see if we can now backfill it
@@ -1083,7 +1120,7 @@ def test_get_insertion_event_backward_extremities_in_room_attempted_event_retry_
             )
         )
         backfill_event_ids = [backfill_point[0] for backfill_point in backfill_points]
-        self.assertListEqual(backfill_event_ids, ["insertion_eventA"])
+        self.assertEqual(backfill_event_ids, ["insertion_eventA"])
 
 
 @attr.s