Repository: matrix-org/synapse

Issue:
Traceback on background process 'user_directory.notify_new_event' threw an exception
### Description

I just upgraded a homeserver from 1.26 to 1.27 and now I'm seeing a lot of these tracebacks in my homeserver.log:

```
2021-02-16 17:36:09,420 - synapse.metrics.background_process_metrics - 211 - ERROR - user_directory.notify_new_event-9 - Background process 'user_directory.notify_new_event' threw an exception
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/metrics/background_process_metrics.py", line 208, in run
    return await maybe_awaitable(func(*args, **kwargs))
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/handlers/user_directory.py", line 110, in process
    await self._unsafe_process()
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/handlers/user_directory.py", line 159, in _unsafe_process
    self.pos, room_max_stream_ordering
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/storage/databases/main/state_deltas.py", line 51, in get_current_state_deltas
    prev_stream_id = int(prev_stream_id)
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType
```

### Version information

- **Homeserver**: Private
- **Version**: 1.27
- **Install method**: Debian package.
- **Platform**: Debian


Discussion:
I am also experiencing the same issue when I upgraded a homeserver running on Ubuntu 18.04 using DEB packages to 1.27.
A similar problem after upgrading from 1.26.0 to 1.27.0 (debian package). Ubuntu server 20.04
```
2021-02-17 15:47:16,958 - synapse.metrics.background_process_metrics - 210 - ERROR - user_directory.notify_new_event-22 - Background process 'user_directory.notify_new_event' thr>
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/metrics/background_process_metrics.py", line 208, in run
    return await maybe_awaitable(func(*args, **kwargs))
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/handlers/user_directory.py", line 110, in process
    await self._unsafe_process()
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/handlers/user_directory.py", line 158, in _unsafe_process
    max_pos, deltas = await self.store.get_current_state_deltas(
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/storage/databases/main/state_deltas.py", line 51, in get_current_state_deltas
    prev_stream_id = int(prev_stream_id)
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

```
yup, same here.
The direct cause seems to be (for me) that the stream_id column in the user_directory_stream_pos table is empty.
Same happened here, upgrading from 1.26 to 1.27
UPDATE: actually rebuilding the user directory index (see: https://github.com/matrix-org/synapse/blob/master/docs/user_directory.md) seemingly fixed the tracebacks.
This seems to be a regression from https://github.com/matrix-org/synapse/pull/9223/files#r564010797.

Resolving patch:
diff --git a/synapse/handlers/user_directory.py b/synapse/handlers/user_directory.py
--- a/synapse/handlers/user_directory.py
+++ b/synapse/handlers/user_directory.py
@@ -143,6 +143,10 @@ async def _unsafe_process(self) -> None:
         if self.pos is None:
             self.pos = await self.store.get_user_directory_stream_pos()
 
+        # If still None then the initial background update hasn't happened yet.
+        if self.pos is None:
+            return None
+
         # Loop round handling deltas until we're up to date
         while True:
             with Measure(self.clock, "user_dir_delta"):
diff --git a/synapse/storage/databases/main/user_directory.py b/synapse/storage/databases/main/user_directory.py
--- a/synapse/storage/databases/main/user_directory.py
+++ b/synapse/storage/databases/main/user_directory.py
@@ -707,7 +707,13 @@ def _get_shared_rooms_for_users_txn(txn):
 
         return {row["room_id"] for row in rows}
 
-    async def get_user_directory_stream_pos(self) -> int:
+    async def get_user_directory_stream_pos(self) -> Optional[int]:
+        """
+        Get the stream ID of the user directory stream.
+
+        Returns:
+            The stream token or None if the initial background update hasn't happened yet.
+        """
         return await self.db_pool.simple_select_one_onecol(
             table="user_directory_stream_pos",
             keyvalues={},


Test patch: