internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: matrix-org/synapse

Issue:
Background process 'sync_partial_state_room' threw an exception
This seems to happen after there is an error contacting a server for state:

```
Traceback (most recent call last):
  File "/home/synapse/src/synapse/synapse/metrics/background_process_metrics.py", line 240, in run
    return await func(*args, **kwargs)
  File "/home/synapse/src/synapse/synapse/handlers/federation.py", line 1597, in _sync_partial_state_room
    if attempt == len(destinations) - 1:
UnboundLocalError: local variable 'destinations' referenced before assignment
```



Discussion:

Resolving patch:
diff --git a/synapse/handlers/federation.py b/synapse/handlers/federation.py
--- a/synapse/handlers/federation.py
+++ b/synapse/handlers/federation.py
@@ -1539,15 +1539,16 @@ async def _sync_partial_state_room(
 
         # Make an infinite iterator of destinations to try. Once we find a working
         # destination, we'll stick with it until it flakes.
+        destinations: Collection[str]
         if initial_destination is not None:
             # Move `initial_destination` to the front of the list.
             destinations = list(other_destinations)
             if initial_destination in destinations:
                 destinations.remove(initial_destination)
             destinations = [initial_destination] + destinations
-            destination_iter = itertools.cycle(destinations)
         else:
-            destination_iter = itertools.cycle(other_destinations)
+            destinations = other_destinations
+        destination_iter = itertools.cycle(destinations)
 
         # `destination` is the current remote homeserver we're pulling from.
         destination = next(destination_iter)


Test patch:
Repository: matrix-org/synapse

Issue:
Faster joins: fix up `/_matrix/client/v3/rooms/{roomId}/joined_members`
` /_matrix/client/v3/rooms/{roomId}/joined_members` should block pending full state


Discussion:

Resolving patch:
diff --git a/synapse/handlers/message.py b/synapse/handlers/message.py
--- a/synapse/handlers/message.py
+++ b/synapse/handlers/message.py
@@ -331,7 +331,11 @@ async def get_joined_members(self, requester: Requester, room_id: str) -> dict:
                     msg="Getting joined members while not being a current member of the room is forbidden.",
                 )
 
-        users_with_profile = await self.store.get_users_in_room_with_profiles(room_id)
+        users_with_profile = (
+            await self._state_storage_controller.get_users_in_room_with_profiles(
+                room_id
+            )
+        )
 
         # If this is an AS, double check that they are allowed to see the members.
         # This can either be because the AS user is in the room or because there
diff --git a/synapse/storage/controllers/state.py b/synapse/storage/controllers/state.py
--- a/synapse/storage/controllers/state.py
+++ b/synapse/storage/controllers/state.py
@@ -30,6 +30,7 @@
 from synapse.api.constants import EventTypes
 from synapse.events import EventBase
 from synapse.logging.opentracing import trace
+from synapse.storage.roommember import ProfileInfo
 from synapse.storage.state import StateFilter
 from synapse.storage.util.partial_state_events_tracker import (
     PartialCurrentStateTracker,
@@ -506,3 +507,15 @@ async def get_current_hosts_in_room(self, room_id: str) -> Set[str]:
         await self._partial_state_room_tracker.await_full_state(room_id)
 
         return await self.stores.main.get_current_hosts_in_room(room_id)
+
+    async def get_users_in_room_with_profiles(
+        self, room_id: str
+    ) -> Dict[str, ProfileInfo]:
+        """
+        Get the current users in the room with their profiles.
+        If the room is currently partial-stated, this will block until the room has
+        full state.
+        """
+        await self._partial_state_room_tracker.await_full_state(room_id)
+
+        return await self.stores.main.get_users_in_room_with_profiles(room_id)
diff --git a/synapse/storage/databases/main/roommember.py b/synapse/storage/databases/main/roommember.py
--- a/synapse/storage/databases/main/roommember.py
+++ b/synapse/storage/databases/main/roommember.py
@@ -283,6 +283,9 @@ async def get_users_in_room_with_profiles(
 
         Returns:
             A mapping from user ID to ProfileInfo.
+
+        Preconditions:
+          - There is full state available for the room (it is not partial-stated).
         """
 
         def _get_users_in_room_with_profiles(


Test patch:
Repository: matrix-org/synapse

Issue:
Faster joins: prevent enabling it with workers
Currently if you enable faster-joins with worker processes, you're going to get horrible cache-coherency issues. We should stop people shooting themselves in the foot like this with a check on the config setting.



Discussion:
(proper support for workers is #12994)

Resolving patch:
diff --git a/synapse/app/generic_worker.py b/synapse/app/generic_worker.py
--- a/synapse/app/generic_worker.py
+++ b/synapse/app/generic_worker.py
@@ -441,6 +441,13 @@ def start(config_options: List[str]) -> None:
         "synapse.app.user_dir",
     )
 
+    if config.experimental.faster_joins_enabled:
+        raise ConfigError(
+            "You have enabled the experimental `faster_joins` config option, but it is "
+            "not compatible with worker deployments yet. Please disable `faster_joins` "
+            "or run Synapse as a single process deployment instead."
+        )
+
     synapse.events.USE_FROZEN_DICTS = config.server.use_frozen_dicts
     synapse.util.caches.TRACK_MEMORY_USAGE = config.caches.track_memory_usage
 


Test patch:
Repository: matrix-org/synapse

Issue:
Docs and warnings refer to nonexistent `custom_templates_directory` setting
The [templates](https://matrix-org.github.io/synapse/v1.64/templates.html#templates) documentation, and various warnings emitted by synapse ([example](https://github.com/matrix-org/synapse/blob/v1.64.0/synapse/config/emailconfig.py#L56)) refer to a config setting called `custom_templates_directory`. The correct name is [`custom_template_directory`](https://matrix-org.github.io/synapse/v1.64/usage/configuration/config_documentation.html#templates-and-custom_template_directory).




Discussion:
Unfortunate. A warning on startup that the `custom_templates_directory` plural is an unknown key might have helped.

Resolving patch:
diff --git a/synapse/config/account_validity.py b/synapse/config/account_validity.py
--- a/synapse/config/account_validity.py
+++ b/synapse/config/account_validity.py
@@ -23,7 +23,7 @@
 This server's configuration file is using the deprecated 'template_dir' setting in the
 'account_validity' section. Support for this setting has been deprecated and will be
 removed in a future version of Synapse. Server admins should instead use the new
-'custom_templates_directory' setting documented here:
+'custom_template_directory' setting documented here:
 https://matrix-org.github.io/synapse/latest/templates.html
 ---------------------------------------------------------------------------------------"""
 
diff --git a/synapse/config/emailconfig.py b/synapse/config/emailconfig.py
--- a/synapse/config/emailconfig.py
+++ b/synapse/config/emailconfig.py
@@ -53,7 +53,7 @@
 This server's configuration file is using the deprecated 'template_dir' setting in the
 'email' section. Support for this setting has been deprecated and will be removed in a
 future version of Synapse. Server admins should instead use the new
-'custom_templates_directory' setting documented here:
+'custom_template_directory' setting documented here:
 https://matrix-org.github.io/synapse/latest/templates.html
 ---------------------------------------------------------------------------------------"""
 
diff --git a/synapse/config/sso.py b/synapse/config/sso.py
--- a/synapse/config/sso.py
+++ b/synapse/config/sso.py
@@ -26,7 +26,7 @@
 This server's configuration file is using the deprecated 'template_dir' setting in the
 'sso' section. Support for this setting has been deprecated and will be removed in a
 future version of Synapse. Server admins should instead use the new
-'custom_templates_directory' setting documented here:
+'custom_template_directory' setting documented here:
 https://matrix-org.github.io/synapse/latest/templates.html
 ---------------------------------------------------------------------------------------"""
 


Test patch:
Repository: matrix-org/synapse

Issue:
Drop unused column `application_services_state.last_txn`
#12680 removed the last reference to `application_services_state.last_txn`. We should drop the unused column.


Discussion:

Resolving patch:
diff --git a/synapse/storage/schema/__init__.py b/synapse/storage/schema/__init__.py
--- a/synapse/storage/schema/__init__.py
+++ b/synapse/storage/schema/__init__.py
@@ -75,6 +75,7 @@
 Changes in SCHEMA_VERSION = 72:
     - event_edges.(room_id, is_state) are no longer written to.
     - Tables related to groups are dropped.
+    - Unused column application_services_state.last_txn is dropped
 """
 
 


Test patch:
Repository: matrix-org/synapse

Issue:
Retention policy purge jobs fail if `run_background_tasks_on` is set to something other than master
If `run_background_tasks_on` is set to anything other than `master`, Synapse will try to run background jobs on a worker, including the message retention policies support's purge jobs. However, these jobs end up calling the `purge_history` function of the storage layer (in the `PurgeEventsStorage` class) which doesn't exist on a worker, therefore the jobs fail with `AttributeError: 'GenericWorkerSlavedStore' object has no attribute 'purge_history'`.

This bug was introduced in https://github.com/matrix-org/synapse/pull/8513

cc @iansinclair 


Discussion:
I guess we either need to move the method to the worker store or backout the changes that run that on a background worker. I think they're potentially heavy though, so if we could run it on a worker that would be best!
Yes I tend to agree - though having something that fiddles with the database as much as `purge_history` does run on a worker sounds a bit scary to me.
Noting that @richvdh self-assigned: I believe you're a bit overloaded at the moment; would it make sense to hand this off to someone else (@DMRobertson?) with you providing review once impl'd?
> Noting that @richvdh self-assigned: 

sorry, at one point it sounded like this was a drop-the-world level of urgency, but that turned out to be incorrect. Yes, it's unlikely I'll have time to look at this in the near future!
Let's tentatively try to address this after FOSDEM next year... noting that retention policies are still an experimental feature themselves.
related: #11165 
Very interested in a fix on this issue, we run matrix under Kubernetes, running purges from the master creates high load cpu spikes for several minutes on each scheduled purge, I'd rather have those spikes on a non critical worker like backgroundtasks. thx.

Resolving patch:
diff --git a/synapse/handlers/pagination.py b/synapse/handlers/pagination.py
--- a/synapse/handlers/pagination.py
+++ b/synapse/handlers/pagination.py
@@ -159,11 +159,9 @@ def __init__(self, hs: "HomeServer"):
         self._retention_allowed_lifetime_max = (
             hs.config.retention.retention_allowed_lifetime_max
         )
+        self._is_master = hs.config.worker.worker_app is None
 
-        if (
-            hs.config.worker.run_background_tasks
-            and hs.config.retention.retention_enabled
-        ):
+        if hs.config.retention.retention_enabled and self._is_master:
             # Run the purge jobs described in the configuration file.
             for job in hs.config.retention.retention_purge_jobs:
                 logger.info("Setting up purge job with config: %s", job)


Test patch:
Repository: matrix-org/synapse

Issue:
url_preview media cache is not expired when background database migrations are running
Normally, any media downloaded as part of a url preview is deleted after a day. However, while a background database migration is running, the process is stopped. If there is a long-running database migration, this can therefore cause the media store to fill up with old preview files.

This logic was added in https://github.com/matrix-org/synapse/pull/2697 to make sure that we didn't try to run the expiry without an index on `local_media_repository.created_ts`; the original logic that needs that index was added in https://github.com/matrix-org/synapse/pull/2478 (in [`get_url_cache_media_before`](https://github.com/matrix-org/synapse/pull/2478/files#diff-5772984c46f21c905ff00a9b6aebcb39a5fa21fec74c046fbd968b4721616a16R268-R273), as amended by https://github.com/matrix-org/synapse/commit/93247a424a5068b088567fa98b6990e47608b7cb), and is still present.

It looks like too broad a constraint: instead, we should use `has_completed_background_update` to check if the *specific* background update we care about has completed. However, given you'd have to be upgrading from before Synapse v1.0.0 (which created the `v54` schema snapshot) to *not* have this index in place... we should probably just drop the check altogether.


Discussion:
TL;DR: let's just remove [this check](https://github.com/matrix-org/synapse/blob/v1.65.0/synapse/rest/media/v1/preview_url_resource.py#L735).
> any media downloaded as part of a url preview is deleted after a day

Just to avoid confusion -- they're deleted after [two days](https://github.com/matrix-org/synapse/blob/1381563988c6dc7a2b8801b736b1f0c663970da8/synapse/rest/media/v1/preview_url_resource.py#L67). I don't think this changes any of the rest of the issue though. Thanks for researching!

Resolving patch:
diff --git a/synapse/rest/media/v1/preview_url_resource.py b/synapse/rest/media/v1/preview_url_resource.py
--- a/synapse/rest/media/v1/preview_url_resource.py
+++ b/synapse/rest/media/v1/preview_url_resource.py
@@ -732,10 +732,6 @@ async def _expire_url_cache_data(self) -> None:
 
         logger.debug("Running url preview cache expiry")
 
-        if not (await self.store.db_pool.updates.has_completed_background_updates()):
-            logger.debug("Still running DB updates; skipping url preview cache expiry")
-            return
-
         def try_remove_parent_dirs(dirs: Iterable[str]) -> None:
             """Attempt to remove the given chain of parent directories
 


Test patch:
Repository: matrix-org/synapse

Issue:
`is_partial_state_room` should have a cache
It has a 40 Hz transaction rate on matrix.org, approximately 4% CPU consumption on federation senders.


Discussion:

Resolving patch:
diff --git a/synapse/storage/databases/main/room.py b/synapse/storage/databases/main/room.py
--- a/synapse/storage/databases/main/room.py
+++ b/synapse/storage/databases/main/room.py
@@ -1185,8 +1185,9 @@ async def clear_partial_state_room(self, room_id: str) -> bool:
             )
             return False
 
-    @staticmethod
-    def _clear_partial_state_room_txn(txn: LoggingTransaction, room_id: str) -> None:
+    def _clear_partial_state_room_txn(
+        self, txn: LoggingTransaction, room_id: str
+    ) -> None:
         DatabasePool.simple_delete_txn(
             txn,
             table="partial_state_rooms_servers",
@@ -1197,7 +1198,9 @@ def _clear_partial_state_room_txn(txn: LoggingTransaction, room_id: str) -> None
             table="partial_state_rooms",
             keyvalues={"room_id": room_id},
         )
+        self._invalidate_cache_and_stream(txn, self.is_partial_state_room, (room_id,))
 
+    @cached()
     async def is_partial_state_room(self, room_id: str) -> bool:
         """Checks if this room has partial state.
 
@@ -1771,9 +1774,8 @@ async def store_partial_state_room(
             servers,
         )
 
-    @staticmethod
     def _store_partial_state_room_txn(
-        txn: LoggingTransaction, room_id: str, servers: Collection[str]
+        self, txn: LoggingTransaction, room_id: str, servers: Collection[str]
     ) -> None:
         DatabasePool.simple_insert_txn(
             txn,
@@ -1788,6 +1790,7 @@ def _store_partial_state_room_txn(
             keys=("room_id", "server_name"),
             values=((room_id, s) for s in servers),
         )
+        self._invalidate_cache_and_stream(txn, self.is_partial_state_room, (room_id,))
 
     async def maybe_store_room_on_outlier_membership(
         self, room_id: str, room_version: RoomVersion


Test patch:
Repository: matrix-org/synapse

Issue:
CI run against latest deps is failing
See https://github.com/matrix-org/synapse/actions/runs/2991362357



Discussion:
```
Run poetry run mypy
synapse/api/filtering.py:142: error: Value of type variable "_F" of function cannot be "Callable[[str], RoomID]"  [type-var]
synapse/api/filtering.py:147: error: Value of type variable "_F" of function cannot be "Callable[[str], UserID]"  [type-var]
```
Likely https://github.com/python/typeshed/pull/8673
I think the functions decorated by `cls_checks`

https://github.com/matrix-org/synapse/blob/c52abc1cfdd9e5480cdb4a03d626fe61cacc6573/synapse/api/filtering.py#L142-L149

are supposed to return a bool, judging by https://github.com/python/typeshed/pull/8673/files#diff-f5e2679481de689eab96a68a36277093752a2cfa26602b4ca82e1178b01a726fL5-R6

It's not clear to me if we actually use the return value e.g. to parse as well as validate?
See also https://github.com/python-jsonschema/jsonschema/pull/958

Resolving patch:
diff --git a/synapse/api/filtering.py b/synapse/api/filtering.py
--- a/synapse/api/filtering.py
+++ b/synapse/api/filtering.py
@@ -140,13 +140,13 @@
 
 
 @FormatChecker.cls_checks("matrix_room_id")
-def matrix_room_id_validator(room_id_str: str) -> RoomID:
-    return RoomID.from_string(room_id_str)
+def matrix_room_id_validator(room_id_str: str) -> bool:
+    return RoomID.is_valid(room_id_str)
 
 
 @FormatChecker.cls_checks("matrix_user_id")
-def matrix_user_id_validator(user_id_str: str) -> UserID:
-    return UserID.from_string(user_id_str)
+def matrix_user_id_validator(user_id_str: str) -> bool:
+    return UserID.is_valid(user_id_str)
 
 
 class Filtering:


Test patch:
Repository: matrix-org/synapse

Issue:
CI run against latest deps is failing
See https://github.com/matrix-org/synapse/actions/runs/2998201194



Discussion:
```
synapse/api/filtering.py:142: error: Value of type variable "_F" of function cannot be "Callable[[str], bool]"  [type-var]
synapse/api/filtering.py:147: error: Value of type variable "_F" of function cannot be "Callable[[str], bool]"  [type-var]
```

Erm, I thought I just fixed this in #13708? Although I suppose I didn't test this against the latest stubs. Oops.

Resolving patch:
diff --git a/synapse/api/filtering.py b/synapse/api/filtering.py
--- a/synapse/api/filtering.py
+++ b/synapse/api/filtering.py
@@ -140,13 +140,13 @@
 
 
 @FormatChecker.cls_checks("matrix_room_id")
-def matrix_room_id_validator(room_id_str: str) -> bool:
-    return RoomID.is_valid(room_id_str)
+def matrix_room_id_validator(room_id: object) -> bool:
+    return isinstance(room_id, str) and RoomID.is_valid(room_id)
 
 
 @FormatChecker.cls_checks("matrix_user_id")
-def matrix_user_id_validator(user_id_str: str) -> bool:
-    return UserID.is_valid(user_id_str)
+def matrix_user_id_validator(user_id: object) -> bool:
+    return isinstance(user_id, str) and UserID.is_valid(user_id)
 
 
 class Filtering:


Test patch:
Repository: matrix-org/synapse

Issue:
Synapse doesn't start if the signing key file contains a newline
### Description

When deploying a synapse instance I copied a `/data/mydomain.signing.key` file which contained the line Synapse had generated on first start and a new line.

Synapse didn't start and complained with 
```
Starting synapse with args -m synapse.app.homeserver --config-path /data/homeserver.yaml

Error in configuration:
  Error reading signing_key: not enough values to unpack (expected 3, got 0)
```

### Steps to reproduce

* Generate a new configuration file for Synapse.
* Add a newline in `/data/yourdomain.signing.key`
* Start the container.

### Homeserver

Container based deployment

### Synapse Version

1.66.0

### Installation Method

Docker (matrixdotorg/synapse)

### Platform

Rocky Linux 8.5 & Podman

### Relevant log output

```shell
Starting synapse with args -m synapse.app.homeserver --config-path /data/homeserver.yaml

Error in configuration:
  Error reading signing_key: not enough values to unpack (expected 3, got 0)
```


### Anything else that would be useful to know?

_No response_


Discussion:

Resolving patch:
diff --git a/synapse/config/key.py b/synapse/config/key.py
--- a/synapse/config/key.py
+++ b/synapse/config/key.py
@@ -217,7 +217,18 @@ def read_signing_keys(self, signing_key_path: str, name: str) -> List[SigningKey
 
         signing_keys = self.read_file(signing_key_path, name)
         try:
-            return read_signing_keys(signing_keys.splitlines(True))
+            loaded_signing_keys = read_signing_keys(
+                [
+                    signing_key_line
+                    for signing_key_line in signing_keys.splitlines(keepends=False)
+                    if signing_key_line.strip()
+                ]
+            )
+
+            if not loaded_signing_keys:
+                raise ConfigError(f"No signing keys in file {signing_key_path}")
+
+            return loaded_signing_keys
         except Exception as e:
             raise ConfigError("Error reading %s: %s" % (name, str(e)))
 


Test patch:
Repository: matrix-org/synapse

Issue:
Missing validation on typing EDUs
When we receive a typing EDU from a remote homeserver, we neglect to check that said homeserver is in the room the typing EDU is for.

There was a check that looked like it was doing the job, except it was really checking that the local homeserver is in the room a second time:
https://github.com/matrix-org/synapse/blob/1a209efdb2a6c51e52dd277de7581099852877ae/synapse/handlers/typing.py#L365-L369

related: #11456


Discussion:

Resolving patch:
diff --git a/synapse/handlers/typing.py b/synapse/handlers/typing.py
--- a/synapse/handlers/typing.py
+++ b/synapse/handlers/typing.py
@@ -362,11 +362,14 @@ async def _recv_edu(self, origin: str, content: JsonDict) -> None:
             )
             return
 
-        domains = await self._storage_controllers.state.get_current_hosts_in_room(
+        # Let's check that the origin server is in the room before accepting the typing
+        # event. We don't want to block waiting on a partial state so take an
+        # approximation if needed.
+        domains = await self._storage_controllers.state.get_current_hosts_in_room_or_partial_state_approximation(
             room_id
         )
 
-        if self.server_name in domains:
+        if user.domain in domains:
             logger.info("Got typing update from %s: %r", user_id, content)
             now = self.clock.time_msec()
             self._member_typing_until[member] = now + FEDERATION_TIMEOUT


Test patch:
diff --git a/tests/handlers/test_typing.py b/tests/handlers/test_typing.py
--- a/tests/handlers/test_typing.py
+++ b/tests/handlers/test_typing.py
@@ -138,6 +138,10 @@ async def get_current_hosts_in_room(room_id: str):
             get_current_hosts_in_room
         )
 
+        hs.get_storage_controllers().state.get_current_hosts_in_room_or_partial_state_approximation = (
+            get_current_hosts_in_room
+        )
+
         async def get_users_in_room(room_id: str):
             return {str(u) for u in self.room_members}