internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: matrix-org/synapse

Issue:
CI run against latest deps is failing
See https://github.com/matrix-org/synapse/actions/runs/2407332978



Discussion:
```
 synapse/config/_util.py:59: error: Subclass of "str" and "int" cannot exist: would have incompatible method signatures  [unreachable]
synapse/config/_util.py:60: error: Statement is unreachable  [unreachable]
```

https://github.com/matrix-org/synapse/blob/4b965c862dc66c0da5d3240add70e9b5f0aa720b/synapse/config/_util.py#L43-L63

I think the underlying change was the recent types-jsonschema release, see https://github.com/python/typeshed/pull/7950. The stubs now [say](https://github.com/sirosen/typeshed/blob/8ddc5934d3e9d06ef24bb9ce0f0d27a783f09237/stubs/jsonschema/jsonschema/exceptions.pyi#L41-L42) that `e.absolute_path` is a `Sequence[str]`.
The stubs are incorrect!

```python
import jsonschema

SCHEMA = {
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "array",
    "items": {"type": "string"},
}


document = ["a", 12345, "c", "d"]

try:
    jsonschema.validate(document, SCHEMA)
except jsonschema.ValidationError as e:
    print(repr(e))
    for path_entry in e.absolute_path:
        print(repr(path_entry), type(path_entry))
```

```
python temp.py
<ValidationError: "12345 is not of type 'string'">
1 <class 'int'>
```
Proposed a fix upstream at https://github.com/python/typeshed/pull/7980 

Resolving patch:
diff --git a/synapse/events/validator.py b/synapse/events/validator.py
--- a/synapse/events/validator.py
+++ b/synapse/events/validator.py
@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import collections.abc
-from typing import Iterable, Type, Union
+from typing import Iterable, Type, Union, cast
 
 import jsonschema
 
@@ -103,7 +103,12 @@ def validate_new(self, event: EventBase, config: HomeServerConfig) -> None:
             except jsonschema.ValidationError as e:
                 if e.path:
                     # example: "users_default": '0' is not of type 'integer'
-                    message = '"' + e.path[-1] + '": ' + e.message  # noqa: B306
+                    # cast safety: path entries can be integers, if we fail to validate
+                    # items in an array. However the POWER_LEVELS_SCHEMA doesn't expect
+                    # to see any arrays.
+                    message = (
+                        '"' + cast(str, e.path[-1]) + '": ' + e.message  # noqa: B306
+                    )
                     # jsonschema.ValidationError.message is a valid attribute
                 else:
                     # example: '0' is not of type 'integer'


Test patch:
Repository: matrix-org/synapse

Issue:
Allow updating password without logging out sessions via admin api
It'd be helpful to allow resetting passwords for users without logging them out of all of their sessions. When a user doesn't have their security key saved properly, as many do, this will prevent them from loosing access to their message history. What I'd imagine is a query parameter for the user update endpoint, that allows choosing between no logout at all, a soft-logout and a real logout, with the default being the real logout to not have a breaking change there.

I think a soft-logout is probably always the desired thing to do here, but there's also cases where no logout at all is the best option.

For additional context: https://github.com/Awesome-Technologies/synapse-admin/issues/268


Discussion:
This is possible for the client endpoint (see MSC2457), so all the infrastructure should be there.

It looks like we just need to read `logout_devices` from the JSON body instead of hard-coding to true:

https://github.com/matrix-org/synapse/blob/e24ff8ebe3d4119d377355402245947f7de61c00/synapse/rest/admin/users.py#L307-L316
It is also possible with this endpoint: https://matrix-org.github.io/synapse/latest/admin_api/user_admin_api.html#reset-password

Resolving patch:
diff --git a/synapse/rest/admin/users.py b/synapse/rest/admin/users.py
--- a/synapse/rest/admin/users.py
+++ b/synapse/rest/admin/users.py
@@ -226,6 +226,13 @@ async def on_PUT(
             if not isinstance(password, str) or len(password) > 512:
                 raise SynapseError(HTTPStatus.BAD_REQUEST, "Invalid password")
 
+        logout_devices = body.get("logout_devices", True)
+        if not isinstance(logout_devices, bool):
+            raise SynapseError(
+                HTTPStatus.BAD_REQUEST,
+                "'logout_devices' parameter is not of type boolean",
+            )
+
         deactivate = body.get("deactivated", False)
         if not isinstance(deactivate, bool):
             raise SynapseError(
@@ -305,7 +312,6 @@ async def on_PUT(
                     await self.store.set_server_admin(target_user, set_admin_to)
 
             if password is not None:
-                logout_devices = True
                 new_password_hash = await self.auth_handler.hash(password)
 
                 await self.set_password_handler.set_password(


Test patch:
Repository: matrix-org/synapse

Issue:
Remove groups/communities support from Synapse
The groups/communities feature in Synapse is unspecced (defined in [MSC1513](https://github.com/matrix-org/matrix-doc/issues/1513), kind of, but is abandoned) -- spaces ([MSC1772](https://github.com/matrix-org/matrix-doc/pull/1772) and friends).

There are still some useful features driven by Groups (namely flair) so it is unclear when this will happen. Note that Element Web has the capability to convert a Group -> Space.

A rough plan of how we might want to do this is:

* [x] Add an experimental config flag for groups that guards the unstable endpoints for groups.
* [x] Add a deprecation notice to our release and upgrade notes. (Should probably be done at the same time as 1.)
* [x] Default the above flag to `false` to disable those endpoints at the end of the deprecation period. The configuration flag should emit a warning if it is enabled.
* [x] (After some time...) Remove the groups code.
	* Endpoints
	* Handlers
	* Update the worker documentation
	* `RelatedGroups` constant
* [x] (After some time...) Bump the schema and remove the database tables for groups.

Related to #8334.


Discussion:
I'm not really sure if there's much benefit to step 3 -- we might want to just jump over it and do step 4 at the end of the deprecation period.
Removal is waiting for Synapse 1.61.
Almost all of the groups related code I could find has now been removed. There's a tiny bit left which will be handled in #12940.

Resolving patch:
diff --git a/synapse/_scripts/synapse_port_db.py b/synapse/_scripts/synapse_port_db.py
--- a/synapse/_scripts/synapse_port_db.py
+++ b/synapse/_scripts/synapse_port_db.py
@@ -166,22 +166,6 @@
     "ui_auth_sessions",
     "ui_auth_sessions_credentials",
     "ui_auth_sessions_ips",
-    # Groups/communities is no longer supported.
-    "group_attestations_remote",
-    "group_attestations_renewals",
-    "group_invites",
-    "group_roles",
-    "group_room_categories",
-    "group_rooms",
-    "group_summary_roles",
-    "group_summary_room_categories",
-    "group_summary_rooms",
-    "group_summary_users",
-    "group_users",
-    "groups",
-    "local_group_membership",
-    "local_group_updates",
-    "remote_profile_cache",
 }
 
 
diff --git a/synapse/storage/schema/__init__.py b/synapse/storage/schema/__init__.py
--- a/synapse/storage/schema/__init__.py
+++ b/synapse/storage/schema/__init__.py
@@ -74,13 +74,14 @@
 
 Changes in SCHEMA_VERSION = 72:
     - event_edges.(room_id, is_state) are no longer written to.
+    - Tables related to groups are dropped.
 """
 
 
 SCHEMA_COMPAT_VERSION = (
-    # We no longer maintain `event_edges.room_id`, so synapses with SCHEMA_VERSION < 71
-    # will break.
-    71
+    # The groups tables are no longer accessible, so synapses with SCHEMA_VERSION < 72
+    # could break.
+    72
 )
 """Limit on how far the synapse codebase can be rolled back without breaking db compat
 


Test patch:
Repository: matrix-org/synapse

Issue:
Default `sync_response_cache_duration` can lead to clients never being able to initial sync
### Description

Large accounts or slow servers can mean that the client will retry `/sync` (without `?since`) until it gets a result. However, the server's default cache time for sync responses seemingly went from some number of minutes to zero. This means that the server processes large `/sync` requests in the background but immediately throws away the result if the client gave up waiting and tried a second request.

There should be a non-zero default on the config value.

### Steps to reproduce

- Be in thousands of rooms
- Have a slow-ish server
- Try to sync your account


### Homeserver

t2l.io & t2bot.io

### Synapse Version

1.60.0

### Installation Method

pip (from PyPI)

### Platform

N/A

### Relevant log output

```shell
N/A
```


### Anything else that would be useful to know?

_No response_


Discussion:
This is sounding similar to (maybe the cause of) #12971.
> However, the server's default cache time for sync responses seemingly went from some number of minutes to zero. 

I don't believe it has ever been non-zero ftr, at least I can't see any evidence of that in the code.

Resolving patch:
diff --git a/synapse/config/cache.py b/synapse/config/cache.py
--- a/synapse/config/cache.py
+++ b/synapse/config/cache.py
@@ -206,7 +206,7 @@ def read_config(self, config: JsonDict, **kwargs: Any) -> None:
             self.cache_autotuning["min_cache_ttl"] = self.parse_duration(min_cache_ttl)
 
         self.sync_response_cache_duration = self.parse_duration(
-            cache_config.get("sync_response_cache_duration", 0)
+            cache_config.get("sync_response_cache_duration", "2m")
         )
 
     def resize_all_caches(self) -> None:


Test patch:
diff --git a/tests/utils.py b/tests/utils.py
--- a/tests/utils.py
+++ b/tests/utils.py
@@ -169,7 +169,7 @@ def default_config(name, parse=False):
         # disable user directory updates, because they get done in the
         # background, which upsets the test runner.
         "update_user_directory": False,
-        "caches": {"global_factor": 1},
+        "caches": {"global_factor": 1, "sync_response_cache_duration": 0},
         "listeners": [{"port": 0, "type": "http"}],
     }
Repository: matrix-org/synapse

Issue:
Update worker-mode Synapse-in-Complement config to use `update_user_directory_from_worker` in lieu of deprecated option
Not urgent by any means, but this kind of logging is noisy and distracts from the real problems.

```
synapse_main | The 'update_user_directory' configuration option is deprecated and will be removed in a future
synapse_main | Synapse version. Please use ``update_user_directory_from_worker: name_of_worker`` instead.
```
There's another option for application services, too.

Let's change the config template.



Discussion:

Resolving patch:
diff --git a/docker/configure_workers_and_start.py b/docker/configure_workers_and_start.py
--- a/docker/configure_workers_and_start.py
+++ b/docker/configure_workers_and_start.py
@@ -52,12 +52,12 @@
         "worker_extra_conf": "",
     },
     "user_dir": {
-        "app": "synapse.app.user_dir",
+        "app": "synapse.app.generic_worker",
         "listener_resources": ["client"],
         "endpoint_patterns": [
             "^/_matrix/client/(api/v1|r0|v3|unstable)/user_directory/search$"
         ],
-        "shared_extra_conf": {"update_user_directory": False},
+        "shared_extra_conf": {"update_user_directory_from_worker": "user_dir1"},
         "worker_extra_conf": "",
     },
     "media_repository": {
@@ -78,7 +78,7 @@
         "app": "synapse.app.generic_worker",
         "listener_resources": [],
         "endpoint_patterns": [],
-        "shared_extra_conf": {"notify_appservices_from_worker": "appservice"},
+        "shared_extra_conf": {"notify_appservices_from_worker": "appservice1"},
         "worker_extra_conf": "",
     },
     "federation_sender": {


Test patch:
Repository: matrix-org/synapse

Issue:
`_run_push_actions_and_persist_event` re-starts a finished logging context when `persist_events_and_notify` raises an exception
When `persist_events_and_notify` raises an exception, we try to clean up push actions in a background task:

https://github.com/matrix-org/synapse/blob/e3163e2e11cf8bffa4cb3e58ac0b86a83eca314c/synapse/handlers/federation_event.py#L1961-L1969

`run_in_background` inherits the current logging context, but since we don't await the background task, we end up closing the logging context before the background task finishes. We ought to either await `remove_push_actions_from_staging` directly, or create a logging context for it.


Discussion:
Actually every call to `run_in_background` where we discard the return value probably has the same issue.
> Actually every call to `run_in_background` where we discard the return value probably has the same issue.

I think `grep -r "  run_in_background" synapse` (note the two spaces before it) would be a quick way to find those. I see a bunch. 😢 

Resolving patch:
diff --git a/synapse/handlers/federation_event.py b/synapse/handlers/federation_event.py
--- a/synapse/handlers/federation_event.py
+++ b/synapse/handlers/federation_event.py
@@ -57,7 +57,7 @@
 from synapse.events import EventBase
 from synapse.events.snapshot import EventContext
 from synapse.federation.federation_client import InvalidResponseError
-from synapse.logging.context import nested_logging_context, run_in_background
+from synapse.logging.context import nested_logging_context
 from synapse.metrics.background_process_metrics import run_as_background_process
 from synapse.replication.http.devices import ReplicationUserDevicesResyncRestServlet
 from synapse.replication.http.federation import (
@@ -1964,9 +1964,7 @@ async def _run_push_actions_and_persist_event(
                 event.room_id, [(event, context)], backfilled=backfilled
             )
         except Exception:
-            run_in_background(
-                self._store.remove_push_actions_from_staging, event.event_id
-            )
+            await self._store.remove_push_actions_from_staging(event.event_id)
             raise
 
     async def persist_events_and_notify(


Test patch:
Repository: matrix-org/synapse

Issue:
We do lots of processing of /send requests for users who have hit their ratelimit
We have a limit on the rate a user is allowed to send events, but it is applied quite late in the processing cycle (specifically, it happens in `persist_and_notify_client_event). 

That means we go to all the effort of getting a lock on the room, checking the state, building the event, making the request to the event persister - all to be told that the user is way over their ratelimit.

This means it's quite easy for a single user to overload an event creator worker with lots of /send requests.


Discussion:
Relatedly: because we limit the number of concurrent sends per room, and this process of constructing the event and calling out to the event persister takes 100ms or so, we can end up with a big queue of requests for a given room, which means we get very slow response times for the events that *are* going through to that room. That in turn causes performance pages.
Jaeger trace for a request which got 429ed:

![image](https://user-images.githubusercontent.com/1389908/173860327-26b82a50-f180-46b1-b763-59f3de40ccb9.png)

I believe the 100 second delay before anything happens is waiting for
```
        async with self.limiter.queue(event_dict["room_id"]):
```
Huh, I really thought that we checked the ratelimiting up front as well (with `update=False`), but I can't find any evidence of that...

Resolving patch:
diff --git a/synapse/handlers/message.py b/synapse/handlers/message.py
--- a/synapse/handlers/message.py
+++ b/synapse/handlers/message.py
@@ -903,6 +903,9 @@ async def create_and_send_nonmember_event(
             await self.clock.sleep(random.randint(1, 10))
             raise ShadowBanError()
 
+        if ratelimit:
+            await self.request_ratelimiter.ratelimit(requester, update=False)
+
         # We limit the number of concurrent event sends in a room so that we
         # don't fork the DAG too much. If we don't limit then we can end up in
         # a situation where event persistence can't keep up, causing


Test patch:
Repository: matrix-org/synapse

Issue:
Make `complement.sh` respect `SYNAPSE_TEST_LOG_LEVEL=DEBUG`
This would be consistent with Trial.

Currently the log level is hard-coded in the logging config file.


Discussion:

Resolving patch:
diff --git a/docker/configure_workers_and_start.py b/docker/configure_workers_and_start.py
--- a/docker/configure_workers_and_start.py
+++ b/docker/configure_workers_and_start.py
@@ -29,6 +29,10 @@
 #   * SYNAPSE_USE_EXPERIMENTAL_FORKING_LAUNCHER: Whether to use the forking launcher,
 #         only intended for usage in Complement at the moment.
 #         No stability guarantees are provided.
+#   * SYNAPSE_LOG_LEVEL: Set this to DEBUG, INFO, WARNING or ERROR to change the
+#         log level. INFO is the default.
+#   * SYNAPSE_LOG_SENSITIVE: If unset, SQL and SQL values won't be logged,
+#         regardless of the SYNAPSE_LOG_LEVEL setting.
 #
 # NOTE: According to Complement's ENTRYPOINT expectations for a homeserver image (as defined
 # in the project's README), this script may be run multiple times, and functionality should
@@ -38,7 +42,7 @@
 import subprocess
 import sys
 from pathlib import Path
-from typing import Any, Dict, List, Mapping, MutableMapping, NoReturn, Set
+from typing import Any, Dict, List, Mapping, MutableMapping, NoReturn, Optional, Set
 
 import yaml
 from jinja2 import Environment, FileSystemLoader
@@ -552,13 +556,17 @@ def generate_worker_log_config(
     Returns: the path to the generated file
     """
     # Check whether we should write worker logs to disk, in addition to the console
-    extra_log_template_args = {}
+    extra_log_template_args: Dict[str, Optional[str]] = {}
     if environ.get("SYNAPSE_WORKERS_WRITE_LOGS_TO_DISK"):
-        extra_log_template_args["LOG_FILE_PATH"] = "{dir}/logs/{name}.log".format(
-            dir=data_dir, name=worker_name
-        )
+        extra_log_template_args["LOG_FILE_PATH"] = f"{data_dir}/logs/{worker_name}.log"
+
+    extra_log_template_args["SYNAPSE_LOG_LEVEL"] = environ.get("SYNAPSE_LOG_LEVEL")
+    extra_log_template_args["SYNAPSE_LOG_SENSITIVE"] = environ.get(
+        "SYNAPSE_LOG_SENSITIVE"
+    )
+
     # Render and write the file
-    log_config_filepath = "/conf/workers/{name}.log.config".format(name=worker_name)
+    log_config_filepath = f"/conf/workers/{worker_name}.log.config"
     convert(
         "/conf/log.config",
         log_config_filepath,


Test patch:
Repository: matrix-org/synapse

Issue:
Log an exception when we try and write malformed server names to `destinations` table.
We are somehow getting malformed entries in the destination table, which while not directly a problem is also not something we want. We should log an exception when we try and write such a row so that we an slowly track down all the places this can happen.


Discussion:

Resolving patch:
diff --git a/synapse/http/matrixfederationclient.py b/synapse/http/matrixfederationclient.py
--- a/synapse/http/matrixfederationclient.py
+++ b/synapse/http/matrixfederationclient.py
@@ -79,6 +79,7 @@
 from synapse.util import json_decoder
 from synapse.util.async_helpers import AwakenableSleeper, timeout_deferred
 from synapse.util.metrics import Measure
+from synapse.util.stringutils import parse_and_validate_server_name
 
 if TYPE_CHECKING:
     from synapse.server import HomeServer
@@ -479,6 +480,14 @@ async def _send_request(
             RequestSendFailed: If there were problems connecting to the
                 remote, due to e.g. DNS failures, connection timeouts etc.
         """
+        # Validate server name and log if it is an invalid destination, this is
+        # partially to help track down code paths where we haven't validated before here
+        try:
+            parse_and_validate_server_name(request.destination)
+        except ValueError:
+            logger.exception(f"Invalid destination: {request.destination}.")
+            raise FederationDeniedError(request.destination)
+
         if timeout:
             _sec_timeout = timeout / 1000
         else:


Test patch:
diff --git a/tests/federation/test_federation_client.py b/tests/federation/test_federation_client.py
--- a/tests/federation/test_federation_client.py
+++ b/tests/federation/test_federation_client.py
@@ -102,7 +102,7 @@ def test_get_room_state(self):
         # now fire off the request
         state_resp, auth_resp = self.get_success(
             self.hs.get_federation_client().get_room_state(
-                "yet_another_server",
+                "yet.another.server",
                 test_room_id,
                 "event_id",
                 RoomVersions.V9,
@@ -112,7 +112,7 @@ def test_get_room_state(self):
         # check the right call got made to the agent
         self._mock_agent.request.assert_called_once_with(
             b"GET",
-            b"matrix://yet_another_server/_matrix/federation/v1/state/%21room_id?event_id=event_id",
+            b"matrix://yet.another.server/_matrix/federation/v1/state/%21room_id?event_id=event_id",
             headers=mock.ANY,
             bodyProducer=None,
         )
Repository: matrix-org/synapse

Issue:
Ubuntu 21.10 (Impish Indri) has reached end of life as of July 14, 2022
See https://lists.ubuntu.com/archives/ubuntu-announce/2022-July/000281.html

I don't think we have good docs for removing a distribution, but should be the opposite of [gitlab.matrix.org/new-vector/internal/-/wikis/Synapse-Debian-Packages#adding-a-new-distribution](https://gitlab.matrix.org/new-vector/internal/-/wikis/Synapse-Debian-Packages#adding-a-new-distribution).


Discussion:
(And for cross-linking, this was added in #10904.)
> I don't think we have good docs for removing a distribution, 

you just need to remove it from the build list: https://github.com/matrix-org/synapse/pull/11961/files.

> but should be the opposite of [gitlab.matrix.org/new-vector/internal/-/wikis/Synapse-Debian-Packages#adding-a-new-distribution](https://gitlab.matrix.org/new-vector/internal/-/wikis/Synapse-Debian-Packages#adding-a-new-distribution).

No, we keep the repository around for old distributions - we just stop publishing to it.

Resolving patch:
diff --git a/scripts-dev/build_debian_packages.py b/scripts-dev/build_debian_packages.py
--- a/scripts-dev/build_debian_packages.py
+++ b/scripts-dev/build_debian_packages.py
@@ -26,7 +26,6 @@
     "debian:bookworm",
     "debian:sid",
     "ubuntu:focal",  # 20.04 LTS (our EOL forced by Py38 on 2024-10-14)
-    "ubuntu:impish",  # 21.10  (EOL 2022-07)
     "ubuntu:jammy",  # 22.04 LTS (EOL 2027-04)
 )
 


Test patch:
Repository: matrix-org/synapse

Issue:
CI run against latest deps is failing
See https://github.com/matrix-org/synapse/actions/runs/2730494056



Discussion:

Resolving patch:
diff --git a/synapse/logging/opentracing.py b/synapse/logging/opentracing.py
--- a/synapse/logging/opentracing.py
+++ b/synapse/logging/opentracing.py
@@ -910,8 +910,8 @@ def tag_args(func: Callable[P, R]) -> Callable[P, R]:
     def _tag_args_inner(*args: P.args, **kwargs: P.kwargs) -> R:
         argspec = inspect.getfullargspec(func)
         for i, arg in enumerate(argspec.args[1:]):
-            set_tag("ARG_" + arg, args[i])  # type: ignore[index]
-        set_tag("args", args[len(argspec.args) :])  # type: ignore[index]
+            set_tag("ARG_" + arg, str(args[i]))  # type: ignore[index]
+        set_tag("args", str(args[len(argspec.args) :]))  # type: ignore[index]
         set_tag("kwargs", str(kwargs))
         return func(*args, **kwargs)
 


Test patch: