internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: matrix-org/synapse

Issue:
KeyError: 'unsigned' in _inject_bundled_aggregations
https://sentry.matrix.org/sentry/synapse-matrixorg/issues/243572/

This seems to happen if a filter is provided which specifies specific fields to return, investigating more now though...


Discussion:

Resolving patch:
diff --git a/synapse/events/utils.py b/synapse/events/utils.py
--- a/synapse/events/utils.py
+++ b/synapse/events/utils.py
@@ -530,9 +530,12 @@ def _inject_bundled_aggregations(
 
         # Include the bundled aggregations in the event.
         if serialized_aggregations:
-            serialized_event["unsigned"].setdefault("m.relations", {}).update(
-                serialized_aggregations
-            )
+            # There is likely already an "unsigned" field, but a filter might
+            # have stripped it off (via the event_fields option). The server is
+            # allowed to return additional fields, so add it back.
+            serialized_event.setdefault("unsigned", {}).setdefault(
+                "m.relations", {}
+            ).update(serialized_aggregations)
 
     def serialize_events(
         self,


Test patch:
diff --git a/tests/rest/client/test_relations.py b/tests/rest/client/test_relations.py
--- a/tests/rest/client/test_relations.py
+++ b/tests/rest/client/test_relations.py
@@ -1267,6 +1267,34 @@ def test_background_update(self) -> None:
             [annotation_event_id_good, thread_event_id],
         )
 
+    def test_bundled_aggregations_with_filter(self) -> None:
+        """
+        If "unsigned" is an omitted field (due to filtering), adding the bundled
+        aggregations should not break.
+
+        Note that the spec allows for a server to return additional fields beyond
+        what is specified.
+        """
+        self._send_relation(RelationTypes.ANNOTATION, "m.reaction", "a")
+
+        # Note that the sync filter does not include "unsigned" as a field.
+        filter = urllib.parse.quote_plus(
+            b'{"event_fields": ["content", "event_id"], "room": {"timeline": {"limit": 3}}}'
+        )
+        channel = self.make_request(
+            "GET", f"/sync?filter={filter}", access_token=self.user_token
+        )
+        self.assertEqual(200, channel.code, channel.json_body)
+
+        # Ensure the timeline is limited, find the parent event.
+        room_timeline = channel.json_body["rooms"]["join"][self.room]["timeline"]
+        self.assertTrue(room_timeline["limited"])
+        parent_event = self._find_event_in_chunk(room_timeline["events"])
+
+        # Ensure there's bundled aggregations on it.
+        self.assertIn("unsigned", parent_event)
+        self.assertIn("m.relations", parent_event["unsigned"])
+
 
 class RelationRedactionTestCase(BaseRelationsTestCase):
     """
Repository: matrix-org/synapse

Issue:
/send_join response includes unstable prefix for event field
The response generated for `/send_join` (and parsing of it on the federation client side) still uses an unstable prefix from [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083): `org.matrix.msc3083.v2.event`.

This should be updated to be just `event`, the stable field name. We probably need to send / handle both for a period of time unfortunately.

See https://github.com/matrix-org/synapse/blob/98c8fc6ce82d9d6b1bd21bf70df6a0e1ce91c1dc/synapse/federation/federation_server.py#L617


Discussion:

Resolving patch:
diff --git a/synapse/federation/federation_server.py b/synapse/federation/federation_server.py
--- a/synapse/federation/federation_server.py
+++ b/synapse/federation/federation_server.py
@@ -687,8 +687,6 @@ async def on_send_join_request(
         time_now = self._clock.time_msec()
         event_json = event.get_pdu_json(time_now)
         resp = {
-            # TODO Remove the unstable prefix when servers have updated.
-            "org.matrix.msc3083.v2.event": event_json,
             "event": event_json,
             "state": [p.get_pdu_json(time_now) for p in state_events],
             "auth_chain": [p.get_pdu_json(time_now) for p in auth_chain_events],
diff --git a/synapse/federation/transport/client.py b/synapse/federation/transport/client.py
--- a/synapse/federation/transport/client.py
+++ b/synapse/federation/transport/client.py
@@ -1379,16 +1379,6 @@ def __init__(self, room_version: RoomVersion, v1_api: bool):
                 prefix + "auth_chain.item",
                 use_float=True,
             ),
-            # TODO Remove the unstable prefix when servers have updated.
-            #
-            # By re-using the same event dictionary this will cause the parsing of
-            # org.matrix.msc3083.v2.event and event to stomp over each other.
-            # Generally this should be fine.
-            ijson.kvitems_coro(
-                _event_parser(self._response.event_dict),
-                prefix + "org.matrix.msc3083.v2.event",
-                use_float=True,
-            ),
             ijson.kvitems_coro(
                 _event_parser(self._response.event_dict),
                 prefix + "event",


Test patch:
Repository: matrix-org/synapse

Issue:
Stabilize support for MSC3288: room type on the store-invite API
Once [MSC3288](https://github.com/matrix-org/matrix-doc/pull/3288) lands we should (at some point) switch to a stable identifier.

There's a single case to switch:

https://github.com/matrix-org/synapse/blob/eb9ddc8c2e807e691fd1820f88f7c0bf43822661/synapse/handlers/identity.py#L879-L880

We might want to send both the stable and unstable identifiers for a bit.


Discussion:
Note that Synapse v1.47.0 (released on 2021-11-17) started sending both the unstable and stable identifier. I think we can drop it now.

Resolving patch:
diff --git a/synapse/handlers/identity.py b/synapse/handlers/identity.py
--- a/synapse/handlers/identity.py
+++ b/synapse/handlers/identity.py
@@ -858,8 +858,6 @@ async def ask_id_server_for_third_party_invite(
 
         if room_type is not None:
             invite_config["room_type"] = room_type
-            # TODO The unstable field is deprecated and should be removed in the future.
-            invite_config["org.matrix.msc3288.room_type"] = room_type
 
         # If a custom web client location is available, include it in the request.
         if self._web_client_location:


Test patch:
Repository: matrix-org/synapse

Issue:
packages for ubuntu 22.04
Ubuntu 22.04 "Jammy Jellyfish" will be released on 2022-04-21, and it has already been in beta for a week. We should add it to the list of distros we build debs for.

cf https://github.com/matrix-org/synapse/issues/9732


Discussion:
It has now been released.
Two parts to this (cf https://gitlab.matrix.org/new-vector/internal/-/wikis/Synapse-Debian-Packages#adding-a-new-distribution)

* add an entry to `scripts-dev/build_debian_packages`, cf #9909
* add support to the `packages.matrix.org` repo, cf https://github.com/matrix-org/pkg-repo-configs/pull/4


Resolving patch:
diff --git a/scripts-dev/build_debian_packages.py b/scripts-dev/build_debian_packages.py
--- a/scripts-dev/build_debian_packages.py
+++ b/scripts-dev/build_debian_packages.py
@@ -26,6 +26,7 @@
     "debian:sid",
     "ubuntu:focal",  # 20.04 LTS (our EOL forced by Py38 on 2024-10-14)
     "ubuntu:impish",  # 21.10  (EOL 2022-07)
+    "ubuntu:jammy",  # 22.04 LTS (EOL 2027-04)
 )
 
 DESC = """\


Test patch:
Repository: matrix-org/synapse

Issue:
'apparently some people use access tokens that are not macaroons'
This comment seems stale, since it now seems like we encourage and use `syt_...` tokens by default which are not macaroons?

https://github.com/matrix-org/synapse/blob/8c7a531e277f98ac6b7981b9738649f3a70feb94/synapse/api/auth.py#L417-L418


Discussion:
yes it is. That comment dates from a period where we were trying to move away from random strings as access tokens, towards macaroons. We've long since aborted that plan.

Resolving patch:
diff --git a/synapse/api/auth.py b/synapse/api/auth.py
--- a/synapse/api/auth.py
+++ b/synapse/api/auth.py
@@ -417,7 +417,8 @@ async def get_user_by_access_token(
         """
 
         if rights == "access":
-            # first look in the database
+            # First look in the database to see if the access token is present
+            # as an opaque token.
             r = await self.store.get_user_by_access_token(token)
             if r:
                 valid_until_ms = r.valid_until_ms
@@ -434,7 +435,8 @@ async def get_user_by_access_token(
 
                 return r
 
-        # otherwise it needs to be a valid macaroon
+        # If the token isn't found in the database, then it could still be a
+        # macaroon, so we check that here.
         try:
             user_id, guest = self._parse_and_validate_macaroon(token, rights)
 
@@ -482,8 +484,12 @@ async def get_user_by_access_token(
             TypeError,
             ValueError,
         ) as e:
-            logger.warning("Invalid macaroon in auth: %s %s", type(e), e)
-            raise InvalidClientTokenError("Invalid macaroon passed.")
+            logger.warning(
+                "Invalid access token in auth: %s %s.",
+                type(e),
+                e,
+            )
+            raise InvalidClientTokenError("Invalid access token passed.")
 
     def _parse_and_validate_macaroon(
         self, token: str, rights: str = "access"
@@ -504,10 +510,7 @@ def _parse_and_validate_macaroon(
         try:
             macaroon = pymacaroons.Macaroon.deserialize(token)
         except Exception:  # deserialize can throw more-or-less anything
-            # doesn't look like a macaroon: treat it as an opaque token which
-            # must be in the database.
-            # TODO: it would be nice to get rid of this, but apparently some
-            # people use access tokens which aren't macaroons
+            # The access token doesn't look like a macaroon.
             raise _InvalidMacaroonException()
 
         try:


Test patch:
Repository: matrix-org/synapse

Issue:
Remove unstable flag for MSC3069: Allow guests to use /account/whoami
For Synapse v1.55.0 (or later) we should remove the unstable identifiers for MSC3069. The stable ones were added in #12021 for v1.54.0.


Discussion:

Resolving patch:
diff --git a/synapse/rest/client/account.py b/synapse/rest/client/account.py
--- a/synapse/rest/client/account.py
+++ b/synapse/rest/client/account.py
@@ -882,9 +882,7 @@ async def on_GET(self, request: SynapseRequest) -> Tuple[int, JsonDict]:
 
         response = {
             "user_id": requester.user.to_string(),
-            # MSC: https://github.com/matrix-org/matrix-doc/pull/3069
             # Entered spec in Matrix 1.2
-            "org.matrix.msc3069.is_guest": bool(requester.is_guest),
             "is_guest": bool(requester.is_guest),
         }
 


Test patch:
diff --git a/tests/rest/client/test_account.py b/tests/rest/client/test_account.py
--- a/tests/rest/client/test_account.py
+++ b/tests/rest/client/test_account.py
@@ -520,8 +520,6 @@ def test_GET_whoami(self) -> None:
             {
                 "user_id": user_id,
                 "device_id": device_id,
-                # MSC3069 entered spec in Matrix 1.2 but maintained compatibility
-                "org.matrix.msc3069.is_guest": False,
                 "is_guest": False,
             },
         )
@@ -540,8 +538,6 @@ def test_GET_whoami_guests(self) -> None:
             {
                 "user_id": user_id,
                 "device_id": device_id,
-                # MSC3069 entered spec in Matrix 1.2 but maintained compatibility
-                "org.matrix.msc3069.is_guest": True,
                 "is_guest": True,
             },
         )
@@ -564,8 +560,6 @@ def test_GET_whoami_appservices(self) -> None:
             whoami,
             {
                 "user_id": user_id,
-                # MSC3069 entered spec in Matrix 1.2 but maintained compatibility
-                "org.matrix.msc3069.is_guest": False,
                 "is_guest": False,
             },
         )
Repository: matrix-org/synapse

Issue:
Read receipts on element web are broken
Broke by #12168. Element web sends a hidden read receipt type, which Synapse doesn't understand and so 400s:

https://github.com/matrix-org/synapse/blob/0ce2201932488e98cab8a7d81788e5bcf8f8dd5e/synapse/rest/client/read_marker.py#L57-L63

This is a problem as it stops the *other* receipt types from being persisted.


We shouldn't be denying unknown receipt types.


Discussion:

Resolving patch:
diff --git a/synapse/rest/client/read_marker.py b/synapse/rest/client/read_marker.py
--- a/synapse/rest/client/read_marker.py
+++ b/synapse/rest/client/read_marker.py
@@ -16,7 +16,6 @@
 from typing import TYPE_CHECKING, Tuple
 
 from synapse.api.constants import ReceiptTypes
-from synapse.api.errors import SynapseError
 from synapse.http.server import HttpServer
 from synapse.http.servlet import RestServlet, parse_json_object_from_request
 from synapse.http.site import SynapseRequest
@@ -50,17 +49,21 @@ async def on_POST(
 
         body = parse_json_object_from_request(request)
 
-        valid_receipt_types = {ReceiptTypes.READ, ReceiptTypes.FULLY_READ}
-        if self.config.experimental.msc2285_enabled:
-            valid_receipt_types.add(ReceiptTypes.READ_PRIVATE)
-
-        if set(body.keys()) > valid_receipt_types:
-            raise SynapseError(
-                400,
-                "Receipt type must be 'm.read', 'org.matrix.msc2285.read.private' or 'm.fully_read'"
-                if self.config.experimental.msc2285_enabled
-                else "Receipt type must be 'm.read' or 'm.fully_read'",
-            )
+        valid_receipt_types = {
+            ReceiptTypes.READ,
+            ReceiptTypes.FULLY_READ,
+            ReceiptTypes.READ_PRIVATE,
+        }
+
+        unrecognized_types = set(body.keys()) - valid_receipt_types
+        if unrecognized_types:
+            # It's fine if there are unrecognized receipt types, but let's log
+            # it to help debug clients that have typoed the receipt type.
+            #
+            # We specifically *don't* error here, as a) it stops us processing
+            # the valid receipts, and b) we need to be extensible on receipt
+            # types.
+            logger.info("Ignoring unrecognized receipt types: %s", unrecognized_types)
 
         read_event_id = body.get(ReceiptTypes.READ, None)
         if read_event_id:


Test patch:
Repository: matrix-org/synapse

Issue:
CI run against latest deps is failing
See https://github.com/matrix-org/synapse/actions/runs/2292542675



Discussion:
Typechecking failure only. Presumably due to https://github.com/python/typeshed/commit/032d937bdf0c1b659b64bdaebe961a8bdac278e2

Resolving patch:
diff --git a/synapse/rest/media/v1/thumbnailer.py b/synapse/rest/media/v1/thumbnailer.py
--- a/synapse/rest/media/v1/thumbnailer.py
+++ b/synapse/rest/media/v1/thumbnailer.py
@@ -121,10 +121,10 @@ def _resize(self, width: int, height: int) -> Image.Image:
         #
         # If the image has transparency, use RGBA instead.
         if self.image.mode in ["1", "L", "P"]:
-            mode = "RGB"
             if self.image.info.get("transparency", None) is not None:
-                mode = "RGBA"
-            self.image = self.image.convert(mode)
+                self.image = self.image.convert("RGBA")
+            else:
+                self.image = self.image.convert("RGB")
         return self.image.resize((width, height), Image.ANTIALIAS)
 
     def scale(self, width: int, height: int, output_type: str) -> BytesIO:


Test patch:
Repository: matrix-org/synapse

Issue:
Recurring `MacaroonInitException: Must supply serialized macaroon.` in sentry
Internal users: https://sentry.matrix.org/sentry/synapse-matrixorg/issues/219508/?query=is%3Aunresolved

First seen in 1.32.2, according to sentry.

```
Invalid session for OIDC callback

MacaroonInitException: Must supply serialized macaroon.
  File "synapse/handlers/oidc.py", line 220, in handle_oidc_callback
    session, state
  File "synapse/handlers/oidc.py", line 1186, in verify_oidc_session_token
    macaroon = pymacaroons.Macaroon.deserialize(session)
  File "pymacaroons/macaroon.py", line 50, in deserialize
    'Must supply serialized macaroon.'
```
Judging from the source code below, it looks like these are errors that we expect to encounter from time to time. Maybe `logger.execption` -> `logger.warning`?

https://github.com/matrix-org/synapse/blob/646324437543c096e737777c81b4fe4b45c3e1a7/synapse/handlers/oidc.py#L217-L229


Discussion:
Discussed this during triage. Consensus: this is sensible, let's do it.

Resolving patch:
diff --git a/synapse/handlers/oidc.py b/synapse/handlers/oidc.py
--- a/synapse/handlers/oidc.py
+++ b/synapse/handlers/oidc.py
@@ -224,7 +224,7 @@ async def handle_oidc_callback(self, request: SynapseRequest) -> None:
             self._sso_handler.render_error(request, "invalid_session", str(e))
             return
         except MacaroonInvalidSignatureException as e:
-            logger.exception("Could not verify session for OIDC callback")
+            logger.warning("Could not verify session for OIDC callback: %s", e)
             self._sso_handler.render_error(request, "mismatching_session", str(e))
             return
 
@@ -827,7 +827,7 @@ async def handle_oidc_callback(
             logger.debug("Exchanging OAuth2 code for a token")
             token = await self._exchange_code(code)
         except OidcError as e:
-            logger.exception("Could not exchange OAuth2 code")
+            logger.warning("Could not exchange OAuth2 code: %s", e)
             self._sso_handler.render_error(request, e.error, e.error_description)
             return
 


Test patch:
Repository: matrix-org/synapse

Issue:
"Re-starting finished log context" due to `RetryDestinationLimiter`
`RetryDestinationLimiter.__exit__` stores retry timings in a background process:

https://github.com/matrix-org/synapse/blob/8dd3e0e084304dfc02ff072a1beaed5266cf4e33/synapse/util/retryutils.py#L267-L268

`store_retry_timings` inherits the current logcontext, which likely comes from a `with LogContext(): ...` further up the stack.
But there's no guarantee that `store_retry_timings` will finish before we exit that context manager further up the stack.


Discussion:

Resolving patch:
diff --git a/synapse/util/retryutils.py b/synapse/util/retryutils.py
--- a/synapse/util/retryutils.py
+++ b/synapse/util/retryutils.py
@@ -16,8 +16,8 @@
 from types import TracebackType
 from typing import TYPE_CHECKING, Any, Optional, Type
 
-import synapse.logging.context
 from synapse.api.errors import CodeMessageException
+from synapse.metrics.background_process_metrics import run_as_background_process
 from synapse.storage import DataStore
 from synapse.util import Clock
 
@@ -265,4 +265,4 @@ async def store_retry_timings() -> None:
                 logger.exception("Failed to store destination_retry_timings")
 
         # we deliberately do this in the background.
-        synapse.logging.context.run_in_background(store_retry_timings)
+        run_as_background_process("store_retry_timings", store_retry_timings)


Test patch:
Repository: matrix-org/synapse

Issue:
Move `USER_IP` replication command to a separate redis pub/sub channel.
Currently all replication commands get sent to all workers via the single pub/sub stream. This can be a lot of traffic, roughly evenly split between `USER_IP` and `RDATA` commands. 

However, only a subset of workers care about `USER_IP` commands, and the rest just immediately drop them on receipt:

https://github.com/matrix-org/synapse/blob/e3a49f4784d5c915355ac9306e60b09433db60b5/synapse/replication/tcp/handler.py#L412-L423

Instead of sending them to the same pub/sub channel, we should send `USER_IP` commands to a separate channel that is only subscribed to by workers that need them.

This would involve changing the stream name we send the `USER_IP` command to here:

https://github.com/matrix-org/synapse/blob/e3a49f4784d5c915355ac9306e60b09433db60b5/synapse/replication/tcp/redis.py#L219-L221

And change the channels we subscribe to:

https://github.com/matrix-org/synapse/blob/e3a49f4784d5c915355ac9306e60b09433db60b5/synapse/replication/tcp/redis.py#L121


Discussion:

Resolving patch:
diff --git a/synapse/replication/tcp/commands.py b/synapse/replication/tcp/commands.py
--- a/synapse/replication/tcp/commands.py
+++ b/synapse/replication/tcp/commands.py
@@ -58,6 +58,15 @@ def get_logcontext_id(self) -> str:
         # by default, we just use the command name.
         return self.NAME
 
+    def redis_channel_name(self, prefix: str) -> str:
+        """
+        Returns the Redis channel name upon which to publish this command.
+
+        Args:
+            prefix: The prefix for the channel.
+        """
+        return prefix
+
 
 SC = TypeVar("SC", bound="_SimpleCommand")
 
@@ -395,6 +404,9 @@ def __repr__(self) -> str:
             f"{self.user_agent!r}, {self.device_id!r}, {self.last_seen})"
         )
 
+    def redis_channel_name(self, prefix: str) -> str:
+        return f"{prefix}/USER_IP"
+
 
 class RemoteServerUpCommand(_SimpleCommand):
     """Sent when a worker has detected that a remote server is no longer
diff --git a/synapse/replication/tcp/redis.py b/synapse/replication/tcp/redis.py
--- a/synapse/replication/tcp/redis.py
+++ b/synapse/replication/tcp/redis.py
@@ -221,10 +221,10 @@ async def _async_send_command(self, cmd: Command) -> None:
         # remote instances.
         tcp_outbound_commands_counter.labels(cmd.NAME, "redis").inc()
 
+        channel_name = cmd.redis_channel_name(self.synapse_stream_prefix)
+
         await make_deferred_yieldable(
-            self.synapse_outbound_redis_connection.publish(
-                self.synapse_stream_prefix, encoded_string
-            )
+            self.synapse_outbound_redis_connection.publish(channel_name, encoded_string)
         )
 
 


Test patch:
Repository: matrix-org/synapse

Issue:
We should remove Caddy from the workerised Complement container image, because it's confusing having both Nginx and Caddy
Our current Complement-with-workers image uses Nginx to route worker requests to the right worker and uses Caddy to do TLS termination.

Whilst it's convenient that Caddy can generate the TLS certificate on-demand (I guess), it's frustrating that it's configured with a very messy and under-documented JSON file.
It's also confusing that we use two reverse-proxies in the same image.

Nginx can terminate TLS just fine and a handful of OpenSSL commands can generate a certificate (this is what the non-workerised version does), so I think it'd be cleaner to get rid of Caddy from the 'architecture' of our worker image.


Discussion:

Resolving patch:
diff --git a/docker/configure_workers_and_start.py b/docker/configure_workers_and_start.py
--- a/docker/configure_workers_and_start.py
+++ b/docker/configure_workers_and_start.py
@@ -21,6 +21,9 @@
 #   * SYNAPSE_REPORT_STATS: Whether to report stats.
 #   * SYNAPSE_WORKER_TYPES: A comma separated list of worker names as specified in WORKER_CONFIG
 #         below. Leave empty for no workers, or set to '*' for all possible workers.
+#   * SYNAPSE_TLS_CERT: Path to a TLS certificate in PEM format.
+#   * SYNAPSE_TLS_KEY: Path to a TLS key. If this and SYNAPSE_TLS_CERT are specified,
+#         Nginx will be configured to serve TLS on port 8448.
 #
 # NOTE: According to Complement's ENTRYPOINT expectations for a homeserver image (as defined
 # in the project's README), this script may be run multiple times, and functionality should
@@ -501,6 +504,8 @@ def generate_worker_files(
         "/etc/nginx/conf.d/matrix-synapse.conf",
         worker_locations=nginx_location_config,
         upstream_directives=nginx_upstream_config,
+        tls_cert_path=os.environ.get("SYNAPSE_TLS_CERT"),
+        tls_key_path=os.environ.get("SYNAPSE_TLS_KEY"),
     )
 
     # Supervisord config


Test patch: