internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: matrix-org/synapse

Issue:
When enabling EDU support for appservices, synapse will start from stream position 0 when fetching presence/receipts
Which is very expensive if you were say, matrix.org. If the appservice hasn't got a stream position in the database, we should go from a reasonable point in time.


Discussion:

Resolving patch:
diff --git a/synapse/handlers/appservice.py b/synapse/handlers/appservice.py
--- a/synapse/handlers/appservice.py
+++ b/synapse/handlers/appservice.py
@@ -226,7 +226,7 @@ async def _notify_interested_services_ephemeral(
         new_token: Optional[int],
         users: Collection[Union[str, UserID]],
     ):
-        logger.info("Checking interested services for %s" % (stream_key))
+        logger.debug("Checking interested services for %s" % (stream_key))
         with Measure(self.clock, "notify_interested_services_ephemeral"):
             for service in services:
                 # Only handle typing if we have the latest token
diff --git a/synapse/handlers/receipts.py b/synapse/handlers/receipts.py
--- a/synapse/handlers/receipts.py
+++ b/synapse/handlers/receipts.py
@@ -158,7 +158,8 @@ async def get_new_events_as(
         if from_key == to_key:
             return [], to_key
 
-        # We first need to fetch all new receipts
+        # Fetch all read receipts for all rooms, up to a limit of 100. This is ordered
+        # by most recent.
         rooms_to_events = await self.store.get_linearized_receipts_for_all_rooms(
             from_key=from_key, to_key=to_key
         )
diff --git a/synapse/storage/databases/main/receipts.py b/synapse/storage/databases/main/receipts.py
--- a/synapse/storage/databases/main/receipts.py
+++ b/synapse/storage/databases/main/receipts.py
@@ -278,7 +278,8 @@ def f(txn):
     async def get_linearized_receipts_for_all_rooms(
         self, to_key: int, from_key: Optional[int] = None
     ) -> Dict[str, JsonDict]:
-        """Get receipts for all rooms between two stream_ids.
+        """Get receipts for all rooms between two stream_ids, up
+        to a limit of the latest 100 read receipts.
 
         Args:
             to_key: Max stream id to fetch receipts upto.
@@ -294,12 +295,16 @@ def f(txn):
                 sql = """
                     SELECT * FROM receipts_linearized WHERE
                     stream_id > ? AND stream_id <= ?
+                    ORDER BY stream_id DESC
+                    LIMIT 100
                 """
                 txn.execute(sql, [from_key, to_key])
             else:
                 sql = """
                     SELECT * FROM receipts_linearized WHERE
                     stream_id <= ?
+                    ORDER BY stream_id DESC
+                    LIMIT 100
                 """
 
                 txn.execute(sql, [to_key])


Test patch:
Repository: matrix-org/synapse

Issue:
remove pin on prometheus-client <0.9.0
we should check that prom-client 0.9.0 doesn't break anything, and then bump the pin.

Indeed, given it's now been a few years since prom-client 0.4.0 broke the world, maybe we should remove the pin altogether?


Discussion:
I don't know what prometheus_client is, but I edited my `python_dependencies.py` file to see if I could run synapse with version 0.9.0, and everything appears to be working fine, so I could open a pull request for that if you'd like. But I'm not a synapse expert, I just run a small server, and all I know is that removing the pin worked so I can use synapse again. Is there any official tests that need to be done, or can we update `python_dependencies.py`?
@jordanbancino If you could create a PR that removes the pin, having the CI run over it and pass would be a great start!

Resolving patch:
diff --git a/synapse/python_dependencies.py b/synapse/python_dependencies.py
--- a/synapse/python_dependencies.py
+++ b/synapse/python_dependencies.py
@@ -40,6 +40,10 @@
 # Note that these both represent runtime dependencies (and the versions
 # installed are checked at runtime).
 #
+# Also note that we replicate these constraints in the Synapse Dockerfile while
+# pre-installing dependencies. If these constraints are updated here, the same
+# change should be made in the Dockerfile.
+#
 # [1] https://pip.pypa.io/en/stable/reference/pip_install/#requirement-specifiers.
 
 REQUIREMENTS = [
@@ -69,14 +73,7 @@
     "msgpack>=0.5.2",
     "phonenumbers>=8.2.0",
     # we use GaugeHistogramMetric, which was added in prom-client 0.4.0.
-    # prom-client has a history of breaking backwards compatibility between
-    # minor versions (https://github.com/prometheus/client_python/issues/317),
-    # so we also pin the minor version.
-    #
-    # Note that we replicate these constraints in the Synapse Dockerfile while
-    # pre-installing dependencies. If these constraints are updated here, the
-    # same change should be made in the Dockerfile.
-    "prometheus_client>=0.4.0,<0.9.0",
+    "prometheus_client>=0.4.0",
     # we use attr.validators.deep_iterable, which arrived in 19.1.0 (Note:
     # Fedora 31 only has 19.1, so if we want to upgrade we should wait until 33
     # is out in November.)


Test patch:
Repository: matrix-org/synapse

Issue:
UnboundLocalError raised when response body exceeds max size
A bug with 1.25.0 was reported in the Synapse Package Maintainer's channel:

> I got a nice Traceback in 1.25.0 after start:
>
>     2021-01-15 20:32:45,345 - synapse.http.matrixfederationclient - 987 - WARNING - GET-25- {GET-O-1} [matrix.org] Requested file is too large > 10485760 bytes
>     2021-01-15 20:32:45,345 - synapse.rest.media.v1.media_repository - 417 - ERROR - GET-25- Failed to fetch remote media matrix.org/cPeSAplLYzzcKlpJjLtwlzrT
>     Traceback (most recent call last):                                                                                     
>       File "/usr/local/lib/python3.7/site-packages/synapse/rest/media/v1/media_repository.py", line 384, in _download_remote_file
>         "allow_remote": "false"                                                                                            
>       File "/usr/local/lib/python3.7/site-packages/synapse/http/matrixfederationclient.py", line 1004, in get_file
>         length,                                                                                                            
>     UnboundLocalError: local variable 'length' referenced before assignment
>
> From a (very) quick glance at the source I would guess that synapse hits `except BodyExceededMaxSize` before the `length = ...` statement in the try-block, which causes the `logger.info` statement to reference a non-existing `length` variable.
> Otherwise things appear to work fine. :D

I suspect this was introduced as part of this commit:

https://github.com/matrix-org/synapse/commit/ff5c4da1289cb5e097902b3e55b771be342c29d6#diff-a53ea02bf6a0cbb98e925b47e71f4ffb149075536543c1eca15366fc65c90aefL977-R983

The diagnosis looks right to me: when `read_body_with_max_size` throws, it skips the `length` assignment statement two lines later. However, we don't raise or return from the exception handler, so we continue through to a logging statement which tries to use the unassigned `length` variable.


Discussion:

Resolving patch:
diff --git a/synapse/http/client.py b/synapse/http/client.py
--- a/synapse/http/client.py
+++ b/synapse/http/client.py
@@ -724,7 +724,7 @@ async def get_file(
                 read_body_with_max_size(response, output_stream, max_size)
             )
         except BodyExceededMaxSize:
-            SynapseError(
+            raise SynapseError(
                 502,
                 "Requested file is too large > %r bytes" % (max_size,),
                 Codes.TOO_LARGE,
diff --git a/synapse/http/matrixfederationclient.py b/synapse/http/matrixfederationclient.py
--- a/synapse/http/matrixfederationclient.py
+++ b/synapse/http/matrixfederationclient.py
@@ -996,7 +996,7 @@ async def get_file(
             logger.warning(
                 "{%s} [%s] %s", request.txn_id, request.destination, msg,
             )
-            SynapseError(502, msg, Codes.TOO_LARGE)
+            raise SynapseError(502, msg, Codes.TOO_LARGE)
         except Exception as e:
             logger.warning(
                 "{%s} [%s] Error reading response: %s",


Test patch:
Repository: matrix-org/synapse

Issue:
CAS should use the public_baseurl instead of a separate cas_service_url
When using CAS you currently need to define a `cas_service_url` in the homeserver configuration. This ends up being a duplicate of the `public_baseurl` configuration and should be fairly easy to get rid of.

Reasons for doing this:
* Not having duplicated config is good. 👍 
* This will match what SAML requires.

We'll need to deprecate the old option before just outright removing it.


Discussion:

Resolving patch:
diff --git a/synapse/config/cas.py b/synapse/config/cas.py
--- a/synapse/config/cas.py
+++ b/synapse/config/cas.py
@@ -30,7 +30,13 @@ def read_config(self, config, **kwargs):
 
         if self.cas_enabled:
             self.cas_server_url = cas_config["server_url"]
-            self.cas_service_url = cas_config["service_url"]
+            public_base_url = cas_config.get("service_url") or self.public_baseurl
+            if public_base_url[-1] != "/":
+                public_base_url += "/"
+            # TODO Update this to a _synapse URL.
+            self.cas_service_url = (
+                public_base_url + "_matrix/client/r0/login/cas/ticket"
+            )
             self.cas_displayname_attribute = cas_config.get("displayname_attribute")
             self.cas_required_attributes = cas_config.get("required_attributes") or {}
         else:
@@ -53,10 +59,6 @@ def generate_config_section(self, config_dir_path, server_name, **kwargs):
           #
           #server_url: "https://cas-server.com"
 
-          # The public URL of the homeserver.
-          #
-          #service_url: "https://homeserver.domain.com:8448"
-
           # The attribute of the CAS response to use as the display name.
           #
           # If unset, no displayname will be set.
diff --git a/synapse/config/oidc_config.py b/synapse/config/oidc_config.py
--- a/synapse/config/oidc_config.py
+++ b/synapse/config/oidc_config.py
@@ -54,8 +54,7 @@ def read_config(self, config, **kwargs):
                     "Multiple OIDC providers have the idp_id %r." % idp_id
                 )
 
-        public_baseurl = self.public_baseurl
-        self.oidc_callback_url = public_baseurl + "_synapse/oidc/callback"
+        self.oidc_callback_url = self.public_baseurl + "_synapse/oidc/callback"
 
     @property
     def oidc_enabled(self) -> bool:
diff --git a/synapse/handlers/cas_handler.py b/synapse/handlers/cas_handler.py
--- a/synapse/handlers/cas_handler.py
+++ b/synapse/handlers/cas_handler.py
@@ -99,11 +99,7 @@ def _build_service_param(self, args: Dict[str, str]) -> str:
         Returns:
             The URL to use as a "service" parameter.
         """
-        return "%s%s?%s" % (
-            self._cas_service_url,
-            "/_matrix/client/r0/login/cas/ticket",
-            urllib.parse.urlencode(args),
-        )
+        return "%s?%s" % (self._cas_service_url, urllib.parse.urlencode(args),)
 
     async def _validate_ticket(
         self, ticket: str, service_args: Dict[str, str]


Test patch:
Repository: matrix-org/synapse

Issue:
OIDC: allow individual endpoints to be overridden without disabling discovery
It's sometimes useful to be able to tweak individual endpoints for the OpenID Connect auth provider, for example to add additional parameters to the "authentication" API. One example is for Google, which [specifies](https://developers.google.com/identity/protocols/oauth2/openid-connect#authenticationuriparameters) a `prompt` parameter telling Google to let the user pick between one of their accounts.

Currently the only way for an admin to add these parameters is by disabling `discovery` and manually configuring *all* of the OAuth2 endpoints. I think we should reverse this and make configured endpoints override any discovered ones.




Discussion:
somewhat related to #9309

Resolving patch:
diff --git a/synapse/handlers/oidc_handler.py b/synapse/handlers/oidc_handler.py
--- a/synapse/handlers/oidc_handler.py
+++ b/synapse/handlers/oidc_handler.py
@@ -383,22 +383,31 @@ async def load_metadata(self, force: bool = False) -> OpenIDProviderMetadata:
         return await self._provider_metadata.get()
 
     async def _load_metadata(self) -> OpenIDProviderMetadata:
-        # init the metadata from our config
-        metadata = OpenIDProviderMetadata(
-            issuer=self._config.issuer,
-            authorization_endpoint=self._config.authorization_endpoint,
-            token_endpoint=self._config.token_endpoint,
-            userinfo_endpoint=self._config.userinfo_endpoint,
-            jwks_uri=self._config.jwks_uri,
-        )
+        # start out with just the issuer (unlike the other settings, discovered issuer
+        # takes precedence over configured issuer, because configured issuer is
+        # required for discovery to take place.)
+        #
+        metadata = OpenIDProviderMetadata(issuer=self._config.issuer)
 
         # load any data from the discovery endpoint, if enabled
         if self._config.discover:
             url = get_well_known_url(self._config.issuer, external=True)
             metadata_response = await self._http_client.get_json(url)
-            # TODO: maybe update the other way around to let user override some values?
             metadata.update(metadata_response)
 
+        # override any discovered data with any settings in our config
+        if self._config.authorization_endpoint:
+            metadata["authorization_endpoint"] = self._config.authorization_endpoint
+
+        if self._config.token_endpoint:
+            metadata["token_endpoint"] = self._config.token_endpoint
+
+        if self._config.userinfo_endpoint:
+            metadata["userinfo_endpoint"] = self._config.userinfo_endpoint
+
+        if self._config.jwks_uri:
+            metadata["jwks_uri"] = self._config.jwks_uri
+
         self._validate_metadata(metadata)
 
         return metadata


Test patch:
Repository: matrix-org/synapse

Issue:
SAML SSO broken on Synapse 1.27
The [1.27 upgrade notes](https://github.com/matrix-org/synapse/blob/v1.27.0/UPGRADE.rst#changes-to-callback-uri-for-oauth2--openid-connect) claim:

> Note: a similar change is being made for SAML2; in this case the old URI `[synapse public baseurl]/_matrix/saml2` is being deprecated, but **will continue to work**, so no immediate changes are required for existing installations.

In fact, as part of the SAML2 request, the expected callback URI is sent to the SAML2 IdP, which will check that it matches what is expected and fail if not.

Hence, upgrading to Synapse 1.27 will break login for anyone using SAML :/.


Discussion:
This is a docs-only fix, yes?
I vote that yes, we fix this through updating the docs and letting people know about it, though there's an alternative approach where we roll back #9289 and attempt a more orderly deprecation.
So this essentially has the same behavior as the OIDC code where we have to jump from one callback URL to another in one go? I think fixing up the docs and removing the old URL sounds OK. At least than OIDC and SAML were done in the same release.

Resolving patch:
diff --git a/synapse/rest/synapse/client/__init__.py b/synapse/rest/synapse/client/__init__.py
--- a/synapse/rest/synapse/client/__init__.py
+++ b/synapse/rest/synapse/client/__init__.py
@@ -54,11 +54,7 @@ def build_synapse_client_resource_tree(hs: "HomeServer") -> Mapping[str, Resourc
     if hs.config.saml2_enabled:
         from synapse.rest.synapse.client.saml2 import SAML2Resource
 
-        res = SAML2Resource(hs)
-        resources["/_synapse/client/saml2"] = res
-
-        # This is also mounted under '/_matrix' for backwards-compatibility.
-        resources["/_matrix/saml2"] = res
+        resources["/_synapse/client/saml2"] = SAML2Resource(hs)
 
     return resources
 


Test patch:
Repository: matrix-org/synapse

Issue:
synapse 1.28rc1 breaks compat with SAML portal
one of our customers has a saml portal which expects synapse to respond to saml callbacks on `/_matrix/saml2/authn_response`. Since #9434, it doesn't.


Discussion:
We can partially revert #9434 easily enough to provide a transition period, but we shouldn't maintain that indefinitely.

Let's arbitrarily say we'll re-apply the change with Synapse 1.32, which will be our first full release in April and coincide with Python 3.5 deprecation.

Resolving patch:
diff --git a/synapse/rest/synapse/client/__init__.py b/synapse/rest/synapse/client/__init__.py
--- a/synapse/rest/synapse/client/__init__.py
+++ b/synapse/rest/synapse/client/__init__.py
@@ -54,7 +54,12 @@ def build_synapse_client_resource_tree(hs: "HomeServer") -> Mapping[str, Resourc
     if hs.config.saml2_enabled:
         from synapse.rest.synapse.client.saml2 import SAML2Resource
 
-        resources["/_synapse/client/saml2"] = SAML2Resource(hs)
+        res = SAML2Resource(hs)
+        resources["/_synapse/client/saml2"] = res
+
+        # This is also mounted under '/_matrix' for backwards-compatibility.
+        # To be removed in Synapse v1.32.0.
+        resources["/_matrix/saml2"] = res
 
     return resources
 


Test patch:
Repository: matrix-org/synapse

Issue:
08delete_pushers_for_deactivated_accounts.sql takes minutes
We should convert it to a bg update.


Discussion:

Resolving patch:
diff --git a/synapse/storage/databases/main/pusher.py b/synapse/storage/databases/main/pusher.py
--- a/synapse/storage/databases/main/pusher.py
+++ b/synapse/storage/databases/main/pusher.py
@@ -39,6 +39,11 @@ def __init__(self, database: DatabasePool, db_conn: Connection, hs: "HomeServer"
             db_conn, "pushers", "id", extra_tables=[("deleted_pushers", "stream_id")]
         )
 
+        self.db_pool.updates.register_background_update_handler(
+            "remove_deactivated_pushers",
+            self._remove_deactivated_pushers,
+        )
+
     def _decode_pushers_rows(self, rows: Iterable[dict]) -> Iterator[PusherConfig]:
         """JSON-decode the data in the rows returned from the `pushers` table
 
@@ -284,6 +289,54 @@ async def set_throttle_params(
             lock=False,
         )
 
+    async def _remove_deactivated_pushers(self, progress: dict, batch_size: int) -> int:
+        """A background update that deletes all pushers for deactivated users.
+
+        Note that we don't proacively tell the pusherpool that we've deleted
+        these (just because its a bit off a faff to do from here), but they will
+        get cleaned up at the next restart
+        """
+
+        last_user = progress.get("last_user", "")
+
+        def _delete_pushers(txn) -> int:
+
+            sql = """
+                SELECT name FROM users
+                WHERE deactivated = ? and name > ?
+                ORDER BY name ASC
+                LIMIT ?
+            """
+
+            txn.execute(sql, (1, last_user, batch_size))
+            users = [row[0] for row in txn]
+
+            self.db_pool.simple_delete_many_txn(
+                txn,
+                table="pushers",
+                column="user_name",
+                iterable=users,
+                keyvalues={},
+            )
+
+            if users:
+                self.db_pool.updates._background_update_progress_txn(
+                    txn, "remove_deactivated_pushers", {"last_user": users[-1]}
+                )
+
+            return len(users)
+
+        number_deleted = await self.db_pool.runInteraction(
+            "_remove_deactivated_pushers", _delete_pushers
+        )
+
+        if number_deleted < batch_size:
+            await self.db_pool.updates._end_background_update(
+                "remove_deactivated_pushers"
+            )
+
+        return number_deleted
+
 
 class PusherStore(PusherWorkerStore):
     def get_pushers_stream_token(self) -> int:


Test patch:
Repository: matrix-org/synapse

Issue:
Deprecated imp module is being used
### Description

The deprecated `imp` module is being imported:
https://github.com/matrix-org/synapse/blob/7b06f85c0e18b62775f12789fdf4adb6a0a47a4b/synapse/storage/prepare_database.py#L16

This causes warnings (e.g. when running tests) and should be replaced with `importlib`.


Discussion:

Resolving patch:
diff --git a/synapse/storage/prepare_database.py b/synapse/storage/prepare_database.py
--- a/synapse/storage/prepare_database.py
+++ b/synapse/storage/prepare_database.py
@@ -13,7 +13,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import imp
+import importlib.util
 import logging
 import os
 import re
@@ -454,8 +454,13 @@ def _upgrade_existing_database(
                     )
 
                 module_name = "synapse.storage.v%d_%s" % (v, root_name)
-                with open(absolute_path) as python_file:
-                    module = imp.load_source(module_name, absolute_path, python_file)  # type: ignore
+
+                spec = importlib.util.spec_from_file_location(
+                    module_name, absolute_path
+                )
+                module = importlib.util.module_from_spec(spec)
+                spec.loader.exec_module(module)  # type: ignore
+
                 logger.info("Running script %s", relative_path)
                 module.run_create(cur, database_engine)  # type: ignore
                 if not is_empty:


Test patch:
Repository: matrix-org/synapse

Issue:
Transaction logging doesn't contain request ID
It instead logs with `runWithConnection`.


Discussion:
This is probably a regression from #9654 or one of the follow-ups (#9857 or #9829).

Resolving patch:
diff --git a/synapse/storage/database.py b/synapse/storage/database.py
--- a/synapse/storage/database.py
+++ b/synapse/storage/database.py
@@ -718,7 +718,9 @@ def inner_func(conn, *args, **kwargs):
             # pool).
             assert not self.engine.in_transaction(conn)
 
-            with LoggingContext("runWithConnection", parent_context) as context:
+            with LoggingContext(
+                str(curr_context), parent_context=parent_context
+            ) as context:
                 sched_duration_sec = monotonic_time() - start_time
                 sql_scheduling_timer.observe(sched_duration_sec)
                 context.add_database_scheduled(sched_duration_sec)


Test patch:
Repository: matrix-org/synapse

Issue:
`Tried to activate scope outside of loggingcontext` from connection setup code
@erikjohnston reports that his homeserver is emitting a `Tried to activate scope outside of loggingcontext` error; investigation reveals it is happening with the `SET bytea_output TO escape` query.


Discussion:

Resolving patch:
diff --git a/synapse/storage/database.py b/synapse/storage/database.py
--- a/synapse/storage/database.py
+++ b/synapse/storage/database.py
@@ -91,12 +91,18 @@ def make_pool(
     db_args = dict(db_config.config.get("args", {}))
     db_args.setdefault("cp_reconnect", True)
 
+    def _on_new_connection(conn):
+        # Ensure we have a logging context so we can correctly track queries,
+        # etc.
+        with LoggingContext("db.on_new_connection"):
+            engine.on_new_connection(
+                LoggingDatabaseConnection(conn, engine, "on_new_connection")
+            )
+
     return adbapi.ConnectionPool(
         db_config.config["name"],
         cp_reactor=reactor,
-        cp_openfun=lambda conn: engine.on_new_connection(
-            LoggingDatabaseConnection(conn, engine, "on_new_connection")
-        ),
+        cp_openfun=_on_new_connection,
         **db_args,
     )
 


Test patch: