internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: matrix-org/synapse

Issue:
SAML login fails since v1.12.0 if custom SAML module raises a RedirectException
### Description

Mozilla SAML module raises a RedirectException to redirect a new user to a username picker: https://github.com/matrix-org/matrix-synapse-saml-mozilla/blob/master/matrix_synapse_saml_mozilla/mapping_provider.py#L162

Since v1.12.0 Synapse is catching this and rendering an error page instead: https://github.com/matrix-org/synapse/pull/7058/files#diff-4a1427aad5021dc55886911e28d9c941R120 

### Version information

Synapse v1.12.0
Matrix SYnapse SAML Mozilla v0.2.5


Discussion:
FYI fix pushed to the Mozilla Modular instance, PR upcoming.

Resolving patch:
diff --git a/synapse/handlers/saml_handler.py b/synapse/handlers/saml_handler.py
--- a/synapse/handlers/saml_handler.py
+++ b/synapse/handlers/saml_handler.py
@@ -26,6 +26,7 @@
 from synapse.http.server import finish_request
 from synapse.http.servlet import parse_string
 from synapse.module_api import ModuleApi
+from synapse.module_api.errors import RedirectException
 from synapse.types import (
     UserID,
     map_username_to_mxid_localpart,
@@ -119,6 +120,9 @@ async def handle_saml_response(self, request):
 
         try:
             user_id = await self._map_saml_response_to_user(resp_bytes, relay_state)
+        except RedirectException:
+            # Raise the exception as per the wishes of the SAML module response
+            raise
         except Exception as e:
             # If decoding the response or mapping it to a user failed, then log the
             # error and tell the user that something went wrong.


Test patch:
Repository: matrix-org/synapse

Issue:
Background process 'expire_url_cache_data' threw an exception
I'm seeing this every few seconds:

```
2020-03-31 12:30:15,350 - synapse.rest.media.v1.preview_url_resource - 401 - INFO - expire_url_cache_data-1 - Running url preview cache expiry
2020-03-31 12:30:15,351 - synapse.metrics.background_process_metrics - 215 - ERROR - expire_url_cache_data-1 - Background process 'expire_url_cache_data' threw an exception
Traceback (most recent call last):
  File "/home/rav/work/synapse/env3/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: []

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rav/work/synapse/env3/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: []

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rav/work/synapse/synapse/metrics/background_process_metrics.py", line 213, in run
    return (yield result)
  File "/home/rav/work/synapse/env3/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/home/rav/work/synapse/synapse/rest/media/v1/preview_url_resource.py", line 430, in _expire_url_cache_data
    await self.store.delete_url_cache(removed_media)
TypeError: object NoneType can't be used in 'await' expression
```



Discussion:
looks like this was introduced in #7110, though I'm a bit surprised by the error message. `delete_url_cache` returns a Deferred, not a None.
scratch that, sometimes it returns a None.

Resolving patch:
diff --git a/synapse/storage/data_stores/main/media_repository.py b/synapse/storage/data_stores/main/media_repository.py
--- a/synapse/storage/data_stores/main/media_repository.py
+++ b/synapse/storage/data_stores/main/media_repository.py
@@ -340,7 +340,7 @@ def _get_expired_url_cache_txn(txn):
             "get_expired_url_cache", _get_expired_url_cache_txn
         )
 
-    def delete_url_cache(self, media_ids):
+    async def delete_url_cache(self, media_ids):
         if len(media_ids) == 0:
             return
 
@@ -349,7 +349,7 @@ def delete_url_cache(self, media_ids):
         def _delete_url_cache_txn(txn):
             txn.executemany(sql, [(media_id,) for media_id in media_ids])
 
-        return self.db.runInteraction("delete_url_cache", _delete_url_cache_txn)
+        return await self.db.runInteraction("delete_url_cache", _delete_url_cache_txn)
 
     def get_url_cache_media_before(self, before_ts):
         sql = (


Test patch:
Repository: matrix-org/synapse

Issue:
Move GET requests to account_data/* onto a worker.



Discussion:

Resolving patch:
diff --git a/synapse/app/generic_worker.py b/synapse/app/generic_worker.py
--- a/synapse/app/generic_worker.py
+++ b/synapse/app/generic_worker.py
@@ -98,6 +98,10 @@
 from synapse.rest.client.v2_alpha import groups, sync, user_directory
 from synapse.rest.client.v2_alpha._base import client_patterns
 from synapse.rest.client.v2_alpha.account import ThreepidRestServlet
+from synapse.rest.client.v2_alpha.account_data import (
+    AccountDataServlet,
+    RoomAccountDataServlet,
+)
 from synapse.rest.client.v2_alpha.keys import KeyChangesServlet, KeyQueryServlet
 from synapse.rest.client.v2_alpha.register import RegisterRestServlet
 from synapse.rest.client.versions import VersionsRestServlet
@@ -475,6 +479,8 @@ def _listen_http(self, listener_config):
                     ProfileDisplaynameRestServlet(self).register(resource)
                     ProfileRestServlet(self).register(resource)
                     KeyUploadServlet(self).register(resource)
+                    AccountDataServlet(self).register(resource)
+                    RoomAccountDataServlet(self).register(resource)
 
                     sync.register_servlets(self, resource)
                     events.register_servlets(self, resource)
diff --git a/synapse/rest/client/v2_alpha/account_data.py b/synapse/rest/client/v2_alpha/account_data.py
--- a/synapse/rest/client/v2_alpha/account_data.py
+++ b/synapse/rest/client/v2_alpha/account_data.py
@@ -38,8 +38,12 @@ def __init__(self, hs):
         self.auth = hs.get_auth()
         self.store = hs.get_datastore()
         self.notifier = hs.get_notifier()
+        self._is_worker = hs.config.worker_app is not None
 
     async def on_PUT(self, request, user_id, account_data_type):
+        if self._is_worker:
+            raise Exception("Cannot handle PUT /account_data on worker")
+
         requester = await self.auth.get_user_by_req(request)
         if user_id != requester.user.to_string():
             raise AuthError(403, "Cannot add account data for other users.")
@@ -86,8 +90,12 @@ def __init__(self, hs):
         self.auth = hs.get_auth()
         self.store = hs.get_datastore()
         self.notifier = hs.get_notifier()
+        self._is_worker = hs.config.worker_app is not None
 
     async def on_PUT(self, request, user_id, room_id, account_data_type):
+        if self._is_worker:
+            raise Exception("Cannot handle PUT /account_data on worker")
+
         requester = await self.auth.get_user_by_req(request)
         if user_id != requester.user.to_string():
             raise AuthError(403, "Cannot add account data for other users.")


Test patch:
Repository: matrix-org/synapse

Issue:
master doesn't send replication traffic unless you have a TCP replication listener
the TCP replication listener is unused if you have redis, so you'd expect to be able to remove it (indeed, the [docs](https://github.com/matrix-org/synapse/blob/85155654c52bb9999b6b4bcbcf84cd68505d4c82/docs/workers.md#experimental-support-for-replication-over-redis) say that you can). But if you do that, nothing builds the `ReplicationStreamer`, so no replication traffic ever gets sent.


Discussion:

Resolving patch:
diff --git a/synapse/replication/tcp/handler.py b/synapse/replication/tcp/handler.py
--- a/synapse/replication/tcp/handler.py
+++ b/synapse/replication/tcp/handler.py
@@ -159,6 +159,9 @@ def start_replication(self, hs):
                 hs.config.redis_port,
             )
 
+            # First let's ensure that we have a ReplicationStreamer started.
+            hs.get_replication_streamer()
+
             # We need two connections to redis, one for the subscription stream and
             # one to send commands to (as you can't send further redis commands to a
             # connection after SUBSCRIBE is called).


Test patch:
Repository: matrix-org/synapse

Issue:
Update SSO UIAuth login identifier to m.login.sso
I'm not sure when exactly we do this, but [MSC2454](https://github.com/matrix-org/matrix-doc/pull/2454) was merged which identified `m.login.sso` as the identifier for SSO + UIAuth. Synapse is currently using `org.matrix.login.sso`. At some point we should switch to the standardized version.


Discussion:
According to [MSC2324](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2324-when-to-ship.md), this should happen after a spec release with the identifier has gone out.

Resolving patch:
diff --git a/synapse/api/constants.py b/synapse/api/constants.py
--- a/synapse/api/constants.py
+++ b/synapse/api/constants.py
@@ -61,7 +61,7 @@ class LoginType(object):
     MSISDN = "m.login.msisdn"
     RECAPTCHA = "m.login.recaptcha"
     TERMS = "m.login.terms"
-    SSO = "org.matrix.login.sso"
+    SSO = "m.login.sso"
     DUMMY = "m.login.dummy"
 
     # Only for C/S API v1


Test patch:
Repository: matrix-org/synapse

Issue:
PhoneHomeStatsTestCase.test_performance_100 seems flaky on SQLite
I've seen it fail quite a few times over the past couple of days in BuildKite, exclusively on SQLite tests, with:

```
Traceback (most recent call last):
  File "/src/tests/test_phone_home.py", line 51, in test_performance_100
    self.assertApproximates(stats["cpu_average"], 100, tolerance=2.5)
  File "/src/.tox/py36/lib/python3.6/site-packages/twisted/trial/_synctest.py", line 564, in assertApproximates
    raise self.failureException(msg or "%s ~== %s" % (first, second))
twisted.trial.unittest.FailTest: 103 ~== 100
```


Discussion:
Related to #6676 maybe?
Oops, yes, I forgot to link this one. Yeah, it might be related.

Resolving patch:
diff --git a/synapse/app/homeserver.py b/synapse/app/homeserver.py
--- a/synapse/app/homeserver.py
+++ b/synapse/app/homeserver.py
@@ -488,6 +488,29 @@ def phone_stats_home(hs, stats, stats_process=_stats_process):
     if uptime < 0:
         uptime = 0
 
+    #
+    # Performance statistics. Keep this early in the function to maintain reliability of `test_performance_100` test.
+    #
+    old = stats_process[0]
+    new = (now, resource.getrusage(resource.RUSAGE_SELF))
+    stats_process[0] = new
+
+    # Get RSS in bytes
+    stats["memory_rss"] = new[1].ru_maxrss
+
+    # Get CPU time in % of a single core, not % of all cores
+    used_cpu_time = (new[1].ru_utime + new[1].ru_stime) - (
+        old[1].ru_utime + old[1].ru_stime
+    )
+    if used_cpu_time == 0 or new[0] == old[0]:
+        stats["cpu_average"] = 0
+    else:
+        stats["cpu_average"] = math.floor(used_cpu_time / (new[0] - old[0]) * 100)
+
+    #
+    # General statistics
+    #
+
     stats["homeserver"] = hs.config.server_name
     stats["server_context"] = hs.config.server_context
     stats["timestamp"] = now
@@ -522,25 +545,6 @@ def phone_stats_home(hs, stats, stats_process=_stats_process):
     stats["cache_factor"] = hs.config.caches.global_factor
     stats["event_cache_size"] = hs.config.caches.event_cache_size
 
-    #
-    # Performance statistics
-    #
-    old = stats_process[0]
-    new = (now, resource.getrusage(resource.RUSAGE_SELF))
-    stats_process[0] = new
-
-    # Get RSS in bytes
-    stats["memory_rss"] = new[1].ru_maxrss
-
-    # Get CPU time in % of a single core, not % of all cores
-    used_cpu_time = (new[1].ru_utime + new[1].ru_stime) - (
-        old[1].ru_utime + old[1].ru_stime
-    )
-    if used_cpu_time == 0 or new[0] == old[0]:
-        stats["cpu_average"] = 0
-    else:
-        stats["cpu_average"] = math.floor(used_cpu_time / (new[0] - old[0]) * 100)
-
     #
     # Database version
     #


Test patch:
Repository: matrix-org/synapse

Issue:
it's possible to join an empty room over federation, resulting in a split-brain room
we will happily accept make_join and send_join requests for an empty room; you then end up in a room on your lonesome. And then the target server thinks there is someone in the room even though there aren't any local members, and it's a horrible mess.


Discussion:

Resolving patch:
diff --git a/synapse/handlers/federation.py b/synapse/handlers/federation.py
--- a/synapse/handlers/federation.py
+++ b/synapse/handlers/federation.py
@@ -44,6 +44,7 @@
     FederationDeniedError,
     FederationError,
     HttpResponseException,
+    NotFoundError,
     RequestSendFailed,
     SynapseError,
 )
@@ -1439,10 +1440,20 @@ async def on_make_join_request(
             )
             raise SynapseError(403, "User not from origin", Codes.FORBIDDEN)
 
-        event_content = {"membership": Membership.JOIN}
-
+        # checking the room version will check that we've actually heard of the room
+        # (and return a 404 otherwise)
         room_version = await self.store.get_room_version_id(room_id)
 
+        # now check that we are *still* in the room
+        is_in_room = await self.auth.check_host_in_room(room_id, self.server_name)
+        if not is_in_room:
+            logger.info(
+                "Got /make_join request for room %s we are no longer in", room_id,
+            )
+            raise NotFoundError("Not an active room on this server")
+
+        event_content = {"membership": Membership.JOIN}
+
         builder = self.event_builder_factory.new(
             room_version,
             {


Test patch:
Repository: matrix-org/synapse

Issue:
synapse returns pointless body for OPTIONS requests
we should just return a 204 No Content


Discussion:
I believe this requires messing with the [`OptionsResource`](https://github.com/matrix-org/synapse/blob/f07061e75f61e159a6d81b2b95888446aa375b6f/synapse/http/server.py#L489-L502) but there might be some other places to touch too.

Resolving patch:
diff --git a/synapse/http/server.py b/synapse/http/server.py
--- a/synapse/http/server.py
+++ b/synapse/http/server.py
@@ -442,21 +442,6 @@ def render_GET(self, request: Request):
         return super().render_GET(request)
 
 
-def _options_handler(request):
-    """Request handler for OPTIONS requests
-
-    This is a request handler suitable for return from
-    _get_handler_for_request. It returns a 200 and an empty body.
-
-    Args:
-        request (twisted.web.http.Request):
-
-    Returns:
-        Tuple[int, dict]: http code, response body.
-    """
-    return 200, {}
-
-
 def _unrecognised_request_handler(request):
     """Request handler for unrecognised requests
 
@@ -490,11 +475,12 @@ class OptionsResource(resource.Resource):
     """Responds to OPTION requests for itself and all children."""
 
     def render_OPTIONS(self, request):
-        code, response_json_object = _options_handler(request)
+        request.setResponseCode(204)
+        request.setHeader(b"Content-Length", b"0")
 
-        return respond_with_json(
-            request, code, response_json_object, send_cors=True, canonical_json=False,
-        )
+        set_cors_headers(request)
+
+        return b""
 
     def getChildWithDefault(self, path, request):
         if request.method == b"OPTIONS":


Test patch:
diff --git a/tests/test_server.py b/tests/test_server.py
--- a/tests/test_server.py
+++ b/tests/test_server.py
@@ -199,10 +199,10 @@ def _make_request(self, method, path):
         return channel
 
     def test_unknown_options_request(self):
-        """An OPTIONS requests to an unknown URL still returns 200 OK."""
+        """An OPTIONS requests to an unknown URL still returns 204 No Content."""
         channel = self._make_request(b"OPTIONS", b"/foo/")
-        self.assertEqual(channel.result["code"], b"200")
-        self.assertEqual(channel.result["body"], b"{}")
+        self.assertEqual(channel.result["code"], b"204")
+        self.assertNotIn("body", channel.result)
 
         # Ensure the correct CORS headers have been added
         self.assertTrue(
@@ -219,10 +219,10 @@ def test_unknown_options_request(self):
         )
 
     def test_known_options_request(self):
-        """An OPTIONS requests to an known URL still returns 200 OK."""
+        """An OPTIONS requests to an known URL still returns 204 No Content."""
         channel = self._make_request(b"OPTIONS", b"/res/")
-        self.assertEqual(channel.result["code"], b"200")
-        self.assertEqual(channel.result["body"], b"{}")
+        self.assertEqual(channel.result["code"], b"204")
+        self.assertNotIn("body", channel.result)
 
         # Ensure the correct CORS headers have been added
         self.assertTrue(
Repository: matrix-org/synapse

Issue:
spell out that .well-known/matrix/client needs an Access-Control-Allow-Origin: * header if it is to ever work with web clients like Riot/Web...



Discussion:
See also: https://github.com/vector-im/riot-web/issues/7760

Resolving patch:
diff --git a/synapse/config/registration.py b/synapse/config/registration.py
--- a/synapse/config/registration.py
+++ b/synapse/config/registration.py
@@ -333,24 +333,6 @@ def generate_config_section(self, generate_secrets=False, **kwargs):
         #
         #default_identity_server: https://matrix.org
 
-        # The list of identity servers trusted to verify third party
-        # identifiers by this server.
-        #
-        # Also defines the ID server which will be called when an account is
-        # deactivated (one will be picked arbitrarily).
-        #
-        # Note: This option is deprecated. Since v0.99.4, Synapse has tracked which identity
-        # server a 3PID has been bound to. For 3PIDs bound before then, Synapse runs a
-        # background migration script, informing itself that the identity server all of its
-        # 3PIDs have been bound to is likely one of the below.
-        #
-        # As of Synapse v1.4.0, all other functionality of this option has been deprecated, and
-        # it is now solely used for the purposes of the background migration script, and can be
-        # removed once it has run.
-        #trusted_third_party_id_servers:
-        #  - matrix.org
-        #  - vector.im
-
         # Handle threepid (email/phone etc) registration and password resets through a set of
         # *trusted* identity servers. Note that this allows the configured identity server to
         # reset passwords for accounts!


Test patch:
Repository: matrix-org/synapse

Issue:
Change default logging to use buffered logging
This would involve using something like: https://docs.python.org/3/library/logging.handlers.html#memoryhandler

We need to check if making this change will change existing systems or only effect new installations.

The goal here is to reduce the amount of IO used, as the current default INFO logging can be fairly high volume and cause disk IO issues.


Discussion:

Resolving patch:
diff --git a/synapse/config/logger.py b/synapse/config/logger.py
--- a/synapse/config/logger.py
+++ b/synapse/config/logger.py
@@ -62,13 +62,29 @@
 
 handlers:
     file:
-        class: logging.handlers.RotatingFileHandler
+        class: logging.handlers.TimedRotatingFileHandler
         formatter: precise
         filename: ${log_file}
-        maxBytes: 104857600
-        backupCount: 10
-        filters: [context]
+        when: midnight
+        backupCount: 3  # Does not include the current log file.
         encoding: utf8
+
+    # Default to buffering writes to log file for efficiency. This means that
+    # will be a delay for INFO/DEBUG logs to get written, but WARNING/ERROR
+    # logs will still be flushed immediately.
+    buffer:
+        class: logging.handlers.MemoryHandler
+        filters: [context]
+        target: file
+        # The capacity is the number of log lines that are buffered before
+        # being written to disk. Increasing this will lead to better
+        # performance, at the expensive of it taking longer for log lines to
+        # be written to disk.
+        capacity: 10
+        flushLevel: 30  # Flush for WARNING logs as well
+
+    # A handler that writes logs to stderr. Unused by default, but can be used
+    # instead of "buffer" and "file" in the logger handlers.
     console:
         class: logging.StreamHandler
         formatter: precise
@@ -80,9 +96,24 @@
         # information such as access tokens.
         level: INFO
 
+    twisted:
+        # We send the twisted logging directly to the file handler,
+        # to work around https://github.com/matrix-org/synapse/issues/3471
+        # when using "buffer" logger. Use "console" to log to stderr instead.
+        handlers: [file]
+        propagate: false
+
 root:
     level: INFO
-    handlers: [file, console]
+
+    # Write logs to the `buffer` handler, which will buffer them together in memory,
+    # then write them to a file.
+    #
+    # Replace "buffer" with "console" to log to stderr instead. (Note that you'll
+    # also need to update the configuation for the `twisted` logger above, in
+    # this case.)
+    #
+    handlers: [buffer]
 
 disable_existing_loggers: false
 """


Test patch:
Repository: matrix-org/synapse

Issue:
Improve MatrixError when user tries to create an account with digits only
Could be nice to send errcode `M_INVALID_USERNAME`, for the moment, the client receives `M_UNKNOWN`:

```json
{"errcode":"M_UNKNOWN","error":"Numeric user IDs are reserved for guest users."}
```

_Originally posted by @bmarty in https://github.com/matrix-org/synapse/pull/7625#r473022751_


Discussion:
A similar fix has been done here if it helps: https://github.com/matrix-org/synapse/pull/6151/files

Resolving patch:
diff --git a/synapse/handlers/register.py b/synapse/handlers/register.py
--- a/synapse/handlers/register.py
+++ b/synapse/handlers/register.py
@@ -124,7 +124,9 @@ async def check_username(
             try:
                 int(localpart)
                 raise SynapseError(
-                    400, "Numeric user IDs are reserved for guest users."
+                    400,
+                    "Numeric user IDs are reserved for guest users.",
+                    errcode=Codes.INVALID_USERNAME,
                 )
             except ValueError:
                 pass


Test patch: