internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: matrix-org/synapse

Issue:
Backfilled events marked as outliers break backfill
A homeserver backfilling from another homeserver was found to be raising an `AssertionError` at:

https://github.com/matrix-org/synapse/blob/30fba6210834a4ecd91badf0c8f3eb278b72e746/synapse/handlers/federation.py#L921-L923

It's currently unclear which event is problematic and why it might be marked as an outlier. The logs of the backfilling server also contained many instances of event signature validation failures. This was due to a remote server changing their signing key without also changing the key ID (or putting the old key in `old_signing_keys`). This may be a red herring however.

The `assert` and a similar assert at:

https://github.com/matrix-org/synapse/blob/30fba6210834a4ecd91badf0c8f3eb278b72e746/synapse/handlers/federation.py#L891-L894

both do not give any information about the failing event.

Thus the first step is to improve these assert statements and figure out why these backfilled events are marked as outliers. After that, we may want to make the backfill code drop these events instead of simple raising, similar to what was done in #8350. 


Discussion:
@anoadragon453 Any chance you have a stack trace for this?
The stack trace is quite unhelpful:

```
AssertionError: null
  File "synapse/handlers/federation.py", line 1097, in try_backfill
    await self.backfill(
  File "synapse/handlers/federation.py", line 920, in backfill
    assert not event.internal_metadata.is_outlier()
```

However you can see a little bit of surrounding logs and other metadata at https://sentry.matrix.org/sentry/synapse-modular/issues/194313/ and https://sentry.matrix.org/sentry/synapse-modular/issues/194314/ (for those that have access).
Deferring for now, as one of our quarterly goals is to tame our Sentry alert rate, we should return to this in due course.

Resolving patch:
diff --git a/synapse/federation/federation_client.py b/synapse/federation/federation_client.py
--- a/synapse/federation/federation_client.py
+++ b/synapse/federation/federation_client.py
@@ -272,7 +272,7 @@ async def backfill(
 
         # Check signatures and hash of pdus, removing any from the list that fail checks
         pdus[:] = await self._check_sigs_and_hash_and_fetch(
-            dest, pdus, outlier=True, room_version=room_version
+            dest, pdus, room_version=room_version
         )
 
         return pdus
diff --git a/synapse/handlers/federation_event.py b/synapse/handlers/federation_event.py
--- a/synapse/handlers/federation_event.py
+++ b/synapse/handlers/federation_event.py
@@ -666,7 +666,9 @@ async def _process_pulled_event(
         logger.info("Processing pulled event %s", event)
 
         # these should not be outliers.
-        assert not event.internal_metadata.is_outlier()
+        assert (
+            not event.internal_metadata.is_outlier()
+        ), "pulled event unexpectedly flagged as outlier"
 
         event_id = event.event_id
 


Test patch:
Repository: matrix-org/synapse

Issue:
Add a capabilties flag for MSC3440
Per https://github.com/matrix-org/matrix-doc/commit/fc81bbd836955876c931c95277249981b3e2778c




Discussion:

Resolving patch:
diff --git a/synapse/rest/client/capabilities.py b/synapse/rest/client/capabilities.py
--- a/synapse/rest/client/capabilities.py
+++ b/synapse/rest/client/capabilities.py
@@ -73,6 +73,9 @@ async def on_GET(self, request: SynapseRequest) -> Tuple[int, JsonDict]:
                 "enabled": self.config.registration.enable_3pid_changes
             }
 
+        if self.config.experimental.msc3440_enabled:
+            response["capabilities"]["io.element.thread"] = {"enabled": True}
+
         return 200, response
 
 


Test patch:
Repository: matrix-org/synapse

Issue:
sample doc in #11561 is malformatted
see https://github.com/matrix-org/synapse/pull/11561#pullrequestreview-851817482


Discussion:

Resolving patch:
diff --git a/synapse/config/api.py b/synapse/config/api.py
--- a/synapse/config/api.py
+++ b/synapse/config/api.py
@@ -61,11 +61,12 @@ def generate_config_section(cls, **kwargs) -> str:
            #additional_event_types:
            #  - org.example.custom.event.type
 
-        # If enabled, puppeted user IP's can also be tracked. By default when
-        # puppeting another user, the user who has created the access token
-        # for puppeting is tracked. If this is enabled, both requests are tracked.
-        # Implicitly enables MAU tracking for puppeted users.
-        #track_puppeted_user_ips: false
+        # By default when puppeting another user, the user who has created the
+        # access token for puppeting is tracked. If this is enabled, both
+        # requests are tracked. Implicitly enables MAU tracking for puppeted users.
+        # Uncomment to also track puppeted user IP's.
+        #
+        #track_puppeted_user_ips: true
         """ % {
             "formatted_default_state_types": formatted_default_state_types
         }


Test patch:
Repository: matrix-org/synapse

Issue:
sample doc in #11561 is malformatted
see https://github.com/matrix-org/synapse/pull/11561#pullrequestreview-851817482


Discussion:
please can we replace "If this is enabled" with "If `track_puppeted_user_ips` is set to `true`"?

also, what exactly does "both requests are tracked" mean? what are the "requests" in this context, and why are there two of them?

Sorry to nit-pick on this, but clear documentation for this file is really important. 

Resolving patch:
diff --git a/synapse/config/api.py b/synapse/config/api.py
--- a/synapse/config/api.py
+++ b/synapse/config/api.py
@@ -61,10 +61,18 @@ def generate_config_section(cls, **kwargs) -> str:
            #additional_event_types:
            #  - org.example.custom.event.type
 
-        # By default when puppeting another user, the user who has created the
-        # access token for puppeting is tracked. If this is enabled, both
-        # requests are tracked. Implicitly enables MAU tracking for puppeted users.
-        # Uncomment to also track puppeted user IP's.
+        # We record the IP address of clients used to access the API for various
+        # reasons, including displaying it to the user in the "Where you're signed in"
+        # dialog.
+        #
+        # By default, when puppeting another user via the admin API, the client IP
+        # address is recorded against the user who created the access token (ie, the
+        # admin user), and *not* the puppeted user.
+        #
+        # Uncomment the following to also record the IP address against the puppeted
+        # user. (This also means that the puppeted user will count as an "active" user
+        # for the purpose of monthly active user tracking - see 'limit_usage_by_mau' etc
+        # above.)
         #
         #track_puppeted_user_ips: true
         """ % {


Test patch:
Repository: matrix-org/synapse

Issue:
AS events should include `prev_content` field
### Description

AS spec states events should be "formatted as per the Client-Server API", but state events sent to ASes don't include the `prev_content` field.

### Version information

Occurs as of 1.50.

### Ref

Sayeth vdh:

> `ApplicationServiceTransactionWorkerStore.get_new_events_for_appservice` is missing `get_prev_content=True` in its call to `get_events_as_list`


Discussion:
Can I work on this issue?


Resolving patch:
diff --git a/synapse/storage/databases/main/appservice.py b/synapse/storage/databases/main/appservice.py
--- a/synapse/storage/databases/main/appservice.py
+++ b/synapse/storage/databases/main/appservice.py
@@ -384,7 +384,7 @@ def get_new_events_for_appservice_txn(txn):
             "get_new_events_for_appservice", get_new_events_for_appservice_txn
         )
 
-        events = await self.get_events_as_list(event_ids)
+        events = await self.get_events_as_list(event_ids, get_prev_content=True)
 
         return upper_bound, events
 


Test patch:
Repository: matrix-org/synapse

Issue:
`HTTPStatus.*` have started to appear in `Processed request` log lines instead of an integer
An example log line from matrix.org:

```
2022-01-24 08:10:36,396 - synapse.access.http.8080 - 445 - INFO - GET-7050353 - xxx.xxx.xxx.xxx - 8080 - {@redacted:matrix.org} Processed request: 0.296sec/0.003sec (0.007sec, 0.000sec) (0.002sec/0.288sec/2) 36141B HTTPStatus.OK "GET /_synapse/admin/v1/event_reports HTTP/1.1" "aiohttp/0.2.1" [0 dbevts]
```

This seems to currently happening for `EventReportsRestServlet`, `RoomStateRestServlet` and `RoomRestServlet`. Perhaps other endpoints are affected, but none observed in the recent logs on matrix.org.

Notice the `HTTPStatus.OK` in the logs, whereas one would expect `200`. This seems to happen as we simply cast the return code to a `str`:

https://github.com/matrix-org/synapse/blob/10a88ba91cb16ccf757984f0a7d41ddf8b4dc07f/synapse/http/site.py#L410

However, `str(HTTPStatus.OK)` unhelpfully resolves to the str `"HTTPStatus.OK"`...

This could have implications for scripts that parse Synapse logs.


Discussion:
Sorry, this has my fingerprints all over it.
`code = str(int(self.code))` would work when `self.code: int` or `self.code: HTTPStatus`. Or maybe it's just easier to ditch the status enum unilaterally?
I do like using HTTPStatus, but perhaps we should `int(HTTPStatus)` before returning from servlets?
If we're starting to lean towards using `HTTPStatus`, we probably should fix the logging code rather than each individual servlet?
`ServletCallback` is currently defined as returning and `int` when returning a status code:

https://github.com/matrix-org/synapse/blob/cbd82d0b2db069400b5d43373838817d8a0209e7/synapse/http/server.py#L202-L207

We could extend this to include `HTTPStatus`, and then update all code that interacts with it to process both correctly.

@babolivier I agree that that would probably be less things to update (and to remember) than doing `int(HTTPStatus.*)` everywhere.
> `ServletCallback` is currently defined as returning and `int` when returning a status code:

FWIW note that `IntEnum` instances will inherit from `int`:

```py
>>> import http
>>> isinstance(http.HTTPStatus.OK, int)
True
```
we've had this before, btw: https://github.com/matrix-org/synapse/issues/7017

Resolving patch:
diff --git a/synapse/http/site.py b/synapse/http/site.py
--- a/synapse/http/site.py
+++ b/synapse/http/site.py
@@ -407,7 +407,10 @@ def _finished_processing(self) -> None:
 
         user_agent = get_request_user_agent(self, "-")
 
-        code = str(self.code)
+        # int(self.code) looks redundant, because self.code is already an int.
+        # But self.code might be an HTTPStatus (which inherits from int)---which has
+        # a different string representation. So ensure we really have an integer.
+        code = str(int(self.code))
         if not self.finished:
             # we didn't send the full response before we gave up (presumably because
             # the connection dropped)


Test patch:
Repository: matrix-org/synapse

Issue:
CI run against Twisted trunk is failing
See https://github.com/matrix-org/synapse/actions/runs/1805228510



Discussion:
This is due to mypy failing:

```
synapse/rest/media/v1/upload_resource.py:92: error: Argument 4 to "create_content" of "MediaRepository" has incompatible type "str"; expected "int"  [arg-type]
```

Resolving patch:
diff --git a/synapse/rest/media/v1/upload_resource.py b/synapse/rest/media/v1/upload_resource.py
--- a/synapse/rest/media/v1/upload_resource.py
+++ b/synapse/rest/media/v1/upload_resource.py
@@ -49,10 +49,14 @@ async def _async_render_OPTIONS(self, request: SynapseRequest) -> None:
 
     async def _async_render_POST(self, request: SynapseRequest) -> None:
         requester = await self.auth.get_user_by_req(request)
-        content_length = request.getHeader("Content-Length")
-        if content_length is None:
+        raw_content_length = request.getHeader("Content-Length")
+        if raw_content_length is None:
             raise SynapseError(msg="Request must specify a Content-Length", code=400)
-        if int(content_length) > self.max_upload_size:
+        try:
+            content_length = int(raw_content_length)
+        except ValueError:
+            raise SynapseError(msg="Content-Length value is invalid", code=400)
+        if content_length > self.max_upload_size:
             raise SynapseError(
                 msg="Upload request body is too large",
                 code=413,
@@ -66,7 +70,8 @@ async def _async_render_POST(self, request: SynapseRequest) -> None:
                 upload_name: Optional[str] = upload_name_bytes.decode("utf8")
             except UnicodeDecodeError:
                 raise SynapseError(
-                    msg="Invalid UTF-8 filename parameter: %r" % (upload_name), code=400
+                    msg="Invalid UTF-8 filename parameter: %r" % (upload_name_bytes,),
+                    code=400,
                 )
 
         # If the name is falsey (e.g. an empty byte string) ensure it is None.


Test patch:
Repository: matrix-org/synapse

Issue:
Remove type-ignore for `IReactorTCP.connectTCP`'s `hostname` argument
The fix proposed in #10895 for #10851 requires a series temporary `# type: ignore`.

We should remove this as soon as we're able: namely, when twisted release a version which includes [this commit](https://github.com/twisted/twisted/commit/ec7445d6361ce04424b3570a81263dc6760f15de).


Discussion:
hi ! Is this issue open for contribution ?

Hi @eshaanagarwal , a contribution would be welcome, but we can't make progress on this until the twisted project makes a new release. I'll remove the good-first-issue label for now to make that a bit clearer.
Note that "new" in this case means a release after 21.7.0 (which is what the current Twisted release is).

Resolving patch:
diff --git a/synapse/handlers/send_email.py b/synapse/handlers/send_email.py
--- a/synapse/handlers/send_email.py
+++ b/synapse/handlers/send_email.py
@@ -106,7 +106,7 @@ def build_sender_factory(**kwargs: Any) -> ESMTPSenderFactory:
         factory = build_sender_factory(hostname=smtphost if enable_tls else None)
 
     reactor.connectTCP(
-        smtphost,  # type: ignore[arg-type]
+        smtphost,
         smtpport,
         factory,
         timeout=30,
diff --git a/synapse/replication/tcp/handler.py b/synapse/replication/tcp/handler.py
--- a/synapse/replication/tcp/handler.py
+++ b/synapse/replication/tcp/handler.py
@@ -318,7 +318,7 @@ def start_replication(self, hs: "HomeServer"):
                 hs, outbound_redis_connection
             )
             hs.get_reactor().connectTCP(
-                hs.config.redis.redis_host,  # type: ignore[arg-type]
+                hs.config.redis.redis_host,
                 hs.config.redis.redis_port,
                 self._factory,
                 timeout=30,
@@ -330,7 +330,7 @@ def start_replication(self, hs: "HomeServer"):
             host = hs.config.worker.worker_replication_host
             port = hs.config.worker.worker_replication_port
             hs.get_reactor().connectTCP(
-                host,  # type: ignore[arg-type]
+                host,
                 port,
                 self._factory,
                 timeout=30,
diff --git a/synapse/replication/tcp/redis.py b/synapse/replication/tcp/redis.py
--- a/synapse/replication/tcp/redis.py
+++ b/synapse/replication/tcp/redis.py
@@ -373,7 +373,7 @@ def lazyConnection(
 
     reactor = hs.get_reactor()
     reactor.connectTCP(
-        host,  # type: ignore[arg-type]
+        host,
         port,
         factory,
         timeout=30,


Test patch:
Repository: matrix-org/synapse

Issue:
Remove unstable flags for MSC3283
#11933 added support for the stable flags in MSC3283, so we should remove the unstable ones in the next release


Discussion:
Where "next release" == 1.54.0, I think?

Resolving patch:
diff --git a/synapse/config/experimental.py b/synapse/config/experimental.py
--- a/synapse/config/experimental.py
+++ b/synapse/config/experimental.py
@@ -41,9 +41,6 @@ def read_config(self, config: JsonDict, **kwargs):
         # MSC3244 (room version capabilities)
         self.msc3244_enabled: bool = experimental.get("msc3244_enabled", True)
 
-        # MSC3283 (set displayname, avatar_url and change 3pid capabilities)
-        self.msc3283_enabled: bool = experimental.get("msc3283_enabled", False)
-
         # MSC3266 (room summary api)
         self.msc3266_enabled: bool = experimental.get("msc3266_enabled", False)
 
diff --git a/synapse/rest/client/capabilities.py b/synapse/rest/client/capabilities.py
--- a/synapse/rest/client/capabilities.py
+++ b/synapse/rest/client/capabilities.py
@@ -72,20 +72,6 @@ async def on_GET(self, request: SynapseRequest) -> Tuple[int, JsonDict]:
                 "org.matrix.msc3244.room_capabilities"
             ] = MSC3244_CAPABILITIES
 
-        # Must be removed in later versions.
-        # Is only included for migration.
-        # Also the parts in `synapse/config/experimental.py`.
-        if self.config.experimental.msc3283_enabled:
-            response["capabilities"]["org.matrix.msc3283.set_displayname"] = {
-                "enabled": self.config.registration.enable_set_displayname
-            }
-            response["capabilities"]["org.matrix.msc3283.set_avatar_url"] = {
-                "enabled": self.config.registration.enable_set_avatar_url
-            }
-            response["capabilities"]["org.matrix.msc3283.3pid_changes"] = {
-                "enabled": self.config.registration.enable_3pid_changes
-            }
-
         if self.config.experimental.msc3440_enabled:
             response["capabilities"]["io.element.thread"] = {"enabled": True}
 


Test patch:
Repository: matrix-org/synapse

Issue:
Implement MSC3589: update default room version to v9
This is a tracking issue to update the default room version to room version 9, per [MSC3589](https://github.com/matrix-org/matrix-doc/pull/3589).

This is currently blocked on the MSC going through the FCP process.


Discussion:

Resolving patch:
diff --git a/synapse/config/server.py b/synapse/config/server.py
--- a/synapse/config/server.py
+++ b/synapse/config/server.py
@@ -146,7 +146,7 @@ def generate_ip_set(
     "fec0::/10",
 ]
 
-DEFAULT_ROOM_VERSION = "6"
+DEFAULT_ROOM_VERSION = "9"
 
 ROOM_COMPLEXITY_TOO_GREAT = (
     "Your homeserver is unable to join rooms this large or complex. "


Test patch:
Repository: matrix-org/synapse

Issue:
Don't use `isinstance(..., Sequence)`
In a couple of places, we validate incoming lists of strings with `isinstance(..., Sequence)`. Unfortunately this lets plain strings slip through.
We have to make these checks more specific.


Discussion:

Resolving patch:
diff --git a/synapse/federation/federation_client.py b/synapse/federation/federation_client.py
--- a/synapse/federation/federation_client.py
+++ b/synapse/federation/federation_client.py
@@ -1428,7 +1428,7 @@ async def send_request(
 
             # Validate children_state of the room.
             children_state = room.pop("children_state", [])
-            if not isinstance(children_state, Sequence):
+            if not isinstance(children_state, list):
                 raise InvalidResponseError("'room.children_state' must be a list")
             if any(not isinstance(e, dict) for e in children_state):
                 raise InvalidResponseError("Invalid event in 'children_state' list")
@@ -1440,14 +1440,14 @@ async def send_request(
 
             # Validate the children rooms.
             children = res.get("children", [])
-            if not isinstance(children, Sequence):
+            if not isinstance(children, list):
                 raise InvalidResponseError("'children' must be a list")
             if any(not isinstance(r, dict) for r in children):
                 raise InvalidResponseError("Invalid room in 'children' list")
 
             # Validate the inaccessible children.
             inaccessible_children = res.get("inaccessible_children", [])
-            if not isinstance(inaccessible_children, Sequence):
+            if not isinstance(inaccessible_children, list):
                 raise InvalidResponseError("'inaccessible_children' must be a list")
             if any(not isinstance(r, str) for r in inaccessible_children):
                 raise InvalidResponseError(
@@ -1630,7 +1630,7 @@ def _validate_hierarchy_event(d: JsonDict) -> None:
         raise ValueError("Invalid event: 'content' must be a dict")
 
     via = content.get("via")
-    if not isinstance(via, Sequence):
+    if not isinstance(via, list):
         raise ValueError("Invalid event: 'via' must be a list")
     if any(not isinstance(v, str) for v in via):
         raise ValueError("Invalid event: 'via' must be a list of strings")
diff --git a/synapse/handlers/room_summary.py b/synapse/handlers/room_summary.py
--- a/synapse/handlers/room_summary.py
+++ b/synapse/handlers/room_summary.py
@@ -857,7 +857,7 @@ def as_json(self) -> JsonDict:
 
 def _has_valid_via(e: EventBase) -> bool:
     via = e.content.get("via")
-    if not via or not isinstance(via, Sequence):
+    if not via or not isinstance(via, list):
         return False
     for v in via:
         if not isinstance(v, str):


Test patch:
Repository: matrix-org/synapse

Issue:
RC versions of dependencies don't satisfy the run-time dependency checker; `Need Twisted>=18.9.0, but got Twisted==21.7.0rc3` (1.54.0rc1 suspected regression)
When deploying `1.54.0rc1` to matrix.org and some personal homeservers that had an RC of Twisted installed, the dependency checker complained:

`Need Twisted>=18.9.0, but got Twisted==21.7.0rc3`

For some reason it appears that being an RC makes the version insufficient, even though the version is higher. Using the non-RC version works fine.

Possibly fall-out from https://github.com/matrix-org/synapse/pull/12088?

I wonder if the same logic as e.g. `pip` is being used, in that it would never select an RC version as being satisfactory unless it was a hard match?


Discussion:

Resolving patch:
diff --git a/synapse/util/check_dependencies.py b/synapse/util/check_dependencies.py
--- a/synapse/util/check_dependencies.py
+++ b/synapse/util/check_dependencies.py
@@ -163,7 +163,8 @@ def check_requirements(extra: Optional[str] = None) -> None:
                 deps_unfulfilled.append(requirement.name)
                 errors.append(_not_installed(requirement, extra))
         else:
-            if not requirement.specifier.contains(dist.version):
+            # We specify prereleases=True to allow prereleases such as RCs.
+            if not requirement.specifier.contains(dist.version, prereleases=True):
                 deps_unfulfilled.append(requirement.name)
                 errors.append(_incorrect_version(requirement, dist.version, extra))
 


Test patch:
diff --git a/tests/util/test_check_dependencies.py b/tests/util/test_check_dependencies.py
--- a/tests/util/test_check_dependencies.py
+++ b/tests/util/test_check_dependencies.py
@@ -27,7 +27,9 @@ def read_text(self, filename):
 
 
 old = DummyDistribution("0.1.2")
+old_release_candidate = DummyDistribution("0.1.2rc3")
 new = DummyDistribution("1.2.3")
+new_release_candidate = DummyDistribution("1.2.3rc4")
 
 # could probably use stdlib TestCase --- no need for twisted here
 
@@ -110,3 +112,20 @@ def test_check_for_extra_dependencies(self) -> None:
             with self.mock_installed_package(new):
                 # should not raise
                 check_requirements("cool-extra")
+
+    def test_release_candidates_satisfy_dependency(self) -> None:
+        """
+        Tests that release candidates count as far as satisfying a dependency
+        is concerned.
+        (Regression test, see #12176.)
+        """
+        with patch(
+            "synapse.util.check_dependencies.metadata.requires",
+            return_value=["dummypkg >= 1"],
+        ):
+            with self.mock_installed_package(old_release_candidate):
+                self.assertRaises(DependencyException, check_requirements)
+
+            with self.mock_installed_package(new_release_candidate):
+                # should not raise
+                check_requirements()