internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: matrix-org/synapse

Issue:
Audit and/or add validation to endpoints in synapse/rest/client/account.py
# Processed

- [x] EmailPasswordRequestTokenRestServlet
- [x] PasswordRestServlet
- [x] DeactivateAccountRestServlet
- [x] EmailThreepidRequestTokenRestServlet
   - The above covered by #13188
- [x] MsisdnThreepidRequestTokenRestServlet
- [x] AccountStatusRestServlet
  - The above covered by #13687
- [x] ThreepidAddRestServlet
- [x] ThreepidBindRestServlet
- [x] ThreepidUnbindRestServlet
- [x] ThreepidDeleteRestServlet
  - The above covered by #13736

# Todo

- [x] AddThreepidMsisdnSubmitTokenServlet
  - The above covered by https://github.com/matrix-org/synapse/pull/13832

# No action

- [x] AddThreepidEmailSubmitTokenServlet, GET only
- [x] WhoamiRestServlet, GET only
- [x] ThreepidRestServlet
  - GET: nothing to do  
  - Going to ignore POST body: it's deprecated and seemingly unused


Discussion:

Resolving patch:
diff --git a/synapse/rest/client/account.py b/synapse/rest/client/account.py
--- a/synapse/rest/client/account.py
+++ b/synapse/rest/client/account.py
@@ -534,6 +534,11 @@ class AddThreepidMsisdnSubmitTokenServlet(RestServlet):
         "/add_threepid/msisdn/submit_token$", releases=(), unstable=True
     )
 
+    class PostBody(RequestBodyModel):
+        client_secret: ClientSecretStr
+        sid: StrictStr
+        token: StrictStr
+
     def __init__(self, hs: "HomeServer"):
         super().__init__()
         self.config = hs.config
@@ -549,16 +554,14 @@ async def on_POST(self, request: Request) -> Tuple[int, JsonDict]:
                 "instead.",
             )
 
-        body = parse_json_object_from_request(request)
-        assert_params_in_dict(body, ["client_secret", "sid", "token"])
-        assert_valid_client_secret(body["client_secret"])
+        body = parse_and_validate_json_object_from_request(request, self.PostBody)
 
         # Proxy submit_token request to msisdn threepid delegate
         response = await self.identity_handler.proxy_msisdn_submit_token(
             self.config.registration.account_threepid_delegate_msisdn,
-            body["client_secret"],
-            body["sid"],
-            body["token"],
+            body.client_secret,
+            body.sid,
+            body.token,
         )
         return 200, response
 
@@ -581,6 +584,10 @@ async def on_GET(self, request: SynapseRequest) -> Tuple[int, JsonDict]:
 
         return 200, {"threepids": threepids}
 
+    # NOTE(dmr): I have chosen not to use Pydantic to parse this request's body, because
+    # the endpoint is deprecated. (If you really want to, you could do this by reusing
+    # ThreePidBindRestServelet.PostBody with an `alias_generator` to handle
+    # `threePidCreds` versus `three_pid_creds`.
     async def on_POST(self, request: SynapseRequest) -> Tuple[int, JsonDict]:
         if not self.hs.config.registration.enable_3pid_changes:
             raise SynapseError(


Test patch:
Repository: matrix-org/synapse

Issue:
CI run against latest deps is failing
See https://github.com/matrix-org/synapse/actions/runs/3125758020



Discussion:
Complement flake, plus mypy:

```
synapse/http/server.py:751: error: Incompatible types in assignment (expression has type "Callable[[Any], bytes]", variable has type "Callable[[object], bytes]")  [assignment]
Found 1 error in 1 file (checked 734 source files)
```

This is due to https://github.com/matrix-org/python-canonicaljson/pull/52

Fix by either marking `encoder: Callable[[Any], bytes]`; by making `_encode_json_bytes` take `object`; or with a `#type: ignore`.

Resolving patch:
diff --git a/synapse/http/server.py b/synapse/http/server.py
--- a/synapse/http/server.py
+++ b/synapse/http/server.py
@@ -705,7 +705,7 @@ def stopProducing(self) -> None:
         self._request = None
 
 
-def _encode_json_bytes(json_object: Any) -> bytes:
+def _encode_json_bytes(json_object: object) -> bytes:
     """
     Encode an object into JSON. Returns an iterator of bytes.
     """
@@ -746,7 +746,7 @@ def respond_with_json(
         return None
 
     if canonical_json:
-        encoder = encode_canonical_json
+        encoder: Callable[[object], bytes] = encode_canonical_json
     else:
         encoder = _encode_json_bytes
 


Test patch:
Repository: matrix-org/synapse

Issue:
`parse_and_validate_server_name(":80")` raises `IndexError` instead of `ValueError`
Currently `parse_server_name("")` raises a ValueError by accident, while `parse_server_name(":80")` returns `("", 80)`.
The latter causes `parse_and_validate_server_name` to raise an IndexError when it tries to inspect the first character of the hostname.

As part of the fix, we should make `parse_server_name` behave consistently. Either `parse_server_name("")` should return `("", None)`, or `parse_server_name(":80")` should raise a ValueError.

I'm mildly in favor of the former, since in all other cases `parse_server_name` behaves like a hostname-port splitter without regard for the validity of the hostname.

https://sentry.matrix.org/sentry/synapse-matrixorg/issues/231342/
https://sentry.matrix.org/sentry/synapse-matrixorg/issues/242910/
https://sentry.matrix.org/sentry/synapse-matrixorg/issues/197400/



Discussion:
hello, I opened a PR for this here https://github.com/matrix-org/synapse/pull/12167

Resolving patch:
diff --git a/synapse/util/stringutils.py b/synapse/util/stringutils.py
--- a/synapse/util/stringutils.py
+++ b/synapse/util/stringutils.py
@@ -86,7 +86,7 @@ def parse_server_name(server_name: str) -> Tuple[str, Optional[int]]:
         ValueError if the server name could not be parsed.
     """
     try:
-        if server_name[-1] == "]":
+        if server_name and server_name[-1] == "]":
             # ipv6 literal, hopefully
             return server_name, None
 
@@ -123,7 +123,7 @@ def parse_and_validate_server_name(server_name: str) -> Tuple[str, Optional[int]
     # that nobody is sneaking IP literals in that look like hostnames, etc.
 
     # look for ipv6 literals
-    if host[0] == "[":
+    if host and host[0] == "[":
         if host[-1] != "]":
             raise ValueError("Mismatched [...] in server name '%s'" % (server_name,))
 


Test patch:
diff --git a/tests/http/test_endpoint.py b/tests/http/test_endpoint.py
--- a/tests/http/test_endpoint.py
+++ b/tests/http/test_endpoint.py
@@ -25,6 +25,8 @@ def test_parse_server_name(self):
             "[0abc:1def::1234]": ("[0abc:1def::1234]", None),
             "1.2.3.4:1": ("1.2.3.4", 1),
             "[0abc:1def::1234]:8080": ("[0abc:1def::1234]", 8080),
+            ":80": ("", 80),
+            "": ("", None),
         }
 
         for i, o in test_data.items():
@@ -42,6 +44,7 @@ def test_validate_bad_server_names(self):
             "newline.com\n",
             ".empty-label.com",
             "1234:5678:80",  # too many colons
+            ":80",
         ]
         for i in test_data:
             try:
Repository: matrix-org/synapse

Issue:
`poetry install` seems to build Rust crate in `debug` mode
We should build it in `release` mode for packages, at the very least.

I *think* wheels are correctly built in release mode


Discussion:
I think we should just set `debug=True` in `build_rust.py` to force `poetry install` to always build in release mode. The default behaviour is to use `debug` for "inplace" builds (which is what poetry install does) and otherwise "release" (so poetry build would be fine).
> think we should just set `debug=True` in `build_rust.py` to force `poetry install` to always build in release mode.

Err, should this be `debug=False`?
Yes :facepalm: 

Resolving patch:
diff --git a/build_rust.py b/build_rust.py
--- a/build_rust.py
+++ b/build_rust.py
@@ -15,6 +15,9 @@ def build(setup_kwargs: Dict[str, Any]) -> None:
         path=cargo_toml_path,
         binding=Binding.PyO3,
         py_limited_api=True,
+        # We force always building in release mode, as we can't tell the
+        # difference between using `poetry` in development vs production.
+        debug=False,
     )
     setup_kwargs.setdefault("rust_extensions", []).append(extension)
     setup_kwargs["zip_safe"] = False


Test patch:
Repository: matrix-org/synapse

Issue:
CI run against Twisted trunk is failing
See https://github.com/matrix-org/synapse/actions/runs/3172535322



Discussion:
```
synapse/handlers/ui_auth/checkers.py:122: error: Item "None" of "Optional[bytes]" has no attribute "decode"  [union-attr]
synapse/handlers/cas.py:133: error: Incompatible types in assignment (expression has type "Optional[bytes]", variable has type "bytes")  [assignment]
```

Would guess this is https://github.com/twisted/twisted/pull/11688. Twisted seem to be adding a bunch of annotations at the moment.

Resolving patch:
diff --git a/synapse/handlers/cas.py b/synapse/handlers/cas.py
--- a/synapse/handlers/cas.py
+++ b/synapse/handlers/cas.py
@@ -130,6 +130,9 @@ async def _validate_ticket(
         except PartialDownloadError as pde:
             # Twisted raises this error if the connection is closed,
             # even if that's being used old-http style to signal end-of-data
+            # Assertion is for mypy's benefit. Error.response is Optional[bytes],
+            # but a PartialDownloadError should always have a non-None response.
+            assert pde.response is not None
             body = pde.response
         except HttpResponseException as e:
             description = (
diff --git a/synapse/handlers/ui_auth/checkers.py b/synapse/handlers/ui_auth/checkers.py
--- a/synapse/handlers/ui_auth/checkers.py
+++ b/synapse/handlers/ui_auth/checkers.py
@@ -119,6 +119,9 @@ async def check_auth(self, authdict: dict, clientip: str) -> Any:
         except PartialDownloadError as pde:
             # Twisted is silly
             data = pde.response
+            # For mypy's benefit. A general Error.response is Optional[bytes], but
+            # a PartialDownloadError.response should be bytes AFAICS.
+            assert data is not None
             resp_body = json_decoder.decode(data.decode("utf-8"))
 
         if "success" in resp_body:


Test patch:
Repository: matrix-org/synapse

Issue:
Support Matrix 1.4
Matrix 1.4 was just released, we should break down a list of things that Synapse needs to support it. See the release notes: https://matrix.org/blog/2022/09/29/matrix-v-1-4-release#client-server-api

Depends on #13953

----

## Client-Server API

* [x] Remove unused policy room sharing mechanism, as per MSC3844.
* [x] Add a `.m.rule.room.server_acl` push rule to match m.room.server_acl events, as per MSC3786. #12601, #13997
* [x] Add `Cross-Origin-Resource-Policy` (CORP) headers to media repository, as per MSC3828. #12944
* [x] Copy a room's `type` when upgrading it, as per MSC3818. #12786
* [x] Add `room_types` filter and `room_type` response to `/publicRooms`, as per MSC3827. #13031
* [x] Add `m.replace` relations (event edits), as per MSC2676. #12793, #12503
* [x] Add `m.read.private` receipts, as per MSC2285.
* [x] Make `m.fully_read` optional on `/read_markers`, as per MSC2285.
* [x] Allow `m.fully_read` markers to be set from `/receipts`, as per MSC2285.
* [x] Add threading via `m.thread` relations, as per MSC3440, MSC3816, MSC3856, and MSC3715. #13394
* [x] Add per-thread notifications and read receipts, as per MSC3771 and MSC3773. #12550

## Server-Server API

* [x] Add per-thread notifications and read receipts, as per MSC3771 and MSC3773. #12550

## Application Service API

* [x] Replace homeserver authorization approach with an `Authorization` header instead of `access_token` when talking to the application service, as per MSC2832. #13996



Discussion:

Resolving patch:
diff --git a/synapse/rest/client/versions.py b/synapse/rest/client/versions.py
--- a/synapse/rest/client/versions.py
+++ b/synapse/rest/client/versions.py
@@ -76,6 +76,7 @@ def on_GET(self, request: Request) -> Tuple[int, JsonDict]:
                     "v1.1",
                     "v1.2",
                     "v1.3",
+                    "v1.4",
                 ],
                 # as per MSC1497:
                 "unstable_features": {


Test patch:
Repository: matrix-org/synapse

Issue:
"Invalid filter json" error message should use an appropriate error code
```json
{
  "errcode": "M_UNKNOWN",
  "error": "Invalid filter JSON"
}
```

From https://github.com/matrix-org/synapse/blob/a6895dd576f96d7fd086fb4128d48ac8a3f098c5/synapse/rest/client/sync.py#L145-L154

We should return `M_NOT_JSON` if the decode fails.

We should probably also defer the `set_timeline_upper_limit` call until `            self.filtering.check_valid_filter(filter_object)` has passed?


Discussion:
I would like to work on it. @DMRobertson could you please elaborate more on defer logic.
> I would like to work on it. @DMRobertson could you please elaborate more on defer logic.

I was thinking that we should move lines 148-150 outside of the `try` block, because they do not related to the decoding of JSON. Probably between lines 153 and 154. But this isn't a hugely important change to make.

Resolving patch:
diff --git a/synapse/rest/client/sync.py b/synapse/rest/client/sync.py
--- a/synapse/rest/client/sync.py
+++ b/synapse/rest/client/sync.py
@@ -146,12 +146,12 @@ async def on_GET(self, request: SynapseRequest) -> Tuple[int, JsonDict]:
         elif filter_id.startswith("{"):
             try:
                 filter_object = json_decoder.decode(filter_id)
-                set_timeline_upper_limit(
-                    filter_object, self.hs.config.server.filter_timeline_limit
-                )
             except Exception:
-                raise SynapseError(400, "Invalid filter JSON")
+                raise SynapseError(400, "Invalid filter JSON", errcode=Codes.NOT_JSON)
             self.filtering.check_valid_filter(filter_object)
+            set_timeline_upper_limit(
+                filter_object, self.hs.config.server.filter_timeline_limit
+            )
             filter_collection = FilterCollection(self.hs, filter_object)
         else:
             try:


Test patch:
Repository: matrix-org/synapse

Issue:
Use `room_stats_state.room_type` column instead of fetching `m.room.create` event
#13031 introduced a `room_type` column in `room_stats_state`. There are now one or two places where we can avoid fetching `m.room.create` events and just use the new column instead.

Searching for `EventContentFields.ROOM_TYPE` or `"room_type"` ought to find them all.

eg. if we make `get_room_with_stats` return `room_type`, we can avoid fetching the `m.room.create` event in `RoomSummaryHandler._build_room_entry`.


Discussion:
We do need to wait until the background update finishes, however. I'm unsure if we have a standard way of doing that?
> eg. if we make `get_room_with_stats` return `room_type`

Looks like #13208 already did this part! 🎉 

Resolving patch:
diff --git a/synapse/handlers/room_summary.py b/synapse/handlers/room_summary.py
--- a/synapse/handlers/room_summary.py
+++ b/synapse/handlers/room_summary.py
@@ -20,7 +20,6 @@
 import attr
 
 from synapse.api.constants import (
-    EventContentFields,
     EventTypes,
     HistoryVisibility,
     JoinRules,
@@ -701,13 +700,6 @@ async def _build_room_entry(self, room_id: str, for_federation: bool) -> JsonDic
         # there should always be an entry
         assert stats is not None, "unable to retrieve stats for %s" % (room_id,)
 
-        current_state_ids = await self._storage_controllers.state.get_current_state_ids(
-            room_id
-        )
-        create_event = await self._store.get_event(
-            current_state_ids[(EventTypes.Create, "")]
-        )
-
         entry = {
             "room_id": stats["room_id"],
             "name": stats["name"],
@@ -720,7 +712,7 @@ async def _build_room_entry(self, room_id: str, for_federation: bool) -> JsonDic
                 stats["history_visibility"] == HistoryVisibility.WORLD_READABLE
             ),
             "guest_can_join": stats["guest_access"] == "can_join",
-            "room_type": create_event.content.get(EventContentFields.ROOM_TYPE),
+            "room_type": stats["room_type"],
         }
 
         if self._msc3266_enabled:
@@ -730,7 +722,11 @@ async def _build_room_entry(self, room_id: str, for_federation: bool) -> JsonDic
         # Federation requests need to provide additional information so the
         # requested server is able to filter the response appropriately.
         if for_federation:
+            current_state_ids = (
+                await self._storage_controllers.state.get_current_state_ids(room_id)
+            )
             room_version = await self._store.get_room_version(room_id)
+
             if await self._event_auth_handler.has_restricted_join_rules(
                 current_state_ids, room_version
             ):


Test patch:
Repository: matrix-org/synapse

Issue:
Outbound federation broken on Python 3.7
When there should be outbound federation traffic:

```
2022-10-21 23:33:28,436 - synapse.metrics.background_process_metrics - 244 - ERROR - process_event_queue_for_federation-6 - Background process 'process_event_queue_for_federation' threw an exception
...
Traceback (most recent call last):
  File "/home/synapse/src/synapse/synapse/metrics/background_process_metrics.py", line 240, in run
    return await func(*args, **kwargs)
  File "/home/synapse/src/synapse/synapse/federation/sender/__init__.py", line 539, in _process_event_queue_loop
    last_id = next(reversed(event_ids))
TypeError: 'dict_keys' object is not reversible
```

Turns out that `dict.keys` is, as the error says, not reversible on Python 3.7.

I think this was introduced by #13813 (and I think it was my rubbish suggestion). 


Discussion:

Resolving patch:
diff --git a/synapse/federation/sender/__init__.py b/synapse/federation/sender/__init__.py
--- a/synapse/federation/sender/__init__.py
+++ b/synapse/federation/sender/__init__.py
@@ -536,8 +536,7 @@ async def handle_room_events(events: List[EventBase]) -> None:
 
                 if event_entries:
                     now = self.clock.time_msec()
-                    last_id = next(reversed(event_ids))
-                    ts = event_to_received_ts[last_id]
+                    ts = max(t for t in event_to_received_ts.values() if t)
                     assert ts is not None
 
                     synapse.metrics.event_processing_lag.labels(


Test patch:
Repository: matrix-org/synapse

Issue:
Faster joins: check compatibility with history import
I suspect the MSC2716 code makes incorrect assumptions about whether state is complete at the insertion points.

https://github.com/matrix-org/synapse/blob/7c6b2204d143550d81e5bf9612c4e69fe0866b4c/synapse/handlers/federation_event.py#L778-L785


Discussion:
cc @MadLittleMods 
Maybe. We need the full state at the event you insert next to (the `/batch_send?prev_event_id`) to inherit from. See:

 - [`synapse/rest/client/room_batch.py#L139-L145`](https://github.com/matrix-org/synapse/blob/3d1d510fa9d6d46e08b347dd8c803e5dd9b8a5fe/synapse/rest/client/room_batch.py#L139-L145)
 - [`synapse/handlers/room_batch.py#L125-L150` -> `get_most_recent_full_state_ids_from_event_id_list`](https://github.com/matrix-org/synapse/blob/7c6b2204d143550d81e5bf9612c4e69fe0866b4c/synapse/handlers/room_batch.py#L125-L150)
We should just disallow history import for partial-state rooms

Resolving patch:
diff --git a/synapse/rest/client/room_batch.py b/synapse/rest/client/room_batch.py
--- a/synapse/rest/client/room_batch.py
+++ b/synapse/rest/client/room_batch.py
@@ -108,6 +108,13 @@ async def on_POST(
                 errcode=Codes.MISSING_PARAM,
             )
 
+        if await self.store.is_partial_state_room(room_id):
+            raise SynapseError(
+                HTTPStatus.BAD_REQUEST,
+                "Cannot insert history batches until we have fully joined the room",
+                errcode=Codes.UNABLE_DUE_TO_PARTIAL_STATE,
+            )
+
         # Verify the batch_id_from_query corresponds to an actual insertion event
         # and have the batch connected.
         if batch_id_from_query:


Test patch:
Repository: matrix-org/synapse

Issue:
Set legacy metric names to default off, pending removal in 1.73
As promised in https://github.com/matrix-org/synapse/pull/14024


Discussion:

Resolving patch:
diff --git a/synapse/config/metrics.py b/synapse/config/metrics.py
--- a/synapse/config/metrics.py
+++ b/synapse/config/metrics.py
@@ -43,7 +43,7 @@ class MetricsConfig(Config):
     def read_config(self, config: JsonDict, **kwargs: Any) -> None:
         self.enable_metrics = config.get("enable_metrics", False)
 
-        self.enable_legacy_metrics = config.get("enable_legacy_metrics", True)
+        self.enable_legacy_metrics = config.get("enable_legacy_metrics", False)
 
         self.report_stats = config.get("report_stats", None)
         self.report_stats_endpoint = config.get(


Test patch:
Repository: matrix-org/synapse

Issue:
Test against PostgreSQL 15
https://www.postgresql.org/docs/15/release-15.html

We should update our CI to test against latest Postgres, which is now 15 instead of 14.


Discussion:
I think the steps to do this are:

1. Update the `tests.yml` file and replace `14` with `15`.
Hold off for a bit, newest thing up there is the RC2 version

https://hub.docker.com/_/postgres
Ready now 😄 
> I think the steps to do this are:
> 
> 1. Update the `tests.yml` file and replace `14` with `15`.

2. Update `.ci/scripts/calculate_jobs.py` and replace `14` with `15`.

Resolving patch:
diff --git a/.ci/scripts/calculate_jobs.py b/.ci/scripts/calculate_jobs.py
--- a/.ci/scripts/calculate_jobs.py
+++ b/.ci/scripts/calculate_jobs.py
@@ -64,7 +64,7 @@ def set_output(key: str, value: str):
         {
             "python-version": "3.11",
             "database": "postgres",
-            "postgres-version": "14",
+            "postgres-version": "15",
             "extras": "all",
         }
     )


Test patch:
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -412,7 +412,7 @@ jobs:
             postgres-version: "10"
 
           - python-version: "3.11"
-            postgres-version: "14"
+            postgres-version: "15"
 
     services:
       postgres:
@@ -430,6 +430,15 @@ jobs:
 
     steps:
       - uses: actions/checkout@v3
+      - name: Add PostgreSQL apt repository
+        # We need a version of pg_dump that can handle the version of
+        # PostgreSQL being tested against. The Ubuntu package repository lags
+        # behind new releases, so we have to use the PostreSQL apt repository.
+        # Steps taken from https://www.postgresql.org/download/linux/ubuntu/
+        run: |
+          sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
+          wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
+          sudo apt-get update
       - run: sudo apt-get -qq install xmlsec1 postgresql-client
       - uses: matrix-org/setup-python-poetry@v1
         with: