internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: matrix-org/synapse

Issue:
Uploading a too large file: wrong error code
When uploading a too large file, synapse responds with a 413 (OK) but with errcode M_UNKNOWN. According to the [spec](https://matrix.org/docs/spec/client_server/latest#post-matrix-media-r0-upload), it should be "M_TOO_LARGE"

Received responseCode: 413
Received responseBody:
`{"errcode":"M_UNKNOWN","error":"Upload request body is too large"}`




Discussion:

Resolving patch:
diff --git a/synapse/rest/media/v1/upload_resource.py b/synapse/rest/media/v1/upload_resource.py
--- a/synapse/rest/media/v1/upload_resource.py
+++ b/synapse/rest/media/v1/upload_resource.py
@@ -17,7 +17,7 @@
 
 from twisted.web.server import NOT_DONE_YET
 
-from synapse.api.errors import SynapseError
+from synapse.api.errors import Codes, SynapseError
 from synapse.http.server import (
     DirectServeResource,
     respond_with_json,
@@ -56,7 +56,11 @@ async def _async_render_POST(self, request):
         if content_length is None:
             raise SynapseError(msg="Request must specify a Content-Length", code=400)
         if int(content_length) > self.max_upload_size:
-            raise SynapseError(msg="Upload request body is too large", code=413)
+            raise SynapseError(
+                msg="Upload request body is too large",
+                code=413,
+                errcode=Codes.TOO_LARGE,
+            )
 
         upload_name = parse_string(request, b"filename", encoding=None)
         if upload_name:


Test patch:
Repository: matrix-org/synapse

Issue:
Logging broken in Docker image
### Description

Synapse in the `matrixdotorg/synapse` Docker image produces debug logs normally in v1.0.0 but after v1.1.0 *only* HTTP request logs.

I believe it is related to the removal of `loggers.synapse.level` in #5547, restoring that restores normal debug logging.

### Steps to reproduce

- `docker run --rm -it -p 8008:8008 -e SYNAPSE_SERVER_NAME=localhost -e SYNAPSE_REPORT_STATS=no -e SYNAPSE_LOG_LEVEL=DEBUG -e SYNAPSE_NO_TLS=true matrixdotorg/synapse:v1.0.0-py3`
- Observe debug logs
- `docker run --rm -it -p 8008:8008 -e SYNAPSE_SERVER_NAME=localhost -e SYNAPSE_REPORT_STATS=no -e SYNAPSE_LOG_LEVEL=DEBUG -e SYNAPSE_NO_TLS=true matrixdotorg/synapse:v1.1.0-py3`
- Observe diminished logs, only HTTP related request logs
- Add `loggers.synapse.level="DEBUG"` to logging config (exact details left as an exercise)
- Run the result
- observe logs again

### Version information

Problem observed 

- **Version**:

`>=1.1.0`

- **Install method**:

Docker.

- **Platform**:

Tested on Docker on Windows and Docker on Debian Linux



Discussion:

Resolving patch:
diff --git a/synapse/config/logger.py b/synapse/config/logger.py
--- a/synapse/config/logger.py
+++ b/synapse/config/logger.py
@@ -68,9 +68,6 @@
         filters: [context]
 
 loggers:
-    synapse:
-        level: INFO
-
     synapse.storage.SQL:
         # beware: increasing this to DEBUG will make synapse log sensitive
         # information such as access tokens.
@@ -79,6 +76,8 @@
 root:
     level: INFO
     handlers: [file, console]
+
+disable_existing_loggers: false
 """
 )
 


Test patch:
Repository: matrix-org/synapse

Issue:
/sync table-scans events
the labels support seems to have introduced a performance regression in the /sync code such that it tries to table-scan events, which is ... suboptimal


Discussion:

Resolving patch:
diff --git a/synapse/storage/data_stores/main/stream.py b/synapse/storage/data_stores/main/stream.py
--- a/synapse/storage/data_stores/main/stream.py
+++ b/synapse/storage/data_stores/main/stream.py
@@ -871,14 +871,38 @@ def _paginate_room_events_txn(
 
         args.append(int(limit))
 
-        sql = (
-            "SELECT DISTINCT event_id, topological_ordering, stream_ordering"
-            " FROM events"
-            " LEFT JOIN event_labels USING (event_id, room_id, topological_ordering)"
-            " WHERE outlier = ? AND room_id = ? AND %(bounds)s"
-            " ORDER BY topological_ordering %(order)s,"
-            " stream_ordering %(order)s LIMIT ?"
-        ) % {"bounds": bounds, "order": order}
+        select_keywords = "SELECT"
+        join_clause = ""
+        if event_filter and event_filter.labels:
+            # If we're not filtering on a label, then joining on event_labels will
+            # return as many row for a single event as the number of labels it has. To
+            # avoid this, only join if we're filtering on at least one label.
+            join_clause = """
+                LEFT JOIN event_labels
+                USING (event_id, room_id, topological_ordering)
+            """
+            if len(event_filter.labels) > 1:
+                # Using DISTINCT in this SELECT query is quite expensive, because it
+                # requires the engine to sort on the entire (not limited) result set,
+                # i.e. the entire events table. We only need to use it when we're
+                # filtering on more than two labels, because that's the only scenario
+                # in which we can possibly to get multiple times the same event ID in
+                # the results.
+                select_keywords += "DISTINCT"
+
+        sql = """
+            %(select_keywords)s event_id, topological_ordering, stream_ordering
+            FROM events
+            %(join_clause)s
+            WHERE outlier = ? AND room_id = ? AND %(bounds)s
+            ORDER BY topological_ordering %(order)s,
+            stream_ordering %(order)s LIMIT ?
+        """ % {
+            "select_keywords": select_keywords,
+            "join_clause": join_clause,
+            "bounds": bounds,
+            "order": order,
+        }
 
         txn.execute(sql, args)
 


Test patch:
Repository: matrix-org/synapse

Issue:
parameter mismatch in register.py _register_email_threepid()
Hit this below error on our homeserver while trying the LDAP integration.

TypeError: _register_email_threepid() takes 4 positional arguments but 5 were given

https://github.com/matrix-org/synapse/blob/9ffcf0f7ba72f16e366f04db6384a9233b1808cb/synapse/handlers/register.py#L260 

I think this should be 
```yield self._register_email_threepid(user_id, threepid_dict, None)```
@hawkowl @anoadragon453 will send a patch if any of you can just validate 

many thanks


Discussion:
@syamgk This indeed seems to be a problem. A patch would be very welcome :)

Resolving patch:
diff --git a/synapse/handlers/register.py b/synapse/handlers/register.py
--- a/synapse/handlers/register.py
+++ b/synapse/handlers/register.py
@@ -266,7 +266,7 @@ def register_user(
             }
 
             # Bind email to new account
-            yield self._register_email_threepid(user_id, threepid_dict, None, False)
+            yield self._register_email_threepid(user_id, threepid_dict, None)
 
         return user_id
 


Test patch:
Repository: matrix-org/synapse

Issue:
Fatal 'Failed to upgrade database' error on startup
As of Synapse 1.7.0, when I start synapse with an old database version, I get this rather cryptic error.


Discussion:
I believe this only affects sqlite. I think it was introduced by #6487, which moves [this code](https://github.com/matrix-org/synapse/pull/6487/files#diff-01aac508ef38061957e6fb961582e5ceL363-L364) which calls `prepare_database` using a connection that was created with `run_new_connection=False` into [Datastore](https://github.com/matrix-org/synapse/pull/6487/files#diff-23406f76811c2c98a6ed1758c0bda2abR34) where it is called on a connection which has had the `on_new_connection` method handler called.

The sqlite engine, for reasons better known to itself, calls `prepare_database` in its `on_new_connection`: https://github.com/matrix-org/synapse/blob/v1.7.0/synapse/storage/engines/sqlite.py#L62.
turns out this is already fixed by #6513 on develop.

Resolving patch:
diff --git a/synapse/storage/engines/sqlite.py b/synapse/storage/engines/sqlite.py
--- a/synapse/storage/engines/sqlite.py
+++ b/synapse/storage/engines/sqlite.py
@@ -25,6 +25,9 @@ class Sqlite3Engine(object):
     def __init__(self, database_module, database_config):
         self.module = database_module
 
+        database = database_config.get("args", {}).get("database")
+        self._is_in_memory = database in (None, ":memory:",)
+
         # The current max state_group, or None if we haven't looked
         # in the DB yet.
         self._current_state_group_id = None
@@ -59,7 +62,12 @@ def convert_param_style(self, sql):
         return sql
 
     def on_new_connection(self, db_conn):
-        prepare_database(db_conn, self, config=None)
+        if self._is_in_memory:
+            # In memory databases need to be rebuilt each time. Ideally we'd
+            # reuse the same connection as we do when starting up, but that
+            # would involve using adbapi before we have started the reactor.
+            prepare_database(db_conn, self, config=None)
+
         db_conn.create_function("rank", 1, _rank)
 
     def is_deadlock(self, error):


Test patch:
Repository: matrix-org/synapse

Issue:
Events are rejected if min_depth is 0 and prev_events refers to missing events
If the server's `min_depth` (i.e. the `min_depth` column from the `room_depth` DB table) for a room is 0 (which means it has seen the first evens for that room), then the server considers it has seen every event in the room and therefore won't ever fetch the `prev_events` for any event sent to it.

This means that, if for some reason it doesn't see an event sent to that room, it's going to reject every event sent to that room that includes the missed event in its `prev_events` because it believes that it has seen every event in the room but doesn't know about the missed events.


Discussion:

Resolving patch:
diff --git a/synapse/handlers/federation.py b/synapse/handlers/federation.py
--- a/synapse/handlers/federation.py
+++ b/synapse/handlers/federation.py
@@ -248,13 +248,13 @@ async def on_receive_pdu(self, origin, pdu, sent_to_us_directly=False) -> None:
             prevs = set(pdu.prev_event_ids())
             seen = await self.store.have_seen_events(prevs)
 
-            if min_depth and pdu.depth < min_depth:
+            if min_depth is not None and pdu.depth < min_depth:
                 # This is so that we don't notify the user about this
                 # message, to work around the fact that some events will
                 # reference really really old events we really don't want to
                 # send to the clients.
                 pdu.internal_metadata.outlier = True
-            elif min_depth and pdu.depth > min_depth:
+            elif min_depth is not None and pdu.depth > min_depth:
                 missing_prevs = prevs - seen
                 if sent_to_us_directly and missing_prevs:
                     # If we're missing stuff, ensure we only fetch stuff one
diff --git a/synapse/storage/data_stores/main/event_federation.py b/synapse/storage/data_stores/main/event_federation.py
--- a/synapse/storage/data_stores/main/event_federation.py
+++ b/synapse/storage/data_stores/main/event_federation.py
@@ -506,7 +506,7 @@ def __init__(self, database: Database, db_conn, hs):
     def _update_min_depth_for_room_txn(self, txn, room_id, depth):
         min_depth = self._get_min_depth_interaction(txn, room_id)
 
-        if min_depth and depth >= min_depth:
+        if min_depth is not None and depth >= min_depth:
             return
 
         self.db.simple_upsert_txn(


Test patch:
Repository: matrix-org/synapse

Issue:
now LL is in CS 0.5, we should remove it from unstable-features in versions?



Discussion:
reliant on https://github.com/vector-im/riot-web/issues/9966
> reliant on vector-im/riot-web#9966

does this mean that we can never remove it, for fear of breaking old versions of riot?
No, it means we should get our ducks in a row and fix riot, wait X number (where X is small) Riot release cycles, then remove it from synapse.

The correct answer would have been Riot being proactive on this.
ftr, Riot's fix landed today which means it should land in a release in ~2 weeks. I would recommend waiting ~8 weeks before removing it from Synapse, but I'm also not a release coordinator. 
set a reminder 8 weeks from now
keywords: lazy-loading lazy loading

Resolving patch:
diff --git a/synapse/rest/client/versions.py b/synapse/rest/client/versions.py
--- a/synapse/rest/client/versions.py
+++ b/synapse/rest/client/versions.py
@@ -52,7 +52,6 @@ def on_GET(self, request):
                 ],
                 # as per MSC1497:
                 "unstable_features": {
-                    "m.lazy_load_members": True,
                     # as per MSC2190, as amended by MSC2264
                     # to be removed in r0.6.0
                     "m.id_access_token": True,


Test patch:
Repository: matrix-org/synapse

Issue:
explicitly warn about semantic whitespace in homeserver.yaml
folks unfamiliar with YAML repeatedly get bitten by this


Discussion:
c.f. https://mastodon.fidonet.io/@joacim/103642104431645730

Resolving patch:
diff --git a/synapse/config/_base.py b/synapse/config/_base.py
--- a/synapse/config/_base.py
+++ b/synapse/config/_base.py
@@ -53,6 +53,18 @@ class ConfigError(Exception):
 """
 
 
+CONFIG_FILE_HEADER = """\
+# Configuration file for Synapse.
+#
+# This is a YAML file: see [1] for a quick introduction. Note in particular
+# that *indentation is important*: all the elements of a list or dictionary
+# should have the same indentation.
+#
+# [1] https://docs.ansible.com/ansible/latest/reference_appendices/YAMLSyntax.html
+
+"""
+
+
 def path_exists(file_path):
     """Check if a file exists
 
@@ -344,7 +356,7 @@ def generate_config(
             str: the yaml config file
         """
 
-        return "\n\n".join(
+        return CONFIG_FILE_HEADER + "\n\n".join(
             dedent(conf)
             for conf in self.invoke_all(
                 "generate_config_section",
@@ -574,8 +586,8 @@ def load_or_generate_config(cls, description, argv):
                 if not path_exists(config_dir_path):
                     os.makedirs(config_dir_path)
                 with open(config_path, "w") as config_file:
-                    config_file.write("# vim:ft=yaml\n\n")
                     config_file.write(config_str)
+                    config_file.write("\n\n# vim:ft=yaml")
 
                 config_dict = yaml.safe_load(config_str)
                 obj.generate_missing_files(config_dict, config_dir_path)


Test patch:
Repository: matrix-org/synapse

Issue:
Warning about incorrect database collation/ctype should have a link on how to fix it
```
[homeserver_1] 2020-02-03 12:13:06,654 - synapse.storage.engines.postgres - 65 - WARNING - None- Database has incorrect collation of 'en_US.UTF-8'. Should be 'C'
[homeserver_1] 2020-02-03 12:13:06,655 - synapse.storage.engines.postgres - 70 - WARNING - None- Database has incorrect ctype of 'en_US.UTF-8'. Should be 'C'                        
```

These unfortunately don't say how I should be fixing it :(


Discussion:
I'm sure you read https://github.com/matrix-org/synapse/blob/develop/UPGRADE.rst#upgrading-to-v1100 carefully?
Can i work on this?
If you like, sure.
> I'm sure you read https://github.com/matrix-org/synapse/blob/develop/UPGRADE.rst#upgrading-to-v1100 carefully?

Not carefully, but did skim it and thought it would be fine. Updated Synapse, restarted, got the message, opened the issue. A bit of goldfish brain ._.

https://github.com/matrix-org/synapse/blob/develop/docs/postgres.md#fixing-incorrect-collate-or-ctype was the thing I was looking for
Should i add [https://github.com/matrix-org/synapse/blob/develop/docs/postgres.md#fixing-incorrect-collate-or-ctype](url) to the warning. I have just added docs/postgres.md 
`logger.warning(
                    "Database has incorrect collation of %r. Should be 'C'\n"
                    "See docs/postgres.md for more information.", collation
                )`

Resolving patch:
diff --git a/synapse/storage/engines/postgres.py b/synapse/storage/engines/postgres.py
--- a/synapse/storage/engines/postgres.py
+++ b/synapse/storage/engines/postgres.py
@@ -53,7 +53,7 @@ def check_database(self, db_conn, allow_outdated_version: bool = False):
             if rows and rows[0][0] != "UTF8":
                 raise IncorrectDatabaseSetup(
                     "Database has incorrect encoding: '%s' instead of 'UTF8'\n"
-                    "See docs/postgres.rst for more information." % (rows[0][0],)
+                    "See docs/postgres.md for more information." % (rows[0][0],)
                 )
 
             txn.execute(
@@ -62,12 +62,16 @@ def check_database(self, db_conn, allow_outdated_version: bool = False):
             collation, ctype = txn.fetchone()
             if collation != "C":
                 logger.warning(
-                    "Database has incorrect collation of %r. Should be 'C'", collation
+                    "Database has incorrect collation of %r. Should be 'C'\n"
+                    "See docs/postgres.md for more information.",
+                    collation,
                 )
 
             if ctype != "C":
                 logger.warning(
-                    "Database has incorrect ctype of %r. Should be 'C'", ctype
+                    "Database has incorrect ctype of %r. Should be 'C'\n"
+                    "See docs/postgres.md for more information.",
+                    ctype,
                 )
 
     def check_new_database(self, txn):


Test patch:
Repository: matrix-org/synapse

Issue:
media repo serves utf-8 files with no charset info, leading to incorrect in-browser rendering
### Description

Certain text files do not render correctly in-browser.

Example being a character sheet I made:
[How It Should Render](https://0x0.st/iTqU.txt) vs [How It Actually Renders](https://matrix-client.matrix.org/_matrix/media/r0/download/matrix.org/pAMkdBaolnxNCwQTuBaDJYrF)

This is not a browser error; it happens in both Chrome and in Firefox.

### Steps to reproduce

- Upload a text file with certain UTF8 characters
- Click on that file
- Watch it render incorrectly

### Version information

This issue has been identified on matrix.org; the above link on how it actually renders is a link to the matrix.org homeserver.


Discussion:

Resolving patch:
diff --git a/synapse/rest/media/v1/_base.py b/synapse/rest/media/v1/_base.py
--- a/synapse/rest/media/v1/_base.py
+++ b/synapse/rest/media/v1/_base.py
@@ -30,6 +30,22 @@
 
 logger = logging.getLogger(__name__)
 
+# list all text content types that will have the charset default to UTF-8 when
+# none is given
+TEXT_CONTENT_TYPES = [
+    "text/css",
+    "text/csv",
+    "text/html",
+    "text/calendar",
+    "text/plain",
+    "text/javascript",
+    "application/json",
+    "application/ld+json",
+    "application/rtf",
+    "image/svg+xml",
+    "text/xml",
+]
+
 
 def parse_media_id(request):
     try:
@@ -96,7 +112,14 @@ def add_file_headers(request, media_type, file_size, upload_name):
     def _quote(x):
         return urllib.parse.quote(x.encode("utf-8"))
 
-    request.setHeader(b"Content-Type", media_type.encode("UTF-8"))
+    # Default to a UTF-8 charset for text content types.
+    # ex, uses UTF-8 for 'text/css' but not 'text/css; charset=UTF-16'
+    if media_type.lower() in TEXT_CONTENT_TYPES:
+        content_type = media_type + "; charset=UTF-8"
+    else:
+        content_type = media_type
+
+    request.setHeader(b"Content-Type", content_type.encode("UTF-8"))
     if upload_name:
         # RFC6266 section 4.1 [1] defines both `filename` and `filename*`.
         #


Test patch:
Repository: matrix-org/synapse

Issue:
AttributeError: 'GenericWorkerSlavedStore' object has no attribute 'get_time_of_last_push_action_before'
Happens when sending out new email notifications. Given it happened on `matrix-org-hotfixes` between Feb 19th and Feb 27th, I assume this was caused by https://github.com/matrix-org/synapse/pull/6964.


Discussion:

Resolving patch:
diff --git a/synapse/push/mailer.py b/synapse/push/mailer.py
--- a/synapse/push/mailer.py
+++ b/synapse/push/mailer.py
@@ -555,10 +555,12 @@ def make_summary_text(
             else:
                 # If the reason room doesn't have a name, say who the messages
                 # are from explicitly to avoid, "messages in the Bob room"
+                room_id = reason["room_id"]
+
                 sender_ids = list(
                     {
                         notif_events[n["event_id"]].sender
-                        for n in notifs_by_room[reason["room_id"]]
+                        for n in notifs_by_room[room_id]
                     }
                 )
 
diff --git a/synapse/storage/data_stores/main/event_push_actions.py b/synapse/storage/data_stores/main/event_push_actions.py
--- a/synapse/storage/data_stores/main/event_push_actions.py
+++ b/synapse/storage/data_stores/main/event_push_actions.py
@@ -608,6 +608,23 @@ def _find_first_stream_ordering_after_ts_txn(txn, ts):
 
         return range_end
 
+    @defer.inlineCallbacks
+    def get_time_of_last_push_action_before(self, stream_ordering):
+        def f(txn):
+            sql = (
+                "SELECT e.received_ts"
+                " FROM event_push_actions AS ep"
+                " JOIN events e ON ep.room_id = e.room_id AND ep.event_id = e.event_id"
+                " WHERE ep.stream_ordering > ?"
+                " ORDER BY ep.stream_ordering ASC"
+                " LIMIT 1"
+            )
+            txn.execute(sql, (stream_ordering,))
+            return txn.fetchone()
+
+        result = yield self.db.runInteraction("get_time_of_last_push_action_before", f)
+        return result[0] if result else None
+
 
 class EventPushActionsStore(EventPushActionsWorkerStore):
     EPA_HIGHLIGHT_INDEX = "epa_highlight_index"
@@ -735,23 +752,6 @@ def f(txn):
             pa["actions"] = _deserialize_action(pa["actions"], pa["highlight"])
         return push_actions
 
-    @defer.inlineCallbacks
-    def get_time_of_last_push_action_before(self, stream_ordering):
-        def f(txn):
-            sql = (
-                "SELECT e.received_ts"
-                " FROM event_push_actions AS ep"
-                " JOIN events e ON ep.room_id = e.room_id AND ep.event_id = e.event_id"
-                " WHERE ep.stream_ordering > ?"
-                " ORDER BY ep.stream_ordering ASC"
-                " LIMIT 1"
-            )
-            txn.execute(sql, (stream_ordering,))
-            return txn.fetchone()
-
-        result = yield self.db.runInteraction("get_time_of_last_push_action_before", f)
-        return result[0] if result else None
-
     @defer.inlineCallbacks
     def get_latest_push_action_stream_ordering(self):
         def f(txn):


Test patch: