internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: matrix-org/synapse

Issue:
Admin API to query the topic and avatar of a given room
It would be useful for a server admin to be able to query the topic and avatar of rooms (without joining). 



Discussion:
It seems like this was already added in #7317: https://github.com/matrix-org/synapse/blob/master/docs/admin_api/rooms.md#draft-room-details-api
I am probably being blind, but I can't see the room topic (not room name) or avatar in the list of fields returned in the JSON response body? Please do say if I'm just not seeing it.

e.g,  from the docs:

```
{
  "room_id": "!mscvqgqpHYjBGDxNym:matrix.org",
  "name": "Music Theory",
  "canonical_alias": "#musictheory:matrix.org",
  "joined_members": 127
  "joined_local_members": 2,
  "version": "1",
  "creator": "@foo:matrix.org",
  "encryption": null,
  "federatable": true,
  "public": true,
  "join_rules": "invite",
  "guest_access": null,
  "history_visibility": "shared",
  "state_events": 93534
}
```
Oops, sorry, I didn't read the issue carefully enough. It should be fairly trivial to add to that room details endpoint though. The implementation seems to use the `room_stats_state` table, which does contain `avatar` and `topic` fields

Resolving patch:
diff --git a/synapse/storage/databases/main/room.py b/synapse/storage/databases/main/room.py
--- a/synapse/storage/databases/main/room.py
+++ b/synapse/storage/databases/main/room.py
@@ -104,7 +104,8 @@ def get_room_with_stats_txn(txn, room_id):
                   curr.local_users_in_room AS joined_local_members, rooms.room_version AS version,
                   rooms.creator, state.encryption, state.is_federatable AS federatable,
                   rooms.is_public AS public, state.join_rules, state.guest_access,
-                  state.history_visibility, curr.current_state_events AS state_events
+                  state.history_visibility, curr.current_state_events AS state_events,
+                  state.avatar, state.topic
                 FROM rooms
                 LEFT JOIN room_stats_state state USING (room_id)
                 LEFT JOIN room_stats_current curr USING (room_id)


Test patch:
diff --git a/tests/rest/admin/test_room.py b/tests/rest/admin/test_room.py
--- a/tests/rest/admin/test_room.py
+++ b/tests/rest/admin/test_room.py
@@ -1174,6 +1174,8 @@ def test_single_room(self):
 
         self.assertIn("room_id", channel.json_body)
         self.assertIn("name", channel.json_body)
+        self.assertIn("topic", channel.json_body)
+        self.assertIn("avatar", channel.json_body)
         self.assertIn("canonical_alias", channel.json_body)
         self.assertIn("joined_members", channel.json_body)
         self.assertIn("joined_local_members", channel.json_body)
Repository: matrix-org/synapse

Issue:
Document versions used for linting
We've pinned `mypy` and `black` in CI, so we should document that in https://github.com/matrix-org/synapse/blob/develop/CONTRIBUTING.md#code-style.

We could also add a `pip install .[lint]` that installed the right packages to make things easier.


Discussion:
We should update tox to use this afterward. 👍 

Resolving patch:
diff --git a/synapse/python_dependencies.py b/synapse/python_dependencies.py
--- a/synapse/python_dependencies.py
+++ b/synapse/python_dependencies.py
@@ -104,13 +104,16 @@
     # hiredis is not a *strict* dependency, but it makes things much faster.
     # (if it is not installed, we fall back to slow code.)
     "redis": ["txredisapi>=1.4.7", "hiredis"],
+    # We pin black so that our tests don't start failing on new releases.
+    "lint": ["isort==5.0.3", "black==19.10b0", "flake8-comprehensions", "flake8"],
 }
 
 ALL_OPTIONAL_REQUIREMENTS = set()  # type: Set[str]
 
 for name, optional_deps in CONDITIONAL_REQUIREMENTS.items():
     # Exclude systemd as it's a system-based requirement.
-    if name not in ["systemd"]:
+    # Exclude lint as it's a dev-based requirement.
+    if name not in ["systemd", "lint"]:
         ALL_OPTIONAL_REQUIREMENTS = set(optional_deps) | ALL_OPTIONAL_REQUIREMENTS
 
 


Test patch:
Repository: matrix-org/synapse

Issue:
Appservices users that sync will be counted towards MAU limits
Which means that you might unhelpfully find you at your limit if you enable a certain encrypted bridges project. I don't believe we want to count appservice users towards the block total, so hopefully this can be filtered out.


Discussion:

Resolving patch:
diff --git a/synapse/storage/databases/main/monthly_active_users.py b/synapse/storage/databases/main/monthly_active_users.py
--- a/synapse/storage/databases/main/monthly_active_users.py
+++ b/synapse/storage/databases/main/monthly_active_users.py
@@ -41,7 +41,14 @@ async def get_monthly_active_count(self) -> int:
         """
 
         def _count_users(txn):
-            sql = "SELECT COALESCE(count(*), 0) FROM monthly_active_users"
+            # Exclude app service users
+            sql = """
+                SELECT COALESCE(count(*), 0)
+                FROM monthly_active_users
+                    LEFT JOIN users
+                    ON monthly_active_users.user_id=users.name
+                WHERE (users.appservice_id IS NULL OR users.appservice_id = '');
+            """
             txn.execute(sql)
             (count,) = txn.fetchone()
             return count


Test patch:
diff --git a/tests/storage/test_monthly_active_users.py b/tests/storage/test_monthly_active_users.py
--- a/tests/storage/test_monthly_active_users.py
+++ b/tests/storage/test_monthly_active_users.py
@@ -137,6 +137,21 @@ def test_can_insert_and_count_mau(self):
         count = self.get_success(self.store.get_monthly_active_count())
         self.assertEqual(count, 1)
 
+    def test_appservice_user_not_counted_in_mau(self):
+        self.get_success(
+            self.store.register_user(
+                user_id="@appservice_user:server", appservice_id="wibble"
+            )
+        )
+        count = self.get_success(self.store.get_monthly_active_count())
+        self.assertEqual(count, 0)
+
+        d = self.store.upsert_monthly_active_user("@appservice_user:server")
+        self.get_success(d)
+
+        count = self.get_success(self.store.get_monthly_active_count())
+        self.assertEqual(count, 0)
+
     def test_user_last_seen_monthly_active(self):
         user_id1 = "@user1:server"
         user_id2 = "@user2:server"
@@ -383,7 +398,7 @@ def test_get_monthly_active_count_by_service(self):
         self.get_success(self.store.upsert_monthly_active_user(appservice2_user1))
 
         count = self.get_success(self.store.get_monthly_active_count())
-        self.assertEqual(count, 4)
+        self.assertEqual(count, 1)
 
         d = self.store.get_monthly_active_count_by_service()
         result = self.get_success(d)
Repository: matrix-org/synapse

Issue:
Add synapse log level to phonehome stats
We'd like to see if our recent improvements to logging result in a reduction in the number of people turning off INFO-level logging on synapse.

To track this at least at a coarse level, we'd like to add the log level for the `synapse` logger to the phonehome stats.


Discussion:

Resolving patch:
diff --git a/synapse/app/phone_stats_home.py b/synapse/app/phone_stats_home.py
--- a/synapse/app/phone_stats_home.py
+++ b/synapse/app/phone_stats_home.py
@@ -113,6 +113,13 @@ async def phone_stats_home(hs, stats, stats_process=_stats_process):
     stats["database_engine"] = hs.get_datastore().db_pool.engine.module.__name__
     stats["database_server_version"] = hs.get_datastore().db_pool.engine.server_version
 
+    #
+    # Logging configuration
+    #
+    synapse_logger = logging.getLogger("synapse")
+    log_level = synapse_logger.getEffectiveLevel()
+    stats["log_level"] = logging.getLevelName(log_level)
+
     logger.info("Reporting stats to %s: %s" % (hs.config.report_stats_endpoint, stats))
     try:
         await hs.get_proxied_http_client().put_json(


Test patch:
Repository: matrix-org/synapse

Issue:
increase default file transfer limit to 50MB (SYN-730)
Submitted by @&#8203;matthew:matrix.org


(Imported from https://matrix.org/jira/browse/SYN-730)


Discussion:

Resolving patch:
diff --git a/synapse/config/repository.py b/synapse/config/repository.py
--- a/synapse/config/repository.py
+++ b/synapse/config/repository.py
@@ -100,7 +100,7 @@ def read_config(self, config, **kwargs):
             "media_instance_running_background_jobs",
         )
 
-        self.max_upload_size = self.parse_size(config.get("max_upload_size", "10M"))
+        self.max_upload_size = self.parse_size(config.get("max_upload_size", "50M"))
         self.max_image_pixels = self.parse_size(config.get("max_image_pixels", "32M"))
         self.max_spider_size = self.parse_size(config.get("max_spider_size", "10M"))
 
@@ -242,7 +242,7 @@ def generate_config_section(self, data_dir_path, **kwargs):
 
         # The largest allowed upload size in bytes
         #
-        #max_upload_size: 10M
+        #max_upload_size: 50M
 
         # Maximum number of pixels that will be thumbnailed
         #


Test patch:
Repository: matrix-org/synapse

Issue:
/displayname returns non-JSON body
### Description

An attempt to call `/profile/{}/displayname` with an empty body results (rightfully) in a 400 error, with plain text "Unable to parse name" in the response. The expected error payload is something about (could also be `M_NOT_JSON` instead of `M_BAD_JSON`):
```json
{
    "errcode": "M_BAD_JSON",
    "error": "Unable to parse name" 
}
```

### Version information

<!-- IMPORTANT: please answer the following questions, to help us narrow down the problem -->

<!-- Was this issue identified on matrix.org or another homeserver? -->
- **Homeserver**: matrix.org


Discussion:
@KitsuneRal What's the request you performed? (i.e. what's the HTTP method you used?)

Both `PUT` and `GET` on `https://matrix-client.matrix.org/_matrix/client/r0/profile/{}/displayname` give me

```
{
    "errcode": "M_INVALID_PARAM",
    "error":"Expected UserID string to start with '@'"
}
```
Oh looks like `{}` is a placeholder for the MXID here.
In which case I can reproduce.
Ok, I wasn't clear: `{}` in the endpoint path is just a shortcut for `{userId}`. The problem ~~(which, embarrasingly, I can't reproduce either)~~ is that with ~~_some_~~  empty JSON payload _in the request body_ I end up with the plain-text body in the 400 response.

Resolving patch:
diff --git a/synapse/rest/client/v1/profile.py b/synapse/rest/client/v1/profile.py
--- a/synapse/rest/client/v1/profile.py
+++ b/synapse/rest/client/v1/profile.py
@@ -59,7 +59,9 @@ async def on_PUT(self, request, user_id):
         try:
             new_name = content["displayname"]
         except Exception:
-            return 400, "Unable to parse name"
+            raise SynapseError(
+                code=400, msg="Unable to parse name", errcode=Codes.BAD_JSON,
+            )
 
         await self.profile_handler.set_displayname(user, requester, new_name, is_admin)
 


Test patch:
Repository: matrix-org/synapse

Issue:
Email notifications for encrypted messages are blank
See original issue: https://github.com/vector-im/element-web/issues/15307


Discussion:

Resolving patch:
diff --git a/synapse/push/mailer.py b/synapse/push/mailer.py
--- a/synapse/push/mailer.py
+++ b/synapse/push/mailer.py
@@ -387,8 +387,8 @@ async def get_notif_vars(self, notif, user_id, notif_event, room_state_ids):
         return ret
 
     async def get_message_vars(self, notif, event, room_state_ids):
-        if event.type != EventTypes.Message:
-            return
+        if event.type != EventTypes.Message and event.type != EventTypes.Encrypted:
+            return None
 
         sender_state_event_id = room_state_ids[("m.room.member", event.sender)]
         sender_state_event = await self.store.get_event(sender_state_event_id)
@@ -399,10 +399,8 @@ async def get_message_vars(self, notif, event, room_state_ids):
         # sender_hash % the number of default images to choose from
         sender_hash = string_ordinal_total(event.sender)
 
-        msgtype = event.content.get("msgtype")
-
         ret = {
-            "msgtype": msgtype,
+            "event_type": event.type,
             "is_historical": event.event_id != notif["event_id"],
             "id": event.event_id,
             "ts": event.origin_server_ts,
@@ -411,6 +409,14 @@ async def get_message_vars(self, notif, event, room_state_ids):
             "sender_hash": sender_hash,
         }
 
+        # Encrypted messages don't have any additional useful information.
+        if event.type == EventTypes.Encrypted:
+            return ret
+
+        msgtype = event.content.get("msgtype")
+
+        ret["msgtype"] = msgtype
+
         if msgtype == "m.text":
             self.add_text_message_vars(ret, event)
         elif msgtype == "m.image":


Test patch:
diff --git a/tests/push/test_email.py b/tests/push/test_email.py
--- a/tests/push/test_email.py
+++ b/tests/push/test_email.py
@@ -158,8 +158,21 @@ def test_multiple_members_email(self):
         # We should get emailed about those messages
         self._check_for_mail()
 
+    def test_encrypted_message(self):
+        room = self.helper.create_room_as(self.user_id, tok=self.access_token)
+        self.helper.invite(
+            room=room, src=self.user_id, tok=self.access_token, targ=self.others[0].id
+        )
+        self.helper.join(room=room, user=self.others[0].id, tok=self.others[0].token)
+
+        # The other user sends some messages
+        self.helper.send_event(room, "m.room.encrypted", {}, tok=self.others[0].token)
+
+        # We should get emailed about that message
+        self._check_for_mail()
+
     def _check_for_mail(self):
-        "Check that the user receives an email notification"
+        """Check that the user receives an email notification"""
 
         # Get the stream ordering before it gets sent
         pushers = self.get_success(
Repository: matrix-org/synapse

Issue:
Increase in errors due to no active logging span
Starting on Oct 1 (which corresponds with a re-deploy of matrix.org) we've been seeing a lot more of the following error:

> There was no active span when trying to inject the span into a byte dict. Did you forget to start one or did a context slip?

https://sentry.matrix.org/sentry/synapse-matrixorg/issues/15795/


Discussion:
I believe that this is a regression in the range of [bbb7ca1f157f43250a2373b194c7dee3dfbfca80 to 7b6f857aa996ec4dde368e11115bf1684770d78f](https://github.com/matrix-org/synapse/compare/bbb7ca1f157f43250a2373b194c7dee3dfbfca80...7b6f857aa996ec4dde368e11115bf1684770d78f).
Based on the message, it seems this is from `inject_active_span_byte_dict`, which is used in only two spots:

https://github.com/matrix-org/synapse/blob/1c262431f9bf768d106bf79a568479fa5a0784a1/synapse/http/matrixfederationclient.py#L408

https://github.com/matrix-org/synapse/blob/866c84da8ddda04f1da94b8e03719147c6d4875e/synapse/replication/http/_base.py#L205

I've seen this on workers not handling federation, which makes me guess this is from replication?

Resolving patch:
diff --git a/synapse/metrics/background_process_metrics.py b/synapse/metrics/background_process_metrics.py
--- a/synapse/metrics/background_process_metrics.py
+++ b/synapse/metrics/background_process_metrics.py
@@ -24,6 +24,7 @@
 from twisted.internet import defer
 
 from synapse.logging.context import LoggingContext, PreserveLoggingContext
+from synapse.logging.opentracing import start_active_span
 
 if TYPE_CHECKING:
     import resource
@@ -197,14 +198,14 @@ async def run():
 
         with BackgroundProcessLoggingContext(desc) as context:
             context.request = "%s-%i" % (desc, count)
-
             try:
-                result = func(*args, **kwargs)
+                with start_active_span(desc, tags={"request_id": context.request}):
+                    result = func(*args, **kwargs)
 
-                if inspect.isawaitable(result):
-                    result = await result
+                    if inspect.isawaitable(result):
+                        result = await result
 
-                return result
+                    return result
             except Exception:
                 logger.exception(
                     "Background process '%s' threw an exception", desc,


Test patch:
Repository: matrix-org/synapse

Issue:
mypy==0.782 will cause mypy-zope dependency regression in future pip releases
## Description

Pip is planning to use a [new resolver by default from `20.3`](https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-resolver-in-20-2-2020), which is more strict in it's dependency handling.

Currently, `tox`'s `mypy` test environment defines this:

```
deps =
    {[base]deps}
    mypy==0.782
    mypy-zope
```

Given that input, `pip>=20.3` will likely either:
1. Choose an older `mypy-zope` package that works with `mypy==0.782`, causing a regression.
2. ~~Reject installing outright, as this stricter dependency resolver will refuse to install incompatible packages.~~



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -102,6 +102,8 @@ def exec_file(path_segments):
     "flake8",
 ]
 
+CONDITIONAL_REQUIREMENTS["mypy"] = ["mypy==0.790", "mypy-zope"]
+
 # Dependencies which are exclusively required by unit test code. This is
 # NOT a list of all modules that are necessary to run the unit tests.
 # Tests assume that all optional dependencies are installed.


Test patch:
Repository: matrix-org/synapse

Issue:
duplicated rows in user_daily_visits
I *think* it's supposed to only contain one row per (user_id, device_id, timestamp), but in practice each different user_agent seems to get its own row.



Discussion:
seems to have been introduced in https://github.com/matrix-org/synapse/pull/8503/files#diff-dd317c99e1c729e92c1e18959071bcc73b19fc5051db0a4cfc0e0c70bbdc7bf5R291-R302.

(might be worth adding a unique constraint to enforce this)

Resolving patch:
diff --git a/synapse/storage/databases/main/metrics.py b/synapse/storage/databases/main/metrics.py
--- a/synapse/storage/databases/main/metrics.py
+++ b/synapse/storage/databases/main/metrics.py
@@ -282,9 +282,10 @@ def _generate_user_daily_visits(txn):
             now = self._clock.time_msec()
 
             # A note on user_agent. Technically a given device can have multiple
-            # user agents, so we need to decide which one to pick. We could have handled this
-            # in number of ways, but given that we don't _that_ much have gone for MAX()
-            # For more details of the other options considered see
+            # user agents, so we need to decide which one to pick. We could have
+            # handled this in number of ways, but given that we don't care
+            # _that_ much we have gone for MAX(). For more details of the other
+            # options considered see
             # https://github.com/matrix-org/synapse/pull/8503#discussion_r502306111
             sql = """
                 INSERT INTO user_daily_visits (user_id, device_id, timestamp, user_agent)
@@ -299,7 +300,7 @@ def _generate_user_daily_visits(txn):
                     WHERE last_seen > ? AND last_seen <= ?
                     AND udv.timestamp IS NULL AND users.is_guest=0
                     AND users.appservice_id IS NULL
-                    GROUP BY u.user_id, u.device_id, u.user_agent
+                    GROUP BY u.user_id, u.device_id
             """
 
             # This means that the day has rolled over but there could still


Test patch:
Repository: matrix-org/synapse

Issue:
Events for newly created rooms are not always sent to appservices
Reproduction flow:
- Have an appservice that autojoins rooms when you invite it.
- Invite the appservice bot to a DM.
- Send a message into the DM
- Notice that the appservice doesn't receive a transaction for the event.

Expected behaviour:
- Appservices should get any and all events for rooms of which they are members of.

An example of this not working can be found by looking for the event `$gyPH0FEqlReAQwWGdpPLz-UdpQ5T2YjwRy4Cpa8ZaHQ` in `!BEPhTttjYKNQfBgWvS:matrix.org`.

Additionally, sending new messages into the room after some time will allow those to be bridged so this smells strongly of stale cache.


Discussion:

Resolving patch:
diff --git a/synapse/appservice/__init__.py b/synapse/appservice/__init__.py
--- a/synapse/appservice/__init__.py
+++ b/synapse/appservice/__init__.py
@@ -19,7 +19,7 @@
 from synapse.api.constants import EventTypes
 from synapse.events import EventBase
 from synapse.types import GroupID, JsonDict, UserID, get_domain_from_id
-from synapse.util.caches.descriptors import cached
+from synapse.util.caches.descriptors import _CacheContext, cached
 
 if TYPE_CHECKING:
     from synapse.appservice.api import ApplicationServiceApi
@@ -164,9 +164,9 @@ async def _matches_user(
         does_match = await self.matches_user_in_member_list(event.room_id, store)
         return does_match
 
-    @cached(num_args=1)
+    @cached(num_args=1, cache_context=True)
     async def matches_user_in_member_list(
-        self, room_id: str, store: "DataStore"
+        self, room_id: str, store: "DataStore", cache_context: _CacheContext,
     ) -> bool:
         """Check if this service is interested a room based upon it's membership
 
@@ -177,7 +177,9 @@ async def matches_user_in_member_list(
         Returns:
             True if this service would like to know about this room.
         """
-        member_list = await store.get_users_in_room(room_id)
+        member_list = await store.get_users_in_room(
+            room_id, on_invalidate=cache_context.invalidate
+        )
 
         # check joined member events
         for user_id in member_list:


Test patch:
Repository: matrix-org/synapse

Issue:
/_synapse/admin/v1/send_server_notice may send a notice to a user via a public room
Docs: https://github.com/matrix-org/synapse/blob/b2b86990705de8a099093ec141ad83e09f182034/docs/admin_api/server_notices.md

The server notice API can be used to send a user a messages from the @ server:example.com user, by specifying a `user_id` as part of the parameters. Synapse will find the first room it shares with the user and post the message. This is fine if the server notices user was only a member of DM rooms, but unfortunately this can not be guaranteed as it may be used to autocreate the `auto_join` rooms.

We should instead look for 1:1 rooms where the target and server notices user are the only participants.

Offending code: https://github.com/matrix-org/synapse/blob/c619253db80c8d1c606dc40756dd3c9e3a55a9fb/synapse/server_notices/server_notices_manager.py#L112-L130




Discussion:

Resolving patch:
diff --git a/synapse/server_notices/server_notices_manager.py b/synapse/server_notices/server_notices_manager.py
--- a/synapse/server_notices/server_notices_manager.py
+++ b/synapse/server_notices/server_notices_manager.py
@@ -119,7 +119,7 @@ async def get_or_create_notice_room_for_user(self, user_id: str) -> str:
             # manages to invite the system user to a room, that doesn't make it
             # the server notices room.
             user_ids = await self._store.get_users_in_room(room.room_id)
-            if self.server_notices_mxid in user_ids:
+            if len(user_ids) <= 2 and self.server_notices_mxid in user_ids:
                 # we found a room which our user shares with the system notice
                 # user
                 logger.info(


Test patch: