internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: matrix-org/synapse

Issue:
sometimes the key notary endpoint doesn't give back the keys you asked for
Example: https://matrix.org/_matrix/key/v2/query/sw1v.org/ed25519:a_BPNW: it returns `sw1v.org/ed25519:auto`, and not the key I want.


Discussion:
Turns out that the spec says that the server should ignore the keyId param for the GET endpoint (https://matrix.org/docs/spec/server_server/r0.1.1.html#get-matrix-key-v2-query-servername-keyid). However: (a) that's braindead, and (b) the same problem happens for the POST endpoint.

Resolving patch:
diff --git a/synapse/crypto/keyring.py b/synapse/crypto/keyring.py
--- a/synapse/crypto/keyring.py
+++ b/synapse/crypto/keyring.py
@@ -394,7 +394,7 @@ def __init__(self, hs):
 
     @defer.inlineCallbacks
     def process_v2_response(
-        self, from_server, response_json, time_added_ms, requested_ids=[]
+        self, from_server, response_json, time_added_ms
     ):
         """Parse a 'Server Keys' structure from the result of a /key request
 
@@ -417,10 +417,6 @@ def process_v2_response(
 
             time_added_ms (int): the timestamp to record in server_keys_json
 
-            requested_ids (iterable[str]): a list of the key IDs that were requested.
-                We will store the json for these key ids as well as any that are
-                actually in the response
-
         Returns:
             Deferred[dict[str, FetchKeyResult]]: map from key_id to result object
         """
@@ -476,11 +472,6 @@ def process_v2_response(
 
         signed_key_json_bytes = encode_canonical_json(signed_key_json)
 
-        # for reasons I don't quite understand, we store this json for the key ids we
-        # requested, as well as those we got.
-        updated_key_ids = set(requested_ids)
-        updated_key_ids.update(verify_keys)
-
         yield logcontext.make_deferred_yieldable(
             defer.gatherResults(
                 [
@@ -493,7 +484,7 @@ def process_v2_response(
                         ts_expires_ms=ts_valid_until_ms,
                         key_json_bytes=signed_key_json_bytes,
                     )
-                    for key_id in updated_key_ids
+                    for key_id in verify_keys
                 ],
                 consumeErrors=True,
             ).addErrback(unwrapFirstError)
@@ -749,7 +740,6 @@ def get_server_verify_key_v2_direct(self, server_name, key_ids):
 
             response_keys = yield self.process_v2_response(
                 from_server=server_name,
-                requested_ids=[requested_key_id],
                 response_json=response,
                 time_added_ms=time_now_ms,
             )


Test patch:
Repository: matrix-org/synapse

Issue:
voiprestservlet is broken
suspect this was during the kill-off-v2 refactor:
```
2019-06-03 17:22:23,223 - synapse.http.server - 112 - ERROR - GET-73899- Failed handle request via 'VoipRestServlet': <XForwardedForRequest at 0x7f2d4e05dfd0 method='GET' uri='/_matrix/client/r0/voip/turnServer' clientproto='HTTP/1.1' site=8008>
Traceback (most recent call last):
  File /opt/synapse/synapse/synapse/http/server.py, line 81, in wrapped_request_handler
    yield h(self, request)
  File /opt/synapse/synapse/synapse/http/server.py, line 316, in _async_render
    callback_return = yield callback(request, **kwargs)
  File /opt/synapse/env3/lib/python3.5/site-packages/twisted/internet/defer.py, line 1418, in _inlineCallbacks
    result = g.send(result)
  File /opt/synapse/synapse/synapse/rest/client/v1/voip.py, line 35, in on_GET
    requester = yield self.auth.get_user_by_req(
AttributeError: 'VoipRestServlet' object has no attribute 'auth'
```



Discussion:
think this is a regression, so marking as a release-blocker


Resolving patch:
diff --git a/synapse/rest/client/v1/voip.py b/synapse/rest/client/v1/voip.py
--- a/synapse/rest/client/v1/voip.py
+++ b/synapse/rest/client/v1/voip.py
@@ -29,6 +29,7 @@ class VoipRestServlet(RestServlet):
     def __init__(self, hs):
         super(VoipRestServlet, self).__init__()
         self.hs = hs
+        self.auth = hs.get_auth()
 
     @defer.inlineCallbacks
     def on_GET(self, request):


Test patch:
Repository: matrix-org/synapse

Issue:
Implement rooms v5
Synapse v1.0 will default to rooms v4, however the intention is that it also ships with rooms v5 which is envisaged to be rooms v4 + https://github.com/matrix-org/synapse/issues/4364

rooms v5 is yet to be MSC'd so does not formally exist at this point, this issue is a place holder.


Discussion:
spec-side tracking issue is https://github.com/matrix-org/matrix-doc/issues/2021 although would recommend tracking progress against either this issue or https://github.com/matrix-org/synapse/issues/4364

Resolving patch:
diff --git a/synapse/api/room_versions.py b/synapse/api/room_versions.py
--- a/synapse/api/room_versions.py
+++ b/synapse/api/room_versions.py
@@ -82,12 +82,12 @@ class RoomVersions(object):
         StateResolutionVersions.V2,
         enforce_key_validity=False,
     )
-    VDH_TEST_KEY_VALIDITY = RoomVersion(
-        "vdh-test-key-validity",
-        RoomDisposition.UNSTABLE,
+    V5 = RoomVersion(
+        "5",
+        RoomDisposition.STABLE,
         EventFormatVersions.V3,
         StateResolutionVersions.V2,
-        enforce_key_validity=False,
+        enforce_key_validity=True,
     )
 
 
@@ -97,6 +97,6 @@ class RoomVersions(object):
         RoomVersions.V2,
         RoomVersions.V3,
         RoomVersions.V4,
-        RoomVersions.VDH_TEST_KEY_VALIDITY,
+        RoomVersions.V5,
     )
 }   # type: dict[str, RoomVersion]


Test patch:
Repository: matrix-org/synapse

Issue:
Sync tightloop due to presence
Occasionally my `/sync` will tightloop, i.e. it will return immediately with a presence update, but the since token matches the next token, and so it makes no progress.

This seems to be due to:

1. Presence handler doesn't early return if from token == current token
2. Current token is based off the ID gen, which may not return the latest ID persisted (as ID gens don't return tokens unless all previous IDs have successfully been persisted)
3. The stream change cache will return all results after the given token, including rows with a token greater than the current token.

The simplest solution here is probably to just early return if `from token == current token`. This is safe as once the persistence of pending presence updates have completed current token will increment.


Discussion:

Resolving patch:
diff --git a/synapse/handlers/presence.py b/synapse/handlers/presence.py
--- a/synapse/handlers/presence.py
+++ b/synapse/handlers/presence.py
@@ -1017,11 +1017,28 @@ def get_new_events(
             if from_key is not None:
                 from_key = int(from_key)
 
+            max_token = self.store.get_current_presence_token()
+            if from_key == max_token:
+                # This is necessary as due to the way stream ID generators work
+                # we may get updates that have a stream ID greater than the max
+                # token (e.g. max_token is N but stream generator may return
+                # results for N+2, due to N+1 not having finished being
+                # persisted yet).
+                #
+                # This is usually fine, as it just means that we may send down
+                # some presence updates multiple times. However, we need to be
+                # careful that the sync stream either actually does make some
+                # progress or doesn't return, otherwise clients will end up
+                # tight looping calling /sync due to it immediately returning
+                # the same token repeatedly.
+                #
+                # Hence this guard where we just return nothing so that the sync
+                # doesn't return. C.f. #5503.
+                defer.returnValue(([], max_token))
+
             presence = self.get_presence_handler()
             stream_change_cache = self.store.presence_stream_cache
 
-            max_token = self.store.get_current_presence_token()
-
             users_interested_in = yield self._get_interested_in(user, explicit_room_id)
 
             user_ids_changed = set()


Test patch:
Repository: matrix-org/synapse

Issue:
scripts-dev/check_event_hash.py doesn't work anymore
It complains about missing a format_version on the event. See https://github.com/matrix-org/synapse/pull/5281 for a not-too-great fix.


Discussion:
Honestly, I'd be more inclined to get rid of it than fix it. I don't really know what it's supposed to do (it doesn't have a single comment), and it's clearly been forgotten. Is it actually fulfilling a useful purpose?
It was useful to check a hash, but admittedly I've only ever used it twice. Being a dev script, I'd support removing it as it's a clear dust collector

Resolving patch:
diff --git a/scripts-dev/check_event_hash.py b/scripts-dev/check_event_hash.py
deleted file mode 100644
--- a/scripts-dev/check_event_hash.py
+++ /dev/null
@@ -1,54 +0,0 @@
-import argparse
-import hashlib
-import json
-import logging
-import sys
-
-from unpaddedbase64 import encode_base64
-
-from synapse.crypto.event_signing import (
-    check_event_content_hash,
-    compute_event_reference_hash,
-)
-
-
-class dictobj(dict):
-    def __init__(self, *args, **kargs):
-        dict.__init__(self, *args, **kargs)
-        self.__dict__ = self
-
-    def get_dict(self):
-        return dict(self)
-
-    def get_full_dict(self):
-        return dict(self)
-
-    def get_pdu_json(self):
-        return dict(self)
-
-
-def main():
-    parser = argparse.ArgumentParser()
-    parser.add_argument(
-        "input_json", nargs="?", type=argparse.FileType("r"), default=sys.stdin
-    )
-    args = parser.parse_args()
-    logging.basicConfig()
-
-    event_json = dictobj(json.load(args.input_json))
-
-    algorithms = {"sha256": hashlib.sha256}
-
-    for alg_name in event_json.hashes:
-        if check_event_content_hash(event_json, algorithms[alg_name]):
-            print("PASS content hash %s" % (alg_name,))
-        else:
-            print("FAIL content hash %s" % (alg_name,))
-
-    for algorithm in algorithms.values():
-        name, h_bytes = compute_event_reference_hash(event_json, algorithm)
-        print("Reference hash %s: %s" % (name, encode_base64(h_bytes)))
-
-
-if __name__ == "__main__":
-    main()


Test patch:
Repository: matrix-org/synapse

Issue:
Consent broken on develop
From riot-web's end to end tests:
```
2019-06-30 18:27:03,267 - synapse.http.server - 177 - ERROR - GET-526 - Failed handle request <XForwardedForRequest at 0x7fbf0a8f6fd0 method='GET' uri='/_matrix/consent?h=f0691bd8ff7161edcd6f5349db69807dd600a49ddc5c5412b8201098dd8065e9&u=charly-4' clientproto='HTTP/1.1' site=5005>
Traceback (most recent call last):
  File "/workdir/matrix-react-end-to-end-tests/synapse/installations/consent/env/lib/python3.5/site-packages/synapse/http/server.py", line 142, in wrapped_request_handler
    return await h(self, request)
  File "/workdir/matrix-react-end-to-end-tests/synapse/installations/consent/env/lib/python3.5/site-packages/synapse/rest/consent/consent_resource.py", line 142, in _async_render_GET
    u = await self.store.get_user_by_id(qualified_user_id)
TypeError: object dict can't be used in 'await' expression
```

Log: https://buildkite.com/matrix-dot-org/matrix-react-sdk/builds/1360#0ec3ad58-40f7-40fa-abb9-36613530047f


Discussion:
Guessing this has been introduced by #5475

Resolving patch:
diff --git a/synapse/rest/consent/consent_resource.py b/synapse/rest/consent/consent_resource.py
--- a/synapse/rest/consent/consent_resource.py
+++ b/synapse/rest/consent/consent_resource.py
@@ -23,6 +23,8 @@
 import jinja2
 from jinja2 import TemplateNotFound
 
+from twisted.internet import defer
+
 from synapse.api.errors import NotFoundError, StoreError, SynapseError
 from synapse.config import ConfigError
 from synapse.http.server import (
@@ -139,7 +141,7 @@ async def _async_render_GET(self, request):
             else:
                 qualified_user_id = UserID(username, self.hs.hostname).to_string()
 
-            u = await self.store.get_user_by_id(qualified_user_id)
+            u = await defer.maybeDeferred(self.store.get_user_by_id, qualified_user_id)
             if u is None:
                 raise NotFoundError("Unknown user")
 


Test patch:
Repository: matrix-org/synapse

Issue:
Synapse queries the IS's /store-invite endpoint with the wrong content-type
Synapse currently sends requests to `/store-invite` using the `application/x-www-form-urlencoded` content-type header: https://github.com/matrix-org/synapse/blob/master/synapse/handlers/room_member.py#L908

The IS spec [says](https://matrix.org/docs/spec/identity_service/r0.2.1#api-standards):

> all API calls use a Content-Type of `application/json`

It currently works because Sydent supports both `application/x-www-form-urlencoded` and `application/json` but this doesn't sound like something we'd want to do.


Discussion:

Resolving patch:
diff --git a/synapse/handlers/room_member.py b/synapse/handlers/room_member.py
--- a/synapse/handlers/room_member.py
+++ b/synapse/handlers/room_member.py
@@ -29,7 +29,7 @@
 import synapse.server
 import synapse.types
 from synapse.api.constants import EventTypes, Membership
-from synapse.api.errors import AuthError, Codes, SynapseError
+from synapse.api.errors import AuthError, Codes, HttpResponseException, SynapseError
 from synapse.types import RoomID, UserID
 from synapse.util.async_helpers import Linearizer
 from synapse.util.distributor import user_joined_room, user_left_room
@@ -872,9 +872,23 @@ def _ask_id_server_for_third_party_invite(
             "sender_avatar_url": inviter_avatar_url,
         }
 
-        data = yield self.simple_http_client.post_urlencoded_get_json(
-            is_url, invite_config
-        )
+        try:
+            data = yield self.simple_http_client.post_json_get_json(
+                is_url, invite_config
+            )
+        except HttpResponseException as e:
+            # Some identity servers may only support application/x-www-form-urlencoded
+            # types. This is especially true with old instances of Sydent, see
+            # https://github.com/matrix-org/sydent/pull/170
+            logger.info(
+                "Failed to POST %s with JSON, falling back to urlencoded form: %s",
+                is_url,
+                e,
+            )
+            data = yield self.simple_http_client.post_urlencoded_get_json(
+                is_url, invite_config
+            )
+
         # TODO: Check for success
         token = data["token"]
         public_keys = data.get("public_keys", [])


Test patch:
Repository: matrix-org/synapse

Issue:
Cache version string in versionstring.py
### Description

https://github.com/matrix-org/synapse/blob/develop/synapse/federation/transport/server.py#L773

Synapse does not cache it's version string on startup, causing requests to that endpoint to cause 4 git processes to be spawned, even if the code is not deployed via git.

https://github.com/matrix-org/synapse/blob/develop/synapse/util/versionstring.py

While we're at it, we probably don't need to do that caching at each of the places get_version_string is called - perhaps that function saves the response forever (during the lifetime of the python process), this will save us checking git multiple times on startup - we just need to try it that 4 times and that's it.

We also tack on the "Synapse/" in lots of the places that we do this as well, maybe just do that in one place too.

### Version information

- **Version**: develop

- **Install method**: 
git clone

- **Platform**: 
linux


Discussion:
This is actually using a non-trivial amount of cpu time on the master. Given how easy it is to fix, we should do it asap

Resolving patch:
diff --git a/synapse/util/versionstring.py b/synapse/util/versionstring.py
--- a/synapse/util/versionstring.py
+++ b/synapse/util/versionstring.py
@@ -22,6 +22,23 @@
 
 
 def get_version_string(module):
+    """Given a module calculate a git-aware version string for it.
+
+    If called on a module not in a git checkout will return `__verison__`.
+
+    Args:
+        module (module)
+
+    Returns:
+        str
+    """
+
+    cached_version = getattr(module, "_synapse_version_string_cache", None)
+    if cached_version:
+        return cached_version
+
+    version_string = module.__version__
+
     try:
         null = open(os.devnull, "w")
         cwd = os.path.dirname(os.path.abspath(module.__file__))
@@ -80,8 +97,10 @@ def get_version_string(module):
                 s for s in (git_branch, git_tag, git_commit, git_dirty) if s
             )
 
-            return "%s (%s)" % (module.__version__, git_version)
+            version_string = "%s (%s)" % (module.__version__, git_version)
     except Exception as e:
         logger.info("Failed to check for git repository: %s", e)
 
-    return module.__version__
+    module._synapse_version_string_cache = version_string
+
+    return version_string


Test patch:
Repository: matrix-org/synapse

Issue:
Add a m.id_access_token flag to unstable_features
For https://github.com/matrix-org/synapse/issues/5786#issuecomment-523879040


Discussion:

Resolving patch:
diff --git a/synapse/rest/client/versions.py b/synapse/rest/client/versions.py
--- a/synapse/rest/client/versions.py
+++ b/synapse/rest/client/versions.py
@@ -44,7 +44,12 @@ def on_GET(self, request):
                     "r0.5.0",
                 ],
                 # as per MSC1497:
-                "unstable_features": {"m.lazy_load_members": True},
+                "unstable_features": {
+                    "m.lazy_load_members": True,
+                    # as per https://github.com/matrix-org/synapse/issues/5927
+                    # to be removed in r0.6.0
+                    "m.id_access_token": True,
+                },
             },
         )
 


Test patch:
Repository: matrix-org/synapse

Issue:
Add a m.id_access_token flag to unstable_features
For https://github.com/matrix-org/synapse/issues/5786#issuecomment-523879040


Discussion:

Resolving patch:
diff --git a/synapse/rest/client/versions.py b/synapse/rest/client/versions.py
--- a/synapse/rest/client/versions.py
+++ b/synapse/rest/client/versions.py
@@ -50,6 +50,9 @@ def on_GET(self, request):
                 # as per MSC1497:
                 "unstable_features": {
                     "m.lazy_load_members": True,
+                    # as per MSC2190, as amended by MSC2264
+                    # to be removed in r0.6.0
+                    "m.id_access_token": True,
                     # Advertise to clients that they need not include an `id_server`
                     # parameter during registration or password reset, as Synapse now decides
                     # itself which identity server to use (or none at all).


Test patch:
Repository: matrix-org/synapse

Issue:
Disable GET /_matrix/client/r0/register/available if registration is disabled
Otherwise we're leaking server usernames when we don't need to be.


Discussion:

Resolving patch:
diff --git a/synapse/rest/client/v2_alpha/register.py b/synapse/rest/client/v2_alpha/register.py
--- a/synapse/rest/client/v2_alpha/register.py
+++ b/synapse/rest/client/v2_alpha/register.py
@@ -328,6 +328,11 @@ def __init__(self, hs):
 
     @defer.inlineCallbacks
     def on_GET(self, request):
+        if not self.hs.config.enable_registration:
+            raise SynapseError(
+                403, "Registration has been disabled", errcode=Codes.FORBIDDEN
+            )
+
         ip = self.hs.get_ip_from_request(request)
         with self.ratelimiter.ratelimit(ip) as wait_deferred:
             yield wait_deferred


Test patch:
Repository: matrix-org/synapse

Issue:
Rejections storage



Discussion:

Resolving patch:
diff --git a/synapse/handlers/room.py b/synapse/handlers/room.py
--- a/synapse/handlers/room.py
+++ b/synapse/handlers/room.py
@@ -28,6 +28,7 @@
 from synapse.api.constants import EventTypes, JoinRules, RoomCreationPreset
 from synapse.api.errors import AuthError, Codes, NotFoundError, StoreError, SynapseError
 from synapse.api.room_versions import KNOWN_ROOM_VERSIONS
+from synapse.http.endpoint import parse_and_validate_server_name
 from synapse.storage.state import StateFilter
 from synapse.types import RoomAlias, RoomID, RoomStreamToken, StreamToken, UserID
 from synapse.util import stringutils
@@ -554,7 +555,8 @@ def create_room(self, requester, config, ratelimit=True, creator_join_profile=No
         invite_list = config.get("invite", [])
         for i in invite_list:
             try:
-                UserID.from_string(i)
+                uid = UserID.from_string(i)
+                parse_and_validate_server_name(uid.domain)
             except Exception:
                 raise SynapseError(400, "Invalid user_id: %s" % (i,))
 


Test patch:
diff --git a/tests/rest/client/v1/test_rooms.py b/tests/rest/client/v1/test_rooms.py
--- a/tests/rest/client/v1/test_rooms.py
+++ b/tests/rest/client/v1/test_rooms.py
@@ -484,6 +484,15 @@ def test_post_room_invalid_content(self):
         self.render(request)
         self.assertEquals(400, channel.code)
 
+    def test_post_room_invitees_invalid_mxid(self):
+        # POST with invalid invitee, see https://github.com/matrix-org/synapse/issues/4088
+        # Note the trailing space in the MXID here!
+        request, channel = self.make_request(
+            "POST", "/createRoom", b'{"invite":["@alice:example.com "]}'
+        )
+        self.render(request)
+        self.assertEquals(400, channel.code)
+
 
 class RoomTopicTestCase(RoomBase):
     """ Tests /rooms/$room_id/topic REST events. """
Repository: matrix-org/synapse

Issue:
Failed to store destination_retry_timings:  integer out of range
it is trying to store the retry interval in an integer field, and 2^31 ms is only 25 days, and it's supposed to go a lot higher than that


Discussion:
we need to fix this before release

Resolving patch:
diff --git a/synapse/util/retryutils.py b/synapse/util/retryutils.py
--- a/synapse/util/retryutils.py
+++ b/synapse/util/retryutils.py
@@ -29,7 +29,7 @@
 RETRY_MULTIPLIER = 5
 
 # a cap on the backoff. (Essentially none)
-MAX_RETRY_INTERVAL = 2 ** 63
+MAX_RETRY_INTERVAL = 2 ** 62
 
 
 class NotRetryingDestination(Exception):


Test patch:
diff --git a/tests/storage/test_transactions.py b/tests/storage/test_transactions.py
--- a/tests/storage/test_transactions.py
+++ b/tests/storage/test_transactions.py
@@ -13,6 +13,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from synapse.util.retryutils import MAX_RETRY_INTERVAL
+
 from tests.unittest import HomeserverTestCase
 
 
@@ -45,3 +47,12 @@ def test_initial_set_transactions(self):
         """
         d = self.store.set_destination_retry_timings("example.com", 1000, 50, 100)
         self.get_success(d)
+
+    def test_large_destination_retry(self):
+        d = self.store.set_destination_retry_timings(
+            "example.com", MAX_RETRY_INTERVAL, MAX_RETRY_INTERVAL, MAX_RETRY_INTERVAL
+        )
+        self.get_success(d)
+
+        d = self.store.get_destination_retry_timings("example.com")
+        self.get_success(d)