internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: matrix-org/synapse

Issue:
Sending AS transactions can wedge if network connectivity drops
The freenode bridge had a 9h outage caused by our cloud provider dropping temporarily. Synapse survived the drop, but in-flight requests got wedged and never timed out. We should have application level timeouts for outbound HTTP requests.


Discussion:

Resolving patch:
diff --git a/synapse/http/client.py b/synapse/http/client.py
--- a/synapse/http/client.py
+++ b/synapse/http/client.py
@@ -19,6 +19,7 @@
     CodeMessageException, MatrixCodeMessageException, SynapseError, Codes,
 )
 from synapse.util.logcontext import preserve_context_over_fn
+from synapse.util import logcontext
 import synapse.metrics
 from synapse.http.endpoint import SpiderEndpoint
 
@@ -72,39 +73,45 @@ def __init__(self, hs):
             contextFactory=hs.get_http_client_context_factory()
         )
         self.user_agent = hs.version_string
+        self.clock = hs.get_clock()
         if hs.config.user_agent_suffix:
             self.user_agent = "%s %s" % (self.user_agent, hs.config.user_agent_suffix,)
 
+    @defer.inlineCallbacks
     def request(self, method, uri, *args, **kwargs):
         # A small wrapper around self.agent.request() so we can easily attach
         # counters to it
         outgoing_requests_counter.inc(method)
-        d = preserve_context_over_fn(
-            self.agent.request,
-            method, uri, *args, **kwargs
-        )
+
+        def send_request():
+            request_deferred = self.agent.request(
+                method, uri, *args, **kwargs
+            )
+
+            return self.clock.time_bound_deferred(
+                request_deferred,
+                time_out=60,
+            )
 
         logger.info("Sending request %s %s", method, uri)
 
-        def _cb(response):
+        try:
+            with logcontext.PreserveLoggingContext():
+                response = yield send_request()
+
             incoming_responses_counter.inc(method, response.code)
             logger.info(
                 "Received response to  %s %s: %s",
                 method, uri, response.code
             )
-            return response
-
-        def _eb(failure):
+            defer.returnValue(response)
+        except Exception as e:
             incoming_responses_counter.inc(method, "ERR")
             logger.info(
                 "Error sending request to  %s %s: %s %s",
-                method, uri, failure.type, failure.getErrorMessage()
+                method, uri, type(e).__name__, e.message
             )
-            return failure
-
-        d.addCallbacks(_cb, _eb)
-
-        return d
+            raise e
 
     @defer.inlineCallbacks
     def post_urlencoded_get_json(self, uri, args={}):


Test patch:
Repository: matrix-org/synapse

Issue:
unit tests fail with TypeError (frozendict is not JSON serializable) in Fedora Rawhide
Hey all,

I'm the package maintainer for Synapse in Fedora. When trying to update to the latest version in Rawhide, I encountered some test failures. From the [full build log](https://kojipkgs.fedoraproject.org//work/tasks/1337/20441337/build.log):
```
[ERROR]
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 1185, in _inlineCallbacks
    result = g.send(result)
  File "/builddir/build/BUILD/synapse-0.22.1/synapse/handlers/message.py", line 506, in handle_new_client_event
    dump = ujson.dumps(event.content)
exceptions.TypeError: <frozendict {'creator': '@some_other_guy:red'}> is not JSON serializable
tests.rest.client.v1.test_rooms.RoomsMemberListTestCase.test_get_member_list_mixed_memberships
tests.rest.client.v1.test_rooms.RoomsMemberListTestCase.test_get_member_list_no_permission
```

All the tests are failing from the same spot. It looks like https://github.com/matrix-org/synapse/commit/fcf01dd88e738e5745c17c7740e0d72b8bbbcb66 introduced the change.


Discussion:
We've had conversation on IRC with @jeremycline and it turns out that in Fedora we have ujson of snapshot 2.0... And it seems commit https://github.com/esnme/ultrajson/commit/53f85b1bd6e4f27a3e4cdc605518c48a6c7e7e9e breaks thing... So just casting frozendict to dict should do the trick when serializing / deserializing.

Resolving patch:
diff --git a/synapse/handlers/message.py b/synapse/handlers/message.py
--- a/synapse/handlers/message.py
+++ b/synapse/handlers/message.py
@@ -26,6 +26,7 @@
 from synapse.util.async import run_on_reactor, ReadWriteLock, Limiter
 from synapse.util.logcontext import preserve_fn
 from synapse.util.metrics import measure_func
+from synapse.util.frozenutils import unfreeze
 from synapse.visibility import filter_events_for_client
 
 from ._base import BaseHandler
@@ -503,7 +504,7 @@ def handle_new_client_event(
 
         # Ensure that we can round trip before trying to persist in db
         try:
-            dump = ujson.dumps(event.content)
+            dump = ujson.dumps(unfreeze(event.content))
             ujson.loads(dump)
         except:
             logger.exception("Failed to encode content: %r", event.content)


Test patch:
Repository: matrix-org/synapse

Issue:
media repo worker doesn't support OPTIONS
which is problematic when used with riot-web develop which preflights url_preview requests due to the auth header


Discussion:

Resolving patch:
diff --git a/synapse/rest/media/v1/preview_url_resource.py b/synapse/rest/media/v1/preview_url_resource.py
--- a/synapse/rest/media/v1/preview_url_resource.py
+++ b/synapse/rest/media/v1/preview_url_resource.py
@@ -25,7 +25,8 @@
 from synapse.util.caches.expiringcache import ExpiringCache
 from synapse.http.client import SpiderHttpClient
 from synapse.http.server import (
-    request_handler, respond_with_json_bytes
+    request_handler, respond_with_json_bytes,
+    respond_with_json,
 )
 from synapse.util.async import ObservableDeferred
 from synapse.util.stringutils import is_ascii
@@ -78,6 +79,9 @@ def __init__(self, hs, media_repo):
             self._expire_url_cache_data, 10 * 1000
         )
 
+    def render_OPTIONS(self, request):
+        return respond_with_json(request, 200, {}, send_cors=True)
+
     def render_GET(self, request):
         self._async_render_GET(request)
         return NOT_DONE_YET


Test patch:
Repository: matrix-org/synapse

Issue:
Logout returns 200 when given an invalid access_token
<!-- 

**IF YOU HAVE SUPPORT QUESTIONS ABOUT RUNNING OR CONFIGURING YOUR OWN HOME SERVER**: 
You will likely get better support more quickly if you ask in ** #matrix:matrix.org ** ;)


This is a bug report template. By following the instructions below and
filling out the sections with your information, you will help the us to get all
the necessary data to fix your issue.

You can also preview your report before submitting it. You may remove sections
that aren't relevant to your particular case.

Text between <!-- and --​> marks will be invisible in the report.

-->

### Description

According to the specification for the [/logout endpoint](https://matrix.org/speculator/spec/HEAD/client_server/unstable.html#post-matrix-client-r0-logout), it invalidates an **existing token**. However, when supplied an invalid/non-existent access_token (such as "42") it still returns a 200 status code.   

### Steps to reproduce

- Send a POST to /_matrix/client/r0/logout with an Authorization header containing an invalid access_token (e.g. "Bearer 42")
- Response is a 200 status code

Ideally this should respond with error code  M_UNKNOWN_TOKEN to provide a more verbose error



Discussion:

Resolving patch:
diff --git a/synapse/rest/client/v1/logout.py b/synapse/rest/client/v1/logout.py
--- a/synapse/rest/client/v1/logout.py
+++ b/synapse/rest/client/v1/logout.py
@@ -44,7 +44,10 @@ def on_POST(self, request):
             requester = yield self.auth.get_user_by_req(request)
         except AuthError:
             # this implies the access token has already been deleted.
-            pass
+            defer.returnValue((401, {
+                "errcode": "M_UNKNOWN_TOKEN",
+                "error": "Access Token unknown or expired"
+            }))
         else:
             if requester.device_id is None:
                 # the acccess token wasn't associated with a device.


Test patch:
Repository: matrix-org/synapse

Issue:
room_id missing from response body for rooms/{roomId}/join endpoint
According to [the specification](https://matrix.org/speculator/spec/HEAD/client_server/unstable.html#post-matrix-client-unstable-rooms-roomid-join) (both r0.2.0 and unstable) the `rooms/{roomId}/join` endpoint should return the room_id for the room that was joined. This is not actually the case on matrix.org though – the response is just an empty object. I have not checked if this applies to `join/{roomIdOrAlias}` too, but I suspect that it does.


Discussion:
I just tested `join/{roomIdOrAlias}` in Riot, joining a room that I was invited to both on matrix.org and on another Synapse homeserver. I then joined a public room on matrix.org (no invite). In all cases, the response body was properly formed, with the room_id.

I did not test `rooms/{roomId}/join`.

Resolving patch:
diff --git a/synapse/rest/client/v1/room.py b/synapse/rest/client/v1/room.py
--- a/synapse/rest/client/v1/room.py
+++ b/synapse/rest/client/v1/room.py
@@ -655,7 +655,12 @@ def on_POST(self, request, room_id, membership_action, txn_id=None):
             content=event_content,
         )
 
-        defer.returnValue((200, {}))
+        return_value = {}
+
+        if membership_action == "join":
+            return_value["room_id"] = room_id
+
+        defer.returnValue((200, return_value))
 
     def _has_3pid_invite_keys(self, content):
         for key in {"id_server", "medium", "address"}:


Test patch:
Repository: matrix-org/synapse

Issue:
Add six as dependency
Just a quick tracking issue to remember to add six as dependency. It is currently used, but it's just an indirect dependency of many other packages. For clarity, it would be good to add it to the dependencies. I'm not sure how to do it myself, the file is non-standard.


Discussion:
just add it to python_dependencies.py iirc
There's multiple fields and sometimes they are the same and sometimes they are not and sometimes the versions are pinned and sometimes not. So I'd rather a person familiar with how and why that's the case do it.

Resolving patch:
diff --git a/synapse/python_dependencies.py b/synapse/python_dependencies.py
--- a/synapse/python_dependencies.py
+++ b/synapse/python_dependencies.py
@@ -1,5 +1,6 @@
 # Copyright 2015, 2016 OpenMarket Ltd
 # Copyright 2017 Vector Creations Ltd
+# Copyright 2018 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -18,6 +19,18 @@
 
 logger = logging.getLogger(__name__)
 
+# this dict maps from python package name to a list of modules we expect it to
+# provide.
+#
+# the key is a "requirement specifier", as used as a parameter to `pip
+# install`[1], or an `install_requires` argument to `setuptools.setup` [2].
+#
+# the value is a sequence of strings; each entry should be the name of the
+# python module, optionally followed by a version assertion which can be either
+# ">=<ver>" or "==<ver>".
+#
+# [1] https://pip.pypa.io/en/stable/reference/pip_install/#requirement-specifiers.
+# [2] https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-dependencies
 REQUIREMENTS = {
     "jsonschema>=2.5.1": ["jsonschema>=2.5.1"],
     "frozendict>=0.4": ["frozendict"],
@@ -26,7 +39,11 @@
     "signedjson>=1.0.0": ["signedjson>=1.0.0"],
     "pynacl>=1.2.1": ["nacl>=1.2.1", "nacl.bindings"],
     "service_identity>=1.0.0": ["service_identity>=1.0.0"],
-    "Twisted>=16.0.0": ["twisted>=16.0.0"],
+
+    # we break under Twisted 18.4
+    # (https://github.com/matrix-org/synapse/issues/3135)
+    "Twisted>=16.0.0,<18.4": ["twisted>=16.0.0"],
+
     "pyopenssl>=0.14": ["OpenSSL>=0.14"],
     "pyyaml": ["yaml"],
     "pyasn1": ["pyasn1"],
@@ -39,6 +56,7 @@
     "pymacaroons-pynacl": ["pymacaroons"],
     "msgpack-python>=0.3.0": ["msgpack"],
     "phonenumbers>=8.2.0": ["phonenumbers"],
+    "six": ["six"],
 }
 CONDITIONAL_REQUIREMENTS = {
     "web_client": {


Test patch:
Repository: matrix-org/synapse

Issue:
make the consent resource provide the current consentedness in a template var
to help with https://github.com/matrix-org/synapse/issues/3197#issuecomment-390944144


Discussion:

Resolving patch:
diff --git a/synapse/rest/consent/consent_resource.py b/synapse/rest/consent/consent_resource.py
--- a/synapse/rest/consent/consent_resource.py
+++ b/synapse/rest/consent/consent_resource.py
@@ -95,8 +95,8 @@ def __init__(self, hs):
         # this is required by the request_handler wrapper
         self.clock = hs.get_clock()
 
-        self._default_consent_verison = hs.config.user_consent_version
-        if self._default_consent_verison is None:
+        self._default_consent_version = hs.config.user_consent_version
+        if self._default_consent_version is None:
             raise ConfigError(
                 "Consent resource is enabled but user_consent section is "
                 "missing in config file.",
@@ -114,7 +114,10 @@ def __init__(self, hs):
             )
 
         loader = jinja2.FileSystemLoader(consent_template_directory)
-        self._jinja_env = jinja2.Environment(loader=loader)
+        self._jinja_env = jinja2.Environment(
+            loader=loader,
+            autoescape=jinja2.select_autoescape(['html', 'htm', 'xml']),
+        )
 
         if hs.config.form_secret is None:
             raise ConfigError(
@@ -129,6 +132,7 @@ def render_GET(self, request):
         return NOT_DONE_YET
 
     @wrap_html_request_handler
+    @defer.inlineCallbacks
     def _async_render_GET(self, request):
         """
         Args:
@@ -136,16 +140,26 @@ def _async_render_GET(self, request):
         """
 
         version = parse_string(request, "v",
-                               default=self._default_consent_verison)
+                               default=self._default_consent_version)
         username = parse_string(request, "u", required=True)
         userhmac = parse_string(request, "h", required=True)
 
         self._check_hash(username, userhmac)
 
+        if username.startswith('@'):
+            qualified_user_id = username
+        else:
+            qualified_user_id = UserID(username, self.hs.hostname).to_string()
+
+        u = yield self.store.get_user_by_id(qualified_user_id)
+        if u is None:
+            raise NotFoundError("Unknown user")
+
         try:
             self._render_template(
                 request, "%s.html" % (version,),
                 user=username, userhmac=userhmac, version=version,
+                has_consented=(u["consent_version"] == version),
             )
         except TemplateNotFound:
             raise NotFoundError("Unknown policy version")


Test patch:
Repository: matrix-org/synapse

Issue:
we suddenly require jinja
this was supposed to be only required if the consent resource was configured


Discussion:

Resolving patch:
diff --git a/synapse/app/homeserver.py b/synapse/app/homeserver.py
--- a/synapse/app/homeserver.py
+++ b/synapse/app/homeserver.py
@@ -41,7 +41,6 @@
 from synapse.replication.http import ReplicationRestResource, REPLICATION_PREFIX
 from synapse.replication.tcp.resource import ReplicationStreamProtocolFactory
 from synapse.rest import ClientRestResource
-from synapse.rest.consent.consent_resource import ConsentResource
 from synapse.rest.key.v1.server_key_resource import LocalKey
 from synapse.rest.key.v2 import KeyApiV2Resource
 from synapse.rest.media.v0.content_repository import ContentRepoResource
@@ -186,6 +185,7 @@ def _configure_named_resource(self, name, compress=False):
             })
 
         if name == "consent":
+            from synapse.rest.consent.consent_resource import ConsentResource
             consent_resource = ConsentResource(self)
             if compress:
                 consent_resource = gz_wrap(consent_resource)


Test patch:
Repository: matrix-org/synapse

Issue:
synapse_federation_client_sent_pdu_destinations:count no longer populated
The metric appears to have gone missing as of 0.31.0


Discussion:

Resolving patch:
diff --git a/synapse/federation/transaction_queue.py b/synapse/federation/transaction_queue.py
--- a/synapse/federation/transaction_queue.py
+++ b/synapse/federation/transaction_queue.py
@@ -42,8 +42,11 @@
 
 logger = logging.getLogger(__name__)
 
-sent_pdus_destination_dist = Counter(
-    "synapse_federation_transaction_queue_sent_pdu_destinations", ""
+sent_pdus_destination_dist_count = Counter(
+    "synapse_federation_client_sent_pdu_destinations:count", ""
+)
+sent_pdus_destination_dist_total = Counter(
+    "synapse_federation_client_sent_pdu_destinations:total", ""
 )
 
 
@@ -280,7 +283,8 @@ def _send_pdu(self, pdu, destinations):
         if not destinations:
             return
 
-        sent_pdus_destination_dist.inc(len(destinations))
+        sent_pdus_destination_dist_total.inc(len(destinations))
+        sent_pdus_destination_dist_count.inc()
 
         for destination in destinations:
             self.pending_pdus_by_dest.setdefault(destination, []).append(


Test patch:
Repository: matrix-org/synapse

Issue:
/directory/list APIs should return 404 not 400 for room not found
As per https://github.com/matrix-org/matrix-doc/pull/1141#discussion_r172503566



Discussion:

Resolving patch:
diff --git a/synapse/rest/client/v1/directory.py b/synapse/rest/client/v1/directory.py
--- a/synapse/rest/client/v1/directory.py
+++ b/synapse/rest/client/v1/directory.py
@@ -18,7 +18,7 @@
 
 from twisted.internet import defer
 
-from synapse.api.errors import AuthError, Codes, SynapseError
+from synapse.api.errors import AuthError, Codes, NotFoundError, SynapseError
 from synapse.http.servlet import parse_json_object_from_request
 from synapse.types import RoomAlias
 
@@ -159,7 +159,7 @@ def __init__(self, hs):
     def on_GET(self, request, room_id):
         room = yield self.store.get_room(room_id)
         if room is None:
-            raise SynapseError(400, "Unknown room")
+            raise NotFoundError("Unknown room")
 
         defer.returnValue((200, {
             "visibility": "public" if room["is_public"] else "private"


Test patch:
Repository: matrix-org/synapse

Issue:
GET /rooms/!whatever/event should be guest accessible
as per https://github.com/matrix-org/matrix-doc/issues/1540#issuecomment-414379939


Discussion:

Resolving patch:
diff --git a/synapse/rest/client/v1/room.py b/synapse/rest/client/v1/room.py
--- a/synapse/rest/client/v1/room.py
+++ b/synapse/rest/client/v1/room.py
@@ -531,7 +531,7 @@ def __init__(self, hs):
 
     @defer.inlineCallbacks
     def on_GET(self, request, room_id, event_id):
-        requester = yield self.auth.get_user_by_req(request)
+        requester = yield self.auth.get_user_by_req(request, allow_guest=True)
         event = yield self.event_handler.get_event(requester.user, room_id, event_id)
 
         time_now = self.clock.time_msec()


Test patch:
Repository: matrix-org/synapse

Issue:
/_matrix/consent fails with 500 error
```
2018-08-21 21:29:30,810 - synapse.http.server - 162 - ERROR - GET-855803 - Failed handle request <XForwardedForRequest at 0x7fb85fdeb518 method='GET' uri='/_matrix/co
  File "synapse/http/server.py", line 126, in wrapped_request_handler
    
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 151, in maybeDeferred
    result = f(*args, **kw)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1613, in unwindGenerator
    return _cancellableInlineCallbacks(gen)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1529, in _cancellableInlineCallbacks
    _inlineCallbacks(None, g, status)
--- <exception caught here> ---
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "synapse/rest/consent/consent_resource.py", line 145, in _async_render_GET
    
  File "synapse/rest/consent/consent_resource.py", line 219, in _check_hash
    
exceptions.TypeError: 'unicode' does not have the buffer interface
```


Discussion:

Resolving patch:
diff --git a/synapse/rest/consent/consent_resource.py b/synapse/rest/consent/consent_resource.py
--- a/synapse/rest/consent/consent_resource.py
+++ b/synapse/rest/consent/consent_resource.py
@@ -140,7 +140,7 @@ def _async_render_GET(self, request):
         version = parse_string(request, "v",
                                default=self._default_consent_version)
         username = parse_string(request, "u", required=True)
-        userhmac = parse_string(request, "h", required=True)
+        userhmac = parse_string(request, "h", required=True, encoding=None)
 
         self._check_hash(username, userhmac)
 
@@ -175,7 +175,7 @@ def _async_render_POST(self, request):
         """
         version = parse_string(request, "v", required=True)
         username = parse_string(request, "u", required=True)
-        userhmac = parse_string(request, "h", required=True)
+        userhmac = parse_string(request, "h", required=True, encoding=None)
 
         self._check_hash(username, userhmac)
 
@@ -210,9 +210,18 @@ def _render_template(self, request, template_name, **template_args):
         finish_request(request)
 
     def _check_hash(self, userid, userhmac):
+        """
+        Args:
+            userid (unicode):
+            userhmac (bytes):
+
+        Raises:
+              SynapseError if the hash doesn't match
+
+        """
         want_mac = hmac.new(
             key=self._hmac_secret,
-            msg=userid,
+            msg=userid.encode('utf-8'),
             digestmod=sha256,
         ).hexdigest()
 


Test patch:
Repository: matrix-org/synapse

Issue:
Unhandled error in deferred: SortedDict.itervalues() is not implemented
This popped up in the log for t2bot.io. Unknown cause and unknown impact. Synapse 0.31.2
```
Traceback (most recent call last):
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
    result = f(*args, **kw)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/federation/send_queue.py", line 109, in _clear_queue
    self._clear_queue_before_pos(position_to_delete)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/federation/send_queue.py", line 122, in _clear_queue_before_pos
    for uids in itervalues(self.presence_changed)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/six.py", line 602, in itervalues
    return d.itervalues(**kw)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/sortedcontainers/sorteddict.py", line 370, in method
    raise AttributeError(message)
AttributeError: SortedDict.itervalues() is not implemented. Use SortedDict.values() instead.
```


Discussion:
Impact is presence to the outside world is affected, sometimes wrong. Doesn't appear to affect inbound PDUs. Would be nice to have it fixed, given it shows up more frequently than it did a month ago.

Resolving patch:
diff --git a/synapse/federation/send_queue.py b/synapse/federation/send_queue.py
--- a/synapse/federation/send_queue.py
+++ b/synapse/federation/send_queue.py
@@ -32,7 +32,7 @@
 import logging
 from collections import namedtuple
 
-from six import iteritems, itervalues
+from six import iteritems
 
 from sortedcontainers import SortedDict
 
@@ -117,7 +117,7 @@ def _clear_queue_before_pos(self, position_to_delete):
 
             user_ids = set(
                 user_id
-                for uids in itervalues(self.presence_changed)
+                for uids in self.presence_changed.values()
                 for user_id in uids
             )
 


Test patch: