internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: matrix-org/synapse

Issue:
Add python3 tag to version number
So that phone home stats can provide a view on py3 adoption


Discussion:
Could we be even more creative and just report the version of python in use? I'd imagine some people will be using py3.4-7

Resolving patch:
diff --git a/synapse/app/homeserver.py b/synapse/app/homeserver.py
--- a/synapse/app/homeserver.py
+++ b/synapse/app/homeserver.py
@@ -457,6 +457,10 @@ def phone_stats_home():
         stats["homeserver"] = hs.config.server_name
         stats["timestamp"] = now
         stats["uptime_seconds"] = uptime
+        version = sys.version_info
+        stats["python_version"] = "{}.{}.{}".format(
+            version.major, version.minor, version.micro
+        )
         stats["total_users"] = yield hs.get_datastore().count_all_users()
 
         total_nonbridged_users = yield hs.get_datastore().count_nonbridged_users()


Test patch:
Repository: matrix-org/synapse

Issue:
sometimes we use an out-of-date server for send_joins
... which in turn means that the joining user gets a very old view of the room state


Discussion:
For example, when one tries to join `#megolm:matrix.org`, matrix.org returns the following list of servers in the room:
* dmp-develop.com
* matrix.freelock.com
* darmstadt.ccc.de
* jki.re
* onedefence.com
* terracrypt.net
* sw1v.org
* matrix.org
* exul.org
* t2l.io

Of these, dmp-develop.com is offline. I don't know why we don't use matrix.freelock.com. Anyway, I've seen various servers use darmstadt.ccc.de. But that server left a long time ago. So the user gets an *ancient* state and timeline snapshot, and it takes ages for backfilling etc to sort it all out.

Of course if the user sends any encrypted messages while the server is trying to catch up, then it will exclude any users who joined the room since darmstadt.ccc.de left.
Another example: if you try to join #riot via hveem.no, you get a room consisting entirely of a single 'create' event. :/

Resolving patch:
diff --git a/synapse/handlers/room_member.py b/synapse/handlers/room_member.py
--- a/synapse/handlers/room_member.py
+++ b/synapse/handlers/room_member.py
@@ -583,6 +583,11 @@ def lookup_room_alias(self, room_alias):
         room_id = mapping["room_id"]
         servers = mapping["servers"]
 
+        # put the server which owns the alias at the front of the server list.
+        if room_alias.domain in servers:
+            servers.remove(room_alias.domain)
+        servers.insert(0, room_alias.domain)
+
         defer.returnValue((RoomID.from_string(room_id), servers))
 
     @defer.inlineCallbacks


Test patch:
Repository: matrix-org/synapse

Issue:
run_as_background_process doesn't catch & log exceptions 
(plus if it does raise in a loopingcall, we throw away the exception)


Discussion:

Resolving patch:
diff --git a/synapse/metrics/background_process_metrics.py b/synapse/metrics/background_process_metrics.py
--- a/synapse/metrics/background_process_metrics.py
+++ b/synapse/metrics/background_process_metrics.py
@@ -13,6 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import logging
 import threading
 
 import six
@@ -23,6 +24,9 @@
 
 from synapse.util.logcontext import LoggingContext, PreserveLoggingContext
 
+logger = logging.getLogger(__name__)
+
+
 _background_process_start_count = Counter(
     "synapse_background_process_start_count",
     "Number of background processes started",
@@ -191,6 +195,8 @@ def run():
 
             try:
                 yield func(*args, **kwargs)
+            except Exception:
+                logger.exception("Background process '%s' threw an exception", desc)
             finally:
                 proc.update_metrics()
 


Test patch:
Repository: matrix-org/synapse

Issue:
Initial lazy loading sync can fail with a 404
The returned error is: `Could not find event $EVENT_ID`, without a stack trace. The event in question has been rejected, so we're probably calling `DataStore.get_event` with that event_id. (Though the more interesting question is where it got the event ID from in the first place) 


Discussion:
The event in question is a canonical alias event, and so a possible offending `get_event` call would be:

https://github.com/matrix-org/synapse/blob/19dc676d1aaf66a45c9a1810132952f7ae3ca2aa/synapse/handlers/sync.py#L575

Though that would imply that the rejected event had somehow gotten into the state.
In fact, it explicitly says `allow_none=False`, and then checks if the event returned is `None`...

Resolving patch:
diff --git a/synapse/handlers/sync.py b/synapse/handlers/sync.py
--- a/synapse/handlers/sync.py
+++ b/synapse/handlers/sync.py
@@ -567,13 +567,13 @@ def compute_summary(self, room_id, sync_config, batch, state, now_token):
         # be a valid name or canonical_alias - i.e. we're checking that they
         # haven't been "deleted" by blatting {} over the top.
         if name_id:
-            name = yield self.store.get_event(name_id, allow_none=False)
+            name = yield self.store.get_event(name_id, allow_none=True)
             if name and name.content:
                 defer.returnValue(summary)
 
         if canonical_alias_id:
             canonical_alias = yield self.store.get_event(
-                canonical_alias_id, allow_none=False,
+                canonical_alias_id, allow_none=True,
             )
             if canonical_alias and canonical_alias.content:
                 defer.returnValue(summary)


Test patch:
Repository: matrix-org/synapse

Issue:
Make loopingcall wrapper log loudly if it gets killed.
If a looping call function errors, then it kills the loop entirely. Currently it throws away the exception logs, so we should make it actually log them.

Related to #3921


Discussion:

Resolving patch:
diff --git a/synapse/handlers/appservice.py b/synapse/handlers/appservice.py
--- a/synapse/handlers/appservice.py
+++ b/synapse/handlers/appservice.py
@@ -28,6 +28,7 @@
     event_processing_loop_room_count,
 )
 from synapse.metrics.background_process_metrics import run_as_background_process
+from synapse.util import log_failure
 from synapse.util.logcontext import make_deferred_yieldable, run_in_background
 from synapse.util.metrics import Measure
 
@@ -36,17 +37,6 @@
 events_processed_counter = Counter("synapse_handlers_appservice_events_processed", "")
 
 
-def log_failure(failure):
-    logger.error(
-        "Application Services Failure",
-        exc_info=(
-            failure.type,
-            failure.value,
-            failure.getTracebackObject()
-        )
-    )
-
-
 class ApplicationServicesHandler(object):
 
     def __init__(self, hs):
@@ -112,7 +102,10 @@ def handle_event(event):
 
                         if not self.started_scheduler:
                             def start_scheduler():
-                                return self.scheduler.start().addErrback(log_failure)
+                                return self.scheduler.start().addErrback(
+                                    log_failure, "Application Services Failure",
+                                )
+
                             run_as_background_process("as_scheduler", start_scheduler)
                             self.started_scheduler = True
 
diff --git a/synapse/util/__init__.py b/synapse/util/__init__.py
--- a/synapse/util/__init__.py
+++ b/synapse/util/__init__.py
@@ -68,7 +68,10 @@ def looping_call(self, f, msec):
         """
         call = task.LoopingCall(f)
         call.clock = self._reactor
-        call.start(msec / 1000.0, now=False)
+        d = call.start(msec / 1000.0, now=False)
+        d.addErrback(
+            log_failure, "Looping call died", consumeErrors=False,
+        )
         return call
 
     def call_later(self, delay, callback, *args, **kwargs):
@@ -109,3 +112,29 @@ def batch_iter(iterable, size):
     sourceiter = iter(iterable)
     # call islice until it returns an empty tuple
     return iter(lambda: tuple(islice(sourceiter, size)), ())
+
+
+def log_failure(failure, msg, consumeErrors=True):
+    """Creates a function suitable for passing to `Deferred.addErrback` that
+    logs any failures that occur.
+
+    Args:
+        msg (str): Message to log
+        consumeErrors (bool): If true consumes the failure, otherwise passes
+            on down the callback chain
+
+    Returns:
+        func(Failure)
+    """
+
+    logger.error(
+        msg,
+        exc_info=(
+            failure.type,
+            failure.value,
+            failure.getTracebackObject()
+        )
+    )
+
+    if not consumeErrors:
+        return failure


Test patch:
Repository: matrix-org/synapse

Issue:
psutil is now a hard dep and should be treated accordingly



Discussion:

Resolving patch:
diff --git a/synapse/app/homeserver.py b/synapse/app/homeserver.py
--- a/synapse/app/homeserver.py
+++ b/synapse/app/homeserver.py
@@ -20,6 +20,7 @@
 
 from six import iteritems
 
+import psutil
 from prometheus_client import Gauge
 
 from twisted.application import service
@@ -502,7 +503,6 @@ def phone_stats_home():
 
     def performance_stats_init():
         try:
-            import psutil
             process = psutil.Process()
             # Ensure we can fetch both, and make the initial request for cpu_percent
             # so the next request will use this as the initial point.
@@ -510,12 +510,9 @@ def performance_stats_init():
             process.cpu_percent(interval=None)
             logger.info("report_stats can use psutil")
             stats_process.append(process)
-        except (ImportError, AttributeError):
-            logger.warn(
-                "report_stats enabled but psutil is not installed or incorrect version."
-                " Disabling reporting of memory/cpu stats."
-                " Ensuring psutil is available will help matrix.org track performance"
-                " changes across releases."
+        except (AttributeError):
+            logger.warning(
+                "Unable to read memory/cpu stats. Disabling reporting."
             )
 
     def generate_user_daily_visit_stats():
diff --git a/synapse/python_dependencies.py b/synapse/python_dependencies.py
--- a/synapse/python_dependencies.py
+++ b/synapse/python_dependencies.py
@@ -53,6 +53,7 @@
     "pillow>=3.1.2": ["PIL"],
     "pydenticon>=0.2": ["pydenticon"],
     "sortedcontainers>=1.4.4": ["sortedcontainers"],
+    "psutil>=2.0.0": ["psutil>=2.0.0"],
     "pysaml2>=3.0.0": ["saml2"],
     "pymacaroons-pynacl>=0.9.3": ["pymacaroons"],
     "msgpack-python>=0.4.2": ["msgpack"],
@@ -79,9 +80,6 @@
     "matrix-synapse-ldap3": {
         "matrix-synapse-ldap3>=0.1": ["ldap_auth_provider"],
     },
-    "psutil": {
-        "psutil>=2.0.0": ["psutil>=2.0.0"],
-    },
     "postgres": {
         "psycopg2>=2.6": ["psycopg2"]
     }


Test patch:
Repository: matrix-org/synapse

Issue:
logcontext warnings during startup
A bunch of mau-related code was introduced in 0.33.3 ish (#3708, #3662, and many others) which runs a bunch of database code during startup without `yield`ing or otherwise managing logcontexts, and hence ends up leaking logcontexts into the reactor and emitting scary-looking warnings in the logs:

```
2018-09-07 11:18:43,101 - synapse.util.logcontext - 431 - WARNING - - Unexpected logging context: main@7fd0fb976c08 is not sentinel
```



Discussion:

Resolving patch:
diff --git a/synapse/app/homeserver.py b/synapse/app/homeserver.py
--- a/synapse/app/homeserver.py
+++ b/synapse/app/homeserver.py
@@ -530,10 +530,13 @@ def generate_user_daily_visit_stats():
     clock.looping_call(generate_user_daily_visit_stats, 5 * 60 * 1000)
 
     # monthly active user limiting functionality
-    clock.looping_call(
-        hs.get_datastore().reap_monthly_active_users, 1000 * 60 * 60
-    )
-    hs.get_datastore().reap_monthly_active_users()
+    def reap_monthly_active_users():
+        return run_as_background_process(
+            "reap_monthly_active_users",
+            hs.get_datastore().reap_monthly_active_users,
+        )
+    clock.looping_call(reap_monthly_active_users, 1000 * 60 * 60)
+    reap_monthly_active_users()
 
     @defer.inlineCallbacks
     def generate_monthly_active_users():
@@ -547,12 +550,23 @@ def generate_monthly_active_users():
         registered_reserved_users_mau_gauge.set(float(reserved_count))
         max_mau_gauge.set(float(hs.config.max_mau_value))
 
-    hs.get_datastore().initialise_reserved_users(
-        hs.config.mau_limits_reserved_threepids
+    def start_generate_monthly_active_users():
+        return run_as_background_process(
+            "generate_monthly_active_users",
+            generate_monthly_active_users,
+        )
+
+    # XXX is this really supposed to be a background process? it looks
+    # like it needs to complete before some of the other stuff runs.
+    run_as_background_process(
+        "initialise_reserved_users",
+        hs.get_datastore().initialise_reserved_users,
+        hs.config.mau_limits_reserved_threepids,
     )
-    generate_monthly_active_users()
+
+    start_generate_monthly_active_users()
     if hs.config.limit_usage_by_mau:
-        clock.looping_call(generate_monthly_active_users, 5 * 60 * 1000)
+        clock.looping_call(start_generate_monthly_active_users, 5 * 60 * 1000)
     # End of monthly active user settings
 
     if hs.config.report_stats:


Test patch:
Repository: matrix-org/synapse

Issue:
we seem to accept events with non-ascii event IDs, but later barf on them
Obviously, we shouldn't accept them in the first place (but a workaround for the current brokenness of rooms which include such events might be to encode as utf8 rather than ascii)


Discussion:
```
Traceback (most recent call last):
  File "/home/matrix/synapse/synapse/handlers/federation.py", line 385, in on_receive_pdu
    state_res_store=StateResolutionStore(self.store),
  File "/home/matrix/.synapse-py3/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/home/matrix/synapse/synapse/state/v1.py", line 108, in resolve_events_with_store
    unconflicted_state, conflicted_state, auth_events, state_map
  File "/home/matrix/synapse/synapse/state/v1.py", line 193, in _resolve_with_state
    conflicted_state, auth_events
  File "/home/matrix/synapse/synapse/state/v1.py", line 240, in _resolve_state_events
    auth_events
  File "/home/matrix/synapse/synapse/state/v1.py", line 256, in _resolve_auth_events
    reverse = [i for i in reversed(_ordered_events(events))]
  File "/home/matrix/synapse/synapse/state/v1.py", line 303, in _ordered_events
    return sorted(events, key=key_func)
  File "/home/matrix/synapse/synapse/state/v1.py", line 301, in key_func
    return -int(e.depth), hashlib.sha1(e.event_id.encode('ascii')).hexdigest()
UnicodeEncodeError: 'ascii' codec can't encode character '\u205f' in position 42: ordinal not in range(128)
```

Resolving patch:
diff --git a/synapse/state/v1.py b/synapse/state/v1.py
--- a/synapse/state/v1.py
+++ b/synapse/state/v1.py
@@ -298,6 +298,8 @@ def _resolve_normal_events(events, auth_events):
 
 def _ordered_events(events):
     def key_func(e):
-        return -int(e.depth), hashlib.sha1(e.event_id.encode('ascii')).hexdigest()
+        # we have to use utf-8 rather than ascii here because it turns out we allow
+        # people to send us events with non-ascii event IDs :/
+        return -int(e.depth), hashlib.sha1(e.event_id.encode('utf-8')).hexdigest()
 
     return sorted(events, key=key_func)


Test patch:
Repository: matrix-org/synapse

Issue:
we should drop sent_transactions
#1188 made `sent_transactions` redundant - we should remove it.


Discussion:

Resolving patch:
diff --git a/synapse/storage/__init__.py b/synapse/storage/__init__.py
--- a/synapse/storage/__init__.py
+++ b/synapse/storage/__init__.py
@@ -119,7 +119,6 @@ def __init__(self, db_conn, hs):
             db_conn, "device_lists_stream", "stream_id",
         )
 
-        self._transaction_id_gen = IdGenerator(db_conn, "sent_transactions", "id")
         self._access_tokens_id_gen = IdGenerator(db_conn, "access_tokens", "id")
         self._event_reports_id_gen = IdGenerator(db_conn, "event_reports", "id")
         self._push_rule_id_gen = IdGenerator(db_conn, "push_rules", "id")
diff --git a/synapse/storage/prepare_database.py b/synapse/storage/prepare_database.py
--- a/synapse/storage/prepare_database.py
+++ b/synapse/storage/prepare_database.py
@@ -25,7 +25,7 @@
 
 # Remember to update this number every time a change is made to database
 # schema files, so the users will be informed on server restarts.
-SCHEMA_VERSION = 52
+SCHEMA_VERSION = 53
 
 dir_path = os.path.abspath(os.path.dirname(__file__))
 
diff --git a/synapse/storage/schema/delta/34/sent_txn_purge.py b/synapse/storage/schema/delta/34/sent_txn_purge.py
deleted file mode 100644
--- a/synapse/storage/schema/delta/34/sent_txn_purge.py
+++ /dev/null
@@ -1,32 +0,0 @@
-# Copyright 2016 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import logging
-
-from synapse.storage.engines import PostgresEngine
-
-logger = logging.getLogger(__name__)
-
-
-def run_create(cur, database_engine, *args, **kwargs):
-    if isinstance(database_engine, PostgresEngine):
-        cur.execute("TRUNCATE sent_transactions")
-    else:
-        cur.execute("DELETE FROM sent_transactions")
-
-    cur.execute("CREATE INDEX sent_transactions_ts ON sent_transactions(ts)")
-
-
-def run_upgrade(cur, database_engine, *args, **kwargs):
-    pass


Test patch:
Repository: matrix-org/synapse

Issue:
GET media/r0/config is missing Access-Control-Allow-Origin according to Riot



Discussion:

Resolving patch:
diff --git a/synapse/rest/media/v1/config_resource.py b/synapse/rest/media/v1/config_resource.py
--- a/synapse/rest/media/v1/config_resource.py
+++ b/synapse/rest/media/v1/config_resource.py
@@ -41,7 +41,7 @@ def render_GET(self, request):
     @defer.inlineCallbacks
     def _async_render_GET(self, request):
         yield self.auth.get_user_by_req(request)
-        respond_with_json(request, 200, self.limits_dict)
+        respond_with_json(request, 200, self.limits_dict, send_cors=True)
 
     def render_OPTIONS(self, request):
         respond_with_json(request, 200, {}, send_cors=True)


Test patch:
Repository: matrix-org/synapse

Issue:
synapse_admin_mau:current metric is not updated when not limiting usage
It's just a constant reading of zero.


Discussion:
#4233

Resolving patch:
diff --git a/synapse/app/homeserver.py b/synapse/app/homeserver.py
--- a/synapse/app/homeserver.py
+++ b/synapse/app/homeserver.py
@@ -537,7 +537,7 @@ def start_generate_monthly_active_users():
         )
 
     start_generate_monthly_active_users()
-    if hs.config.limit_usage_by_mau:
+    if hs.config.limit_usage_by_mau or hs.config.mau_stats_only:
         clock.looping_call(start_generate_monthly_active_users, 5 * 60 * 1000)
     # End of monthly active user settings
 


Test patch:
Repository: matrix-org/synapse

Issue:
Consent templates loaded from the wrong location
Synapse version: `develop` at time of writing.

`homeserver.yaml` bits:
```yaml
user_consent:
  template_dir: ./res/templates/privacy
  version: 1.0
  server_notice_content:
    msgtype: m.text
    body: >-
      To continue using this homeserver you must review and agree to the
      terms and conditions at %(consent_uri)s
  send_server_notice_to_guests: false
  block_events_error: >-
    To continue using this homeserver you must review and agree to the
    terms and conditions at %(consent_uri)s
  require_at_registration: true
  policy_name: "Terms and Conditions"

```

Error:
```
2019-01-28 21:20:04,020 - twisted - 242 - ERROR -  - ConfigError: Could not find template directory '/res/templates/privacy'
```

It shouldn't be trying to load the absolute path at all with this configuration.

See also #4500 


Discussion:
this sounds like a regression which we need to fix before the release this week

Resolving patch:
diff --git a/synapse/config/consent_config.py b/synapse/config/consent_config.py
--- a/synapse/config/consent_config.py
+++ b/synapse/config/consent_config.py
@@ -13,6 +13,10 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from os import path
+
+from synapse.config import ConfigError
+
 from ._base import Config
 
 DEFAULT_CONFIG = """\
@@ -85,7 +89,15 @@ def read_config(self, config):
         if consent_config is None:
             return
         self.user_consent_version = str(consent_config["version"])
-        self.user_consent_template_dir = consent_config["template_dir"]
+        self.user_consent_template_dir = self.abspath(
+            consent_config["template_dir"]
+        )
+        if not path.isdir(self.user_consent_template_dir):
+            raise ConfigError(
+                "Could not find template directory '%s'" % (
+                    self.user_consent_template_dir,
+                ),
+            )
         self.user_consent_server_notice_content = consent_config.get(
             "server_notice_content",
         )
diff --git a/synapse/rest/consent/consent_resource.py b/synapse/rest/consent/consent_resource.py
--- a/synapse/rest/consent/consent_resource.py
+++ b/synapse/rest/consent/consent_resource.py
@@ -101,16 +101,7 @@ def __init__(self, hs):
                 "missing in config file.",
             )
 
-        # daemonize changes the cwd to /, so make the path absolute now.
-        consent_template_directory = path.abspath(
-            hs.config.user_consent_template_dir,
-        )
-        if not path.isdir(consent_template_directory):
-            raise ConfigError(
-                "Could not find template directory '%s'" % (
-                    consent_template_directory,
-                ),
-            )
+        consent_template_directory = hs.config.user_consent_template_dir
 
         loader = jinja2.FileSystemLoader(consent_template_directory)
         self._jinja_env = jinja2.Environment(


Test patch:
Repository: matrix-org/synapse

Issue:
default acme config doesn't work on py2
```
2019-02-04 17:45:46,170 - twisted - 245 - ERROR - - Traceback (most recent call last):
2019-02-04 17:45:46,170 - twisted - 245 - ERROR - -   File "/usr/local/lib/python2.7/site-packages/synapse/app/homeserver.py", line 418, in start
2019-02-04 17:45:46,171 - twisted - 245 - ERROR - -     yield acme.start_listening()
2019-02-04 17:45:46,171 - twisted - 245 - ERROR - - TypeError: expected unicode for text, got 'https://acme-v01.api.letsencrypt.org/directory'
2019-02-04 17:45:46,191 - twisted - 245 - CRITICAL - - Unhandled error in Deferred:
2019-02-04 17:45:46,192 - twisted - 245 - CRITICAL - - 
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/python2.7/site-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python2.7/site-packages/synapse/app/homeserver.py", line 450, in start
    sys.exit(1)
SystemExit: 1
```


Discussion:

Resolving patch:
diff --git a/synapse/config/tls.py b/synapse/config/tls.py
--- a/synapse/config/tls.py
+++ b/synapse/config/tls.py
@@ -37,7 +37,7 @@ def read_config(self, config):
 
         self.acme_enabled = acme_config.get("enabled", False)
         self.acme_url = acme_config.get(
-            "url", "https://acme-v01.api.letsencrypt.org/directory"
+            "url", u"https://acme-v01.api.letsencrypt.org/directory"
         )
         self.acme_port = acme_config.get("port", 80)
         self.acme_bind_addresses = acme_config.get("bind_addresses", ['::', '0.0.0.0'])


Test patch: