Repository: matrix-org/synapse

Issue:
/keys/query fails when fetching federating keys from a server which refuses connection
### Description

I can not send a message in a federated encrypted room (#megolm:matrix.org in particular)

### Steps to reproduce

- Try sending a message
- (Client fetches then keys and triggers that bug on the server with that)

### Version information

- **Version**:        current develop (8efe773)
- **Platform**:       debian

### Logs:
```
2018-03-24 13:00:30,625 - synapse.http.matrixfederationclient - 223 - WARNING - POST-2286 - {POST-O-4871} Sending request failed to sijmenschoon.nl: POST matrix://sijmenschoon.nl/_matrix/federation/v1/user/keys/query: ConnectionRefusedError('Connection refused',)
2018-03-24 13:00:30,627 - synapse.http.outbound - 247 - INFO - POST-2286 - {POST-O-4871} [sijmenschoon.nl] Result: ConnectionRefusedError('Connection refused',) 
2018-03-24 13:00:30,629 - synapse.http.server - 191 - ERROR - POST-2286 - Failed handle request synapse.http.server._async_render on <synapse.rest.ClientRestResource object at 0x7f759a5b5a90>: <XForwardedForRequest at 0x7f758b4bf098 method=POST uri=/_matrix/client/unstable/keys/query?access_token=<redacted> clientproto=HTTP/1.1 site=8008>: Traceback (most recent call last):
  File "/srv/synapse/env/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/srv/synapse/env/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1442, in gotResult
    _inlineCallbacks(r, g, deferred)
  File "/srv/synapse/env/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/srv/synapse/env/local/lib/python2.7/site-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
--- <exception caught here> ---
  File "synapse/http/server.py", line 162, in wrapped_request_handler
    yield request_handler(self, request, request_metrics)
  File "/srv/synapse/env/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
    result = g.send(result)
  File "synapse/http/server.py", line 310, in _async_render
    self._send_response(request, code, response)
  File "synapse/http/server.py", line 352, in _send_response
    canonical_json=self.canonical_json,
  File "synapse/http/server.py", line 464, in respond_with_json
    json_bytes = simplejson.dumps(json_object)
  File "/srv/synapse/env/local/lib/python2.7/site-packages/simplejson/__init__.py", line 382, in dumps
    return _default_encoder.encode(obj)
  File "/srv/synapse/env/local/lib/python2.7/site-packages/simplejson/encoder.py", line 291, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/srv/synapse/env/local/lib/python2.7/site-packages/simplejson/encoder.py", line 373, in iterencode
    return _iterencode(o, 0)
  File "/srv/synapse/env/local/lib/python2.7/site-packages/simplejson/encoder.py", line 268, in default
    o.__class__.__name__)
exceptions.TypeError: Object of type Message is not JSON serializable
2018-03-24 13:00:30,631 - synapse.access.http.8008 - 93 - INFO - POST-2286 - 188.174.84.78 - 8008 - {@krombel:msg-net.de} Processed request: 655ms (32ms, 0ms) (1ms/22ms/3) 55B 500 "POST /_matrix/client/unstable/keys/query?access_token=<redacted> HTTP/1.1" "Riot.im/0.8.4..."
```
For reference:
This is how I expected it to look like (as well "Connection refused"):
```
2018-03-24 13:00:34,962 - synapse.access.http.8008 - 59 - INFO - POST-2294 - 188.174.84.78 - 8008 - Received request: POST /_matrix/client/unstable/keys/query?access_token=<redacted>
2018-03-24 13:00:34,972 - synapse.http.outbound - 165 - INFO - POST-2294 - {POST-O-4873} [nina.ninja] Sending request: POST matrix://nina.ninja/_matrix/federation/v1/user/keys/query
2018-03-24 13:00:34,974 - synapse.http.endpoint - 264 - INFO -  - Connecting to 178.62.155.221:8448
2018-03-24 13:00:34,982 - synapse.http.outbound - 165 - INFO - POST-2294 - {POST-O-4874} [onedefence.com] Sending request: POST matrix://onedefence.com/_matrix/federation/v1/user/keys/query
2018-03-24 13:00:35,088 - synapse.http.outbound - 247 - INFO - POST-2294 - {POST-O-4874} [onedefence.com] Result: 404 Not Found
2018-03-24 13:00:35,289 - synapse.http.matrixfederationclient - 223 - WARNING - POST-2294 - {POST-O-4873} Sending request failed to nina.ninja: POST matrix://nina.ninja/_matrix/federation/v1/user/keys/query: ConnectionRefusedError('Connection refused',)
2018-03-24 13:00:35,290 - synapse.http.outbound - 247 - INFO - POST-2294 - {POST-O-4873} [nina.ninja] Result: ConnectionRefusedError('Connection refused',)
2018-03-24 13:00:35,293 - synapse.access.http.8008 - 93 - INFO - POST-2294 - 188.174.84.78 - 8008 - {@krombel:msg-net.de} Processed request: 330ms (16ms, 7ms) (2ms/4ms/2) 1264B 200 "POST /_matrix/client/unstable/keys/query?access_token=<redacted> HTTP/1.1" "Riot.im/0.8.4..."
```
After some time of waiting I could request the keys without getting into this error


Discussion:
I don't think you have all of the ujson -> simplejson changes in that version. See how many more places have to be changed: https://github.com/matrix-org/synapse/commit/f609acc109af5b981c5662932392b60fa44ec5f0
It might be possible, that merging hotfix to develop might fix this.
But AFAIK simply exchanging the json library should not cause that error... 
0.27.0-rc2 will include the full switch over to simple json, should be out later today. Suggest you give it a try and see if the problem persists for you.
I updated to current develop but the error persists...
I updated the log (as some line numbers have changed) with a newer excerpt
@krombel: I'm very confused. You say:

> I now have tried it again (after some time as you can see in the timestamps).

The timestamps are only 4 seconds apart?

> This one did not fail and I see the following logs:

So the problem is fixed?

The second snipped is from another homeserver - although there is also a "Connection" refused"
I cannot find another try to claim keys from sijmenschoon.nl so it did not fail in second situation

I edited the issue description as I have missed that the server_name is different as well.
BTW: I cannot find another request for sijmenschoon.nl in my logs
I tried now again and I have a different error which might help to find the real issue as they are quite similar:

```
2018-03-27 16:55:32,419 - synapse.http.outbound - 165 - INFO - POST-36492 - {POST-O-90925} [vurpo.fi] Sending request: POST matrix://vurpo.fi/_matrix/federation/v1/user/keys/claim

2018-03-27 16:55:33,186 - synapse.federation.transport.server - 258 - INFO - PUT-36495 - Received txn 1521895195710 from vurpo.fi. (PDUs: 0, EDUs: 1, failures: 0)
2018-03-27 16:55:33,187 - synapse.util.async - 230 - INFO - PUT-36495 - Acquired uncontended linearizer lock 'fed_txn_handler' for key ('vurpo.fi', u'1521895195710')
2018-03-27 16:55:33,194 - synapse.http.matrixfederationclient - 223 - WARNING - POST-36492 - {POST-O-90925} Sending request failed to vurpo.fi: POST matrix://vurpo.fi/_matrix/federation/v1/user/keys/claim: ResponseNeverReceived:[ConnectionDone()]
2018-03-27 16:55:33,197 - synapse.http.outbound - 247 - INFO - POST-36492 - {POST-O-90925} [vurpo.fi] Result: ResponseNeverReceived:[ConnectionDone()]
2018-03-27 16:55:33,202 - synapse.http.server - 191 - ERROR - POST-36492 - Failed handle request synapse.http.server._async_render on <synapse.rest.ClientRestResource object at 0x7f9486a93a10>: <XForwardedForRequest at 0x7f94202d07e8 method=POST uri=/_matrix/client/unstable/keys/claim?access_token=<redacted> clientproto=HTTP/1.1 site=8008>: Traceback (most recent call last):
  File "/srv/synapse/env/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/srv/synapse/env/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1442, in gotResult
    _inlineCallbacks(r, g, deferred)
  File "/srv/synapse/env/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/srv/synapse/env/local/lib/python2.7/site-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
--- <exception caught here> ---
  File "synapse/http/server.py", line 162, in wrapped_request_handler
    yield request_handler(self, request, request_metrics)
  File "/srv/synapse/env/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
    result = g.send(result)
  File "synapse/http/server.py", line 310, in _async_render
    self._send_response(request, code, response)
  File "synapse/http/server.py", line 352, in _send_response
    canonical_json=self.canonical_json,
  File "synapse/http/server.py", line 464, in respond_with_json
    json_bytes = simplejson.dumps(json_object)
  File "/srv/synapse/env/local/lib/python2.7/site-packages/simplejson/__init__.py", line 382, in dumps
    return _default_encoder.encode(obj)
  File "/srv/synapse/env/local/lib/python2.7/site-packages/simplejson/encoder.py", line 291, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/srv/synapse/env/local/lib/python2.7/site-packages/simplejson/encoder.py", line 373, in iterencode
    return _iterencode(o, 0)
  File "/srv/synapse/env/local/lib/python2.7/site-packages/simplejson/encoder.py", line 268, in default
    o.__class__.__name__)
exceptions.TypeError: Object of type Failure is not JSON serializable
2018-03-27 16:55:33,207 - synapse.access.http.8008 - 93 - INFO - POST-36492 - 212.204.110.133 - 8008 - {@krombel:msg-net.de} Processed request: 797ms (11ms, 3ms) (2ms/5ms/2) 55B 500 "POST /_matrix/client/unstable/keys/claim?access_token=<redacted> HTTP/1.1" "Riot.im/0.8.4"
2018-03-27 16:55:32,419 - synapse.http.outbound - 165 - INFO - POST-36492 - {POST-O-90925} [vurpo.fi] Sending request: POST matrix://vurpo.fi/_matrix/federation/v1/user/keys/claim
```
ok this looks like fallout of the switch from ujson to simplejson.

I've basically fixed it in the process of trying to figure out what's going on so I'll PR a fix

Resolving patch:
diff --git a/synapse/handlers/e2e_keys.py b/synapse/handlers/e2e_keys.py
--- a/synapse/handlers/e2e_keys.py
+++ b/synapse/handlers/e2e_keys.py
@@ -1,5 +1,6 @@
 # -*- coding: utf-8 -*-
 # Copyright 2016 OpenMarket Ltd
+# Copyright 2018 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -134,23 +135,8 @@ def do_remote_query(destination):
                     if user_id in destination_query:
                         results[user_id] = keys
 
-            except CodeMessageException as e:
-                failures[destination] = {
-                    "status": e.code, "message": e.message
-                }
-            except NotRetryingDestination as e:
-                failures[destination] = {
-                    "status": 503, "message": "Not ready for retry",
-                }
-            except FederationDeniedError as e:
-                failures[destination] = {
-                    "status": 403, "message": "Federation Denied",
-                }
             except Exception as e:
-                # include ConnectionRefused and other errors
-                failures[destination] = {
-                    "status": 503, "message": e.message
-                }
+                failures[destination] = _exception_to_failure(e)
 
         yield make_deferred_yieldable(defer.gatherResults([
             preserve_fn(do_remote_query)(destination)
@@ -252,19 +238,8 @@ def claim_client_keys(destination):
                 for user_id, keys in remote_result["one_time_keys"].items():
                     if user_id in device_keys:
                         json_result[user_id] = keys
-            except CodeMessageException as e:
-                failures[destination] = {
-                    "status": e.code, "message": e.message
-                }
-            except NotRetryingDestination as e:
-                failures[destination] = {
-                    "status": 503, "message": "Not ready for retry",
-                }
             except Exception as e:
-                # include ConnectionRefused and other errors
-                failures[destination] = {
-                    "status": 503, "message": e.message
-                }
+                failures[destination] = _exception_to_failure(e)
 
         yield make_deferred_yieldable(defer.gatherResults([
             preserve_fn(claim_client_keys)(destination)
@@ -362,6 +337,31 @@ def _upload_one_time_keys_for_user(self, user_id, device_id, time_now,
         )
 
 
+def _exception_to_failure(e):
+    if isinstance(e, CodeMessageException):
+        return {
+            "status": e.code, "message": e.message,
+        }
+
+    if isinstance(e, NotRetryingDestination):
+        return {
+            "status": 503, "message": "Not ready for retry",
+        }
+
+    if isinstance(e, FederationDeniedError):
+        return {
+            "status": 403, "message": "Federation Denied",
+        }
+
+    # include ConnectionRefused and other errors
+    #
+    # Note that some Exceptions (notably twisted's ResponseFailed etc) don't
+    # give a string for e.message, which simplejson then fails to serialize.
+    return {
+        "status": 503, "message": str(e.message),
+    }
+
+
 def _one_time_keys_match(old_key_json, new_key):
     old_key = json.loads(old_key_json)
 


Test patch: