Repository: matrix-org/synapse

Issue:
crypto.keyring: TypeError '<' not supported between instances of 'NoneType' and 'int'
### Description

From the logs:

```
2019-06-07_11:31:30.41626     return g.throw(self.type, self.value, self.tb)
2019-06-07_11:31:30.41626   File "/var/local/matrix/venv/lib/python3.7/site-packages/synapse/crypto/keyring.py", line 350, in do_iterations
2019-06-07_11:31:30.41626     yield self._attempt_key_fetches_with_fetcher(f, remaining_requests)
2019-06-07_11:31:30.41627   File "/var/local/matrix/venv/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
2019-06-07_11:31:30.41627     result = g.send(result)
2019-06-07_11:31:30.41627   File "/var/local/matrix/venv/lib/python3.7/site-packages/synapse/crypto/keyring.py", line 425, in _attempt_key_fetches_with_fetcher
2019-06-07_11:31:30.41627     < verify_request.minimum_valid_until_ts
2019-06-07_11:31:30.41628 TypeError: '<' not supported between instances of 'NoneType' and 'int'
```


### Steps to reproduce

- Start matrix 1.0.0rc1
- Check the logs
- Sometimes this will show up

### Version information

- **Version**: 1.0.0rc1

- **Install method**: pip

### Patch

It looks like issue was introduced here:
https://github.com/matrix-org/synapse/blob/8dc1eb66972a2d2aa98d5c852f667a0854932980/synapse/crypto/keyring.py#L419

Where the expectation is: "either `fetch_key_result = result_keys.get(key_id)` results in a `None` var, or in real values  and it is not being the case, with `fetch_key_result.valid_until_ts` being `None`

Either Fetcher is to be fixed or L419 should check that *both* `fetch_key_result` and `fetch_key_result.valid_until_ts` have a valid value.

FWIW, the second option can be: replacing L419 with
`if not (fetch_key_result and fetch_key_result.valid_until_ts):`

Without knowing the details of Fetcher it's hard for me to know which way a fix should go.


Discussion:
I was encountering many of the same TypeError exceptions after upgrading to 1.0.0rc1. I then applied the above patch and after restarting those exceptions are gone. So the patch seems to work for me.

This seems most likely due to `ts_valid_until_ms` being null in the database. I think the correct thing here would be to treat it as `0`, i.e. if we set `min_valid_until_ms` then we try and refetch, otherwise we just accept it.

Resolving patch:
diff --git a/synapse/storage/keys.py b/synapse/storage/keys.py
--- a/synapse/storage/keys.py
+++ b/synapse/storage/keys.py
@@ -80,6 +80,14 @@ def _get_keys(txn, batch):
 
             for row in txn:
                 server_name, key_id, key_bytes, ts_valid_until_ms = row
+
+                if ts_valid_until_ms is None:
+                    # Old keys may be stored with a ts_valid_until_ms of null,
+                    # in which case we treat this as if it was set to `0`, i.e.
+                    # it won't match key requests that define a minimum
+                    # `ts_valid_until_ms`.
+                    ts_valid_until_ms = 0
+
                 res = FetchKeyResult(
                     verify_key=decode_verify_key_bytes(key_id, bytes(key_bytes)),
                     valid_until_ts=ts_valid_until_ms,


Test patch:
diff --git a/tests/crypto/test_keyring.py b/tests/crypto/test_keyring.py
--- a/tests/crypto/test_keyring.py
+++ b/tests/crypto/test_keyring.py
@@ -25,7 +25,11 @@
 
 from synapse.api.errors import SynapseError
 from synapse.crypto import keyring
-from synapse.crypto.keyring import PerspectivesKeyFetcher, ServerKeyFetcher
+from synapse.crypto.keyring import (
+    PerspectivesKeyFetcher,
+    ServerKeyFetcher,
+    StoreKeyFetcher,
+)
 from synapse.storage.keys import FetchKeyResult
 from synapse.util import logcontext
 from synapse.util.logcontext import LoggingContext
@@ -219,6 +223,50 @@ def test_verify_json_for_server(self):
         # self.assertFalse(d.called)
         self.get_success(d)
 
+    def test_verify_json_for_server_with_null_valid_until_ms(self):
+        """Tests that we correctly handle key requests for keys we've stored
+        with a null `ts_valid_until_ms`
+        """
+        mock_fetcher = keyring.KeyFetcher()
+        mock_fetcher.get_keys = Mock(return_value=defer.succeed({}))
+
+        kr = keyring.Keyring(
+            self.hs, key_fetchers=(StoreKeyFetcher(self.hs), mock_fetcher)
+        )
+
+        key1 = signedjson.key.generate_signing_key(1)
+        r = self.hs.datastore.store_server_verify_keys(
+            "server9",
+            time.time() * 1000,
+            [("server9", get_key_id(key1), FetchKeyResult(get_verify_key(key1), None))],
+        )
+        self.get_success(r)
+
+        json1 = {}
+        signedjson.sign.sign_json(json1, "server9", key1)
+
+        # should fail immediately on an unsigned object
+        d = _verify_json_for_server(kr, "server9", {}, 0, "test unsigned")
+        self.failureResultOf(d, SynapseError)
+
+        # should fail on a signed object with a non-zero minimum_valid_until_ms,
+        # as it tries to refetch the keys and fails.
+        d = _verify_json_for_server(
+            kr, "server9", json1, 500, "test signed non-zero min"
+        )
+        self.get_failure(d, SynapseError)
+
+        # We expect the keyring tried to refetch the key once.
+        mock_fetcher.get_keys.assert_called_once_with(
+            {"server9": {get_key_id(key1): 500}}
+        )
+
+        # should succeed on a signed object with a 0 minimum_valid_until_ms
+        d = _verify_json_for_server(
+            kr, "server9", json1, 0, "test signed with zero min"
+        )
+        self.get_success(d)
+
     def test_verify_json_dedupes_key_requests(self):
         """Two requests for the same key should be deduped."""
         key1 = signedjson.key.generate_signing_key(1)