Repository: matrix-org/synapse

Issue:
CI run against Twisted trunk is failing
See https://github.com/matrix-org/synapse/actions/runs/5961068352



Discussion:
```
synapse/util/manhole.py:101: error: Argument 1 to "Portal" has incompatible type "TerminalRealm"; expected "IRealm"  [arg-type]
```
I think renaming this has caused the github action to open #15490 rather than edit this issue.
https://github.com/twisted/twisted/pull/11813 is a PR to twisted that should fix this, currently awaiting another round of review.
If that doesn't land upstream I guess we can `cast()` it ourselves.
This seems to [now be failing with](https://github.com/matrix-org/synapse/actions/runs/5505588001/jobs/10033144632):

```
Invalid git url "git+https://github.com/twisted/twisted.git#"
```
> This seems to [now be failing with](https://github.com/matrix-org/synapse/actions/runs/5505588001/jobs/10033144632):
> 
> ```
> Invalid git url "git+https://github.com/twisted/twisted.git#"
> ```

I would guess this is a screwup of mine here:

https://github.com/matrix-org/synapse/blob/5e82b07d2c79df9623e5567f935e74395c2e5492/.github/workflows/twisted_trunk.yml#L52

see #15302
It seems that the input I added in #15302 is only processed when the workflow is run due to a workflow_dispatch event, i.e. by one of us triggering it manually in the GitHub web UI. 

Instead, inputs seem evaluate to null in the github actions [language](https://evanhalley.dev/post/github-actions-workflow-with-inputs/) and one has to use `|| default_val` to work around this. (See e.g. https://evanhalley.dev/post/github-actions-workflow-with-inputs/)
Now failing linter with:

```
synapse/logging/context.py:846: error: Argument 1 to "addBoth" of "Deferred" has incompatible type "Callable[[ResultT, LoggingContext], ResultT]"; expected "Callable[[Union[R, Failure], LoggingContext], Failure]"  [arg-type]
synapse/logging/context.py:846: note: This is likely because "_set_context_cb" has named arguments: "result". Consider marking them positional-only
synapse/logging/context.py:846: error: Argument 2 to "addBoth" of "Deferred" has incompatible type "Union[LoggingContext, _Sentinel]"; expected "LoggingContext"  [arg-type]
synapse/logging/context.py:873: error: Argument 1 to "addBoth" of "Deferred" has incompatible type "Callable[[ResultT, LoggingContext], ResultT]"; expected "Callable[[Union[T, Failure], LoggingContext], Failure]"  [arg-type]
synapse/logging/context.py:873: note: This is likely because "_set_context_cb" has named arguments: "result". Consider marking them positional-only
synapse/logging/context.py:873: error: Argument 2 to "addBoth" of "Deferred" has incompatible type "Union[LoggingContext, _Sentinel]"; expected "LoggingContext"  [arg-type]
synapse/util/caches/deferred_cache.py:473: error: Argument 1 to "addCallback" of "Deferred" has incompatible type "Callable[[Dict[KT, VT]], Optional[VT]]"; expected "Callable[[Dict[KT, VT]], Failure]"  [arg-type]
synapse/util/caches/deferred_cache.py:473: error: Incompatible return value type (got "Optional[VT]", expected "Failure")  [return-value]
synapse/handlers/initial_sync.py:203: error: Argument 1 to "addCallback" of "Deferred" has incompatible type "Callable[[Dict[str, Mapping[Tuple[str, str], EventBase]]], Mapping[Tuple[str, str], EventBase]]"; expected "Callable[[Dict[str, Mapping[Tuple[str, str], EventBase]]], Failure]"  [arg-type]
synapse/handlers/initial_sync.py:203: error: Redundant cast to "Mapping[Tuple[str, str], EventBase]"  [redundant-cast]
synapse/handlers/initial_sync.py:203: error: Incompatible return value type (got "Mapping[Tuple[str, str], EventBase]", expected "Failure")  [return-value]
synapse/handlers/message.py:1478: error: Need more than 0 values to unpack (2 expected)  [misc]
synapse/handlers/message.py:1480: error: Argument 1 to "gather_results" has incompatible type "Tuple[Deferred[EventBase], Deferred[Optional[Failure]]]"; expected "Tuple[]"  [arg-type]
synapse/handlers/message.py:1496: error: Cannot determine type of "result"  [has-type]
tests/util/test_async_helpers.py:71: error: Argument 1 to "addCallback" of "Deferred" has incompatible type "Callable[[ObservableDeferred[int], int], ObservableDeferred[int]]"; expected "Callable[[int, int], Failure]"  [arg-type]
tests/util/test_async_helpers.py:71: note: This is likely because "check_val" has named arguments: "res". Consider marking them positional-only
tests/util/test_async_helpers.py:72: error: Argument 1 to "addCallback" of "Deferred" has incompatible type "Callable[[ObservableDeferred[int], int], ObservableDeferred[int]]"; expected "Callable[[int, int], Failure]"  [arg-type]
tests/util/test_async_helpers.py:72: note: This is likely because "check_val" has named arguments: "res". Consider marking them positional-only
tests/util/test_async_helpers.py:102: error: Argument 1 to "addErrback" of "Deferred" has incompatible type "Callable[[ObservableDeferred[str], int], None]"; expected "Callable[[Failure, int], Deferred[<nothing>]]"  [arg-type]
tests/util/test_async_helpers.py:102: note: This is likely because "check_val" has named arguments: "res". Consider marking them positional-only
tests/util/test_async_helpers.py:103: error: Argument 1 to "addErrback" of "Deferred" has incompatible type "Callable[[ObservableDeferred[str], int], None]"; expected "Callable[[Failure, int], Deferred[<nothing>]]"  [arg-type]
tests/util/test_async_helpers.py:103: note: This is likely because "check_val" has named arguments: "res". Consider marking them positional-only
```

Resolving patch:
diff --git a/synapse/handlers/message.py b/synapse/handlers/message.py
--- a/synapse/handlers/message.py
+++ b/synapse/handlers/message.py
@@ -1474,23 +1474,23 @@ async def handle_new_client_event(
 
         # We now persist the event (and update the cache in parallel, since we
         # don't want to block on it).
-        event, context = events_and_context[0]
+        #
+        # Note: mypy gets confused if we inline dl and check with twisted#11770.
+        # Some kind of bug in mypy's deduction?
+        deferreds = (
+            run_in_background(
+                self._persist_events,
+                requester=requester,
+                events_and_context=events_and_context,
+                ratelimit=ratelimit,
+                extra_users=extra_users,
+            ),
+            run_in_background(
+                self.cache_joined_hosts_for_events, events_and_context
+            ).addErrback(log_failure, "cache_joined_hosts_for_event failed"),
+        )
         result, _ = await make_deferred_yieldable(
-            gather_results(
-                (
-                    run_in_background(
-                        self._persist_events,
-                        requester=requester,
-                        events_and_context=events_and_context,
-                        ratelimit=ratelimit,
-                        extra_users=extra_users,
-                    ),
-                    run_in_background(
-                        self.cache_joined_hosts_for_events, events_and_context
-                    ).addErrback(log_failure, "cache_joined_hosts_for_event failed"),
-                ),
-                consumeErrors=True,
-            )
+            gather_results(deferreds, consumeErrors=True)
         ).addErrback(unwrapFirstError)
 
         return result
diff --git a/synapse/logging/context.py b/synapse/logging/context.py
--- a/synapse/logging/context.py
+++ b/synapse/logging/context.py
@@ -809,23 +809,24 @@ def run_in_background(  # type: ignore[misc]
 
     # `res` may be a coroutine, `Deferred`, some other kind of awaitable, or a plain
     # value. Convert it to a `Deferred`.
+    d: "defer.Deferred[R]"
     if isinstance(res, typing.Coroutine):
         # Wrap the coroutine in a `Deferred`.
-        res = defer.ensureDeferred(res)
+        d = defer.ensureDeferred(res)
     elif isinstance(res, defer.Deferred):
-        pass
+        d = res
     elif isinstance(res, Awaitable):
         # `res` is probably some kind of completed awaitable, such as a `DoneAwaitable`
         # or `Future` from `make_awaitable`.
-        res = defer.ensureDeferred(_unwrap_awaitable(res))
+        d = defer.ensureDeferred(_unwrap_awaitable(res))
     else:
         # `res` is a plain value. Wrap it in a `Deferred`.
-        res = defer.succeed(res)
+        d = defer.succeed(res)
 
-    if res.called and not res.paused:
+    if d.called and not d.paused:
         # The function should have maintained the logcontext, so we can
         # optimise out the messing about
-        return res
+        return d
 
     # The function may have reset the context before returning, so
     # we need to restore it now.
@@ -843,8 +844,8 @@ def run_in_background(  # type: ignore[misc]
     # which is supposed to have a single entry and exit point. But
     # by spawning off another deferred, we are effectively
     # adding a new exit point.)
-    res.addBoth(_set_context_cb, ctx)
-    return res
+    d.addBoth(_set_context_cb, ctx)
+    return d
 
 
 T = TypeVar("T")
@@ -877,7 +878,7 @@ def make_deferred_yieldable(deferred: "defer.Deferred[T]") -> "defer.Deferred[T]
 ResultT = TypeVar("ResultT")
 
 
-def _set_context_cb(result: ResultT, context: LoggingContext) -> ResultT:
+def _set_context_cb(result: ResultT, context: LoggingContextOrSentinel) -> ResultT:
     """A callback function which just sets the logging context"""
     set_current_context(context)
     return result
diff --git a/synapse/util/caches/deferred_cache.py b/synapse/util/caches/deferred_cache.py
--- a/synapse/util/caches/deferred_cache.py
+++ b/synapse/util/caches/deferred_cache.py
@@ -470,7 +470,7 @@ def __init__(self) -> None:
     def deferred(self, key: KT) -> "defer.Deferred[VT]":
         if not self._deferred:
             self._deferred = ObservableDeferred(defer.Deferred(), consumeErrors=True)
-        return self._deferred.observe().addCallback(lambda res: res.get(key))
+        return self._deferred.observe().addCallback(lambda res: res[key])
 
     def add_invalidation_callback(
         self, key: KT, callback: Optional[Callable[[], None]]


Test patch:
diff --git a/tests/util/test_async_helpers.py b/tests/util/test_async_helpers.py
--- a/tests/util/test_async_helpers.py
+++ b/tests/util/test_async_helpers.py
@@ -60,11 +60,9 @@ def check_called_first(res: int) -> int:
         observer1.addBoth(check_called_first)
 
         # store the results
-        results: List[Optional[ObservableDeferred[int]]] = [None, None]
+        results: List[Optional[int]] = [None, None]
 
-        def check_val(
-            res: ObservableDeferred[int], idx: int
-        ) -> ObservableDeferred[int]:
+        def check_val(res: int, idx: int) -> int:
             results[idx] = res
             return res
 
@@ -93,14 +91,14 @@ def check_called_first(res: int) -> int:
         observer1.addBoth(check_called_first)
 
         # store the results
-        results: List[Optional[ObservableDeferred[str]]] = [None, None]
+        results: List[Optional[Failure]] = [None, None]
 
-        def check_val(res: ObservableDeferred[str], idx: int) -> None:
+        def check_failure(res: Failure, idx: int) -> None:
             results[idx] = res
             return None
 
-        observer1.addErrback(check_val, 0)
-        observer2.addErrback(check_val, 1)
+        observer1.addErrback(check_failure, 0)
+        observer2.addErrback(check_failure, 1)
 
         try:
             raise Exception("gah!")