Repository: matrix-org/synapse

Issue:
`_do_execute` (database function) could be more precisely typed with better support for `ParamSpec`
Sprouted from #12311.

Once Mypy supports Concatenate, it could be written as
```python
     def _do_execute(
         self,
         func: Callable[Concatenate[str, P], R],
         sql: str,
         *args: P.args,
         **kwargs: P.kwargs,
     ) -> R:
```
so long as we also pass kwargs.


Discussion:
> Once Mypy supports Concatenate

Which seems to be python/mypy#11847, according to @squahtx 
related: #11711
Blocked on #12585
I've just given this a quick try.

I was a bit surprised that we need to include `**kwargs: P.kwargs` in the outer function definition, but that does seem to be consistent with PEP 612:

> A function of type Callable[P, R] can be called with (*args, **kwargs) if and only if args has the type P.args and kwargs has the type P.kwargs, and that those types both originated from the same function declaration.

I was then surprised by the next error from looking at `args[0]`:

> error: Value of type "P.args" is not indexable  [index]

https://github.com/matrix-org/synapse/blob/a50fb411b366576c3d4796b04c45385bdb7a6897/synapse/storage/database.py#L341-L352

This is again consistent with PEP 612:

> Inside the function, args has the type P.args, not Tuple[P.args, ...] as would be with a normal annotation (and likewise with the **kwargs)

Given that this is already caught at runtime, I'd be inclined to type-ignore this.
Additionally: This doesn't actually give us a huge amount of type safety. Using a ParamSpec allows us to ensure that the `args` (and now `kwargs)` we pass to `_do_ execute` are suitable for `func`. In order for this to actually enforce anything, we need meaningful annotations where `func` is defined and of the arguments where they're passed.

There are four call sites at present, and none of them really have those meaningful annotations AFAICS.

## execute and executemany

https://github.com/matrix-org/synapse/blob/a50fb411b366576c3d4796b04c45385bdb7a6897/synapse/storage/database.py#L332-L336

`*args: Any` means that we don't get checking here, even though `self.txn.execute(many)` have annotations.

## execute_batch

https://github.com/matrix-org/synapse/blob/a50fb411b366576c3d4796b04c45385bdb7a6897/synapse/storage/database.py#L299-L310

Here the lambda expression has no annotations. Proof by example:
```
$ cat temp.py
def accepts_string(s: str): print(s)
f = lambda x: accepts_string(x)
reveal_type(f)

2022-05-07 21:38:24 ✔  $ mypy temp.py
temp.py:3: note: Revealed type is "def (x: Any) -> Any"
Success: no issues found in 1 source file
```

It's also worth noting that the stubs for `execute_batch` aren't very strict either:

```
# from reveal_type(execute_batch)
Revealed type is "def (cur: Any, sql: Any, argslist: Any, page_size: builtins.int =)"
```

This is using the locked version of types-psycopg2, but it's no stricter in the latest release (2.9.13).

# execute_values

Very similar to execute_batch.

https://github.com/matrix-org/synapse/blob/a50fb411b366576c3d4796b04c45385bdb7a6897/synapse/storage/database.py#L315-L330

The underlying postgres function has a few more annotations, but not any that would give us more checking.

```
"def (cur: Any, sql: Any, argslist: Any, template: Union[Any, None] =, page_size: builtins.int =, fetch: builtins.bool =) -> Any
```
```
> I was then surprised by the next error from looking at `args[0]`:
> 
> > error: Value of type "P.args" is not indexable  [index]

The next version of mypy will include a fix for this: https://github.com/python/mypy/pull/12668 
> The next version of mypy will include a fix for this: [python/mypy#12668](https://github.com/python/mypy/pull/12668)

Good to know. I found https://github.com/python/mypy/issues/11826 by searching, but assumed that the behaviour there was intentional.

Resolving patch:
diff --git a/synapse/storage/database.py b/synapse/storage/database.py
--- a/synapse/storage/database.py
+++ b/synapse/storage/database.py
@@ -38,7 +38,7 @@
 
 import attr
 from prometheus_client import Histogram
-from typing_extensions import Literal
+from typing_extensions import Concatenate, Literal, ParamSpec
 
 from twisted.enterprise import adbapi
 
@@ -194,7 +194,7 @@ def __getattr__(self, name):
 # The type of entry which goes on our after_callbacks and exception_callbacks lists.
 _CallbackListEntry = Tuple[Callable[..., object], Iterable[Any], Dict[str, Any]]
 
-
+P = ParamSpec("P")
 R = TypeVar("R")
 
 
@@ -339,7 +339,13 @@ def _make_sql_one_line(self, sql: str) -> str:
         "Strip newlines out of SQL so that the loggers in the DB are on one line"
         return " ".join(line.strip() for line in sql.splitlines() if line.strip())
 
-    def _do_execute(self, func: Callable[..., R], sql: str, *args: Any) -> R:
+    def _do_execute(
+        self,
+        func: Callable[Concatenate[str, P], R],
+        sql: str,
+        *args: P.args,
+        **kwargs: P.kwargs,
+    ) -> R:
         sql = self._make_sql_one_line(sql)
 
         # TODO(paul): Maybe use 'info' and 'debug' for values?
@@ -348,7 +354,10 @@ def _do_execute(self, func: Callable[..., R], sql: str, *args: Any) -> R:
         sql = self.database_engine.convert_param_style(sql)
         if args:
             try:
-                sql_logger.debug("[SQL values] {%s} %r", self.name, args[0])
+                # The type-ignore should be redundant once mypy releases a version with
+                # https://github.com/python/mypy/pull/12668. (`args` might be empty,
+                # (but we'll catch the index error if so.)
+                sql_logger.debug("[SQL values] {%s} %r", self.name, args[0])  # type: ignore[index]
             except Exception:
                 # Don't let logging failures stop SQL from working
                 pass
@@ -363,7 +372,7 @@ def _do_execute(self, func: Callable[..., R], sql: str, *args: Any) -> R:
                     opentracing.tags.DATABASE_STATEMENT: sql,
                 },
             ):
-                return func(sql, *args)
+                return func(sql, *args, **kwargs)
         except Exception as e:
             sql_logger.debug("[SQL FAIL] {%s} %s", self.name, e)
             raise


Test patch: