Repository: hylang/hy

Issue:
Support Python 3.12's new features for type parameters
See [PEP 695](https://peps.python.org/pep-0695/).


Discussion:
[At the AST level](https://peps.python.org/pep-0695/#ast-changes), there are three changes we need to support:

- The `type_params` attribute of `FunctionDef` (Hy core macros `defn`, `defn/a`, `fn`, `fn/a`)
- The `type_params` attribute of `ClassDef` (Hy core macro `defclass`)
- The node type `TypeAlias` (probably requires a new Hy core macro)

I propose the following syntax:

```
(defn [Decorator1 Decorator2] #^ int fun-name (tp T1 T2) [p1 p2 …]
  body…)
(fn #^ int (tp T1 T2) [p1 p2 …]
  body…)
(defclass [Decorator1 Decorator2] ClassName (tp T1 T2) [Base1 Base2]
  body…)
(deftype AliasName (tp T1 T2) rvalue)
```

A more complex `tp` form would look like:

```
(tp A B #* C #** D #* E F)
```

I chose the name `tp` to be short yet mnemonic; it stands for "type parameter". Other options in place of `(tp T1 T2)` include `(T1 T2)`, `#(T1 T2)`, and `{T1 T2}`, but a more legible syntax for an obscure feature like this one seems appropriate to me. We don't actually need a macro named `tp`, although we do need a macro `deftype`. We probably don't want to name that macro `type` because then it would shadow the built-in Python function `type`.

Any objections?
Hmm, if we use a macro form, I think we should use a more verbose name than `tp`, since it fits with names such as `annotate` and `unpack-iterable`; then (like `annotate`) we could have a reader macro for convenience if it makes sense.

I'm not totally clear how your `tp` examples translate to Python. Do your examples include type constraints? Or would those be handled using  `annotate`/`#^` within the `tp` form?

I also think the name being parametrized should be inside the macro form as well, since conceptually we're parametrizing that type name (c.f. `annotate`, `of`).
You keep saying "the macro form", but I'm saying that `tp` doesn't need to be a macro. It would only be recognized in these specific positions in calls to certain core macros, where no macro calls are allowed. That's why it might as well be short: it won't pollute any namespaces.

> I'm not totally clear how your `tp` examples translate to Python.

Like this:

```
@Decorator1
@Decorator2
def fun_name[T1, T2](p1, p2, …) -> int:
   body…

@Decorator1
@Decorator2
class ClassName[T1, T2](Base1, Base2):
    body…

type AliasName[T1, T2] = rvalue
```

> Do your examples include type constraints?

No.

> Or would those be handled using `annotate`/`#^` within the `tp` form?

Right, that should work without any explicit support. I think.

> I also think the name being parametrized should be inside the macro form as well, since conceptually we're parametrizing that type name (c.f. `annotate`, `of`).

You mean like this?

```
(defn [Decorator1 Decorator2] #^ int (tp fun-name T1 T2) [p1 p2 …]
  body…)
```

To me it seems both less visually clear and harder to write a model pattern for (because it will compete with the nesting of `annotate`), but I can probably do it if you're sure you want it.
Ahh ok, I understand what you mean now about not making it a macro form. Yeah in that case we don't need the name inside the form as well.
So if it's a form that's only found in these specific cases, then maybe we could make it a keyword? Kind of like how we have keyword parameters for `require`:
```hy
(defn [Dec1 Dec2] :tp [T1 T2] #^ int fun-name [p1 p2]
  body...)
```
This also lets us put the type parameters before the return type annotation for functions, which makes things like this make more sense:
```hy
;; def foo[T](...) -> T: ...
(defn :tp [T] #^ T foo [...] ...)
```

Correspondingly,
```hy
(fn :tp [T1 T2] #^ int [p1 p2 …]
  body…)
(defclass [Decorator1 Decorator2] :tp [T1 T2] ClassName [Base1 Base2]
  body…)
(deftype :tp [T1 T2] AliasName rvalue)
```
That seems reasonable.
I'm working on this. It's just been slow going.

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -68,6 +68,7 @@
 intersphinx_mapping = dict(
     py=("https://docs.python.org/3/", None),
     py3_10=("https://docs.python.org/3.10/", None),
+    py3_12=("https://docs.python.org/3.12/", None),
     hyrule=("https://hyrule.readthedocs.io/en/master/", None),
 )
 
diff --git a/hy/core/result_macros.py b/hy/core/result_macros.py
--- a/hy/core/result_macros.py
+++ b/hy/core/result_macros.py
@@ -79,6 +79,30 @@ def dotted(name):
     return Expression(map(Symbol, [".", *name.split(".")]))
 
 
+type_params = sym(":tp") + brackets(many(
+    maybe_annotated(SYM) | unpack("either", Symbol)))
+
+def digest_type_params(compiler, tp):
+    "Return a `type_params` attribute for `FunctionDef` etc."
+
+    if tp:
+        if not PY3_12:
+            compiler._syntax_error(tp, "`:tp` requires Python 3.12 or later")
+        tp, = tp
+    elif not PY3_12:
+        return {}
+
+    return dict(type_params = [
+        asty.TypeVarTuple(x[1], name = mangle(x[1]))
+            if is_unpack("iterable", x) else
+        asty.ParamSpec(x[1], name = mangle(x[1]))
+            if is_unpack("mapping", x) else
+        asty.TypeVar(x[0],
+               name = mangle(x[0]),
+               bound = x[1] and compiler.compile(x[1]).force_expr)
+        for x in (tp or [])])
+
+
 # ------------------------------------------------
 # * Fundamentals
 # ------------------------------------------------
@@ -1410,8 +1434,9 @@ def compile_try_expression(compiler, expr, root, body, catchers, orelse, finalbo
 )
 
 
-@pattern_macro(["fn", "fn/a"], [maybe_annotated(lambda_list), many(FORM)])
-def compile_function_lambda(compiler, expr, root, params, body):
+@pattern_macro(["fn", "fn/a"],
+    [maybe(type_params), maybe_annotated(lambda_list), many(FORM)])
+def compile_function_lambda(compiler, expr, root, tp, params, body):
     params, returns = params
     posonly, args, rest, kwonly, kwargs = params
     has_annotations = returns is not None or any(
@@ -1423,13 +1448,13 @@ def compile_function_lambda(compiler, expr, root, params, body):
         body = compiler._compile_branch(body)
 
     # Compile to lambda if we can
-    if not has_annotations and not body.stmts and root != "fn/a":
+    if not (has_annotations or tp or body.stmts or root == "fn/a"):
         return ret + asty.Lambda(expr, args=args, body=body.force_expr)
 
     # Otherwise create a standard function
     node = asty.AsyncFunctionDef if root == "fn/a" else asty.FunctionDef
     name = compiler.get_anon_var()
-    ret += compile_function_node(compiler, expr, node, [], name, args, returns, body)
+    ret += compile_function_node(compiler, expr, node, [], tp, name, args, returns, body)
 
     # return its name as the final expr
     return ret + Result(expr=ret.temp_variables[0])
@@ -1437,9 +1462,9 @@ def compile_function_lambda(compiler, expr, root, params, body):
 
 @pattern_macro(
     ["defn", "defn/a"],
-    [maybe(brackets(many(FORM))), maybe_annotated(SYM), lambda_list, many(FORM)],
+    [maybe(brackets(many(FORM))), maybe(type_params), maybe_annotated(SYM), lambda_list, many(FORM)],
 )
-def compile_function_def(compiler, expr, root, decorators, name, params, body):
+def compile_function_def(compiler, expr, root, decorators, tp, name, params, body):
     name, returns = name
     node = asty.FunctionDef if root == "defn" else asty.AsyncFunctionDef
     decorators, ret, _ = compiler._compile_collect(decorators[0] if decorators else [])
@@ -1451,11 +1476,11 @@ def compile_function_def(compiler, expr, root, decorators, name, params, body):
         body = compiler._compile_branch(body)
 
     return ret + compile_function_node(
-        compiler, expr, node, decorators, name, args, returns, body
+        compiler, expr, node, decorators, tp, name, args, returns, body
     )
 
 
-def compile_function_node(compiler, expr, node, decorators, name, args, returns, body):
+def compile_function_node(compiler, expr, node, decorators, tp, name, args, returns, body):
     ret = Result()
 
     if body.expr:
@@ -1468,7 +1493,7 @@ def compile_function_node(compiler, expr, node, decorators, name, args, returns,
         body=body.stmts or [asty.Pass(expr)],
         decorator_list=decorators,
         returns=compiler.compile(returns).force_expr if returns is not None else None,
-        **({"type_params": []} if PY3_12 else {}),
+        **digest_type_params(compiler, tp),
     )
 
     ast_name = asty.Name(expr, id=name, ctx=ast.Load())
@@ -1648,11 +1673,12 @@ def compile_yield_from_or_await_expression(compiler, expr, root, arg):
     "defclass",
     [
         maybe(brackets(many(FORM))),
+        maybe(type_params),
         SYM,
         maybe(brackets(many(FORM)) + maybe(STR) + many(FORM)),
     ],
 )
-def compile_class_expression(compiler, expr, root, decorators, name, rest):
+def compile_class_expression(compiler, expr, root, decorators, tp, name, rest):
     base_list, docstring, body = rest or ([[]], None, [])
 
     decorators, ret, _ = compiler._compile_collect(decorators[0] if decorators else [])
@@ -1682,7 +1708,7 @@ def compile_class_expression(compiler, expr, root, decorators, name, rest):
         kwargs=None,
         bases=bases_expr,
         body=bodyr.stmts or [asty.Pass(expr)],
-        **({"type_params": []} if PY3_12 else {}),
+        **digest_type_params(compiler, tp)
     )
 
 
@@ -1926,6 +1952,14 @@ def compile_let(compiler, expr, root, bindings, body):
         return res + compiler.compile(mkexpr("do", *body).replace(expr))
 
 
+@pattern_macro(((3, 12), "deftype"), [maybe(type_params), SYM, FORM])
+def compile_deftype(compiler, expr, root, tp, name, value):
+    return asty.TypeAlias(expr,
+       name = asty.Name(name, id = mangle(name), ctx = ast.Store()),
+       value = compiler.compile(value).force_expr,
+        **digest_type_params(compiler, tp))
+
+
 @pattern_macro(
     "pragma unquote unquote-splice unpack-mapping except except* finally else".split(),
     [many(FORM)],
diff --git a/hy/model_patterns.py b/hy/model_patterns.py
--- a/hy/model_patterns.py
+++ b/hy/model_patterns.py
@@ -103,13 +103,15 @@ def notpexpr(*disallowed_heads):
     )
 
 
-def unpack(kind):
+def unpack(kind, content_type = None):
     "Parse an unpacking form, returning it unchanged."
-    return some(
-        lambda x: isinstance(x, Expression)
-        and len(x) > 0
-        and x[0] == Symbol("unpack-" + kind)
-    )
+    return some(lambda x:
+        isinstance(x, Expression) and
+        len(x) > 0 and
+        x[0] in [Symbol("unpack-" + tail) for tail in
+            (["iterable", "mapping"] if kind == "either" else [kind])] and
+        (content_type is None or
+            (len(x) == 2 and isinstance(x[1], content_type))))
 
 
 def times(lo, hi, parser):


Test patch:
diff --git a/tests/native_tests/defclass.hy b/tests/native_tests/defclass.hy
--- a/tests/native_tests/defclass.hy
+++ b/tests/native_tests/defclass.hy
@@ -148,3 +148,11 @@
 
   (defclass Quest [QuestBase :swallow "african"])
   (assert (= (. (Quest) swallow) "african")))
+
+
+(do-mac (when hy._compat.PY3_12 '(defn test-type-params []
+  (import tests.resources.tp :as ttp)
+  (defclass :tp [#^ int A  #** B] C)
+  (assert (= (ttp.show C) [
+    [ttp.TypeVar "A" int #()]
+    [ttp.ParamSpec "B" None #()]])))))
diff --git a/tests/native_tests/deftype.hy b/tests/native_tests/deftype.hy
new file mode 100644
--- /dev/null
+++ b/tests/native_tests/deftype.hy
@@ -0,0 +1,21 @@
+(do-mac (when hy._compat.PY3_12 '(do
+
+(import tests.resources.tp :as ttp)
+
+
+(defn test-deftype []
+
+  (deftype Foo int)
+  (assert (is (type Foo) ttp.TypeAliasType))
+  (assert (= Foo.__value__) int)
+
+  (deftype Foo (| int bool))
+  (assert (is (type Foo.__value__ hy.M.types.UnionType)))
+
+  (deftype :tp [#^ int A  #** B] Foo int)
+  (assert (= (ttp.show Foo) [
+    [ttp.TypeVar "A" int #()]
+    [ttp.ParamSpec "B" None #()]])))))
+
+
+)
diff --git a/tests/native_tests/functions.hy b/tests/native_tests/functions.hy
--- a/tests/native_tests/functions.hy
+++ b/tests/native_tests/functions.hy
@@ -96,6 +96,55 @@
     (assert (= (. f __annotations__ [k]) v))))
 
 
+(do-mac (when hy._compat.PY3_12 '(defn test-type-params []
+  (import tests.resources.tp :as ttp)
+
+  (defn foo [])
+  (assert (= (ttp.show foo) []))
+
+  ; `defn` with a type parameter
+  (defn :tp [T] #^ T foo [#^ T x]
+    (+ x 1))
+  (assert (= (foo 3) 4))
+  (assert (= (ttp.show foo) [[ttp.TypeVar "T" None #()]]))
+
+  ; `fn` with a type parameter
+  (setv foo (fn :tp [T] #^ T [#^ T x]
+    (+ x 2)))
+  (assert (= (foo 3) 5))
+  (assert (= (ttp.show foo) [[ttp.TypeVar "T" None #()]]))
+
+  ; Bounds and constraints
+  (defn :tp [#^ int T] foo [])
+  (assert (= (ttp.show foo) [[ttp.TypeVar "T" int #()]]))
+  (defn :tp [#^ #(int str) T] foo [])
+  (assert (= (ttp.show foo) [[ttp.TypeVar "T" None #(int str)]]))
+
+  ; `TypeVarTuple`s and `ParamSpec`s
+  (defn :tp [#* T] foo [])
+  (assert (= (ttp.show foo) [[ttp.TypeVarTuple "T" None #()]]))
+  (defn :tp [#** T] foo [])
+  (assert (= (ttp.show foo) [[ttp.ParamSpec "T" None #()]]))
+
+  ; A more complex case
+  (defn
+    :tp [A  #^ int B  #* C  #** D  #* E  #^ #(bool float) F]
+    foo [])
+  (assert (= (ttp.show foo) [
+    [ttp.TypeVar "A" None #()]
+    [ttp.TypeVar "B" int #()]
+    [ttp.TypeVarTuple "C" None #()]
+    [ttp.ParamSpec "D" None #()]
+    [ttp.TypeVarTuple "E" None #()]
+    [ttp.TypeVar "F" None #(bool float)]]))
+
+  ; Illegal attempts to annotate unpacking
+  (with [(pytest.raises hy.errors.HySyntaxError)]
+    (hy.eval '(defn :tp [#^ int #* T] foo [])))
+  (with [(pytest.raises hy.errors.HySyntaxError)]
+    (hy.eval '(defn :tp [#^ #(int str) #** T] foo []))))))
+
+
 (defn test-lambda-keyword-lists []
   (defn foo [x #* xs #** kw] [x xs kw])
   (assert (= (foo 10 20 30) [10 #(20 30) {}])))
diff --git a/tests/resources/tp.hy b/tests/resources/tp.hy
new file mode 100644
--- /dev/null
+++ b/tests/resources/tp.hy
@@ -0,0 +1,10 @@
+"Helpers for testing type parameters."
+
+(import typing [TypeVar TypeVarTuple ParamSpec TypeAliasType])
+
+(defn show [f]
+  (lfor
+    tp f.__type_params__
+    [(type tp) tp.__name__
+      (getattr tp "__bound__" None)
+      (getattr tp "__constraints__" #())]))