Repository: cython/cython

Issue:
Classes lack __annotations__ (PEP-526)
With python 3.7 dataclasses were introduced.
They work just fine in Cython except for their standard __init__ function.

How to reproduce:
test.pyx
```
from dataclasses import dataclass

@dataclass
class TestClass:
	x: int
	y: int
	some_string: str
```
This file compiles to Cython code just fine.
However, when ran from the python it raises an error:
test2.py
```
import test

test_class = test.TestClass(1, 2, 'string')
```
This raises the following:
```
Traceback (most recent call last):
  File ".\test2.py", line 3, in <module>
    test_class = test.TestClass(1, 2, 'string')
TypeError: __init__() takes 1 positional argument but 4 were given
```

if you rename test.pyx to test.py and do not compile it, no such error occurs.

my current workaround:
setting each property after initialization separately:
test.pyx remains unchanged.
test.py
```
import test

test_class = test.TestClass()
test_class.x = 1
test_class.y = 2
test_class.some_string = 'string'
```






Discussion:
I don't think we currently create a `__annotations__` for classes, although the syntax tree has the necessary information available. See the description in
https://www.python.org/dev/peps/pep-0526/#runtime-effects-of-type-annotations

PR welcome.
I'm also stumbling over this currently, especially Python 3.6-style NamedTuples not working.

I don't have any experience with how Cython works under the hood, but I'd be willing to give fixing this a try if someone could point me where to start. Is there some relevant documentation somewhere?
Having similliar issue with dataclasses:
```
@dataclass
class Test:
    name: str
```
When using regular dataclass when I import the file with the dataclass, I get:
```
TypeError: 'name' is a field but has no type annotation
```

I guess that's because cython does not creation annotations for classes.
As always, the most important bit are tests. I would add new ones in a file `tests/run/pyclass_annotations_pep526.py` (with a `# tag: pure36` to prevent Py<3.6 from running in uncompiled). Look at the other `pyclass` tests to see how it's done, we basically use doctests (which are executed in Python and not compiled, as opposed to the rest of the file).

I first thought that this could be implemented after executing the class body, but it turns out that [PEP 526](https://www.python.org/dev/peps/pep-0526/#runtime-effects-of-type-annotations) wants the `__annotations__` dict to be available before that, so creating the empty dict and assigning it in `__Pyx_Py3MetaclassPrepare()` would be the right way.

Then each `NameNode` inside of the class body (`entry.is_pyclass_attr`) which holds a `self.annotation` will have to generate C code (in `generate_assignment_code()`) to look up the `__annotations__` dict in the class scope and assign its name to its annotation `.py_result()` on an assignment (or creation of the name). I don't actually know if that happens before or after the assignment to the name (which might fail, but could be shadowed by a try-except). Probably something to test in CPython. Thus: tests.
Also note that there are probably relevant tests for PEP-526 in CPython's own test suite, which could be reused.
I unfortunately didn't have much time these past few weeks to look at this. I've come up with a couple of test cases, however, that should get us pretty far on the way to full support of `__annotations__`. You can check them out [here](https://github.com/ra-kete/cython/blob/pyclass-annotations-pep526/tests/run/pyclass_annotations_pep526.py).

I have two main questions now:
1. Do we need full compliance with CPython or would you also consider merging something incomplete if it enables the (correct) use of dataclasses and NamedTuples? I think it should be pretty straightforward to add the annotations to the `__annotations__` dict (passing test case `Annotations` above), which should be enough for the aforementioned use cases. I'm not sure how hard it would be to satisfy some of the other test cases though.
2. @scoder's explanation above was quite helpful (thanks!), but it seems that `NameNode.generate_assignment_code` is only invoked for actual assignments. So if we have a type annotated class variable that doesn't get a value assigned (i.e. just `foo: int`), we cannot add that to `__annotations__` this way. Is there some function that's invoked for class variable declarations regardless of assignment?

You could handle 2) in `ExprStatNode`, which wraps non-statement expressions, such as arbitrarily sprinkled `NameNode`s.

Regarding compliance, I don't think there is all that much missing. I certainly wouldn't want a random hack that makes it mostly work but would have to be replaced completely when doing it right. I'd rather take at least some steps in the right direction.
Oh, and in fact, `ExprStatNode` marks its expression as unused, so that the non-assignment `NameNode.generate_code()` should be able to know that it only needs to store its annotation and be done. Although it could be that these nodes are discarded somewhere in the compiler pipeline. In that case, they should stay if they have an annotation to share.
Regarding namedtuples specifically, there is also #536 that would probably still prevent the usage even if this ticket was resolved.  PRs welcome.
enabling annotations on a class would be very beneficial for me:
- dataclasses
- pydantic classes (very similar, but available in python:3.6)
- (nice to have) namedtuple, but as said this would need more fixes.

@scoder : i would be willing to help, but find it difficult to assess if this is a reasonable goal for me. i have quite some python experience and a little bit of C++ experience. can you give any indication on the amount of work this might take? (days, weeks, months?). i understand that this is very difficult to say upfront.

maybe also i can contribute in another way by researching some specifics / adding preliminary tests?
@ra-kete : you're link to tests no longer works. can you provide them again?
With PEP-563 in place, I think this now got much easier since annotations are now plain and simple strings. We might get away with building the dict in Python and adding it to the class dict as is, similar to what we do for function annotations:
https://github.com/cython/cython/blob/bcae4d1f31135bdcbedadc83e751f13e18de9d0c/Cython/Compiler/ExprNodes.py#L9371-L9377

So, I think most of the work will be collecting (from CPython)/adapting/writing tests, then building the `__annotations__` dict from whatever there is in the class body, and then possibly a little bit of generated C code in the `Py3ClassNode` to add it to the class's namespace mapping.

@GliderGeek want to give it a try?
thanks for the quick reply. i will try to get some proper time to work on this.
in the meantime i have tried a little workaround:

```python
class Annotated:
    x: int
    y: int 

    __annotations__ = {'x': int, 'y': int}
```

this seems to work properly. is this what you meant by "get away with building the dict in Python and adding it to the class dict "?
Sorry, with "in Python", I meant "in the Python code of the Cython compiler", i.e. in AST nodes (that was very unclear), as opposed to "in hand written C code". As done for functions.

But the end result will then be generated C code that implements pretty much what you spelled out above, yes. (Although PEP 563 would make it `__annotations__ = {'x': 'int', 'y': 'int'}`).
> @ra-kete : you're link to tests no longer works. can you provide them again?

Sorry, I can't. Cleaned up my repos some time ago and forgot about this issue, so those tests are gone for good, unfortunately.
Popping in here - I've got a fork where I've attempted your recommendation, @scoder.

I'm very unclear as to exactly what kind of `StatNode` objects to expect in the class_def body, so this is my best guess:

https://github.com/seandstewart/cython/blob/master/Cython/Compiler/ExprNodes.py#L9060

I've also added a test here, based on your recommendation:

https://github.com/seandstewart/cython/blob/master/tests/run/pyclass_annotations_pep526.py
@seandstewart cool! I was thinking that the annotations should be collected by the nodes themselves, but looking through the constraints in PEP-526 again, it seems that even a simple solution like this might work. The test is ok as well, but should be extended to cover more declaration cases from PEP-526. Note that you don't strictly need a test function. You can use the docstring of the class as well as a doctest.

Here's what CPython uses:
https://github.com/python/cpython/blob/master/Lib/test/ann_module.py
https://github.com/python/cpython/blob/master/Lib/test/ann_module2.py
https://github.com/python/cpython/blob/master/Lib/test/ann_module3.py

Test for those are here:
https://github.com/python/cpython/blob/master/Lib/test/test_grammar.py
https://github.com/python/cpython/blob/57572b103ebd8732ac21922f0051a7f140d0e405/Lib/test/test_parser.py#L149

I've just updated the `test_grammar.py` copy that we use in Cython from Py3.9, but excluding the tests that use the external modules. While they could be integrated, I think it's better to create a separate test set from the annotation modules.
Hey @scoder - thanks for the pointers!

I've made some progress here. I've got the compiler actually extracting the correct node and building the list of annotations. However, I've hit a snag. When passing in the simple `annotation.string`, building the annotations_dict seems to work fine, however `annotations_dict.py_result()` ends up `None`. I tried passing the `AnnotationNode` directly to the dict instead (this feels more compliant, since the pure-python result will either be a string object or the class itself, depending upon how it's declared) but then `annotations_dict.analyse_types(code)` fails.

The relevant portions:

https://github.com/seandstewart/cython/blob/master/Cython/Compiler/ExprNodes.py#L9060-L9077
https://github.com/seandstewart/cython/blob/master/Cython/Compiler/ExprNodes.py#L9096-L9098

I think this would better fit into `Nodes.PyClassDefNode` instead of `ExprNodes.Py3ClassNode`.

The creation of the `DictNode()` should be done at the end of `PyClassDefNode.analyse_declarations()`, which analyses the declared names and variables in the class, long before the code generation happens. The result can be prepended (?) to the `self.body.stats` list (`self.body` is a `StatListNode`), as an assignment expressed in a `SingleAssignmentNode` (see the usages in `Parsing.py`), which should assign the `DictNode` result to an `IdentifierStringNode` for the `__annotations__` key (in Python: `__annotations__ = {…}`). The rest should then happen more or less automatically from the syntax tree nodes.

I also think that it's probably a good idea to _move_ the annotations over from their owning nodes into the `DictNode` values, i.e. set them to `None`, to prevent duplicate node references in the tree. I don't think they are used elsewhere.

This also seems worth a new method. :)
Great, yes, I think this is definitely the preferred approach, and will be in-line with CPython’s behavior.

One caveat - annotations with assignments should remain in the class body, whereas annotations without assignments can be safely removed, I think.

I’ll play with CPython’s AST and see how they handle this.
Oh, I just meant to set `.annotation = None`, not the whole statement node.  The node can just stay there, it won't hurt.
@seandstewart I don't think there's a PR for this yet, right? Could you open one, so that we can move this forward?
Hey @scoder  - apologies for the radio silence here, I haven't been able to work on side projects for a while. I'll try and put something together soon, I hadn't quite got the tests passing, IIRC.

Resolving patch:
diff --git a/Cython/Compiler/ExprNodes.py b/Cython/Compiler/ExprNodes.py
--- a/Cython/Compiler/ExprNodes.py
+++ b/Cython/Compiler/ExprNodes.py
@@ -28,7 +28,7 @@
 from . import StringEncoding
 from . import Naming
 from . import Nodes
-from .Nodes import Node, utility_code_for_imports
+from .Nodes import Node, utility_code_for_imports, SingleAssignmentNode, PassStatNode
 from . import PyrexTypes
 from .PyrexTypes import py_object_type, c_long_type, typecast, error_type, \
     unspecified_type
@@ -458,6 +458,9 @@ class ExprNode(Node):
 
     child_attrs = property(fget=operator.attrgetter('subexprs'))
 
+    def analyse_annotations(self, env):
+        pass
+
     def not_implemented(self, method_name):
         print_call_chain(method_name, "not implemented")
         raise InternalError(
@@ -1973,32 +1976,35 @@ def coerce_to(self, dst_type, env):
     def declare_from_annotation(self, env, as_target=False):
         """Implements PEP 526 annotation typing in a fairly relaxed way.
 
-        Annotations are ignored for global variables, Python class attributes and already declared variables.
-        String literals are allowed and ignored.
-        The ambiguous Python types 'int' and 'long' are ignored and the 'cython.int' form must be used instead.
+        Annotations are ignored for global variables.
+        All other annotations are stored on the entry in the symbol table.
+        String literals are allowed and not evaluated.
+        The ambiguous Python types 'int' and 'long' are not evaluated - the 'cython.int' form must be used instead.
         """
-        if not env.directives['annotation_typing']:
-            return
-        if env.is_module_scope or env.is_py_class_scope:
-            # annotations never create global cdef names and Python classes don't support them anyway
-            return
         name = self.name
-        if self.entry or env.lookup_here(name) is not None:
-            # already declared => ignore annotation
-            return
-
         annotation = self.annotation
-        if annotation.expr.is_string_literal:
-            # name: "description" => not a type, but still a declared variable or attribute
-            atype = None
-        else:
-            _, atype = annotation.analyse_type_annotation(env)
-        if atype is None:
-            atype = unspecified_type if as_target and env.directives['infer_types'] != False else py_object_type
-        if atype.is_fused and env.fused_to_specific:
-            atype = atype.specialize(env.fused_to_specific)
-        self.entry = env.declare_var(name, atype, self.pos, is_cdef=not as_target)
-        self.entry.annotation = annotation
+        entry = self.entry or env.lookup_here(name)
+        if not entry:
+            # annotations never create global cdef names
+            if env.is_module_scope:
+                return
+            if (
+                # name: "description" => not a type, but still a declared variable or attribute
+                annotation.expr.is_string_literal
+                # don't do type analysis from annotations if not asked to, but still collect the annotation
+                or not env.directives['annotation_typing']
+            ):
+                atype = None
+            else:
+                _, atype = annotation.analyse_type_annotation(env)
+            if atype is None:
+                atype = unspecified_type if as_target and env.directives['infer_types'] != False else py_object_type
+            if atype.is_fused and env.fused_to_specific:
+                atype = atype.specialize(env.fused_to_specific)
+            entry = self.entry = env.declare_var(name, atype, self.pos, is_cdef=not as_target)
+        # Even if the entry already exists, make sure we're supplying an annotation if we can.
+        if annotation and not entry.annotation:
+            entry.annotation = annotation
 
     def analyse_as_module(self, env):
         # Try to interpret this as a reference to a cimported module.
@@ -9037,6 +9043,9 @@ class ClassNode(ExprNode, ModuleNameMixin):
     type = py_object_type
     is_temp = True
 
+    def analyse_annotations(self, env):
+        pass
+
     def infer_type(self, env):
         # TODO: could return 'type' in some cases
         return py_object_type
@@ -9107,6 +9116,26 @@ def may_be_none(self):
 
     gil_message = "Constructing Python class"
 
+    def analyse_annotations(self, env):
+        from .AutoDocTransforms import AnnotationWriter
+        position = self.class_def_node.pos
+        dict_items = [
+            DictItemNode(
+                entry.pos,
+                key=IdentifierStringNode(entry.pos, value=entry.name),
+                value=entry.annotation.string
+            )
+            for entry in env.entries.values() if entry.annotation
+        ]
+        # Annotations dict shouldn't exist for classes which don't declare any.
+        if dict_items:
+            annotations_dict = DictNode(position, key_value_pairs=dict_items)
+            lhs = NameNode(position, name=StringEncoding.EncodedString(u"__annotations__"))
+            lhs.entry = env.lookup_here(lhs.name) or env.declare_var(lhs.name, dict_type, position)
+            node = SingleAssignmentNode(position, lhs=lhs, rhs=annotations_dict)
+            node.analyse_declarations(env)
+            self.class_def_node.body.stats.insert(0, node)
+
     def generate_result_code(self, code):
         code.globalstate.use_utility_code(UtilityCode.load_cached("Py3ClassCreate", "ObjectHandling.c"))
         cname = code.intern_identifier(self.name)
diff --git a/Cython/Compiler/Nodes.py b/Cython/Compiler/Nodes.py
--- a/Cython/Compiler/Nodes.py
+++ b/Cython/Compiler/Nodes.py
@@ -5,6 +5,7 @@
 from __future__ import absolute_import
 
 import cython
+
 cython.declare(sys=object, os=object, copy=object,
                Builtin=object, error=object, warning=object, Naming=object, PyrexTypes=object,
                py_object_type=object, ModuleScope=object, LocalScope=object, ClosureScope=object,
@@ -4852,6 +4853,7 @@ def analyse_declarations(self, env):
         if self.doc_node:
             self.doc_node.analyse_target_declaration(cenv)
         self.body.analyse_declarations(cenv)
+        self.class_result.analyse_annotations(cenv)
 
     def analyse_expressions(self, env):
         if self.bases:


Test patch:
diff --git a/tests/run/cython3.pyx b/tests/run/cython3.pyx
--- a/tests/run/cython3.pyx
+++ b/tests/run/cython3.pyx
@@ -616,15 +616,15 @@ def annotation_syntax(a: "test new test", b : "other" = 2, *args: "ARGS", **kwar
     >>> len(annotation_syntax.__annotations__)
     5
     >>> print(annotation_syntax.__annotations__['a'])
-    u'test new test'
+    'test new test'
     >>> print(annotation_syntax.__annotations__['b'])
-    u'other'
+    'other'
     >>> print(annotation_syntax.__annotations__['args'])
-    u'ARGS'
+    'ARGS'
     >>> print(annotation_syntax.__annotations__['kwargs'])
-    u'KWARGS'
+    'KWARGS'
     >>> print(annotation_syntax.__annotations__['return'])
-    u'ret'
+    'ret'
     """
     result : int = a + b
 
@@ -648,10 +648,10 @@ async def async_def_annotations(x: 'int') -> 'float':
     >>> ret, arg = sorted(async_def_annotations.__annotations__.items())
     >>> print(ret[0]); print(ret[1])
     return
-    u'float'
+    'float'
     >>> print(arg[0]); print(arg[1])
     x
-    u'int'
+    'int'
     """
     return float(x)
 
diff --git a/tests/run/decorators_T593.pyx b/tests/run/decorators_T593.pyx
--- a/tests/run/decorators_T593.pyx
+++ b/tests/run/decorators_T593.pyx
@@ -110,7 +110,7 @@ class Base(type):
 
 class Bar(metaclass=Base):
    """
-   >>> [n for n in Bar._order if n != "__qualname__"]
+   >>> [n for n in Bar._order if n not in {"__qualname__", "__annotations__"}]
    ['__module__', '__doc__', 'bar']
    """
    @property
diff --git a/tests/run/locals_T732.pyx b/tests/run/locals_T732.pyx
--- a/tests/run/locals_T732.pyx
+++ b/tests/run/locals_T732.pyx
@@ -23,7 +23,7 @@ def test_class_locals_and_dir():
     >>> klass = test_class_locals_and_dir()
     >>> 'visible' in klass.locs and 'not_visible' not in klass.locs
     True
-    >>> [n for n in klass.names if n != "__qualname__"]
+    >>> [n for n in klass.names if n not in {"__qualname__", "__annotations__"}]
     ['__module__', 'visible']
     """
     not_visible = 1234
diff --git a/tests/run/metaclass.pyx b/tests/run/metaclass.pyx
--- a/tests/run/metaclass.pyx
+++ b/tests/run/metaclass.pyx
@@ -69,7 +69,7 @@ class Py3ClassMCOnly(object, metaclass=Py3MetaclassPlusAttr):
     321
     >>> obj.metaclass_was_here
     True
-    >>> [n for n in obj._order if n != "__qualname__"]
+    >>> [n for n in obj._order if n not in {"__qualname__", "__annotations__"}]
     ['__module__', '__doc__', 'bar', 'metaclass_was_here']
     """
     bar = 321
@@ -81,7 +81,7 @@ class Py3InheritedMetaclass(Py3ClassMCOnly):
     345
     >>> obj.metaclass_was_here
     True
-    >>> [n for n in obj._order if n != "__qualname__"]
+    >>> [n for n in obj._order if n not in {"__qualname__", "__annotations__"}]
     ['__module__', '__doc__', 'bar', 'metaclass_was_here']
     """
     bar = 345
@@ -109,7 +109,7 @@ class Py3Foo(object, metaclass=Py3Base, foo=123):
     123
     >>> obj.bar
     321
-    >>> [n for n in obj._order if n != "__qualname__"]
+    >>> [n for n in obj._order if n not in {"__qualname__", "__annotations__"}]
     ['__module__', '__doc__', 'bar', 'foo']
     """
     bar = 321
@@ -122,7 +122,7 @@ class Py3FooInherited(Py3Foo, foo=567):
     567
     >>> obj.bar
     321
-    >>> [n for n in obj._order if n != "__qualname__"]
+    >>> [n for n in obj._order if n not in {"__qualname__", "__annotations__"}]
     ['__module__', '__doc__', 'bar', 'foo']
     """
     bar = 321
diff --git a/tests/run/pep526_variable_annotations.py b/tests/run/pep526_variable_annotations.py
--- a/tests/run/pep526_variable_annotations.py
+++ b/tests/run/pep526_variable_annotations.py
@@ -156,6 +156,7 @@ def iter_declared_dict_arg(d : Dict[float, float]):
 37:19: Unknown type declaration in annotation, ignoring
 38:12: Unknown type declaration in annotation, ignoring
 39:18: Unknown type declaration in annotation, ignoring
+57:19: Unknown type declaration in annotation, ignoring
 73:19: Unknown type declaration in annotation, ignoring
 # FIXME: these are sort-of evaluated now, so the warning is misleading
 126:21: Unknown type declaration in annotation, ignoring
diff --git a/tests/run/pyclass_annotations_pep526.py b/tests/run/pyclass_annotations_pep526.py
new file mode 100644
--- /dev/null
+++ b/tests/run/pyclass_annotations_pep526.py
@@ -0,0 +1,47 @@
+# cython: language_level=3
+# mode: run
+# tag: pure3.7, pep526, pep484
+
+from __future__ import annotations
+
+try:
+    from typing import ClassVar
+except ImportError:  # Py<=3.5
+    ClassVar = {int: int}
+
+
+class PyAnnotatedClass:
+    """
+    >>> PyAnnotatedClass.__annotations__["CLASS_VAR"]
+    'ClassVar[int]'
+    >>> PyAnnotatedClass.__annotations__["obj"]
+    'str'
+    >>> PyAnnotatedClass.__annotations__["literal"]
+    "'int'"
+    >>> PyAnnotatedClass.__annotations__["recurse"]
+    "'PyAnnotatedClass'"
+    >>> PyAnnotatedClass.__annotations__["default"]
+    'bool'
+    >>> PyAnnotatedClass.CLASS_VAR
+    1
+    >>> PyAnnotatedClass.default
+    False
+    >>> PyAnnotatedClass.obj
+    Traceback (most recent call last):
+      ...
+    AttributeError: type object 'PyAnnotatedClass' has no attribute 'obj'
+    """
+    CLASS_VAR: ClassVar[int] = 1
+    obj: str
+    literal: "int"
+    recurse: "PyAnnotatedClass"
+    default: bool = False
+
+
+class PyVanillaClass:
+    """
+    >>> PyVanillaClass.__annotations__
+    Traceback (most recent call last):
+      ...
+    AttributeError: type object 'PyVanillaClass' has no attribute '__annotations__'
+    """
diff --git a/tests/run/test_grammar.py b/tests/run/test_grammar.py
--- a/tests/run/test_grammar.py
+++ b/tests/run/test_grammar.py
@@ -477,8 +477,7 @@ class C:
             def __init__(self, x):
                 self.x: int = x
 
-        # FIXME: implement class annotations
-        #self.assertEqual(C.__annotations__, {'_C__foo': int, 's': str})
+        self.assertEqual(C.__annotations__, {'_C__foo': 'int', 's': 'str'})
         with self.assertRaises(NameError):
             class CBad:
                 no_such_name_defined.attr: int = 0
@@ -487,7 +486,7 @@ class Cbad2(C):
                 x: int
                 x.y: list = []
 
-    @skip("Class annotations not implemented")
+    @skip("Not currently supported: https://github.com/cython/cython/issues/3839")
     def test_var_annot_metaclass_semantics(self):
         class CMeta(type):
             @classmethod