Repository: cython/cython
Pull request: Add support for PEP 526 `__annotations__` in class body.
This is an attempt to resolve #2552 and follows recommendations by @scoder.

Looking forward to your feedback!

Problem statement:
Classes lack __annotations__ (PEP-526)
With python 3.7 dataclasses were introduced.
They work just fine in Cython except for their standard __init__ function.

How to reproduce:
test.pyx
```
from dataclasses import dataclass

@dataclass
class TestClass:
	x: int
	y: int
	some_string: str
```
This file compiles to Cython code just fine.
However, when ran from the python it raises an error:
test2.py
```
import test

test_class = test.TestClass(1, 2, 'string')
```
This raises the following:
```
Traceback (most recent call last):
  File ".\test2.py", line 3, in <module>
    test_class = test.TestClass(1, 2, 'string')
TypeError: __init__() takes 1 positional argument but 4 were given
```

if you rename test.pyx to test.py and do not compile it, no such error occurs.

my current workaround:
setting each property after initialization separately:
test.pyx remains unchanged.
test.py
```
import test

test_class = test.TestClass()
test_class.x = 1
test_class.y = 2
test_class.some_string = 'string'
```

Hints from issue discussion:
I don't think we currently create a `__annotations__` for classes, although the syntax tree has the necessary information available. See the description in
https://www.python.org/dev/peps/pep-0526/#runtime-effects-of-type-annotations

PR welcome.
I'm also stumbling over this currently, especially Python 3.6-style NamedTuples not working.

I don't have any experience with how Cython works under the hood, but I'd be willing to give fixing this a try if someone could point me where to start. Is there some relevant documentation somewhere?
Having similliar issue with dataclasses:
```
@dataclass
class Test:
    name: str
```
When using regular dataclass when I import the file with the dataclass, I get:
```
TypeError: 'name' is a field but has no type annotation
```

I guess that's because cython does not creation annotations for classes.
As always, the most important bit are tests. I would add new ones in a file `tests/run/pyclass_annotations_pep526.py` (with a `# tag: pure36` to prevent Py<3.6 from running in uncompiled). Look at the other `pyclass` tests to see how it's done, we basically use doctests (which are executed in Python and not compiled, as opposed to the rest of the file).

I first thought that this could be implemented after executing the class body, but it turns out that [PEP 526](https://www.python.org/dev/peps/pep-0526/#runtime-effects-of-type-annotations) wants the `__annotations__` dict to be available before that, so creating the empty dict and assigning it in `__Pyx_Py3MetaclassPrepare()` would be the right way.

Then each `NameNode` inside of the class body (`entry.is_pyclass_attr`) which holds a `self.annotation` will have to generate C code (in `generate_assignment_code()`) to look up the `__annotations__` dict in the class scope and assign its name to its annotation `.py_result()` on an assignment (or creation of the name). I don't actually know if that happens before or after the assignment to the name (which might fail, but could be shadowed by a try-except). Probably something to test in CPython. Thus: tests.
Also note that there are probably relevant tests for PEP-526 in CPython's own test suite, which could be reused.
I unfortunately didn't have much time these past few weeks to look at this. I've come up with a couple of test cases, however, that should get us pretty far on the way to full support of `__annotations__`. You can check them out [here](https://github.com/ra-kete/cython/blob/pyclass-annotations-pep526/tests/run/pyclass_annotations_pep526.py).

I have two main questions now:
1. Do we need full compliance with CPython or would you also consider merging something incomplete if it enables the (correct) use of dataclasses and NamedTuples? I think it should be pretty straightforward to add the annotations to the `__annotations__` dict (passing test case `Annotations` above), which should be enough for the aforementioned use cases. I'm not sure how hard it would be to satisfy some of the other test cases though.
2. @scoder's explanation above was quite helpful (thanks!), but it seems that `NameNode.generate_assignment_code` is only invoked for actual assignments. So if we have a type annotated class variable that doesn't get a value assigned (i.e. just `foo: int`), we cannot add that to `__annotations__` this way. Is there some function that's invoked for class variable declarations regardless of assignment?

You could handle 2) in `ExprStatNode`, which wraps non-statement expressions, such as arbitrarily sprinkled `NameNode`s.

Regarding compliance, I don't think there is all that much missing. I certainly wouldn't want a random hack that makes it mostly work but would have to be replaced completely when doing it right. I'd rather take at least some steps in the right direction.
Oh, and in fact, `ExprStatNode` marks its expression as unused, so that the non-assignment `NameNode.generate_code()` should be able to know that it only needs to store its annotation and be done. Although it could be that these nodes are discarded somewhere in the compiler pipeline. In that case, they should stay if they have an annotation to share.
Regarding namedtuples specifically, there is also #536 that would probably still prevent the usage even if this ticket was resolved.  PRs welcome.
enabling annotations on a class would be very beneficial for me:
- dataclasses
- pydantic classes (very similar, but available in python:3.6)
- (nice to have) namedtuple, but as said this would need more fixes.

@scoder : i would be willing to help, but find it difficult to assess if this is a reasonable goal for me. i have quite some python experience and a little bit of C++ experience. can you give any indication on the amount of work this might take? (days, weeks, months?). i understand that this is very difficult to say upfront.

maybe also i can contribute in another way by researching some specifics / adding preliminary tests?

Commit message: Update cython3.pyx annotations tests for patch addressing `u` prefix in strings

Patch to review:
diff --git a/Cython/Compiler/ExprNodes.py b/Cython/Compiler/ExprNodes.py
index 5ce3032da43..8e7e80ee30d 100644
--- a/Cython/Compiler/ExprNodes.py
+++ b/Cython/Compiler/ExprNodes.py
@@ -28,7 +28,7 @@
 from . import StringEncoding
 from . import Naming
 from . import Nodes
-from .Nodes import Node, utility_code_for_imports
+from .Nodes import Node, utility_code_for_imports, SingleAssignmentNode, PassStatNode
 from . import PyrexTypes
 from .PyrexTypes import py_object_type, c_long_type, typecast, error_type, \
     unspecified_type
@@ -458,6 +458,9 @@ class ExprNode(Node):
 
     child_attrs = property(fget=operator.attrgetter('subexprs'))
 
+    def analyse_annotations(self, env):
+        pass
+
     def not_implemented(self, method_name):
         print_call_chain(method_name, "not implemented")
         raise InternalError(
@@ -1973,32 +1976,35 @@ def coerce_to(self, dst_type, env):
     def declare_from_annotation(self, env, as_target=False):
         """Implements PEP 526 annotation typing in a fairly relaxed way.
 
-        Annotations are ignored for global variables, Python class attributes and already declared variables.
-        String literals are allowed and ignored.
-        The ambiguous Python types 'int' and 'long' are ignored and the 'cython.int' form must be used instead.
+        Annotations are ignored for global variables.
+        All other annotations are stored on the entry in the symbol table.
+        String literals are allowed and not evaluated.
+        The ambiguous Python types 'int' and 'long' are not evaluated - the 'cython.int' form must be used instead.
         """
-        if not env.directives['annotation_typing']:
-            return
-        if env.is_module_scope or env.is_py_class_scope:
-            # annotations never create global cdef names and Python classes don't support them anyway
-            return
         name = self.name
-        if self.entry or env.lookup_here(name) is not None:
-            # already declared => ignore annotation
-            return
-
         annotation = self.annotation
-        if annotation.expr.is_string_literal:
-            # name: "description" => not a type, but still a declared variable or attribute
-            atype = None
-        else:
-            _, atype = annotation.analyse_type_annotation(env)
-        if atype is None:
-            atype = unspecified_type if as_target and env.directives['infer_types'] != False else py_object_type
-        if atype.is_fused and env.fused_to_specific:
-            atype = atype.specialize(env.fused_to_specific)
-        self.entry = env.declare_var(name, atype, self.pos, is_cdef=not as_target)
-        self.entry.annotation = annotation
+        entry = self.entry or env.lookup_here(name)
+        if not entry:
+            # annotations never create global cdef names
+            if env.is_module_scope:
+                return
+            if (
+                # name: "description" => not a type, but still a declared variable or attribute
+                annotation.expr.is_string_literal
+                # don't do type analysis from annotations if not asked to, but still collect the annotation
+                or not env.directives['annotation_typing']
+            ):
+                atype = None
+            else:
+                _, atype = annotation.analyse_type_annotation(env)
+            if atype is None:
+                atype = unspecified_type if as_target and env.directives['infer_types'] != False else py_object_type
+            if atype.is_fused and env.fused_to_specific:
+                atype = atype.specialize(env.fused_to_specific)
+            entry = self.entry = env.declare_var(name, atype, self.pos, is_cdef=not as_target)
+        # Even if the entry already exists, make sure we're supplying an annotation if we can.
+        if annotation and not entry.annotation:
+            entry.annotation = annotation
 
     def analyse_as_module(self, env):
         # Try to interpret this as a reference to a cimported module.
@@ -9037,6 +9043,9 @@ class ClassNode(ExprNode, ModuleNameMixin):
     type = py_object_type
     is_temp = True
 
+    def analyse_annotations(self, env):
+        pass
+
     def infer_type(self, env):
         # TODO: could return 'type' in some cases
         return py_object_type
@@ -9107,6 +9116,26 @@ def may_be_none(self):
 
     gil_message = "Constructing Python class"
 
+    def analyse_annotations(self, env):
+        from .AutoDocTransforms import AnnotationWriter
+        position = self.class_def_node.pos
+        dict_items = [
+            DictItemNode(
+                entry.pos,
+                key=IdentifierStringNode(entry.pos, value=entry.name),
+                value=entry.annotation.string
+            )
+            for entry in env.entries.values() if entry.annotation
+        ]
+        # Annotations dict shouldn't exist for classes which don't declare any.
+        if dict_items:
+            annotations_dict = DictNode(position, key_value_pairs=dict_items)
+            lhs = NameNode(position, name=StringEncoding.EncodedString(u"__annotations__"))
+            lhs.entry = env.lookup_here(lhs.name) or env.declare_var(lhs.name, dict_type, position)
+            node = SingleAssignmentNode(position, lhs=lhs, rhs=annotations_dict)
+            node.analyse_declarations(env)
+            self.class_def_node.body.stats.insert(0, node)
+
     def generate_result_code(self, code):
         code.globalstate.use_utility_code(UtilityCode.load_cached("Py3ClassCreate", "ObjectHandling.c"))
         cname = code.intern_identifier(self.name)
diff --git a/Cython/Compiler/Nodes.py b/Cython/Compiler/Nodes.py
index 7e0961c1561..431c067b217 100644
--- a/Cython/Compiler/Nodes.py
+++ b/Cython/Compiler/Nodes.py
@@ -5,6 +5,7 @@
 from __future__ import absolute_import
 
 import cython
+
 cython.declare(sys=object, os=object, copy=object,
                Builtin=object, error=object, warning=object, Naming=object, PyrexTypes=object,
                py_object_type=object, ModuleScope=object, LocalScope=object, ClosureScope=object,
@@ -4852,6 +4853,7 @@ def analyse_declarations(self, env):
         if self.doc_node:
             self.doc_node.analyse_target_declaration(cenv)
         self.body.analyse_declarations(cenv)
+        self.class_result.analyse_annotations(cenv)
 
     def analyse_expressions(self, env):
         if self.bases:
diff --git a/tests/run/cython3.pyx b/tests/run/cython3.pyx
index f0c18bcece6..dec2226b331 100644
--- a/tests/run/cython3.pyx
+++ b/tests/run/cython3.pyx
@@ -616,15 +616,15 @@ def annotation_syntax(a: "test new test", b : "other" = 2, *args: "ARGS", **kwar
     >>> len(annotation_syntax.__annotations__)
     5
     >>> print(annotation_syntax.__annotations__['a'])
-    u'test new test'
+    'test new test'
     >>> print(annotation_syntax.__annotations__['b'])
-    u'other'
+    'other'
     >>> print(annotation_syntax.__annotations__['args'])
-    u'ARGS'
+    'ARGS'
     >>> print(annotation_syntax.__annotations__['kwargs'])
-    u'KWARGS'
+    'KWARGS'
     >>> print(annotation_syntax.__annotations__['return'])
-    u'ret'
+    'ret'
     """
     result : int = a + b
 
@@ -648,10 +648,10 @@ async def async_def_annotations(x: 'int') -> 'float':
     >>> ret, arg = sorted(async_def_annotations.__annotations__.items())
     >>> print(ret[0]); print(ret[1])
     return
-    u'float'
+    'float'
     >>> print(arg[0]); print(arg[1])
     x
-    u'int'
+    'int'
     """
     return float(x)
 
diff --git a/tests/run/decorators_T593.pyx b/tests/run/decorators_T593.pyx
index 2d87c4c3199..d63bcb7040f 100644
--- a/tests/run/decorators_T593.pyx
+++ b/tests/run/decorators_T593.pyx
@@ -110,7 +110,7 @@ class Base(type):
 
 class Bar(metaclass=Base):
    """
-   >>> [n for n in Bar._order if n != "__qualname__"]
+   >>> [n for n in Bar._order if n not in {"__qualname__", "__annotations__"}]
    ['__module__', '__doc__', 'bar']
    """
    @property
diff --git a/tests/run/locals_T732.pyx b/tests/run/locals_T732.pyx
index 6bae1d23676..6b330752312 100644
--- a/tests/run/locals_T732.pyx
+++ b/tests/run/locals_T732.pyx
@@ -23,7 +23,7 @@ def test_class_locals_and_dir():
     >>> klass = test_class_locals_and_dir()
     >>> 'visible' in klass.locs and 'not_visible' not in klass.locs
     True
-    >>> [n for n in klass.names if n != "__qualname__"]
+    >>> [n for n in klass.names if n not in {"__qualname__", "__annotations__"}]
     ['__module__', 'visible']
     """
     not_visible = 1234
diff --git a/tests/run/metaclass.pyx b/tests/run/metaclass.pyx
index 26ac9af46fa..9e4db4c8bfc 100644
--- a/tests/run/metaclass.pyx
+++ b/tests/run/metaclass.pyx
@@ -69,7 +69,7 @@ class Py3ClassMCOnly(object, metaclass=Py3MetaclassPlusAttr):
     321
     >>> obj.metaclass_was_here
     True
-    >>> [n for n in obj._order if n != "__qualname__"]
+    >>> [n for n in obj._order if n not in {"__qualname__", "__annotations__"}]
     ['__module__', '__doc__', 'bar', 'metaclass_was_here']
     """
     bar = 321
@@ -81,7 +81,7 @@ class Py3InheritedMetaclass(Py3ClassMCOnly):
     345
     >>> obj.metaclass_was_here
     True
-    >>> [n for n in obj._order if n != "__qualname__"]
+    >>> [n for n in obj._order if n not in {"__qualname__", "__annotations__"}]
     ['__module__', '__doc__', 'bar', 'metaclass_was_here']
     """
     bar = 345
@@ -109,7 +109,7 @@ class Py3Foo(object, metaclass=Py3Base, foo=123):
     123
     >>> obj.bar
     321
-    >>> [n for n in obj._order if n != "__qualname__"]
+    >>> [n for n in obj._order if n not in {"__qualname__", "__annotations__"}]
     ['__module__', '__doc__', 'bar', 'foo']
     """
     bar = 321
@@ -122,7 +122,7 @@ class Py3FooInherited(Py3Foo, foo=567):
     567
     >>> obj.bar
     321
-    >>> [n for n in obj._order if n != "__qualname__"]
+    >>> [n for n in obj._order if n not in {"__qualname__", "__annotations__"}]
     ['__module__', '__doc__', 'bar', 'foo']
     """
     bar = 321
diff --git a/tests/run/pep526_variable_annotations.py b/tests/run/pep526_variable_annotations.py
index 9a43f35f6fc..8a29e9decab 100644
--- a/tests/run/pep526_variable_annotations.py
+++ b/tests/run/pep526_variable_annotations.py
@@ -156,6 +156,7 @@ def iter_declared_dict_arg(d : Dict[float, float]):
 37:19: Unknown type declaration in annotation, ignoring
 38:12: Unknown type declaration in annotation, ignoring
 39:18: Unknown type declaration in annotation, ignoring
+57:19: Unknown type declaration in annotation, ignoring
 73:19: Unknown type declaration in annotation, ignoring
 # FIXME: these are sort-of evaluated now, so the warning is misleading
 126:21: Unknown type declaration in annotation, ignoring
diff --git a/tests/run/pyclass_annotations_pep526.py b/tests/run/pyclass_annotations_pep526.py
new file mode 100644
index 00000000000..f70b959dedf
--- /dev/null
+++ b/tests/run/pyclass_annotations_pep526.py
@@ -0,0 +1,50 @@
+# cython: language_level=3
+# mode: run
+# tag: pure3.7, pep526, pep484
+
+from __future__ import annotations
+
+try:
+    from typing import ClassVar
+except ImportError:  # Py3.5
+    try:
+        from typing import Optional as ClassVar  # Good enough for jazz.
+    except ImportError:   # Py<=3.4
+        from collections import defaultdict as ClassVar  # Now we're hacking!
+
+
+class PyAnnotatedClass:
+    """
+    >>> PyAnnotatedClass.__annotations__["CLASS_VAR"]
+    'ClassVar[int]'
+    >>> PyAnnotatedClass.__annotations__["obj"]
+    'str'
+    >>> PyAnnotatedClass.__annotations__["literal"]
+    "'int'"
+    >>> PyAnnotatedClass.__annotations__["recurse"]
+    "'PyAnnotatedClass'"
+    >>> PyAnnotatedClass.__annotations__["default"]
+    'bool'
+    >>> PyAnnotatedClass.CLASS_VAR
+    1
+    >>> PyAnnotatedClass.default
+    False
+    >>> PyAnnotatedClass.obj
+    Traceback (most recent call last):
+      ...
+    AttributeError: type object 'PyAnnotatedClass' has no attribute 'obj'
+    """
+    CLASS_VAR: ClassVar[int] = 1
+    obj: str
+    literal: "int"
+    recurse: "PyAnnotatedClass"
+    default: bool = False
+
+
+class PyVanillaClass:
+    """
+    >>> PyVanillaClass.__annotations__
+    Traceback (most recent call last):
+      ...
+    AttributeError: type object 'PyVanillaClass' has no attribute '__annotations__'
+    """
diff --git a/tests/run/test_grammar.py b/tests/run/test_grammar.py
index b1058ab299b..c41b75f5559 100644
--- a/tests/run/test_grammar.py
+++ b/tests/run/test_grammar.py
@@ -477,8 +477,7 @@ class C:
             def __init__(self, x):
                 self.x: int = x
 
-        # FIXME: implement class annotations
-        #self.assertEqual(C.__annotations__, {'_C__foo': int, 's': str})
+        self.assertEqual(C.__annotations__, {'_C__foo': 'int', 's': 'str'})
         with self.assertRaises(NameError):
             class CBad:
                 no_such_name_defined.attr: int = 0
@@ -487,7 +486,7 @@ class Cbad2(C):
                 x: int
                 x.y: list = []
 
-    @skip("Class annotations not implemented")
+    @skip("Not currently supported: https://github.com/cython/cython/issues/3839")
     def test_var_annot_metaclass_semantics(self):
         class CMeta(type):
             @classmethod


Code review comments:
diff_hunk: @@ -0,0 +1,50 @@
+# cython: language_level=3
+# mode: run
+# tag: pure3.7, pep526, pep484
+
+from __future__ import annotations
+
+try:
+    from typing import ClassVar
+except ImportError:  # Py3.5
+    try:
+        from typing import Optional as ClassVar  # Good enough for jazz.
+    except ImportError:   # Py<=3.4
+        from collections import defaultdict as ClassVar  # Now we're hacking!
path: tests/run/pyclass_annotations_pep526.py
text: @user1:
I don't think the last bit works (`defaultdict[int]`), but I also don't think we have to bend it so much. Isn't this enough?
```suggestion
try:
    from typing import ClassVar
except ImportError:  # Py<=3.5
    ClassVar = {int: int}
```
Actually, even `ClassVar = None` would probably work, but IDEs might be unhappy about it, so…