Repository: cython/cython

Issue:
Casting int to cpdef enum
Simplified code:
```
cpdef enum MyEnum:
    MY_VAL=2

def my_fn():
    return <MyEnum>2
```

I'd expect my_fn() to return MyEnum.MY_VAL, but just returns the int value.


Discussion:
Encountered the same thing.

FWIW this works:

```cython
cpdef enum MyEnum:
    MY_VAL = 2

def my_fn():
    return MyEnum(2)
```

That said, would not have expected needing the explicit construction of `MyEnum`.
Guessing there would need to be some alternative handling [here]( https://github.com/cython/cython/blob/796fd06da1fa1d5481ce43a6b6c901bc87f0ce9a/Cython/Compiler/PyrexTypes.py#L495 ) to handle `cpdef enum` types conversion. Noting this in part as that code path seems to be used to generate the `int` coercion C code seen with this example.
Maybe this could be done by checking if it is an instance of [`__Pyx_EnumBase`]( https://github.com/cython/cython/blob/efdf996dced075682d86b7e53a12b56b5ebfea38/Cython/Utility/CpdefEnums.pyx#L22 ).

What do you think @scoder? 🙂
It is worth noting that using the construction workaround in comment ( https://github.com/cython/cython/issues/2732#issuecomment-1176893773 ) comes with fairly significant overhead.

```cython
cpdef enum MyEnum:
    MY_VAL = 2

def my_fn1():
    return MyEnum.MY_VAL

def my_fn2():
    return MyEnum(MyEnum.MY_VAL)
```

```python
In [3]: %timeit my_fn1()
27.6 ns ± 0.394 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

In [4]: %timeit my_fn2()
398 ns ± 5.33 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)
```
A more performant approach might be mapping preexisting `MyEnum` Python objects to `MyEnum` C values like so.

```cython
cpdef enum MyEnum:
    ZERO = 0
    ONE = 1


cdef inline _convert_MyEnum_c_to_py(MyEnum c_enum):
    if c_enum == MyEnum.ZERO:
        return (<object>MyEnum).ZERO
    elif c_enum == MyEnum.ONE:
        return (<object>MyEnum).ONE
    else:
        raise ValueError(f"Unknown `MyEnum` value: {c_enum}")

 
def my_fn1():
    cdef MyEnum c_enum = MyEnum.ONE
    cdef object py_enum = <object>c_enum
    return py_enum


def my_fn2():
    cdef MyEnum c_enum = MyEnum.ONE
    cdef object py_enum = _convert_MyEnum_c_to_py(c_enum)
    return py_enum
```

```python
In [3]: %timeit my_fn1()
28.1 ns ± 1.42 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

In [4]: %timeit my_fn2()
79.9 ns ± 0.537 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)
```

This still adds some overhead due to the global lookup for `MyEnum` and attribute access, but it is unclear how this would be avoided given the nature of the check.
> Guessing there would need to be some alternative handling [here](https://github.com/cython/cython/blob/796fd06da1fa1d5481ce43a6b6c901bc87f0ce9a/Cython/Compiler/PyrexTypes.py#L495) to handle `cpdef enum` types conversion. Noting this in part as that code path seems to be used to generate the `int` coercion C code seen with this example.

Doesn't look to go through there when I set a breakpoint there. (Although I suspect you're broadly right about where to look)
> A more performant approach might be mapping preexisting MyEnum Python objects to MyEnum C values like so.

It looks to me like `__Pyx_EnumBase` tries to do this already so I'm slightly surprised there's so much benefit. Possibly "ordered_dict" is the wrong structure?
I agree that this behaviour should be changed. Returning the corresponding Enum object is the expected result. In the case of an invalid enum integer value, I'd also expect an exception, probably a `ValueError` as shown in https://github.com/cython/cython/issues/2732#issuecomment-1178383584. Cython can probably do something more efficient internally than the generic attribute lookup in `(<object>MyEnum).ZERO`.
> > Guessing there would need to be some alternative handling [here](https://github.com/cython/cython/blob/796fd06da1fa1d5481ce43a6b6c901bc87f0ce9a/Cython/Compiler/PyrexTypes.py#L495) to handle `cpdef enum` types conversion. Noting this in part as that code path seems to be used to generate the `int` coercion C code seen with this example.
> 
> Doesn't look to go through there when I set a breakpoint there. (Although I suspect you're broadly right about where to look)

Interesting came to that conclusion by looking at the generated C code and comparing. In particular was seeing references to that conversion code when doing `<object>c_enum`

That said, using the debugger seems like the more reliable way to go

> > A more performant approach might be mapping preexisting MyEnum Python objects to MyEnum C values like so.
> 
> It looks to me like `__Pyx_EnumBase` tries to do this already so I'm slightly surprised there's so much benefit. Possibly "ordered_dict" is the wrong structure?

Yeah was noticing that. Think the tricky thing is we want to lookup by `int` in this case, but the keys are `str`s for each value. Wasn't seeing a mapping that went from `int`s to `IntEnum` values, but may be overlooking something
> > it looks to me like `__Pyx_EnumBase` tries to do this already so I'm slightly surprised there's so much benefit. Possibly "ordered_dict" is the wrong structure?
> 
> Yeah was noticing that. Think the tricky thing is we want to lookup by `int` in this case, but the keys are `str`s for each value. Wasn't seeing a mapping that went from `int`s to `IntEnum` values, but may be overlooking something

So having looked in more detail, it uses Python's `IntEnum` on most modern versions (not the class that's listed in the utility code). If you look at that's constructor it actually have it's own int -> enum map, but presumably you're just losing time on boxing/unboxing the C int.
> I agree that this behaviour should be changed. Returning the corresponding Enum object is the expected result. In the case of an invalid enum integer value, I'd also expect an exception, probably a `ValueError` as shown in [#2732 (comment)](https://github.com/cython/cython/issues/2732#issuecomment-1178383584). Cython can probably do something more efficient internally than the generic attribute lookup in `(<object>MyEnum).ZERO`.

Thanks Stefan 🙏

Do you have a sense of where it would make sense to add conversion code like this?

Agree there is likely a better way to handle accessing the right `IntEnum` value. Do you have thoughts on what would be the best way to approach this?

Also guessing we can do some templating to simplify the branch logic (if that is the right way to go).
> Do you have a sense of where it would make sense to add conversion code like this?

(Not Stefan but) I have a 50% written change with the conversion code in as a `cdef __Pyx_Enum_{{name}}_to_py(int c_val)` in `CpdefEnum.pyx` (and called from the `PyrexTypes.CEnumType.create_to_py_utility_code` function).

> Agree there is likely a better way to handle accessing the right IntEnum value. Do you have thoughts on what would be the best way to approach this?

I'm not sure there's a better way to approach this - Cython doesn't look to have much special knowledge of the Python enum

> Also guessing we can do some templating to simplify the branch logic (if that is the right way to go).

yes

Resolving patch:
diff --git a/Cython/Compiler/PyrexTypes.py b/Cython/Compiler/PyrexTypes.py
--- a/Cython/Compiler/PyrexTypes.py
+++ b/Cython/Compiler/PyrexTypes.py
@@ -2787,6 +2787,12 @@ def from_py_call_code(self, source_code, result_code, error_pos, code,
             source_code, result_code, self.size)
         return code.error_goto_if_neg(call_code, error_pos)
 
+    def error_condition(self, result_code):
+        # It isn't possible to use CArrays as return type so the error_condition
+        # is irrelevant. Returning a falsy value does avoid an error when getting
+        # from_py_call_code from a typedef.
+        return ""
+
 
 class CPtrType(CPointerBaseType):
     #  base_type     CType              Reference type
@@ -4257,6 +4263,25 @@ def create_from_py_utility_code(self, env):
     def create_to_py_utility_code(self, env):
         if self.to_py_function is not None:
             return True
+        if self.entry.create_wrapper:
+            from .UtilityCode import CythonUtilityCode
+            self.to_py_function = "__Pyx_Enum_%s_to_py" % self.name
+            if self.entry.scope != env.global_scope():
+                module_name = self.entry.scope.qualified_name
+            else:
+                module_name = None
+            env.use_utility_code(CythonUtilityCode.load(
+                "EnumTypeToPy", "CpdefEnums.pyx",
+                context={"funcname": self.to_py_function,
+                        "name": self.name,
+                        "items": tuple(self.values),
+                        "underlying_type": self.underlying_type.empty_declaration_code(),
+                        "module_name": module_name,
+                        "is_flag": False,
+                        },
+                outer_module_scope=self.entry.scope  # ensure that "name" is findable
+            ))
+            return True
         if self.underlying_type.create_to_py_utility_code(env):
             # Using a C++11 lambda here, which is fine since
             # scoped enums are a C++11 feature
@@ -4383,15 +4408,47 @@ def specialize(self, values):
 
     def create_type_wrapper(self, env):
         from .UtilityCode import CythonUtilityCode
+        # Generate "int"-like conversion function
+        old_to_py_function = self.to_py_function
+        self.to_py_function = None
+        CIntLike.create_to_py_utility_code(self, env)
+        enum_to_pyint_func = self.to_py_function
+        self.to_py_function = old_to_py_function  # we don't actually want to overwrite this
+
         env.use_utility_code(CythonUtilityCode.load(
             "EnumType", "CpdefEnums.pyx",
             context={"name": self.name,
                      "items": tuple(self.values),
                      "enum_doc": self.doc,
+                     "enum_to_pyint_func": enum_to_pyint_func,
                      "static_modname": env.qualified_name,
                      },
             outer_module_scope=env.global_scope()))
 
+    def create_to_py_utility_code(self, env):
+        if self.to_py_function is not None:
+            return self.to_py_function
+        if not self.entry.create_wrapper:
+            return super(CEnumType, self).create_to_py_utility_code(env)
+        from .UtilityCode import CythonUtilityCode
+        self.to_py_function = "__Pyx_Enum_%s_to_py" % self.name
+        if self.entry.scope != env.global_scope():
+            module_name = self.entry.scope.qualified_name
+        else:
+            module_name = None
+        env.use_utility_code(CythonUtilityCode.load(
+            "EnumTypeToPy", "CpdefEnums.pyx",
+            context={"funcname": self.to_py_function,
+                    "name": self.name,
+                    "items": tuple(self.values),
+                    "underlying_type": "int",
+                    "module_name": module_name,
+                    "is_flag": True,
+                    },
+            outer_module_scope=self.entry.scope  # ensure that "name" is findable
+        ))
+        return True
+
 
 class CTupleType(CType):
     # components [PyrexType]
diff --git a/Cython/Compiler/UtilityCode.py b/Cython/Compiler/UtilityCode.py
--- a/Cython/Compiler/UtilityCode.py
+++ b/Cython/Compiler/UtilityCode.py
@@ -173,8 +173,15 @@ def scope_transform(module_node):
         if self.context_types:
             # inject types into module scope
             def scope_transform(module_node):
+                dummy_entry = object()
                 for name, type in self.context_types.items():
+                    # Restore the old type entry after declaring the type.
+                    # We need to access types in the scope, but this shouldn't alter the entry
+                    # that is visible from everywhere else
+                    old_type_entry = getattr(type, "entry", dummy_entry)
                     entry = module_node.scope.declare_type(name, type, None, visibility='extern')
+                    if old_type_entry is not dummy_entry:
+                        type.entry = old_type_entry
                     entry.in_cinclude = True
                 return module_node
 


Test patch:
diff --git a/tests/run/cpdef_enums.pyx b/tests/run/cpdef_enums.pyx
--- a/tests/run/cpdef_enums.pyx
+++ b/tests/run/cpdef_enums.pyx
@@ -132,13 +132,28 @@ def check_docs():
     """
     pass
 
+
+def to_from_py_conversion(PxdEnum val):
+    """
+    >>> to_from_py_conversion(RANK_1) is PxdEnum.RANK_1
+    True
+
+    C enums are commonly enough used as flags that it seems reasonable
+    to allow it in Cython
+    >>> to_from_py_conversion(RANK_1 | RANK_2) == (RANK_1 | RANK_2)
+    True
+    """
+    return val
+
+
 def test_pickle():
     """
     >>> from pickle import loads, dumps
     >>> import sys
 
     Pickling enums won't work without the enum module, so disable the test
-    >>> if sys.version_info < (3, 4):
+    (now requires  3.6 for IntFlag)
+    >>> if sys.version_info < (3, 6):
     ...     loads = dumps = lambda x: x
     >>> loads(dumps(PyxEnum.TWO)) == PyxEnum.TWO
     True
diff --git a/tests/run/cpdef_enums_import.srctree b/tests/run/cpdef_enums_import.srctree
--- a/tests/run/cpdef_enums_import.srctree
+++ b/tests/run/cpdef_enums_import.srctree
@@ -28,13 +28,23 @@ cpdef enum NamedEnumType:
 
 cpdef foo()
 
+######## enums_without_pyx.pxd #####
+
+cpdef enum EnumTypeNotInPyx:
+    AnotherEnumValue = 500
+
 ######## no_enums.pyx ########
 
 from enums cimport *
+from enums_without_pyx cimport *
 
 def get_named_enum_value():
     return NamedEnumType.NamedEnumValue
 
+def get_named_without_pyx():
+    # This'll generate a warning but return a c int
+    return EnumTypeNotInPyx.AnotherEnumValue
+
 ######## import_enums_test.py ########
 
 # We can import enums with a star import.
@@ -51,3 +61,6 @@ assert 'FOO' not in dir(no_enums)
 assert 'foo' not in dir(no_enums)
 
 assert no_enums.get_named_enum_value() == NamedEnumType.NamedEnumValue
+# In this case the enum isn't accessible from Python (by design)
+# but the conversion to Python goes through a reasonable fallback
+assert no_enums.get_named_without_pyx() == 500
diff --git a/tests/run/cpdef_scoped_enums.pyx b/tests/run/cpdef_scoped_enums.pyx
--- a/tests/run/cpdef_scoped_enums.pyx
+++ b/tests/run/cpdef_scoped_enums.pyx
@@ -42,6 +42,28 @@ def test_enum_doc():
     pass
 
 
+def to_from_py_conversion(Enum1 val):
+    """
+    >>> to_from_py_conversion(Enum1.Item1) is Enum1.Item1
+    True
+
+    Scoped enums should not be used as flags, and therefore attempts to set them
+    with arbitrary values should fail
+    >>> to_from_py_conversion(500)
+    Traceback (most recent call last):
+    ...
+    ValueError: 500 is not a valid Enum1
+
+    # Note that the ability to bitwise-or together the two numbers is inherited
+    from the Python enum (so not in Cython's remit to prevent)
+    >>> to_from_py_conversion(Enum1.Item1 | Enum1.Item2)
+    Traceback (most recent call last):
+    ...
+    ValueError: 3 is not a valid Enum1
+    """
+    return val
+
+
 def test_pickle():
     """
     >>> from pickle import loads, dumps
diff --git a/tests/run/cpp_stl_conversion.pyx b/tests/run/cpp_stl_conversion.pyx
--- a/tests/run/cpp_stl_conversion.pyx
+++ b/tests/run/cpp_stl_conversion.pyx
@@ -270,7 +270,7 @@ cpdef enum Color:
 def test_enum_map(o):
     """
     >>> test_enum_map({RED: GREEN})
-    {0: 1}
+    {<Color.RED: 0>: <Color.GREEN: 1>}
     """
     cdef map[Color, Color] m = o
     return m