Repository: cython/cython
Pull request: Fix some issues with duplicated utilitycode
Fixes https://github.com/cython/cython/issues/3716
Hopefully fixes https://github.com/cython/cython/issues/3741 (but I can't produce a working test for it; @KmolYuan could you look at my C++ testcase and tell me what I've missed!)
Fixes https://github.com/cython/cython/issues/3734

Problem statement:
Redefinition error when putting functions in a dict
When trying to compile some Python code with Cython, I got the following kind of errors:

```
error: redefinition of XXX
```

I've managed to reduce my code to a reproducible example:

```
# cython: language_level=3

cdef f1(x, r):
    pass

cdef f2(x1, r):
    pass

cdef map = {
    "f1": f1,
    "f2": f2,
}
```

When trying to compile it with `Cython version 0.29.20`, I get the following output:

```
Compiling /home/lothiraldan/cython-bug/test2.pyx because it changed.
[1/1] Cythonizing /home/lothiraldan/cython-bug/test2.pyx
building 'test2' extension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/lothiraldan/.pyenv/versions/3.7.5/include/python3.7m -c /home/lothiraldan/cython-bug/test2.c -o build/temp.linux-x86_64-3.7/home/lothiraldan/cython-bug/test2.o
/home/lothiraldan/cython-bug/test2.c:1406:13: error: redefinition of ‘__pyx_doc_11cfunc_dot_to_py_46__Pyx_CFunc_object____object____object___to_py_wrap’
 1406 | static char __pyx_doc_11cfunc_dot_to_py_46__Pyx_CFunc_object____object____object___to_py_wrap[] = "wrap(x1, r)";
      |             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/lothiraldan/cython-bug/test2.c:1213:13: note: previous definition of ‘__pyx_doc_11cfunc_dot_to_py_46__Pyx_CFunc_object____object____object___to_py_wrap’ was here
 1213 | static char __pyx_doc_11cfunc_dot_to_py_46__Pyx_CFunc_object____object____object___to_py_wrap[] = "wrap(x, r)";
      |             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/lothiraldan/cython-bug/test2.c:1407:20: error: redefinition of ‘__pyx_mdef_11cfunc_dot_to_py_46__Pyx_CFunc_object____object____object___to_py_1wrap’
 1407 | static PyMethodDef __pyx_mdef_11cfunc_dot_to_py_46__Pyx_CFunc_object____object____object___to_py_1wrap = {"wrap", (PyCFunction)(void*)(PyCFunctionWithKeywords)__pyx_pw_11cfunc_dot_to_py_46__Pyx_CFunc_object____object____object___to_py_1wrap, METH_VARARGS|METH_KEYWORDS, __pyx_doc_11cfunc_dot_to_py_46__Pyx_CFunc_object____object____object___to_py_wrap};
      |                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/lothiraldan/cython-bug/test2.c:1214:20: note: previous definition of ‘__pyx_mdef_11cfunc_dot_to_py_46__Pyx_CFunc_object____object____object___to_py_1wrap’ was here
 1214 | static PyMethodDef __pyx_mdef_11cfunc_dot_to_py_46__Pyx_CFunc_object____object____object___to_py_1wrap = {"wrap", (PyCFunction)(void*)(PyCFunctionWithKeywords)__pyx_pw_11cfunc_dot_to_py_46__Pyx_CFunc_object____object____object___to_py_1wrap, METH_VARARGS|METH_KEYWORDS, __pyx_doc_11cfunc_dot_to_py_46__Pyx_CFunc_object____object____object___to_py_wrap};
      |                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/lothiraldan/cython-bug/test2.c:1408:18: error: redefinition of ‘__pyx_pw_11cfunc_dot_to_py_46__Pyx_CFunc_object____object____object___to_py_1wrap’
 1408 | static PyObject *__pyx_pw_11cfunc_dot_to_py_46__Pyx_CFunc_object____object____object___to_py_1wrap(PyObject *__pyx_self, PyObject *__pyx_args, PyObject *__pyx_kwds) {
      |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/lothiraldan/cython-bug/test2.c:1215:18: note: previous definition of ‘__pyx_pw_11cfunc_dot_to_py_46__Pyx_CFunc_object____object____object___to_py_1wrap’ was here
 1215 | static PyObject *__pyx_pw_11cfunc_dot_to_py_46__Pyx_CFunc_object____object____object___to_py_1wrap(PyObject *__pyx_self, PyObject *__pyx_args, PyObject *__pyx_kwds) {
      |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/lothiraldan/cython-bug/test2.c:1470:18: error: redefinition of ‘__pyx_pf_11cfunc_dot_to_py_46__Pyx_CFunc_object____object____object___to_py_wrap’
 1470 | static PyObject *__pyx_pf_11cfunc_dot_to_py_46__Pyx_CFunc_object____object____object___to_py_wrap(PyObject *__pyx_self, PyObject *__pyx_v_x1, PyObject *__pyx_v_r) {
      |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/lothiraldan/cython-bug/test2.c:1277:18: note: previous definition of ‘__pyx_pf_11cfunc_dot_to_py_46__Pyx_CFunc_object____object____object___to_py_wrap’ was here
 1277 | static PyObject *__pyx_pf_11cfunc_dot_to_py_46__Pyx_CFunc_object____object____object___to_py_wrap(PyObject *__pyx_self, PyObject *__pyx_v_x, PyObject *__pyx_v_r) {
      |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/lothiraldan/cython-bug/test2.c:1524:18: error: redefinition of ‘__Pyx_CFunc_object____object____object___to_py’
 1524 | static PyObject *__Pyx_CFunc_object____object____object___to_py(PyObject *(*__pyx_v_f)(PyObject *, PyObject *)) {
      |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/lothiraldan/cython-bug/test2.c:1331:18: note: previous definition of ‘__Pyx_CFunc_object____object____object___to_py’ was here
 1331 | static PyObject *__Pyx_CFunc_object____object____object___to_py(PyObject *(*__pyx_v_f)(PyObject *, PyObject *)) {
      |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/lothiraldan/cython-bug/test2.c:1470:18: warning: ‘__pyx_pf_11cfunc_dot_to_py_46__Pyx_CFunc_object____object____object___to_py_wrap’ defined but not used [-Wunused-function]
 1470 | static PyObject *__pyx_pf_11cfunc_dot_to_py_46__Pyx_CFunc_object____object____object___to_py_wrap(PyObject *__pyx_self, PyObject *__pyx_v_x1, PyObject *__pyx_v_r) {
      |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/lothiraldan/cython-bug/test2.c:1408:18: warning: ‘__pyx_pw_11cfunc_dot_to_py_46__Pyx_CFunc_object____object____object___to_py_1wrap’ defined but not used [-Wunused-function]
 1408 | static PyObject *__pyx_pw_11cfunc_dot_to_py_46__Pyx_CFunc_object____object____object___to_py_1wrap(PyObject *__pyx_self, PyObject *__pyx_args, PyObject *__pyx_kwds) {
      |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/lothiraldan/cython-bug/test2.c:1331:18: warning: ‘__Pyx_CFunc_object____object____object___to_py’ defined but not used [-Wunused-function]
 1331 | static PyObject *__Pyx_CFunc_object____object____object___to_py(PyObject *(*__pyx_v_f)(PyObject *, PyObject *)) {
      |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
error: command 'gcc' failed with exit status 1
```

I only get the issue when I the functions have a the same of parameters but different parameters names. If I add type to some parameter of one function, I also don't get an error.

I used the workaround for now, so it's not an urgent issue for me.

gcc compile time error with cimport_from_pyx
The following setup with 2 pyx files and the corresponding setup.py to build them results in a gcc compile time error:

```
#>>> moda.pyx

cdef class ClassA:
    cdef int[2] a

#>>> modb.pyx

from moda cimport ClassA

cdef class ClassB(ClassA):
    cdef int[2] b

#>>> setup.py

from setuptools import setup
from Cython.Build import cythonize
import Cython.Compiler.Options

Cython.Compiler.Options.cimport_from_pyx = True

setup(
        ext_modules = cythonize(["moda.pyx", "modb.pyx"],
            compiler_directives={'language_level': 3})
)
```

The error is as follows:

```
$ python setup.py build
running build
running build_ext
building 'modb' extension
gcc -pthread -B /home/clemens/miniconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/clemens/miniconda3/include/python3.7m -c modb.c -o build/temp.linux-x86_64-3.7/modb.o
modb.c:2585:32: error: redefinition of ‘__Pyx_carray_to_py_int’
 static CYTHON_INLINE PyObject *__Pyx_carray_to_py_int(int *__pyx_v_v, Py_ssize_t __pyx_v_length) {
                                ^~~~~~~~~~~~~~~~~~~~~~
modb.c:2038:32: note: previous definition of ‘__Pyx_carray_to_py_int’ was here
 static CYTHON_INLINE PyObject *__Pyx_carray_to_py_int(int *__pyx_v_v, Py_ssize_t __pyx_v_length) {
                                ^~~~~~~~~~~~~~~~~~~~~~
modb.c:2696:32: error: redefinition of ‘__Pyx_carray_to_tuple_int’
 static CYTHON_INLINE PyObject *__Pyx_carray_to_tuple_int(int *__pyx_v_v, Py_ssize_t __pyx_v_length) {
                                ^~~~~~~~~~~~~~~~~~~~~~~~~
modb.c:2149:32: note: previous definition of ‘__Pyx_carray_to_tuple_int’ was here
 static CYTHON_INLINE PyObject *__Pyx_carray_to_tuple_int(int *__pyx_v_v, Py_ssize_t __pyx_v_length) {
                                ^~~~~~~~~~~~~~~~~~~~~~~~~
modb.c:2804:12: error: redefinition of ‘__Pyx_carray_from_py_int’
 static int __Pyx_carray_from_py_int(PyObject *__pyx_v_o, int *__pyx_v_v, Py_ssize_t __pyx_v_length) {
            ^~~~~~~~~~~~~~~~~~~~~~~~
modb.c:2257:12: note: previous definition of ‘__Pyx_carray_from_py_int’ was here
 static int __Pyx_carray_from_py_int(PyObject *__pyx_v_o, int *__pyx_v_v, Py_ssize_t __pyx_v_length) {
            ^~~~~~~~~~~~~~~~~~~~~~~~
error: command 'gcc' failed with exit status 1
```

The issue seems to be that certain utility functions are getting inserted into the C source file multiple times. The error goes away if `cimport_from_pyx` is disabled and a moda.pxd file is generated manually.

This is Python 3.7.4, Cython 0.29.20, Linux (WSL) with gcc 7.5.0.

Conversion function redefinition error when handling C++ types
I have a extension class which including C++ member (`std::vector<my_struct>`, and `my_struct` has some `std::pair<int, int>` members).
And there also has a `std::pair<int, int>` type definition in the same source.

I'm found that the conversion function (`std::pair<int, int>` to `tuple[int, int]`) is defined twice unless added `@cython.auto_pickle(False)` on my extension class.

```
11510:18: error: redefinition of 'PyObject* __pyx_convert_pair_to_py_int____int(const std::pair<int, int>&)'
 static PyObject *__pyx_convert_pair_to_py_int____int(std::pair<int,int>  const &__pyx_v_p) {
                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10055:18: note: 'PyObject* __pyx_convert_pair_to_py_int____int(const std::pair<int, int>&)' previously defined here
 static PyObject *__pyx_convert_pair_to_py_int____int(std::pair<int,int>  const &__pyx_v_p) {
                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```

I'm not clearly know how Cython checks the tool functions of `ctypedef`.
In another case My extension members are sharing a same `ctypedef` with local variables, but this error is not occurred.

Hints from issue discussion:
It probably give an error but it seems like a relatively pointless thing to do: you're not putting a bunch of `cdef` functions in a `dict` - you're putting a bunch of Python wrappers round `cdef` functions into a `dict`. It's pretty much equivalent to if you'd just declared the functions as `def/cpdef`
It doesn't look hugely easy to fix. The utility code is generated https://github.com/cython/cython/blob/9c26b302791515f608d4b200f692dcec9f066229/Cython/Compiler/PyrexTypes.py#L3279. Earlier, it tries to reuse `self.to_py_function` of the function type, however each `cdef` function has its own instance of the type.

-----------------------------

An alternative way of fixing it is https://github.com/cython/cython/blob/9c26b302791515f608d4b200f692dcec9f066229/Cython/Compiler/Nodes.py#L3384 the `DefNodeWrapper` tries to get a unique prefix. However, it only gets a prefix from the utility code module, not the overall module.
When I'm using `vector[pair[int, int]]` and `map[int, bint]` in the same file, the error comes again.
Regardless creating type alias or using original form can't solve the problem.

Maybe somewhere Cython treats `bint` as a different type when generating conversion functions.
This looks like substantially the same issue as https://github.com/cython/cython/issues/3716 and https://github.com/cython/cython/issues/3734. The problem is where Cython utility code is itself generating utility code. Each lot of 2nd-level generated utility code is generated without access to the full module scope so can end up sharing names.

I'll try to have a look at it in the near future. Although it looked trick to solve last time I looked
My case is little complex, which mixed a lot of functions and cannot extracted to a small one.
I'm not actually know what cause the duplicated code in my module.

The repo is here, change this line from `Status &` to `object` can reproduce the error: (build by `setup.py`)
https://github.com/KmolYuan/pyslvs/blob/master/pyslvs/topo_config.pyx#L187

But this error does not bother me at the moment.

Commit message: Node -> StatNode

Patch to review:
diff --git a/Cython/Compiler/Pipeline.py b/Cython/Compiler/Pipeline.py
index b7f512e7ea5..8372aaa7920 100644
--- a/Cython/Compiler/Pipeline.py
+++ b/Cython/Compiler/Pipeline.py
@@ -285,11 +285,26 @@ def create_pyx_as_pxd_pipeline(context, result):
                                            FlattenInListTransform,
                                            WithTransform
                                            ])
+    from .Visitor import VisitorTransform
+    class SetInPxdTransform(VisitorTransform):
+        # a number of nodes have an in_pxd attribute which affects AnalyseDeclarationsTransform
+        # (for example controlling pickling generation)
+        # set this
+        def visit_StatNode(self, node):
+            if hasattr(node, "in_pxd"):
+                node.in_pxd = True
+            self.visitchildren(node)
+            return node
+
+        def visit_Node(self, node):
+            self.visitchildren(node)
+            return node
+
     for stage in pyx_pipeline:
         pipeline.append(stage)
         if isinstance(stage, AnalyseDeclarationsTransform):
-            # This is the last stage we need.
-            break
+            pipeline.insert(-1, SetInPxdTransform())
+            break  # This is the last stage we need.
     def fake_pxd(root):
         for entry in root.scope.entries.values():
             if not entry.in_cinclude:
diff --git a/Cython/Compiler/PyrexTypes.py b/Cython/Compiler/PyrexTypes.py
index dc63675c314..8c6ef5bfa69 100644
--- a/Cython/Compiler/PyrexTypes.py
+++ b/Cython/Compiler/PyrexTypes.py
@@ -3219,8 +3219,16 @@ def create_to_py_utility_code(self, env):
         if not self.can_coerce_to_pyobject(env):
             return False
         from .UtilityCode import CythonUtilityCode
+
+        # include argument names into the c function name to ensure cname is unique
+        # between functions with identical types but different argument names
+        from .Symtab import punycodify_name
+        def arg_name_part(arg):
+            return "%s%s" % (len(arg.name), punycodify_name(arg.name)) if arg.name else "0"
+        arg_names = [ arg_name_part(arg) for arg in self.args ]
+        arg_names = "_".join(arg_names)
         safe_typename = re.sub('[^a-zA-Z0-9]', '__', self.declaration_code("", pyrex=1))
-        to_py_function = "__Pyx_CFunc_%s_to_py" % safe_typename
+        to_py_function = "__Pyx_CFunc_%s_to_py_%s" % (safe_typename, arg_names)
 
         for arg in self.args:
             if not arg.type.is_pyobject and not arg.type.create_from_py_utility_code(env):
diff --git a/tests/run/duplicate_utilitycode.pyx b/tests/run/duplicate_utilitycode.pyx
new file mode 100644
index 00000000000..5c57a4a337e
--- /dev/null
+++ b/tests/run/duplicate_utilitycode.pyx
@@ -0,0 +1,27 @@
+# mode: run
+
+# There were a few cases where duplicate utility code definitions (i.e. with the same name)
+# could be generated, causing C compile errors. This file tests them.
+
+from __future__ import print_function
+
+cdef f1(x, r):
+    return "f1"
+
+cdef f2(x1, r):
+    return "f2"
+
+def make_map():
+    """
+    https://github.com/cython/cython/issues/3716
+    This is testing the generation of wrappers for f1 and f2
+    >>> for k, f in make_map().items():
+    ...    print(k == f(0, 0))  # in both cases the functions should just return their name
+    True
+    True
+    """
+    cdef map = {
+        "f1": f1,
+        "f2": f2,
+    }
+    return map
diff --git a/tests/run/duplicate_utilitycode_from_pyx.srctree b/tests/run/duplicate_utilitycode_from_pyx.srctree
new file mode 100644
index 00000000000..c539af7a60a
--- /dev/null
+++ b/tests/run/duplicate_utilitycode_from_pyx.srctree
@@ -0,0 +1,29 @@
+
+
+PYTHON setup.py build_ext --inplace
+PYTHON -c "import modb; modb.ClassB()"
+
+#################### moda.pyx ####################
+
+cdef class ClassA:
+    cdef int[2] a
+
+#################### modb.pyx #####################
+
+from moda cimport ClassA
+
+cdef class ClassB(ClassA):
+    cdef int[2] b
+
+###################### setup.py ###################
+
+from setuptools import setup
+from Cython.Build import cythonize
+import Cython.Compiler.Options
+
+Cython.Compiler.Options.cimport_from_pyx = True
+
+setup(
+        ext_modules = cythonize(["moda.pyx", "modb.pyx"],
+            compiler_directives={'language_level': 3})
+)


Code review comments:
diff_hunk: @@ -285,11 +285,26 @@ def create_pyx_as_pxd_pipeline(context, result):
                                            FlattenInListTransform,
                                            WithTransform
                                            ])
+    from .Visitor import VisitorTransform
+    class SetInPxdTransform(VisitorTransform):
+        # a number of nodes have an in_pxd attribute which affects AnalyseDeclarationsTransform
+        # (for example controlling pickling generation)
+        # set this
+        def visit_StatNode(self, node):
+            if hasattr(node, "in_pxd"):
+                node.in_pxd = True
+            self.visitchildren(node)
+            return node
+
+        def visit_Node(self, node):
+            self.visitchildren(node)
+            return node
path: Cython/Compiler/Pipeline.py
text: @user1:
This is a bit quicker:
```suggestion
        visit_Node = VisitorTransform.recurse_to_children
```

diff_hunk: @@ -285,11 +285,26 @@ def create_pyx_as_pxd_pipeline(context, result):
                                            FlattenInListTransform,
                                            WithTransform
                                            ])
+    from .Visitor import VisitorTransform
+    class SetInPxdTransform(VisitorTransform):
+        # a number of nodes have an in_pxd attribute which affects AnalyseDeclarationsTransform
+        # (for example controlling pickling generation)
+        # set this
path: Cython/Compiler/Pipeline.py
text: @user1:
```suggestion
        # A number of nodes have an "in_pxd" attribute which affects AnalyseDeclarationsTransform
        # (for example controlling pickling generation). Set it, to make sure we don't mix them up with
        # the importing main module.
        # FIXME: This should be done closer to the parsing step.
```