Repository: cython/cython

Issue:
[BUG] Infinite loop/crash on __add__
<!--
**PLEASE READ THIS FIRST:**
- Do not use the bug and feature tracker for support requests. Use the `cython-users` mailing list instead.
- Did you search for similar issues already? Please do, it helps to save us precious time that we otherwise could not invest into development.
- Did you try the latest master branch or pre-release? It might already have what you want to report. Also see the [Changelog](https://github.com/cython/cython/blob/master/CHANGES.rst) regarding recent changes.
-->

**Describe the bug**
It's possible to get an infinite loop for an add (and presumably other operators) operator that returns `NotImplemented` when used with a derived class.

**To Reproduce**

```cython
cdef class Base:
    def __add__(self, other):
        return NotImplemented

class Derived(Base):
    pass
```
and to test:
```
>>> import hmmm
>>> x = hmmm.Base()
>>> y = hmmm.Derived()
>>> y+x   # works OK as expected
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'Derived' and 'hmmm.Base'
>>> x+y
```

For `x+y` I get an infinite loop but [Pandas reports segmentation fault](https://github.com/pandas-dev/pandas/issues/34213#issuecomment-842694795) - probably just to do with tail-call recursion optimization or not.

**Expected behavior**

Should just be `NotImplementedError`

**Environment (please complete the following information):**
 - OS: Linux
 - Python version 3.8.9
 - Cython version 0e80efb82480b777057770ce2006a6b46ec46028 (close to current master)

**Additional context**

I think this is to do with the changes to support `__radd__` etc.. I think it's somewhere in:
https://github.com/cython/cython/blob/034fc26c1c25b5b69581464b73136038bb201ce4/Cython/Utility/ExtensionTypes.c#L383
However, I haven't traced the error through in great detail so not 100% sure exactly what's gone wrong.



Discussion:
I was playing with this issue recently. I did git bisect and the issue first occurred in following commit:
```
7ac25eea68798f46f79606cfa40a82fabd51534e is the first bad commit
commit 7ac25eea68798f46f79606cfa40a82fabd51534e
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   Wed Jun 17 13:13:28 2020 -0700

    Change the default of the "c_api_binop_methods" directive to False. (GH-3644)

    This is a backwards incompatible change that enables Python semantics for special methods by default, while the directive allows users to opt out of it and go back to the previous C-ish Cython semantics.

    See https://github.com/cython/cython/issues/2056

 Cython/Compiler/Options.py       |  2 +-
 tests/run/unicode_formatting.pyx | 10 +++++-----
 2 files changed, 6 insertions(+), 6 deletions(-)
bisect run success
```

I have checked the commit and the change is only changing default value of `c_api_binop_methods`. I have double-checked and it seems that also in master setting `c_api_binop_methods=True` is fixing the issue:
```
Traceback (most recent call last):
  File "/home/matus/dev/cython/t.py", line 2, in <module>
    print(test.Derived() + test.Derived())
TypeError: unsupported operand type(s) for +: 'Derived' and 'Derived'
```

I have also tried to find commit before bisected commit but I was not able to find any usable. Not sure whether my findings are useful or not...
Thanks. I think the commit you identify is definitely the point where is changed. I'm not sure bisecting further back is too useful.

I am looking into it (slightly slowly...). You can get it not to crash by using `specific_known_type->tp_base` rather than `Py_TYPE(left_or_right)->tp_base`, which is an improvement but doesn't quite bring it in line with Python behaviour. I suspect we'll have to compromise a bit on how closely we can hit Python behaviour though.

Resolving patch:
diff --git a/Cython/Compiler/ModuleNode.py b/Cython/Compiler/ModuleNode.py
--- a/Cython/Compiler/ModuleNode.py
+++ b/Cython/Compiler/ModuleNode.py
@@ -2109,9 +2109,9 @@ def call_slot_method(method_name, reverse):
                     "right, left" if reverse else "left, right",
                     extra_arg)
             else:
-                return '%s_maybe_call_slot(%s, left, right %s)' % (
+                return '%s_maybe_call_slot(%s->tp_base, left, right %s)' % (
                     func_name,
-                    'Py_TYPE(right)->tp_base' if reverse else 'Py_TYPE(left)->tp_base',
+                    scope.parent_type.typeptr_cname,
                     extra_arg)
 
         if get_slot_method_cname(slot.left_slot.method_name) and not get_slot_method_cname(slot.right_slot.method_name):
@@ -2122,13 +2122,16 @@ def call_slot_method(method_name, reverse):
                 slot.right_slot.method_name,
             ))
 
+        overloads_left = int(bool(get_slot_method_cname(slot.left_slot.method_name)))
+        overloads_right = int(bool(get_slot_method_cname(slot.right_slot.method_name)))
         code.putln(
             TempitaUtilityCode.load_as_string(
                 "BinopSlot", "ExtensionTypes.c",
                 context={
                     "func_name": func_name,
                     "slot_name": slot.slot_name,
-                    "overloads_left": int(bool(get_slot_method_cname(slot.left_slot.method_name))),
+                    "overloads_left": overloads_left,
+                    "overloads_right": overloads_right,
                     "call_left": call_slot_method(slot.left_slot.method_name, reverse=False),
                     "call_right": call_slot_method(slot.right_slot.method_name, reverse=True),
                     "type_cname": scope.parent_type.typeptr_cname,


Test patch:
diff --git a/tests/run/binop_reverse_methods_GH2056.pyx b/tests/run/binop_reverse_methods_GH2056.pyx
--- a/tests/run/binop_reverse_methods_GH2056.pyx
+++ b/tests/run/binop_reverse_methods_GH2056.pyx
@@ -9,6 +9,15 @@ class Base(object):
     >>> 2 + Base()
     'Base.__radd__(Base(), 2)'
 
+    >>> Base(implemented=False) + 2  #doctest: +ELLIPSIS
+    Traceback (most recent call last):
+    ...
+    TypeError: unsupported operand type...
+    >>> 2 + Base(implemented=False)  #doctest: +ELLIPSIS
+    Traceback (most recent call last):
+    ...
+    TypeError: unsupported operand type...
+
     >>> Base() ** 2
     'Base.__pow__(Base(), 2, None)'
     >>> 2 ** Base()
@@ -16,17 +25,34 @@ class Base(object):
     >>> pow(Base(), 2, 100)
     'Base.__pow__(Base(), 2, 100)'
     """
+    implemented: cython.bint
+
+    def __init__(self, *, implemented=True):
+        self.implemented = implemented
+
     def __add__(self, other):
-        return "Base.__add__(%s, %s)" % (self, other)
+        if (<Base>self).implemented:
+            return "Base.__add__(%s, %s)" % (self, other)
+        else:
+            return NotImplemented
 
     def __radd__(self, other):
-        return "Base.__radd__(%s, %s)" % (self, other)
+        if (<Base>self).implemented:
+            return "Base.__radd__(%s, %s)" % (self, other)
+        else:
+            return NotImplemented
 
     def __pow__(self, other, mod):
-        return "Base.__pow__(%s, %s, %s)" % (self, other, mod)
+        if (<Base>self).implemented:
+            return "Base.__pow__(%s, %s, %s)" % (self, other, mod)
+        else:
+            return NotImplemented
 
     def __rpow__(self, other, mod):
-        return "Base.__rpow__(%s, %s, %s)" % (self, other, mod)
+        if (<Base>self).implemented:
+            return "Base.__rpow__(%s, %s, %s)" % (self, other, mod)
+        else:
+            return NotImplemented
 
     def __repr__(self):
         return "%s()" % (self.__class__.__name__)
@@ -44,9 +70,27 @@ class OverloadLeft(Base):
     'OverloadLeft.__add__(OverloadLeft(), Base())'
     >>> Base() + OverloadLeft()
     'Base.__add__(Base(), OverloadLeft())'
+
+    >>> OverloadLeft(implemented=False) + Base(implemented=False)  #doctest: +ELLIPSIS
+    Traceback (most recent call last):
+    ...
+    TypeError: unsupported operand type...
+    >>> Base(implemented=False) + OverloadLeft(implemented=False)  #doctest: +ELLIPSIS
+    Traceback (most recent call last):
+    ...
+    TypeError: unsupported operand type...
     """
+    derived_implemented: cython.bint
+
+    def __init__(self, *, implemented=True):
+        super().__init__(implemented=implemented)
+        self.derived_implemented = implemented
+
     def __add__(self, other):
-        return "OverloadLeft.__add__(%s, %s)" % (self, other)
+        if (<OverloadLeft>self).derived_implemented:
+            return "OverloadLeft.__add__(%s, %s)" % (self, other)
+        else:
+            return NotImplemented
 
 
 @cython.c_api_binop_methods(False)
@@ -62,9 +106,27 @@ class OverloadRight(Base):
     'Base.__add__(OverloadRight(), Base())'
     >>> Base() + OverloadRight()
     'OverloadRight.__radd__(OverloadRight(), Base())'
+
+    >>> OverloadRight(implemented=False) + Base(implemented=False)  #doctest: +ELLIPSIS
+    Traceback (most recent call last):
+    ...
+    TypeError: unsupported operand type...
+    >>> Base(implemented=False) + OverloadRight(implemented=False)  #doctest: +ELLIPSIS
+    Traceback (most recent call last):
+    ...
+    TypeError: unsupported operand type...
     """
+    derived_implemented: cython.bint
+
+    def __init__(self, *, implemented=True):
+        super().__init__(implemented=implemented)
+        self.derived_implemented = implemented
+
     def __radd__(self, other):
-        return "OverloadRight.__radd__(%s, %s)" % (self, other)
+        if (<OverloadRight>self).derived_implemented:
+            return "OverloadRight.__radd__(%s, %s)" % (self, other)
+        else:
+            return NotImplemented
 
 @cython.c_api_binop_methods(True)
 @cython.cclass
@@ -79,7 +141,30 @@ class OverloadCApi(Base):
     'OverloadCApi.__add__(OverloadCApi(), Base())'
     >>> Base() + OverloadCApi()
     'OverloadCApi.__add__(Base(), OverloadCApi())'
+
+    >>> OverloadCApi(derived_implemented=False) + 2 #doctest: +ELLIPSIS
+    Traceback (most recent call last):
+    ...
+    TypeError: unsupported operand type...
+    >>> 2 + OverloadCApi(derived_implemented=False) #doctest: +ELLIPSIS
+    Traceback (most recent call last):
+    ...
+    TypeError: unsupported operand type...
     """
+    derived_implemented: cython.bint
+
+    def __init__(self, *, derived_implemented=True):
+        super().__init__(implemented=True)
+        self.derived_implemented = derived_implemented
+
     def __add__(self, other):
-        return "OverloadCApi.__add__(%s, %s)" % (self, other)
+        if isinstance(self, OverloadCApi):
+            derived_implemented = (<OverloadCApi>self).derived_implemented
+        else:
+            derived_implemented = (<OverloadCApi>other).derived_implemented
+        if derived_implemented:
+            return "OverloadCApi.__add__(%s, %s)" % (self, other)
+        else:
+            return NotImplemented
 
+# TODO: Test a class that only defines the `__r...__()` methods.