Repository: pandas-dev/pandas
Pull request: remove ops div class to solve #21374
- [x] closes #21374
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] Existing tests updated/extended
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst` file if fixing a bug or adding a new feature.

`core.computation.div.ops.Div` class that was introduced to manage unexpected upcast from float32 to float64 looks to be responsible of the imaginary part of complexes being dropped. It seams that numpy and numexpr have evolved since #12388 and this fix is no more needed.
This PR removes `core.computation.div.ops.Div` class and extends tests to cover for both floats and complex.


Problem statement:
pd.eval() discards imaginary part in division "/"
#### Code Sample

```python
data = {"a": [1 + 2j], "b": [1 + 1j]}
df = pd.DataFrame(data = data)
df.eval("a/b")

/usr/local/lib64/python3.6/site-packages/pandas/core/dtypes/cast.py:730: ComplexWarning: Casting complex values to real discards the imaginary part
  return arr.astype(dtype, copy=True)

0    1.0
dtype: float64
```
#### Problem description
The output type was coerced into a float. This also happens by assigning the result to another existing column:

```python
data = {"a": [1 + 2j], "b": [1 + 1j], "c": [1j]}
df = pd.DataFrame(data = data)
df.eval("c = a/b")

/usr/local/lib64/python3.6/site-packages/pandas/core/dtypes/cast.py:730: ComplexWarning: Casting complex values to real discards the imaginary part
  return arr.astype(dtype, copy=True)
Out[82]: 
        a       b    c
0  (1+2j)  (1+1j)  1.0
```
And even if the operation is in place:

```python
data = {"a": [1 + 2j], "b": [1 + 1j], "c": [1j]}
df = pd.DataFrame(data = data)
df.eval("c = a/b", inplace = True)
/usr/local/lib64/python3.6/site-packages/pandas/core/dtypes/cast.py:730: ComplexWarning: Casting complex values to real discards the imaginary part
  return arr.astype(dtype, copy=True)

df
        a       b    c
0  (1+2j)  (1+1j)  1.0
```

#### Expected Output
The expected output is
```python
df["a"]/df["b"]

0    (1.5+0.5j)
dtype: complex128
```
The problem seems to happen only with the "/" operator. In fact, the correct result can be obtained by replacing the division with a multiplication and a negative exponent:

```python
df.eval("a*b**(-1)")

0    (1.5+0.5j)
dtype: complex128
```

#### Output of ``pd.show_versions()``
<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.5.final.0
python-bits: 64
OS: Linux
OS-release: 4.16.12-300.fc28.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.utf8
LOCALE: en_US.UTF-8

pandas: 0.23.0
pytest: None
pip: 9.0.3
setuptools: 39.2.0
Cython: None
numpy: 1.14.4
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 6.4.0
sphinx: None
patsy: 0.4.1
dateutil: 2.7.3
pytz: 2018.4
blosc: None
bottleneck: 1.2.1
tables: 3.4.2
numexpr: 2.6.1
feather: None
matplotlib: 2.2.2
openpyxl: None
xlrd: 1.0.0
xlwt: 1.1.2
xlsxwriter: None
lxml: 4.1.1
bs4: 4.6.0
html5lib: 0.999999999
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

</details>

Hints from issue discussion:
@fillipe-gsm : How odd!  Investigation and patch are welcome!
@gfyoung @fillipe-gsm maybe the problem lies in the usage of `numpy.imag` casting (if we are using it) , have a look at it's documentation here : https://docs.scipy.org/doc/numpy/reference/generated/numpy.imag.html

It also mentions that " If *val* is real, the type of *val* is used for the output. If val has complex elements, the returned type is float. " 
Can it happen that "/" operator triggers the type caster and the "**(-1)" doesn't?
@gfyoung @uds5501 
I did some throghout investigation line by line starting in the `pandas.core.computation.eval` module, and I may have found some lead in the `Div` class inside the `pandas.core.computation.ops` module, which seems to be created specially to handle divisions. Given the odd behavior happening only with the "/" operator, this makes some sense. Since it is a small code, here it is:

```python
class Div(BinOp):

    """Div operator to special case casting.

    Parameters
    ----------
    lhs, rhs : Term or Op
        The Terms or Ops in the ``/`` expression.
    truediv : bool
        Whether or not to use true division. With Python 3 this happens
        regardless of the value of ``truediv``.
    """

    def __init__(self, lhs, rhs, truediv, *args, **kwargs):
        super(Div, self).__init__('/', lhs, rhs, *args, **kwargs)

        if not isnumeric(lhs.return_type) or not isnumeric(rhs.return_type):
            raise TypeError("unsupported operand type(s) for {0}:"
                            " '{1}' and '{2}'".format(self.op,
                                                      lhs.return_type,
                                                      rhs.return_type))

        if truediv or PY3:
            # do not upcast float32s to float64 un-necessarily
            acceptable_dtypes = [np.float32, np.float_]
            _cast_inplace(com.flatten(self), acceptable_dtypes, np.float_)
```

It seems that this class is simply not ready to handle complex numbers given the instruction `acceptable_dtypes = [np.float32, np.float_]`. Thus, I tried to replace this line with `acceptable_dtypes = [np.float32, np.float_, np.complex_, np.complex64]`. Apparently, this solves the problem:

```python
data = {"a": [1 + 2j], "b": [1 + 1j]}
 df = pd.DataFrame(data = data)
 df.eval("a/b")
  
0    (1.5+0.5j)
dtype: complex128
```

What do you guys think?
@fillipe-gsm : That seems reasonable to me.  A PR for this would be great!
This is quite an old issue but it was not solved and it looks similar to issues I encountered working on https://github.com/hgrecco/pint-pandas/issues/137 and #58748 :

- current issue is `eval` of `complex` leads to `float` when `complex` is expected
- one part of #58748 is `eval` of `ExtensionArray` leads to `float` when  `ExtensionArray[float]` is expected
- final issue with https://github.com/hgrecco/pint-pandas/issues/137  is `eval` of `PintArray` (sort of ExtensionArray) leads to `float` when `PintArray` is expected with proper units

This makes me questioning the use of `_cast_inplace` function that was introduced by cc1025a62019215a0fa38a891e07e6ca6ba656f1 to resolve #12388. It looks it was designed to manage only floats at the time. Now complex and ExtensionArray and other shall be managed.

A specific test was introduced at the time. It is now here: https://github.com/pandas-dev/pandas/blob/3b48b17e52f3f3837b9ba8551c932f44633b5ff8/pandas/tests/computation/test_eval.py#L761
I tried to run this test removing the used of `_cast_inplace` and the test ran successfully. Changes in numpy or expression backend seams to deprecate the use of `_cast_inplace`. When it is removed complex and ExtensionArray are computed properly @jreback @jennolsen84 do you have any opinion on the matter?

Since `TypeError: unsupported operan types (...)` is also manage somewhere else  `Div` class of `pandas.core.computation.ops` would not be necessary anymore and we could remove it.
https://github.com/pandas-dev/pandas/blob/3b48b17e52f3f3837b9ba8551c932f44633b5ff8/pandas/core/computation/ops.py#L515



note that test `pandas.tests.frame.test_query_eval.test_extension_array_eval` that may be added by #58793 shall then be modified to pass (resulting dtype is no more `float` but `pandas.arrays.FloatingArray`

```diff
-    expected = Series([0.25, 0.40, 0.50])
+    expected = Series(pd.array([0.25, 0.40, 0.50]))
```
I worked a bit on this and I am facing an issue when updating `test_binop_typecasting` to cover for complex (it covers floats only when it could be upgraded to int and complex). complex64 are upcasted to complex128 when using `numexpr` engine.

Commit message: adding specific test

Patch to review:
diff --git a/doc/source/whatsnew/v3.0.0.rst b/doc/source/whatsnew/v3.0.0.rst
index d9d2330f8f11b..b655032305be9 100644
--- a/doc/source/whatsnew/v3.0.0.rst
+++ b/doc/source/whatsnew/v3.0.0.rst
@@ -615,6 +615,7 @@ Other
 ^^^^^
 - Bug in :class:`DataFrame` when passing a ``dict`` with a NA scalar and ``columns`` that would always return ``np.nan`` (:issue:`57205`)
 - Bug in :func:`eval` on :class:`ExtensionArray` on including division ``/`` failed with a ``TypeError``. (:issue:`58748`)
+- Bug in :func:`eval` on :class:`complex` including division ``/`` discards imaginary part. (:issue:`21374`)
 - Bug in :func:`eval` where the names of the :class:`Series` were not preserved when using ``engine="numexpr"``. (:issue:`10239`)
 - Bug in :func:`unique` on :class:`Index` not always returning :class:`Index` (:issue:`57043`)
 - Bug in :meth:`DataFrame.apply` where passing ``engine="numba"`` ignored ``args`` passed to the applied function (:issue:`58712`)
diff --git a/pandas/_testing/__init__.py b/pandas/_testing/__init__.py
index fb8ca8aad3428..1cd91ee5b120c 100644
--- a/pandas/_testing/__init__.py
+++ b/pandas/_testing/__init__.py
@@ -107,6 +107,7 @@
 
 COMPLEX_DTYPES: list[Dtype] = [complex, "complex64", "complex128"]
 STRING_DTYPES: list[Dtype] = [str, "str", "U"]
+COMPLEX_FLOAT_DTYPES: list[Dtype] = [*COMPLEX_DTYPES, *FLOAT_NUMPY_DTYPES]
 
 DATETIME64_DTYPES: list[Dtype] = ["datetime64[ns]", "M8[ns]"]
 TIMEDELTA64_DTYPES: list[Dtype] = ["timedelta64[ns]", "m8[ns]"]
diff --git a/pandas/conftest.py b/pandas/conftest.py
index c3bfc8c06ad8a..70e729dfb98a4 100644
--- a/pandas/conftest.py
+++ b/pandas/conftest.py
@@ -1448,6 +1448,21 @@ def complex_dtype(request):
     return request.param
 
 
+@pytest.fixture(params=tm.COMPLEX_FLOAT_DTYPES)
+def complex_or_float_dtype(request):
+    """
+    Parameterized fixture for complex and numpy float dtypes.
+
+    * complex
+    * 'complex64'
+    * 'complex128'
+    * float
+    * 'float32'
+    * 'float64'
+    """
+    return request.param
+
+
 @pytest.fixture(params=tm.SIGNED_INT_NUMPY_DTYPES)
 def any_signed_int_numpy_dtype(request):
     """
diff --git a/pandas/core/computation/expr.py b/pandas/core/computation/expr.py
index b287cd542068d..b074e768e0842 100644
--- a/pandas/core/computation/expr.py
+++ b/pandas/core/computation/expr.py
@@ -32,7 +32,6 @@
     UNARY_OPS_SYMS,
     BinOp,
     Constant,
-    Div,
     FuncNode,
     Op,
     Term,
@@ -374,7 +373,7 @@ class BaseExprVisitor(ast.NodeVisitor):
         "Add",
         "Sub",
         "Mult",
-        None,
+        "Div",
         "Pow",
         "FloorDiv",
         "Mod",
@@ -537,9 +536,6 @@ def visit_BinOp(self, node, **kwargs):
         left, right = self._maybe_downcast_constants(left, right)
         return self._maybe_evaluate_binop(op, op_class, left, right)
 
-    def visit_Div(self, node, **kwargs):
-        return lambda lhs, rhs: Div(lhs, rhs)
-
     def visit_UnaryOp(self, node, **kwargs):
         op = self.visit(node.op)
         operand = self.visit(node.operand)
diff --git a/pandas/core/computation/ops.py b/pandas/core/computation/ops.py
index 056325fd2e4ab..a1a5f77f8539e 100644
--- a/pandas/core/computation/ops.py
+++ b/pandas/core/computation/ops.py
@@ -18,7 +18,6 @@
 
 from pandas.core.dtypes.common import (
     is_list_like,
-    is_numeric_dtype,
     is_scalar,
 )
 
@@ -328,31 +327,6 @@ def _not_in(x, y):
     _binary_ops_dict.update(d)
 
 
-def _cast_inplace(terms, acceptable_dtypes, dtype) -> None:
-    """
-    Cast an expression inplace.
-
-    Parameters
-    ----------
-    terms : Op
-        The expression that should cast.
-    acceptable_dtypes : list of acceptable numpy.dtype
-        Will not cast if term's dtype in this list.
-    dtype : str or numpy.dtype
-        The dtype to cast to.
-    """
-    dt = np.dtype(dtype)
-    for term in terms:
-        if term.type in acceptable_dtypes:
-            continue
-
-        try:
-            new_value = term.value.astype(dt)
-        except AttributeError:
-            new_value = dt.type(term.value)
-        term.update(new_value)
-
-
 def is_term(obj) -> bool:
     return isinstance(obj, Term)
 
@@ -509,32 +483,6 @@ def _disallow_scalar_only_bool_ops(self) -> None:
             raise NotImplementedError("cannot evaluate scalar only bool ops")
 
 
-class Div(BinOp):
-    """
-    Div operator to special case casting.
-
-    Parameters
-    ----------
-    lhs, rhs : Term or Op
-        The Terms or Ops in the ``/`` expression.
-    """
-
-    def __init__(self, lhs, rhs) -> None:
-        super().__init__("/", lhs, rhs)
-
-        if not is_numeric_dtype(lhs.return_type) or not is_numeric_dtype(
-            rhs.return_type
-        ):
-            raise TypeError(
-                f"unsupported operand type(s) for {self.op}: "
-                f"'{lhs.return_type}' and '{rhs.return_type}'"
-            )
-
-        # do not upcast float32s to float64 un-necessarily
-        acceptable_dtypes = [np.float32, np.float64]
-        _cast_inplace(com.flatten(self), acceptable_dtypes, np.float64)
-
-
 UNARY_OPS_SYMS = ("+", "-", "~", "not")
 _unary_ops_funcs = (operator.pos, operator.neg, operator.invert, operator.invert)
 _unary_ops_dict = dict(zip(UNARY_OPS_SYMS, _unary_ops_funcs))
diff --git a/pandas/tests/computation/test_eval.py b/pandas/tests/computation/test_eval.py
index d52f33fe80434..3206cf339adb4 100644
--- a/pandas/tests/computation/test_eval.py
+++ b/pandas/tests/computation/test_eval.py
@@ -758,16 +758,25 @@ class TestTypeCasting:
     # maybe someday... numexpr has too many upcasting rules now
     # chain(*(np.core.sctypes[x] for x in ['uint', 'int', 'float']))
     @pytest.mark.parametrize("left_right", [("df", "3"), ("3", "df")])
-    def test_binop_typecasting(self, engine, parser, op, float_numpy_dtype, left_right):
-        df = DataFrame(
-            np.random.default_rng(2).standard_normal((5, 3)), dtype=float_numpy_dtype
-        )
+    def test_binop_typecasting(
+        self, engine, parser, op, complex_or_float_dtype, left_right, request
+    ):
+        # GH#21374
+        dtype = complex_or_float_dtype
+        df = DataFrame(np.random.default_rng(2).standard_normal((5, 3)), dtype=dtype)
         left, right = left_right
         s = f"{left} {op} {right}"
         res = pd.eval(s, engine=engine, parser=parser)
-        assert df.values.dtype == float_numpy_dtype
-        assert res.values.dtype == float_numpy_dtype
-        tm.assert_frame_equal(res, eval(s))
+        if dtype == "complex64" and engine == "numexpr":
+            mark = pytest.mark.xfail(
+                reason="numexpr issue with complex that are upcast "
+                "to complex 128 "
+                "https://github.com/pydata/numexpr/issues/492"
+            )
+            request.node.add_marker(mark)
+        assert df.values.dtype == dtype
+        assert res.values.dtype == dtype
+        tm.assert_frame_equal(res, eval(s), check_exact=False)
 
 
 # -------------------------------------
diff --git a/pandas/tests/frame/test_query_eval.py b/pandas/tests/frame/test_query_eval.py
index ff1bf5632e920..a1cca3b4174f6 100644
--- a/pandas/tests/frame/test_query_eval.py
+++ b/pandas/tests/frame/test_query_eval.py
@@ -202,11 +202,23 @@ def test_eval_simple(self, engine, parser):
         expected = df["a"]
         tm.assert_series_equal(expected, res)
 
-    def test_extension_array_eval(self, engine, parser):
+    def test_extension_array_eval(self, engine, parser, request):
         # GH#58748
+        if engine == "numexpr":
+            mark = pytest.mark.xfail(
+                reason="numexpr does not support extension array dtypes"
+            )
+            request.node.add_marker(mark)
         df = DataFrame({"a": pd.array([1, 2, 3]), "b": pd.array([4, 5, 6])})
         result = df.eval("a / b", engine=engine, parser=parser)
-        expected = Series([0.25, 0.40, 0.50])
+        expected = Series(pd.array([0.25, 0.40, 0.50]))
+        tm.assert_series_equal(result, expected)
+
+    def test_complex_eval(self, engine, parser):
+        # GH#21374
+        df = DataFrame({"a": [1 + 2j], "b": [1 + 1j]})
+        result = df.eval("a/b", engine=engine, parser=parser)
+        expected = Series([1.5 + 0.5j])
         tm.assert_series_equal(result, expected)
 
 


Code review comments:
diff_hunk: @@ -758,16 +758,25 @@ class TestTypeCasting:
     # maybe someday... numexpr has too many upcasting rules now
     # chain(*(np.core.sctypes[x] for x in ['uint', 'int', 'float']))
     @pytest.mark.parametrize("left_right", [("df", "3"), ("3", "df")])
-    def test_binop_typecasting(self, engine, parser, op, float_numpy_dtype, left_right):
-        df = DataFrame(
-            np.random.default_rng(2).standard_normal((5, 3)), dtype=float_numpy_dtype
-        )
+    def test_binop_typecasting(
+        self, engine, parser, op, complex_or_float_dtype, left_right, request
+    ):
+        # GH#21374
+        dtype = complex_or_float_dtype
+        df = DataFrame(np.random.default_rng(2).standard_normal((5, 3)), dtype=dtype)
         left, right = left_right
         s = f"{left} {op} {right}"
         res = pd.eval(s, engine=engine, parser=parser)
-        assert df.values.dtype == float_numpy_dtype
-        assert res.values.dtype == float_numpy_dtype
-        tm.assert_frame_equal(res, eval(s))
+        if dtype == "complex64" and engine == "numexpr":
+            mark = pytest.mark.xfail(
+                reason="numexpr issue with complex that are upcast "
+                "to complex 128 "
+                "https://github.com/pydata/numexpr/issues/492"
+            )
+            request.node.add_marker(mark)
path: pandas/tests/computation/test_eval.py
text: @user1:
```suggestion
            request.applymarker(mark)
```

diff_hunk: @@ -202,11 +202,23 @@ def test_eval_simple(self, engine, parser):
         expected = df["a"]
         tm.assert_series_equal(expected, res)
 
-    def test_extension_array_eval(self, engine, parser):
+    def test_extension_array_eval(self, engine, parser, request):
         # GH#58748
+        if engine == "numexpr":
+            mark = pytest.mark.xfail(
+                reason="numexpr does not support extension array dtypes"
+            )
+            request.node.add_marker(mark)
path: pandas/tests/frame/test_query_eval.py
text: @user1:
```suggestion
            request.applymarker(mark)
```