Repository: modin-project/modin
Pull request: FIX-#4411: Fix binary_op between datetime64 Series and pandas timedelta
<!--
Thank you for your contribution!
Please review the contributing docs: https://modin.readthedocs.io/en/latest/development/contributing.html
if you have questions about contributing.
-->

## What do these changes do?

This PR addresses #4411 which tried to add a pandas.timedeltas to a datetime64 Series. The bug ended up being tied up with how we validated binary operations, so I took the liberty to clean up code in `_validate_other` since I didn't see any of the dtype flags used anywhere else in the code. I also added in a test to validate the original posted issue. 

- [x] commit message follows format outlined [here](https://modin.readthedocs.io/en/latest/development/contributing.html#commit-message-formatting)
- [x] passes `flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py`
- [x] passes `black --check modin/ asv_bench/benchmarks scripts/doc_checker.py`
- [x] signed commit with `git commit -s` <!-- you can amend your commit with a signature via `git commit -amend -s` -->
- [x] Resolves #4411 
- [x] tests added and passing
- [x] module layout described at `docs/development/architecture.rst` is up-to-date <!-- if you have added, renamed or removed files or directories please update the documentation accordingly -->
- [x] added (Issue Number: PR title (PR Number)) and github username to release notes for next major release <!-- e.g. DOCS-#4077: Add release notes template to docs folder (#4078) -->


Problem statement:
modin datetime Series fails when touches pandas timedeltas
### System information
- **OS Platform and Distribution**: Ubuntu 18.04.5 LTS
- **Modin version**: 0.14.0
- **Python version**: 3.8.5
- **Code to reproduce**:
    `modin.pandas.date_range('2000-12-12','2000-12-13').to_series().pipe(modin.pandas.Series) + modin.pandas.to_timedelta([1,2], unit='h')`

i ran into this problem when read excel file and parsed a column with `modin.pandas.to_datetime` (and `pandas.to_datetime` aswell!), the dataframe coulmn became a `modin.pandas.Series`. This series errors out everything i try to do with it.


Edit: i tried to create a DataFrame directly to check whether it is the read_excel that is the culprit. Spoiler: it isnt
```
df = modin.pandas.DataFrame(modin.pandas.date_range('2000-12-12','2000-12-13'))
df[0] + modin.pandas.to_timedelta([1,2], unit='h')
```
`TypeError: Cannot do operation non-numeric dtypes`

Hints from issue discussion:
@iliya-malecki thank you for reporting this bug. In can reproduce it at Modin commit 5825856a4ee19ff0dd5d1beaf6090bf601d0b411. Here's a copy-pastable script to reproduce the bug:

```python
import modin.pandas as pd
import numpy as np
import pandas

# Make a modin series of dates
s = pd.Series(np.datetime64("2000-12-12"))
# Make a pandas.core.indexes.timedeltas.TimedeltaIndex
deltas = pd.to_timedelta([1], unit="h")
# Adding the Modin series to the deltas gives TypeError
s + deltas
# But adding the pandas series to the deltas gives 
# 0   2000-12-12 01:00:00
# dtype: datetime64[ns]
print(pandas.Series(np.datetime64("2000-12-12")) + deltas)
```

The Modin team will try to fix this bug.

Commit message: Added release notes

Patch to review:
diff --git a/docs/release_notes/release_notes-0.16.0.rst b/docs/release_notes/release_notes-0.16.0.rst
index a3f1deeb9f1..aba3fe35671 100644
--- a/docs/release_notes/release_notes-0.16.0.rst
+++ b/docs/release_notes/release_notes-0.16.0.rst
@@ -11,6 +11,7 @@ Key Features and Updates
   * FIX-#4059: Add cell-wise execution for binary ops, fix bin ops for empty dataframes (#4391)
   * FIX-#4589: Pin protobuf<4.0.0 to fix ray (#4590)
   * FIX-#4577: Set attribute of Modin dataframe to updated value (#4588)
+  * FIX-#4411: Fix binary_op between datetime64 Series and pandas timedelta (#4592)
 * Performance enhancements
   * PERF-#4182: Add cell-wise execution for binary ops, fix bin ops for empty dataframes (#4391)
 * Benchmarking enhancements
diff --git a/modin/pandas/base.py b/modin/pandas/base.py
index 943c97e2652..503a18107ae 100644
--- a/modin/pandas/base.py
+++ b/modin/pandas/base.py
@@ -222,10 +222,6 @@ def _validate_other(
         self,
         other,
         axis,
-        numeric_only=False,
-        numeric_or_time_only=False,
-        numeric_or_object_only=False,
-        comparison_dtypes_only=False,
         compare_index=False,
     ):
         """
@@ -300,44 +296,17 @@ def _validate_other(
             if not self.index.equals(other.index):
                 raise TypeError("Cannot perform operation with non-equal index")
         # Do dtype checking.
-        if numeric_only:
-            if not all(
-                is_numeric_dtype(self_dtype) and is_numeric_dtype(other_dtype)
-                for self_dtype, other_dtype in zip(self._get_dtypes(), other_dtypes)
-            ):
-                raise TypeError("Cannot do operation on non-numeric dtypes")
-        elif numeric_or_object_only:
-            if not all(
-                (is_numeric_dtype(self_dtype) and is_numeric_dtype(other_dtype))
-                or (is_object_dtype(self_dtype) and is_object_dtype(other_dtype))
-                for self_dtype, other_dtype in zip(self._get_dtypes(), other_dtypes)
-            ):
-                raise TypeError("Cannot do operation non-numeric dtypes")
-        elif comparison_dtypes_only:
-            if not all(
-                (is_numeric_dtype(self_dtype) and is_numeric_dtype(other_dtype))
-                or (
-                    is_datetime_or_timedelta_dtype(self_dtype)
-                    and is_datetime_or_timedelta_dtype(other_dtype)
-                )
-                or is_dtype_equal(self_dtype, other_dtype)
-                for self_dtype, other_dtype in zip(self._get_dtypes(), other_dtypes)
-            ):
-                raise TypeError(
-                    "Cannot do operation non-numeric objects with numeric objects"
-                )
-        elif numeric_or_time_only:
-            if not all(
-                (is_numeric_dtype(self_dtype) and is_numeric_dtype(other_dtype))
-                or (
-                    is_datetime_or_timedelta_dtype(self_dtype)
-                    and is_datetime_or_timedelta_dtype(other_dtype)
-                )
-                for self_dtype, other_dtype in zip(self._get_dtypes(), other_dtypes)
-            ):
-                raise TypeError(
-                    "Cannot do operation non-numeric objects with numeric objects"
-                )
+        if not all(
+            (is_numeric_dtype(self_dtype) and is_numeric_dtype(other_dtype))
+            or (is_object_dtype(self_dtype) and is_object_dtype(other_dtype))
+            or (
+                is_datetime_or_timedelta_dtype(self_dtype)
+                and is_datetime_or_timedelta_dtype(other_dtype)
+            )
+            or is_dtype_equal(self_dtype, other_dtype)
+            for self_dtype, other_dtype in zip(self._get_dtypes(), other_dtypes)
+        ):
+            raise TypeError("Cannot do operation with improper dtypes")
         return result
 
     def _validate_function(self, func, on_invalid=None):
@@ -416,7 +385,7 @@ def _binary_op(self, op, other, **kwargs):
             return self._default_to_pandas(
                 getattr(self._pandas_class, op), other, **kwargs
             )
-        other = self._validate_other(other, axis, numeric_or_object_only=True)
+        other = self._validate_other(other, axis)
         exclude_list = [
             "__add__",
             "__radd__",
diff --git a/modin/pandas/test/test_series.py b/modin/pandas/test/test_series.py
index 04c7142f7b6..9a70a66feb1 100644
--- a/modin/pandas/test/test_series.py
+++ b/modin/pandas/test/test_series.py
@@ -1141,6 +1141,20 @@ def test_between_time():
     )
 
 
+def test_time_ops():
+    # Make a pandas.core.indexes.timedeltas.TimedeltaIndex
+    deltas = pd.to_timedelta([1], unit="h")
+    modin_series = pd.Series(np.datetime64("2000-12-12")) + deltas
+    pandas_series = pandas.Series(np.datetime64("2000-12-12")) + deltas
+
+    df_equals(modin_series, pandas_series)
+
+    modin_series = pd.Series(np.datetime64("2000-12-12")) - deltas
+    pandas_series = pandas.Series(np.datetime64("2000-12-12")) - deltas
+
+    df_equals(modin_series, pandas_series)
+
+
 @pytest.mark.parametrize("data", test_data_values, ids=test_data_keys)
 def test_bfill(data):
     modin_series, pandas_series = create_test_series(data)


Code review comments:
diff_hunk: @@ -1141,6 +1141,20 @@ def test_between_time():
     )
 
 
+def test_time_ops():
+    # Make a pandas.core.indexes.timedeltas.TimedeltaIndex
+    deltas = pd.to_timedelta([1], unit="h")
+    modin_series = pd.Series(np.datetime64("2000-12-12")) + deltas
+    pandas_series = pandas.Series(np.datetime64("2000-12-12")) + deltas
+
+    df_equals(modin_series, pandas_series)
path: modin/pandas/test/test_series.py
text: @user1:
```suggestion
    test_series = create_test_series(np.datetime64("2000-12-12"))
    eval_general(*test_series, lambda s: s + deltas)
```

@author:
Wow this is super clean, I'll try to make use of these idiom properly in the future.

diff_hunk: @@ -1141,6 +1141,20 @@ def test_between_time():
     )
 
 
+def test_time_ops():
+    # Make a pandas.core.indexes.timedeltas.TimedeltaIndex
+    deltas = pd.to_timedelta([1], unit="h")
+    modin_series = pd.Series(np.datetime64("2000-12-12")) + deltas
+    pandas_series = pandas.Series(np.datetime64("2000-12-12")) + deltas
+
+    df_equals(modin_series, pandas_series)
+
+    modin_series = pd.Series(np.datetime64("2000-12-12")) - deltas
+    pandas_series = pandas.Series(np.datetime64("2000-12-12")) - deltas
+
+    df_equals(modin_series, pandas_series)
path: modin/pandas/test/test_series.py
text: @user1:
```suggestion
    eval_general(*test_series, lambda s: s - deltas)
```