Repository: modin-project/modin
Pull request: Correctly apply Series functions element-wise for correct cases
* Resolves #597
* Mimic the error handling/cases of pandas
* Add checks for UDFs which use args/kwargs
* Add `str` to the tests for `apply`
* Remove those from `DataFrame.apply` tests because of #469

<!--
Thank you for your contribution!
-->

## What do these changes do?

<!-- Please give a short brief about these changes. -->

## Related issue number

<!-- Are there any issues opened that will be resolved by merging this change? -->

- [x] passes `flake8 modin`
- [x] passes `black --check modin`
- [x] tests added and passing


Problem statement:
Series.apply uses DataFrame.apply logic in incorrect order
<!--
General questions should be asked on the mailing list modin-dev@googlegroups.com.

Before submitting an issue, please fill out the following form.
-->

### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: OSX
- **Modin installed from (source or binary)**: source
- **Modin version**: master
- **Python version**: 3.6
- **Exact command to reproduce**: series.apply(str)

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
`Series` and `DataFrame` differ on the ways that they apply functions. Modin is using the DataFrame logic always instead of checking if the function should apply element-wise.
<!-- Describe the problem clearly here. -->

### Source code / logs
```python
import modin.pandas as pd
import pandas
print(pd.Series([1,2,3,4]).apply(str))
print(pandas.Series([1,2,3,4]).apply(str))
```
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->

Hints from issue discussion:

Commit message: Correctly apply Series functions element-wise for correct cases

* Resolves #597
* Mimic the error handling/cases of pandas
* Add checks for UDFs which use args/kwargs
* Add `str` to the tests for `apply`
* Remove those from `DataFrame.apply` tests because of #469

Patch to review:
diff --git a/modin/pandas/series.py b/modin/pandas/series.py
index e81a78af684..8e5dc467b90 100644
--- a/modin/pandas/series.py
+++ b/modin/pandas/series.py
@@ -4,6 +4,7 @@
 
 import numpy as np
 import pandas
+from pandas.compat import string_types
 from pandas.core.common import is_bool_indexer
 from pandas.core.dtypes.common import is_dict_like, is_list_like, is_scalar
 import sys
@@ -357,12 +358,25 @@ def apply(self, func, convert_dtype=True, args=(), **kwds):
             apply_func = "agg"
         else:
             apply_func = "apply"
-            # Add this because it only applies for `apply` specifically.
-            kwds["convert_dtype"] = convert_dtype
-        query_compiler = super(Series, self).apply(func, *args, **kwds)
-        # Sometimes we can return a scalar here
-        if not isinstance(query_compiler, type(self._query_compiler)):
-            return query_compiler
+
+        if isinstance(func, string_types) or is_list_like(func):
+            query_compiler = super(Series, self).apply(func, *args, **kwds)
+            # Sometimes we can return a scalar here
+            if not isinstance(query_compiler, type(self._query_compiler)):
+                return query_compiler
+        else:
+            # handle ufuncs and lambdas
+            if kwds or args and not isinstance(func, np.ufunc):
+
+                def f(x):
+                    return func(x, *args, **kwds)
+
+            else:
+                f = func
+            with np.errstate(all="ignore"):
+                if isinstance(f, np.ufunc):
+                    return f(self)
+                query_compiler = self.map(f)._query_compiler
         # This is the simplest way to determine the return type, but there are checks
         # in pandas that verify that some results are created. This is a challenge for
         # empty DataFrames, but fortunately they only happen when the `func` type is
@@ -371,7 +385,7 @@ def apply(self, func, convert_dtype=True, args=(), **kwds):
         # type.
         return_type = type(
             getattr(getattr(pandas, self.__name__)(index=self.index), apply_func)(
-                func, *args, **kwds
+                func, convert_dtype=convert_dtype, *args, **kwds
             )
         ).__name__
         if return_type not in ["DataFrame", "Series"]:
diff --git a/modin/pandas/test/test_dataframe.py b/modin/pandas/test/test_dataframe.py
index 13b1ae3698e..2b16f71563c 100644
--- a/modin/pandas/test/test_dataframe.py
+++ b/modin/pandas/test/test_dataframe.py
@@ -44,6 +44,10 @@
     int_arg_values,
 )
 
+# TODO remove once modin-project/modin#469 is resolved
+agg_func_keys.remove("str")
+agg_func_values.remove(str)
+
 pd.DEFAULT_NPARTITIONS = 4
 
 # Force matplotlib to not use any Xwindows backend.
diff --git a/modin/pandas/test/utils.py b/modin/pandas/test/utils.py
index 725e711c032..a019b0b4461 100644
--- a/modin/pandas/test/utils.py
+++ b/modin/pandas/test/utils.py
@@ -138,6 +138,7 @@
 agg_func = {
     "sum": "sum",
     "df sum": lambda df: df.sum(),
+    "str": str,
     "sum mean": ["sum", "mean"],
     "sum sum": ["sum", "sum"],
     "sum df sum": ["sum", lambda df: df.sum()],


Code review comments:
diff_hunk: @@ -357,12 +358,25 @@ def apply(self, func, convert_dtype=True, args=(), **kwds):
             apply_func = "agg"
         else:
             apply_func = "apply"
-            # Add this because it only applies for `apply` specifically.
-            kwds["convert_dtype"] = convert_dtype
-        query_compiler = super(Series, self).apply(func, *args, **kwds)
-        # Sometimes we can return a scalar here
-        if not isinstance(query_compiler, type(self._query_compiler)):
-            return query_compiler
+
+        if isinstance(func, string_types) or is_list_like(func):
+            query_compiler = super(Series, self).apply(func, *args, **kwds)
+            # Sometimes we can return a scalar here
+            if not isinstance(query_compiler, type(self._query_compiler)):
path: modin/pandas/series.py
text: @user1:
Should this be moved to after the `else` block if a UDF or lambda returns a scalar?

@author:
They cannot return scalars. Those functions must be applied element-wise which will always return the `Series` type.