Repository: modin-project/modin

Issue:
loc doesnt support a callable
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: `Linux 21e69ce492bc 4.19.76-linuxkit #1 SMP Tue May 26 11:42:35 UTC 2020 x86_64 GNU/Linux` running as a docker container on `python:3.8.3-slim`
- **Modin version** (`modin.__version__`): `37b7d2c4c5b3ac04408f362b3989de30dd3df1a2`
- **Python version**: 3.8.3
- **Code we can use to reproduce**:

### Describe the problem
I'm not entirely sure if this is my misunderstanding, but essentially I have some code that filters frames using a callable in a `loc`.

This is what I'm expecting to see:

```
>>> import pandas
>>> foo = pandas.DataFrame({"col": [1,2,3,4]})
>>> foo
   col
0    1
1    2
2    3
3    4
>>> foo.loc[lambda val: val.col%2==0]
   col
1    2
3    4
```

but this is what I'm seeing with modin[ray]:

```
Python 3.8.3 (default, Jun  9 2020, 17:49:41) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import modin.pandas as pandas
>>> foo = pandas.DataFrame({"col": [1,2,3,4]})
UserWarning: Distributing <class 'dict'> object. This may take some time.
>>> foo
   col
0    1
1    2
2    3
3    4
>>> foo.loc[lambda val: val.col%2==0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/.cache/pypoetry/virtualenvs/onepassword-etl-NcjVOHQt-py3.8/src/modin/modin/pandas/indexing.py", line 252, in __getitem__
    row_lookup, col_lookup = self._compute_lookup(row_loc, col_loc)
  File "/root/.cache/pypoetry/virtualenvs/onepassword-etl-NcjVOHQt-py3.8/src/modin/modin/pandas/indexing.py", line 345, in _compute_lookup
    row_lookup = self.qc.index.get_indexer_for(row_loc)
  File "/root/.cache/pypoetry/virtualenvs/onepassword-etl-NcjVOHQt-py3.8/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 4502, in get_indexer_for
    return self.get_indexer(target, **kwargs)
  File "/root/.cache/pypoetry/virtualenvs/onepassword-etl-NcjVOHQt-py3.8/lib/python3.8/site-packages/pandas/core/indexes/range.py", line 358, in get_indexer
    return super().get_indexer(
  File "/root/.cache/pypoetry/virtualenvs/onepassword-etl-NcjVOHQt-py3.8/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 2709, in get_indexer
    target = ensure_index(target)
  File "/root/.cache/pypoetry/virtualenvs/onepassword-etl-NcjVOHQt-py3.8/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 5358, in ensure_index
    return Index(index_like)
  File "/root/.cache/pypoetry/virtualenvs/onepassword-etl-NcjVOHQt-py3.8/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 438, in __new__
    subarr = com.asarray_tuplesafe(data, dtype=object)
  File "/root/.cache/pypoetry/virtualenvs/onepassword-etl-NcjVOHQt-py3.8/lib/python3.8/site-packages/pandas/core/common.py", line 222, in asarray_tuplesafe
    values = list(values)
TypeError: 'function' object is not iterable
>>> 
```

in my case I'm trying to drop records from a source column, and then merge the result with the original, using the loc lambda feature to find the dropped columns. I can find a workaround for my case. unless I'm just using modin wrong?


Discussion:
Hi @orenmazor thanks for the report! Modin is definitely supposed to be working for this case, I looked through the tests and we are testing operations like `df.loc[df.col%2==0]` but not `df.loc[lambda df: df.col%2==0]`. This is a one line fix I can put in tonight.

You are definitely not using Modin wrong! Thanks again for the report!
fantastic. thank you! 

Resolving patch:
diff --git a/modin/pandas/indexing.py b/modin/pandas/indexing.py
--- a/modin/pandas/indexing.py
+++ b/modin/pandas/indexing.py
@@ -236,6 +236,8 @@ class _LocIndexer(_LocationIndexerBase):
     """An indexer for modin_df.loc[] functionality"""
 
     def __getitem__(self, key):
+        if callable(key):
+            return self.__getitem__(key(self.df))
         row_loc, col_loc, ndim, self.row_scaler, self.col_scaler = _parse_tuple(key)
         if isinstance(row_loc, slice) and row_loc == slice(None):
             # If we're only slicing columns, handle the case with `__getitem__`
@@ -361,6 +363,8 @@ class _iLocIndexer(_LocationIndexerBase):
     """An indexer for modin_df.iloc[] functionality"""
 
     def __getitem__(self, key):
+        if callable(key):
+            return self.__getitem__(key(self.df))
         row_loc, col_loc, ndim, self.row_scaler, self.col_scaler = _parse_tuple(key)
         self._check_dtypes(row_loc)
         self._check_dtypes(col_loc)


Test patch:
diff --git a/modin/pandas/test/test_dataframe.py b/modin/pandas/test/test_dataframe.py
--- a/modin/pandas/test/test_dataframe.py
+++ b/modin/pandas/test/test_dataframe.py
@@ -4394,6 +4394,12 @@ def test_iloc(self, request, data):
             modin_df.iloc[:, 0] = modin_df.iloc[:, 1]
             pandas_df.iloc[:, 0] = pandas_df.iloc[:, 1]
             df_equals(modin_df, pandas_df)
+
+            # From issue #1775
+            df_equals(
+                modin_df.iloc[lambda df: df.index.get_indexer_for(df.index[:5])],
+                pandas_df.iloc[lambda df: df.index.get_indexer_for(df.index[:5])],
+            )
         else:
             with pytest.raises(IndexError):
                 modin_df.iloc[0, 1]
@@ -4489,6 +4495,12 @@ def test_loc(self, request, data):
             pandas_df_copy.loc[[1, 2]] = 42
             df_equals(modin_df_copy, pandas_df_copy)
 
+            # From issue #1775
+            df_equals(
+                modin_df.loc[lambda df: df.iloc[:, 0].isin(list(range(1000)))],
+                pandas_df.loc[lambda df: df.iloc[:, 0].isin(list(range(1000)))],
+            )
+
         # From issue #1374
         with pytest.raises(KeyError):
             modin_df.loc["NO_EXIST"]