Repository: modin-project/modin

Issue:
KeyError on groupby().min()
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux Ubuntu16.04
- **Modin installed from (source or binary)**: binary
- **Modin version**: 0.5.4
- **Python version**: 3.6.8
- **Exact command to reproduce**: python mortgage_modin.py

### Describe the problem
I get a weird 'Key Error' from a groupby.min(), Any idea?
```
  File "mortgage_modin.py", line 192, in create_delinq_features
    delinq_180 = delinq_pdf.query('current_loan_delinquency_status >= 6')[['loan_id', 'monthly_reporting_period']].groupby('loan_id').min()
  File "/miniconda3/envs/ray/lib/python3.6/site-packages/modin/pandas/groupby.py", line 183, in min
    lambda df: df.min(**kwargs), None, numeric_only=False
  File "/miniconda3/envs/ray/lib/python3.6/site-packages/modin/pandas/groupby.py", line 440, in _groupby_reduce
    numeric_only=numeric_only,
  File "/miniconda3/envs/ray/lib/python3.6/site-packages/modin/backends/pandas/query_compiler.py", line 2707, in groupby_reduce
    new_data = self.data.groupby_reduce(axis, by.data, _map, _reduce)
  File "/miniconda3/envs/ray/lib/python3.6/site-packages/modin/engines/ray/pandas_on_ray/frame/partition_manager.py", line 58, in groupby_reduce
    return new_partitions.map_across_full_axis(axis, reduce_func)
  File "/miniconda3/envs/ray/lib/python3.6/site-packages/modin/engines/base/frame/partition_manager.py", line 335, in map_across_full_axis
    partitions = self.column_partitions if not axis else self.row_partitions
  File "/miniconda3/envs/ray/lib/python3.6/site-packages/modin/engines/base/frame/partition_manager.py", line 121, in column_partitions
    return [self._column_partitions_class(col) for col in self.partitions.T]
  File "/miniconda3/envs/ray/lib/python3.6/site-packages/modin/engines/base/frame/partition_manager.py", line 74, in _get_partitions
    for i in range(len(self._partitions_cache))
  File "/miniconda3/envs/ray/lib/python3.6/site-packages/modin/engines/base/frame/partition_manager.py", line 74, in <listcomp>
    for i in range(len(self._partitions_cache))
  File "/miniconda3/envs/ray/lib/python3.6/site-packages/modin/engines/base/frame/partition_manager.py", line 72, in <listcomp>
    if self.block_lengths[i] != 0 and self.block_widths[j] != 0
  File "/miniconda3/envs/ray/lib/python3.6/site-packages/modin/engines/ray/generic/frame/partition_manager.py", line 44, in block_lengths
    handle_ray_task_error(e)
  File "/miniconda3/envs/ray/lib/python3.6/site-packages/modin/engines/ray/utils.py", line 13, in handle_ray_task_error
    raise getattr(builtins, s.split(":")[0])("".join(s.split(":")[1:]))
KeyError: ' 836858391918'
```

### Source code / logs
You can find the sources here: https://github.com/fschlimb/scale-out-benchs/blob/master/mortgage/mortgage_modin.py
input data from here: https://github.com/rapidsai/notebooks-extended/tree/master/data/mortgage




Discussion:
Hi @fschlimb, thanks for the report!

I wasn't able to get the script you provided to run with the datasets provided. What's strange in this case is that the previous 2 instances of `groupby.min()` worked fine, and looking at the code there's nothing fundamental that should change between these runs.

Would you be able to just run the following:
```python
delinq_180 = delinq_pdf.query('current_loan_delinquency_status >= 6')
print(delinq_180.head())
delinq_180 = delinq_180[['loan_id', 'monthly_reporting_period']]
print(delinq_180.head())
delinq_180 = delinq_180.groupby('loan_id').min()
print(delinq_180.head())
```

Printing the head in-between will reveal any spurious errors that happen in the tasks and give us a better sense about which part of the line is causing the issue. Thanks again for the detailed report!
@devin-petersohn: thanks for looking int this.

Sorry that the input data doesn't work, I probably made a few modifications, in particular I craeted a smaller subset to allow faster debugging...Attached the files I am using.
[input.tar.gz](https://github.com/modin-project/modin/files/3457387/input.tar.gz)

Here's the output before it fails in `delinq_180 = delinq_180.groupby('loan_id').min()`:

```
UserWarning: `DataFrame.merge` defaulting to pandas implementation.
To request implementation, send an email to feature_requests@modin.org.
UserWarning: Distributing <class 'pandas.core.frame.DataFrame'> object. This may take some time.
UserWarning: User-defined function verification is still under development in Modin. The function provided is not verified.
          loan_id monthly_reporting_period  current_loan_delinquency_status
182  695373409106               2005-02-01                               20
186  876671185391               2014-03-01                                8
399  899915988958               2010-07-01                               22
611  577520595057               2006-06-01                               12
646  782469237171               2002-08-01                               16
          loan_id monthly_reporting_period
182  695373409106               2005-02-01
186  876671185391               2014-03-01
399  899915988958               2010-07-01
611  577520595057               2006-06-01
646  782469237171               2002-08-01
```

Thanks @fschlimb, sorry for the delay I was traveling last week.

This does narrow down the search quite a bit. The `KeyError` itself could be caused by an issue with the indexing in the previous line, I will dig into it and see.

Thanks for the additional input data, that does help!
@fschlimb Thanks for the patience, we have been rewriting the internals for efficiency and better metadata tracking and have fixed your issue in the process. This was done in #721. 

After running your script I ran into an arrow serialization issue, which is new. In order to make it work I had to change the following:
```python
# delinq_merge['delinquency_90'] = delinq_merge['delinquency_90'].fillna(np.dtype('datetime64[ms]').type('1970-01-01').astype('datetime64[ms]'))
delinq_merge['delinquency_90'] = delinq_merge['delinquency_90'].fillna('1970-01-01').astype('datetime64[ms]')
```

This issue isn't something we can fix in Modin, but there's no issue in running it with pandas so there may be an issue serializing `np.datetime` upstream. I'm going to open a new issue to track that issue separately.

Thank you again for the thorough reproducible result. It helped a lot!
Thank you for the update!

Unfortunately I still see the same issue after updating to
`modin                     0.6.0                    pypi_0    pypi`

From looking at the commit log this should include this PR. No?

That commit was included in the release. I am travelling this week and have limited access to wifi/stable machine, so I will have to take another look next week.

I was able to test it and it didn't have the `KeyError` after that PR, but I was able to reproduce it before that PR.

Just out of curiosity, how many cores do you have on your machine?
@devin-petersohn My machine has 16 cores (8 cores x 2 sockets).
If this is hard to fix, do you have an idea how I could work around the issue?
I don't think the issue will be difficult to fix, it seems like the metadata is off. The workaround might be to swap the `query` and `__getitem__` parts of the line:

```python
delinq_180 = delinq_180[['loan_id', 'monthly_reporting_period']].query('current_loan_delinquency_status >= 6').groupby('loan_id').min()
```

I will report back once I am back in my home country. Thanks again for your patience!
Quick update: I was able to reproduce it by starting with 16 CPUs but it does not happen on 8. I will have to dig into this a bit more because given the partition size it shouldn't be any different. Thanks again for the report!

Resolving patch:
diff --git a/modin/backends/pandas/query_compiler.py b/modin/backends/pandas/query_compiler.py
--- a/modin/backends/pandas/query_compiler.py
+++ b/modin/backends/pandas/query_compiler.py
@@ -1413,7 +1413,8 @@ def groupby_reduce(
 
         def _map(df, other):
             return map_func(
-                df.groupby(by=other.squeeze(), axis=axis, **groupby_args), **map_args
+                df.groupby(by=other.squeeze(axis=axis ^ 1), axis=axis, **groupby_args),
+                **map_args
             ).reset_index(drop=False)
 
         if reduce_func is not None:


Test patch: