internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: modin-project/modin

Issue:
Issue when processing space-separated data
<!--
General questions should be asked on the mailing list modin-dev@googlegroups.com.

Before submitting an issue, please fill out the following form.
-->

### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux Ubuntu 16.04
- **Modin installed from (source or binary)**: pip3 install modin
- **Modin version**: 0.2.5
- **Python version**: 3.5.2
- **Exact command to reproduce**: df = pd.read_csv(filename, header=None, delim_whitespace=True)

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->
I have a space-separated data file.
Using pandas, I can separate the values:

```
Processing file: data/data_1_1_36598.csv
                  0                     1                 2   3           4           5    6           7
0           0.169881  192.168.2.2:36598  192.168.2.3:80  32  0xdb0d77fb  0xdb0d44fb   10  2147483647 

```

But when using modin, the values are not separated into different columns:
```
Processing file: data/data_1_1_36598.csv
                                                         0
0        0.169881371 192.168.2.2:36598 192.168.2.3...
```

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->


Discussion:
HI @DavyCao thanks for posting this. I am able to reproduce this with the values you provided. 

It works for some datasets, but not others. This is an interesting one, I will get it fixed. Thanks again for finding this!

Resolving patch:
diff --git a/modin/engines/ray/pandas_on_ray/io.py b/modin/engines/ray/pandas_on_ray/io.py
--- a/modin/engines/ray/pandas_on_ray/io.py
+++ b/modin/engines/ray/pandas_on_ray/io.py
@@ -353,13 +353,13 @@ def _read(cls, filepath_or_buffer, **kwargs):
         # number of nodes in the cluster.
         try:
             args, _, _, defaults, _, _, _ = inspect.getfullargspec(cls.read_csv)
-            defaults = dict(zip(args[1:], defaults))
+            defaults = dict(zip(args[2:], defaults))
             filtered_kwargs = {
                 kw: kwargs[kw]
                 for kw in kwargs
                 if kw in defaults
                 and not isinstance(kwargs[kw], type(defaults[kw]))
-                and kwargs[kw] != defaults[kw]
+                or kwargs[kw] != defaults[kw]
             }
         # This happens on Python2, we will just default to serializing the entire dictionary
         except AttributeError:


Test patch:
Repository: modin-project/modin

Issue:
Invalid Query Compiler object: <class 'modin.data_management.query_compiler.pandas_query_compiler.PandasQueryCompiler'>
### Describe the problem
<!-- Describe the problem clearly here. -->
an inplace operation after a `loc` or `iloc` causes a Exception to be thrown.

### Source code / logs
```python
import modin.pandas as pd 
df = pd.DataFrame(index=list("ABCD"), columns=list("ABCD"))
df.loc[['A']].fillna(0, inplace=True)
```
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:

Resolving patch:
diff --git a/modin/pandas/dataframe.py b/modin/pandas/dataframe.py
--- a/modin/pandas/dataframe.py
+++ b/modin/pandas/dataframe.py
@@ -4845,8 +4845,9 @@ def iloc(self):
 
     def _create_dataframe_from_compiler(self, new_query_compiler, inplace=False):
         """Returns or updates a DataFrame given new query_compiler"""
-        assert isinstance(
-            new_query_compiler, type(self._query_compiler)
+        assert (
+            isinstance(new_query_compiler, type(self._query_compiler))
+            or type(new_query_compiler) in self._query_compiler.__class__.__bases__
         ), "Invalid Query Compiler object: {}".format(type(new_query_compiler))
         if not inplace:
             return DataFrame(query_compiler=new_query_compiler)


Test patch:
Repository: modin-project/modin

Issue:
AttributeError: module 'modin.pandas' has no attribute 'melt'
### Describe the problem

Modin does not have an equivilent of `pandas.melt` although we do have `df.melt`.

### Source code / logs
```python
import modin.pandas as pd
...
melted = pd.melt(frame=data_new,id_vars = 'Name', value_vars= ['Attack','Defense'])
```
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:

Resolving patch:
diff --git a/modin/pandas/__init__.py b/modin/pandas/__init__.py
--- a/modin/pandas/__init__.py
+++ b/modin/pandas/__init__.py
@@ -58,7 +58,7 @@
     read_gbq,
     read_table,
 )
-from .reshape import get_dummies
+from .reshape import get_dummies, melt
 from .general import isna, merge, pivot_table
 from .. import __execution_engine__ as execution_engine
 
@@ -154,6 +154,7 @@
     "PeriodIndex",
     "Categorical",
     "__version__",
+    "melt",
 ]
 
 del pandas
diff --git a/modin/pandas/reshape.py b/modin/pandas/reshape.py
--- a/modin/pandas/reshape.py
+++ b/modin/pandas/reshape.py
@@ -61,3 +61,20 @@ def get_dummies(
         drop_first=drop_first,
     )
     return DataFrame(query_compiler=new_manager)
+
+
+def melt(
+    frame,
+    id_vars=None,
+    value_vars=None,
+    var_name=None,
+    value_name="value",
+    col_level=None,
+):
+    return frame.melt(
+        id_vars=id_vars,
+        value_vars=value_vars,
+        var_name=var_name,
+        value_name=value_name,
+        col_level=col_level,
+    )


Test patch:
Repository: modin-project/modin

Issue:
groupby.mean() throws pandas.core.base.DataError

### Describe the problem
<!-- Describe the problem clearly here. -->
When there are partitions that contain all non-numeric data, `DataError` is thrown on numeric operations.

### Source code / logs
```python
data = {'col0': list("ABCD"), 'col1': [1,2,3,4]}
df = pd.DataFrame(data)
df.groupby("col0").mean()
```

<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:

Resolving patch:
diff --git a/modin/data_management/query_compiler/pandas_query_compiler.py b/modin/data_management/query_compiler/pandas_query_compiler.py
--- a/modin/data_management/query_compiler/pandas_query_compiler.py
+++ b/modin/data_management/query_compiler/pandas_query_compiler.py
@@ -14,6 +14,7 @@
     is_bool_dtype,
 )
 from pandas.core.index import _ensure_index
+from pandas.core.base import DataError
 
 from modin.error_message import ErrorMessage
 from modin.engines.base.block_partitions import BaseBlockPartitions
@@ -2425,7 +2426,14 @@ def groupby_agg_builder(df):
                 df.index = remote_index
             else:
                 df.columns = remote_index
-            return agg_func(df.groupby(by=by, axis=axis, **groupby_args), **agg_args)
+            grouped_df = df.groupby(by=by, axis=axis, **groupby_args)
+            try:
+                return agg_func(grouped_df, **agg_args)
+            # This happens when the partition is filled with non-numeric data and a
+            # numeric operation is done. We need to build the index here to avoid issues
+            # with extracting the index.
+            except DataError:
+                return pandas.DataFrame(index=grouped_df.count().index)
 
         func_prepared = self._prepare_method(lambda df: groupby_agg_builder(df))
         result_data = self.map_across_full_axis(axis, func_prepared)


Test patch:
Repository: modin-project/modin

Issue:
TypeError: 'function' object is not subscriptable

### Describe the problem
<!-- Describe the problem clearly here. -->
Getting this error on master when `df.iloc['col'].iloc[0]`. The problem is in how certain operations are handled on the new `SeriesView` object.

### Source code / logs
```
import modin.pandas as pd
import numpy as np
frame_data = np.random.randint(0, 100, size=(2**10, 2**8))
df = pd.DataFrame(frame_data).add_prefix("col")
df.loc[1].at["col1"]
```


Discussion:

Resolving patch:
diff --git a/modin/pandas/series.py b/modin/pandas/series.py
--- a/modin/pandas/series.py
+++ b/modin/pandas/series.py
@@ -182,7 +182,15 @@ def __getattribute__(self, item):
                 has_inplace_param = callable(method) and "inplace" in str(
                     inspect.getargspec(method)
                 )
-            if callable(method) and has_inplace_param and self.parent_df is not None:
+            # Certain operations like `at`, `loc`, `iloc`, etc. are callable because in
+            # pandas they are equivalent to classes. They are verified here because they
+            # cannot be overridden with the functions below. This generally solves the
+            # problem where the instance property is callable, but the class property is
+            # not.
+            needs_override = callable(method) and callable(
+                getattr(type(self.series), item)
+            )
+            if needs_override and has_inplace_param and self.parent_df is not None:
 
                 def inplace_handler(*args, **kwargs):
                     """Replaces the default behavior of methods with inplace kwarg.
@@ -226,7 +234,7 @@ def inplace_handler(*args, **kwargs):
 
                 # We replace the method with `inplace_handler` for inplace operations
                 method = inplace_handler
-            elif callable(method):
+            elif needs_override:
 
                 def other_handler(*args, **kwargs):
                     """Replaces the method's args and kwargs with the Series object.


Test patch:
Repository: modin-project/modin

Issue:
rfloordiv returning wrong results
### Describe the problem
Calling `rfloordiv` returns the wrong results

### Source code / logs
```
import modin.pandas as pd
import pandas
data = {
    "col3": [np.NaN, 2, np.NaN, 0],
    "col4": [3, 4, np.NaN, 1],
    "col1": [np.NaN, np.NaN, np.NaN, 5],
    "col2": [6, np.NaN, np.NaN, 10],
}
modin_df = pd.DataFrame(data)
pandas_df = pandas.DataFrame(data)
modin_df.rfloordiv(4)
```


Discussion:

Resolving patch:
diff --git a/modin/data_management/query_compiler/pandas_query_compiler.py b/modin/data_management/query_compiler/pandas_query_compiler.py
--- a/modin/data_management/query_compiler/pandas_query_compiler.py
+++ b/modin/data_management/query_compiler/pandas_query_compiler.py
@@ -619,14 +619,26 @@ def rdiv(self, other, **kwargs):
         func = pandas.DataFrame.rdiv
         return self._inter_df_op_handler(func, other, **kwargs)
 
+    def rfloordiv(self, other, **kwargs):
+        """Floordivs this manager with other object (manager or scalar).
+
+        Args:
+            other: The other object (manager or scalar).
+
+        Returns:
+            New DataManager with floordiv-ed data and index.
+        """
+        func = pandas.DataFrame.rfloordiv
+        return self._inter_df_op_handler(func, other, **kwargs)
+
     def rmod(self, other, **kwargs):
-        """Mods other object (manager or scalar) with this manager.
+        """Mods this manager with other object (manager or scalar).
 
         Args:
             other: The other object (manager or scalar).
 
         Returns:
-            New DataManager with divided data and new index.
+            New DataManager with mod data and index.
         """
         func = pandas.DataFrame.rmod
         return self._inter_df_op_handler(func, other, **kwargs)
diff --git a/modin/pandas/dataframe.py b/modin/pandas/dataframe.py
--- a/modin/pandas/dataframe.py
+++ b/modin/pandas/dataframe.py
@@ -3197,7 +3197,21 @@ def reset_index(
         return self._create_dataframe_from_compiler(new_query_compiler, inplace)
 
     def rfloordiv(self, other, axis="columns", level=None, fill_value=None):
-        return self.floordiv(other, axis, level, fill_value)
+        if level is not None:
+            if isinstance(other, DataFrame):
+                other = other._query_compiler.to_pandas()
+            return self._default_to_pandas(
+                pandas.DataFrame.rfloordiv,
+                other,
+                axis=axis,
+                level=level,
+                fill_value=fill_value,
+            )
+        other = self._validate_other(other, axis, numeric_only=True)
+        new_query_compiler = self._query_compiler.rfloordiv(
+            other=other, axis=axis, level=level, fill_value=fill_value
+        )
+        return self._create_dataframe_from_compiler(new_query_compiler)
 
     def rmod(self, other, axis="columns", level=None, fill_value=None):
         """Mod this DataFrame against another DataFrame/Series/scalar.


Test patch:
Repository: modin-project/modin

Issue:
pow returns an IndexError
### Describe the problem
`pow` unexpectedly throws an `IndexError: index out of bounds` error

### Source code / logs
```
import modin.pandas as pd
data = {
    "col3": [0, 1, 2, 3, 4, 5, 6, 7],
    "col4": [8, 9, 10, 11, 12, 13, 14, 15],
    "col5": [16, 17, 18, 19, 20, 21, 22, 23],
    "col1": [24, 25, 26, 27, 28, 29, 30, 31],
    "col2": [32, 33, 34, 35, 36, 37, 38, 39],
}
modin_df = pd.DataFrame(data)
modin_df.pow(4)
```


Discussion:

Resolving patch:
diff --git a/modin/pandas/dataframe.py b/modin/pandas/dataframe.py
--- a/modin/pandas/dataframe.py
+++ b/modin/pandas/dataframe.py
@@ -2682,32 +2682,6 @@ def pow(self, other, axis="columns", level=None, fill_value=None):
                 fill_value=fill_value,
             )
         other = self._validate_other(other, axis, numeric_only=True)
-        # Check to make sure integers are not raised to negative integer powers
-        if isinstance(other, type(self._query_compiler)):
-            other_dtypes = other.dtypes
-        elif is_list_like(other):
-            other_dtypes = [type(x) for x in other]
-        else:
-            other_dtypes = [
-                type(other)
-                for _ in range(len(self.index) if axis else len(self.columns))
-            ]
-        for i in range(len(other_dtypes)):
-            if is_integer_dtype(other_dtypes[i]) and is_integer_dtype(self.dtypes[i]):
-                if isinstance(other, type(self._query_compiler)):
-                    continue
-                    # TODO: Come back to this when we have an efficient way to preprocess data
-                    # if old_other.iloc[:, i].lt(0).any():
-                    #     raise ValueError("Integers to negative integer powers are not allowed.")
-                elif is_list_like(other):
-                    if any(x < 0 for x in other):
-                        raise ValueError(
-                            "Integers to negative integer powers are not allowed."
-                        )
-                elif other < 0:
-                    raise ValueError(
-                        "Integers to negative integer powers are not allowed."
-                    )
         new_query_compiler = self._query_compiler.pow(
             other=other, axis=axis, level=level, fill_value=fill_value
         )


Test patch:
Repository: modin-project/modin

Issue:
query can return incorrect final index if complete partitions are filtered

### Describe the problem
<!-- Describe the problem clearly here. -->
Query can return incorrect final index if one or more complete partitions are filtered. See code below

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->
```python
data = {
    "col3": [np.NaN, 2, np.NaN, 0],
    "col4": [3, 4, np.NaN, 1],
    "col1": [np.NaN, np.NaN, np.NaN, 5],
    "col2": [6, np.NaN, np.NaN, 10],
}
df.query("col1 < col2")
```


Discussion:

Resolving patch:
diff --git a/modin/engines/base/block_partitions.py b/modin/engines/base/block_partitions.py
--- a/modin/engines/base/block_partitions.py
+++ b/modin/engines/base/block_partitions.py
@@ -496,9 +496,13 @@ def get_indices(self, axis=0, index_func=None, old_blocks=None):
         if axis == 0:
             func = self.preprocess_func(index_func)
             # We grab the first column of blocks and extract the indices
+            # Note: We use _partitions_cache in the context of this function to make
+            # sure that none of the partitions are modified or filtered out before we
+            # get the index information.
+            # DO NOT CHANGE TO self.partitions under any circumstance.
             new_indices = (
-                [idx.apply(func).get() for idx in self.partitions.T[0]]
-                if len(self.partitions.T)
+                [idx.apply(func).get() for idx in self._partitions_cache.T[0]]
+                if len(self._partitions_cache.T)
                 else []
             )
             # This is important because sometimes we have resized the data. The new
@@ -511,8 +515,8 @@ def get_indices(self, axis=0, index_func=None, old_blocks=None):
         else:
             func = self.preprocess_func(index_func)
             new_indices = (
-                [idx.apply(func).get() for idx in self.partitions[0]]
-                if len(self.partitions)
+                [idx.apply(func).get() for idx in self._partitions_cache[0]]
+                if len(self._partitions_cache)
                 else []
             )
 


Test patch:
Repository: modin-project/modin

Issue:
sum and prod not matching pandas

### Describe the problem
<!-- Describe the problem clearly here. -->
`NaN` values can occur instead of 0's because of the `min_count` parameter.

### Source code / logs
```python
import pandas
import modin.pandas as pd
from numpy import nan

data = {'col1': [nan, nan, nan, 5], 'col2': [6, nan, nan, 10], 'col3': [nan, 2, nan, 0], 'col4': [3, 4, nan, 1]}
pd.DataFrame(data).sum(axis=1)
pandas.DataFrame(data).sum(axis=1)
```
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:

Resolving patch:
diff --git a/modin/data_management/query_compiler/pandas_query_compiler.py b/modin/data_management/query_compiler/pandas_query_compiler.py
--- a/modin/data_management/query_compiler/pandas_query_compiler.py
+++ b/modin/data_management/query_compiler/pandas_query_compiler.py
@@ -1045,7 +1045,7 @@ def _process_sum_prod(self, func, **kwargs):
         """
         axis = kwargs.get("axis", 0)
         numeric_only = kwargs.get("numeric_only", None) if not axis else True
-        min_count = kwargs.get("min_count", 1)
+        min_count = kwargs.get("min_count", 0)
         reduce_index = self.columns if axis else self.index
 
         if numeric_only:
@@ -1062,7 +1062,7 @@ def sum_prod_builder(df, **kwargs):
 
         map_func = self._prepare_method(sum_prod_builder, **kwargs)
 
-        if min_count == 1:
+        if min_count <= 1:
             return self.full_reduce(axis, map_func, numeric_only=numeric_only)
         elif min_count > len(reduce_index):
             return pandas.Series(
diff --git a/modin/pandas/dataframe.py b/modin/pandas/dataframe.py
--- a/modin/pandas/dataframe.py
+++ b/modin/pandas/dataframe.py
@@ -387,7 +387,7 @@ def sum(
         skipna=True,
         level=None,
         numeric_only=None,
-        min_count=1,
+        min_count=0,
         **kwargs
     ):
         """Perform a sum across the DataFrame.


Test patch:
Repository: modin-project/modin

Issue:
python2 error when doing some operations on SeriesView object

### Describe the problem
<!-- Describe the problem clearly here. -->
Getting the following exception: 
`TypeError: <pandas.core.indexing._AtIndexer object at 0x7f0a3f9c2d60> is not a Python function`

### Source code / logs
`modin_df.loc[0].at[key1]`
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:

Resolving patch:
diff --git a/modin/pandas/series.py b/modin/pandas/series.py
--- a/modin/pandas/series.py
+++ b/modin/pandas/series.py
@@ -173,24 +173,25 @@ def __getattribute__(self, item):
         ]
         if item not in default_behaviors:
             method = self.series.__getattribute__(item)
-            try:
-                has_inplace_param = callable(method) and "inplace" in str(
-                    inspect.signature(method)
-                )
-            # This will occur on Python2
-            except AttributeError:
-                has_inplace_param = callable(method) and "inplace" in str(
-                    inspect.getargspec(method)
-                )
             # Certain operations like `at`, `loc`, `iloc`, etc. are callable because in
             # pandas they are equivalent to classes. They are verified here because they
             # cannot be overridden with the functions below. This generally solves the
             # problem where the instance property is callable, but the class property is
             # not.
-            needs_override = callable(method) and callable(
+            is_callable = callable(method) and callable(
                 getattr(type(self.series), item)
             )
-            if needs_override and has_inplace_param and self.parent_df is not None:
+            try:
+                has_inplace_param = is_callable and "inplace" in str(
+                    inspect.signature(method)
+                )
+            # This will occur on Python2
+            except AttributeError:
+                has_inplace_param = is_callable and "inplace" in str(
+                    inspect.getargspec(method)
+                )
+
+            if is_callable and has_inplace_param and self.parent_df is not None:
 
                 def inplace_handler(*args, **kwargs):
                     """Replaces the default behavior of methods with inplace kwarg.
@@ -234,7 +235,7 @@ def inplace_handler(*args, **kwargs):
 
                 # We replace the method with `inplace_handler` for inplace operations
                 method = inplace_handler
-            elif needs_override:
+            elif is_callable:
 
                 def other_handler(*args, **kwargs):
                     """Replaces the method's args and kwargs with the Series object.


Test patch: