internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: modin-project/modin

Issue:
concat with transposed DataFrames throws ValueError

### Describe the problem
<!-- Describe the problem clearly here. -->
`concat` with a transposed QueryCompiler throws `ValueError`.

This is particularly problematic for `Series` which heavily uses `tranpose`.

### Source code / logs

```python
total = df_train.isnull().sum().sort_values(ascending=False)
percent = (df_train.isnull().sum()/df_train.isnull().count()).sort_values(ascending=False)
missing_data = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])
```
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:

Resolving patch:
diff --git a/modin/backends/pandas/query_compiler.py b/modin/backends/pandas/query_compiler.py
--- a/modin/backends/pandas/query_compiler.py
+++ b/modin/backends/pandas/query_compiler.py
@@ -321,6 +321,17 @@ def concat(self, axis, other, **kwargs):
     def _append_list_of_managers(self, others, axis, **kwargs):
         if not isinstance(others, list):
             others = [others]
+        if self._is_transposed:
+            # If others are transposed, we handle that behavior correctly in
+            # `copartition`, but it is not handled correctly in the case that `self` is
+            # transposed.
+            return (
+                self.transpose()
+                ._append_list_of_managers(
+                    [o.transpose() for o in others], axis ^ 1, **kwargs
+                )
+                .transpose()
+            )
         assert all(
             isinstance(other, type(self)) for other in others
         ), "Different Manager objects are being used. This is not allowed"


Test patch:
Repository: modin-project/modin

Issue:
pd.concat with axis=1 and keys when all objects are Series creates MutliIndex

### Describe the problem
<!-- Describe the problem clearly here. -->
`pd.concat` with `axis=1` and `keys` is not `None` when all objects are Series incorrectly creates `MutliIndex`. 

### Source code / logs

```python
>>> missing_data = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])
>>> print(missing_data.columns)
MultiIndex(levels=[['Percent', 'Total'], [0]],
           codes=[[1, 0], [0, 0]])
```
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:

Resolving patch:
diff --git a/modin/pandas/concat.py b/modin/pandas/concat.py
--- a/modin/pandas/concat.py
+++ b/modin/pandas/concat.py
@@ -90,12 +90,15 @@ def concat(
     ]
     objs = [obj._query_compiler for obj in objs if len(obj.index) or len(obj.columns)]
     if keys is not None:
-        objs = [objs[i] for i in range(min(len(objs), len(keys)))]
-        new_idx_labels = {
-            k: v.index if axis == 0 else v.columns for k, v in zip(keys, objs)
-        }
-        tuples = [(k, o) for k, obj in new_idx_labels.items() for o in obj]
-        new_idx = pandas.MultiIndex.from_tuples(tuples)
+        if all_series:
+            new_idx = keys
+        else:
+            objs = [objs[i] for i in range(min(len(objs), len(keys)))]
+            new_idx_labels = {
+                k: v.index if axis == 0 else v.columns for k, v in zip(keys, objs)
+            }
+            tuples = [(k, o) for k, obj in new_idx_labels.items() for o in obj]
+            new_idx = pandas.MultiIndex.from_tuples(tuples)
     else:
         new_idx = None
     new_query_compiler = objs[0].concat(


Test patch:
Repository: modin-project/modin

Issue:
to_pandas triggers infinite loop
### Describe the problem
`to_pandas` triggers an infinite loop if it gets called on by an empty query_compiler

### Source code / logs
`all_data_na = all_data_na.drop(all_data_na.index)`




Discussion:

Resolving patch:
diff --git a/modin/backends/pandas/query_compiler.py b/modin/backends/pandas/query_compiler.py
--- a/modin/backends/pandas/query_compiler.py
+++ b/modin/backends/pandas/query_compiler.py
@@ -49,9 +49,12 @@ def dtype_builder(df):
             )
             reduce_func = self._build_mapreduce_func(dtype_builder)
             # For now we will use a pandas Series for the dtypes.
-            self._dtype_cache = (
-                self._full_reduce(0, map_func, reduce_func).to_pandas().iloc[0]
-            )
+            if len(self.columns) > 0:
+                self._dtype_cache = (
+                    self._full_reduce(0, map_func, reduce_func).to_pandas().iloc[0]
+                )
+            else:
+                self._dtype_cache = pandas.Series([])
             # reset name to None because we use "__reduced__" internally
             self._dtype_cache.name = None
         return self._dtype_cache


Test patch:
Repository: modin-project/modin

Issue:
MultiLevelIndex errors on `reset_index`

### Describe the problem
<!-- Describe the problem clearly here. -->
The functionality is tested against a fresh dataframe, but the partitions will not be properly aligned in the case that there were some reshaping/filtering operations done before the `reset_index`

For now, we will default to pandas for MultiLevelIndex until we can get a comprehensive implementation in place for MultiLevel functionality, which is planned for late August/early September.
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:

Resolving patch:
diff --git a/modin/pandas/base.py b/modin/pandas/base.py
--- a/modin/pandas/base.py
+++ b/modin/pandas/base.py
@@ -2186,15 +2186,15 @@ def reset_index(
         """
         inplace = validate_bool_kwarg(inplace, "inplace")
         # TODO Implement level
-        if level is not None:
+        if level is not None or isinstance(self.index, pandas.MultiIndex):
             new_query_compiler = self._default_to_pandas(
                 "reset_index",
                 level=level,
                 drop=drop,
-                inplace=inplace,
+                inplace=False,
                 col_level=col_level,
                 col_fill=col_fill,
-            )
+            )._query_compiler
         # Error checking for matching Pandas. Pandas does not allow you to
         # insert a dropped index into a DataFrame if these columns already
         # exist.


Test patch:
Repository: modin-project/modin

Issue:
UserWarning: `DataFrame.<lambda>`
<!--
General questions should be asked on the mailing list modin-dev@googlegroups.com.

Before submitting an issue, please fill out the following form.
-->

### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux Ubuntu 16.04
- **Modin installed from (source or binary)**: binary
- **Modin version**: 0.5.3
- **Python version**: 3.6.8
- **Exact command to reproduce**: -

### Describe the problem
My code does not contain any lambda expression, modin still issues the following warning:

```UserWarning: `DataFrame.<lambda>` defaulting to pandas implementation.```

I'd like to understand where this come from to understand potential performance implications.
Any idea?

### Source code / logs
pandas functions used:
```
* read_csv
* df.merge
* df = df[expr]
* df.groupby
* group.count().unstack()
* group.sum()
* df.values
* series.isin
```


Discussion:
This is probably just an issue with the error message, I will take a look into it.

We default to pandas implementation for operations not yet supported and for all operations on empty dataframes. For larger dataframes there is a non-trivial cost to converting a partitioned dataframe object to a pandas DataFrame, and we are constantly working on improving the coverage for the API: https://modin.readthedocs.io/en/latest/UsingPandasonRay/dataframe_supported.html.
Thanks!
Could this be caused by using `df.groupy(["c1", "c2"])`?

I get an extra warning for using `pd.merge(df1, df2, left_on=..., right_on=..., ...)`.

Resolving patch:
diff --git a/modin/pandas/groupby.py b/modin/pandas/groupby.py
--- a/modin/pandas/groupby.py
+++ b/modin/pandas/groupby.py
@@ -490,6 +490,7 @@ def _default_to_pandas(self, f, **kwargs):
         else:
             by = self._by
 
-        return self._df._default_to_pandas(
-            lambda df: f(df.groupby(by=by, axis=self._axis, **self._kwargs)), **kwargs
-        )
+        def groupby_on_multiple_columns(df):
+            return f(df.groupby(by=by, axis=self._axis, **self._kwargs), **kwargs)
+
+        return self._df._default_to_pandas(groupby_on_multiple_columns)


Test patch:
Repository: modin-project/modin

Issue:
KeyError when specifying a list of labels from MultiIndex
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux Debian 10 unstable
- **Modin installed from (source or binary)**: source
- **Modin version**: 0.6.0
- **Python version**: 3.7
- **Exact command to reproduce**:

Upon fixing of #758 testing with list of labels leads to KeyError.

File test.dat:
```
p0,p1,data0,data1,data2
A,0,10,11,12
B,1,20,21,22
C,2,30,31,32
D,3,40,41,42
E,4,50,51,52
```

### Describe the problem
Performing a groupby with a list of labels raises a KeyError

```
In [1]: import modin.pandas
   ...: import pandas as pd
   ...: 
   ...: # Working example with Pandas
   ...: csv1=pd.read_csv('test.dat',header=0,index_col=[0,1])
   ...: g1=csv1.groupby(['p0', 'p1'])
   ...: 
   ...: # Testing with modin.pandas
   ...: csv2=modin.pandas.read_csv('test.dat',header=0,index_col=[0,1])
   ...: g2=csv2.groupby(['p0', 'p1'])
   ...: 
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-1-57918f70663a> in <module>()
      8 # Testing with modin.pandas
      9 csv2=modin.pandas.read_csv('test.dat',header=0,index_col=[0,1])
---> 10 g2=csv2.groupby(['p0', 'p1'])
~/.local/lib/python3.7/site-packages/modin/pandas/dataframe.py in groupby(self, by, axis, level, as_index, sort, group_keys, squeeze, observed, **kwargs)
    370                 pass
    371             elif mismatch:
--> 372                 raise KeyError(next(x for x in by if x not in self))
    373 
    374         if by is None and level is not None and axis == 0:

KeyError: 'p0'

```




Discussion:

Resolving patch:
diff --git a/modin/pandas/dataframe.py b/modin/pandas/dataframe.py
--- a/modin/pandas/dataframe.py
+++ b/modin/pandas/dataframe.py
@@ -361,10 +361,12 @@ def groupby(
             if isinstance(by, Series):
                 idx_name = by.name
                 by = by.values
-            mismatch = (
-                len(by) != len(self) if axis == 0 else len(by) != len(self.columns)
-            )
-            if mismatch and all(obj in self for obj in by):
+            mismatch = len(by) != len(self.axes[axis])
+            if mismatch and all(
+                obj in self
+                or (hasattr(self.index, "names") and obj in self.index.names)
+                for obj in by
+            ):
                 # In the future, we will need to add logic to handle this, but for now
                 # we default to pandas in this case.
                 pass


Test patch:
diff --git a/modin/pandas/test/test_groupby.py b/modin/pandas/test/test_groupby.py
--- a/modin/pandas/test/test_groupby.py
+++ b/modin/pandas/test/test_groupby.py
@@ -819,3 +819,8 @@ def test_groupby_multiindex():
     ray_df_equals_pandas(
         modin_df.groupby(by="four").count(), pandas_df.groupby(by="four").count()
     )
+
+    by = ["one", "two"]
+    ray_df_equals_pandas(
+        modin_df.groupby(by=by).count(), pandas_df.groupby(by=by).count()
+    )
Repository: modin-project/modin

Issue:
pyarrow is a dependency but is not in `install_requires`

### Describe the problem
<!-- Describe the problem clearly here. -->
The source comes from this file: https://github.com/modin-project/modin/blob/master/modin/experimental/engines/pyarrow_on_ray/io.py#L4-L5

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:

Resolving patch:
diff --git a/modin/experimental/engines/pyarrow_on_ray/io.py b/modin/experimental/engines/pyarrow_on_ray/io.py
--- a/modin/experimental/engines/pyarrow_on_ray/io.py
+++ b/modin/experimental/engines/pyarrow_on_ray/io.py
@@ -1,8 +1,6 @@
 from io import BytesIO
 
 import pandas
-import pyarrow as pa
-import pyarrow.csv as csv
 
 from modin.backends.pyarrow.query_compiler import PyarrowQueryCompiler
 from modin.data_management.utils import get_default_chunksize
@@ -15,6 +13,8 @@
 
 if __execution_engine__ == "Ray":
     import ray
+    import pyarrow as pa
+    import pyarrow.csv as csv
 
     @ray.remote
     def _read_csv_with_offset_pyarrow_on_ray(


Test patch:
Repository: modin-project/modin

Issue:
DataFrame.apply() functionality not working.
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:
- **pip install Modin**:
- **Modin version 0.6.0**:
- **Python version 3.7**:
- **dataframe.apply(.., axis=1)**:

### Describe the problem
<!-- The DataFrame.apply() functionality is not currently working. -->

![image](https://user-images.githubusercontent.com/25496815/64928533-5cb34c80-d7ce-11e9-8281-e8f9ee2bb007.png)

![image](https://user-images.githubusercontent.com/25496815/64928527-486f4f80-d7ce-11e9-8e62-f95df7bbb21b.png)





Discussion:
```python
def add(a, b, c): 
    return a + b + c 
data = { 
            'A':[1, 2, 3],  
            'B':[4, 5, 6],  
            'C':[7, 8, 9] } 
       
df = pd.DataFrame(data) 
print("Original DataFrame:\n", df) 

df['add'] = df.apply(lambda row : add(row['A'], 
                 row['B'], row['C']), axis = 1) 

print('\nAfter Applying Function: ') 
print(df)
```

Resolving patch:
diff --git a/modin/engines/base/frame/data.py b/modin/engines/base/frame/data.py
--- a/modin/engines/base/frame/data.py
+++ b/modin/engines/base/frame/data.py
@@ -775,7 +775,7 @@ def _apply_full_axis(
             A new dataframe.
         """
         new_partitions = self._frame_mgr_cls.map_axis_partitions(
-            axis, self._partitions, func
+            axis, self._partitions, self._build_mapreduce_func(axis, func)
         )
         # Index objects for new object creation. This is shorter than if..else
         if new_columns is None:


Test patch:
diff --git a/modin/pandas/test/test_dataframe.py b/modin/pandas/test/test_dataframe.py
--- a/modin/pandas/test/test_dataframe.py
+++ b/modin/pandas/test/test_dataframe.py
@@ -988,6 +988,23 @@ def test_apply(self, request, data, func, axis):
             modin_result = modin_df.apply(func, axis)
             df_equals(modin_result, pandas_result)
 
+    def test_apply_metadata(self):
+        def add(a, b, c):
+            return a + b + c
+
+        data = {"A": [1, 2, 3], "B": [4, 5, 6], "C": [7, 8, 9]}
+
+        modin_df = pd.DataFrame(data)
+        modin_df["add"] = modin_df.apply(
+            lambda row: add(row["A"], row["B"], row["C"]), axis=1
+        )
+
+        pandas_df = pandas.DataFrame(data)
+        pandas_df["add"] = pandas_df.apply(
+            lambda row: add(row["A"], row["B"], row["C"]), axis=1
+        )
+        df_equals(modin_df, pandas_df)
+
     @pytest.mark.parametrize("data", test_data_values, ids=test_data_keys)
     @pytest.mark.parametrize("func", agg_func_values, ids=agg_func_keys)
     @pytest.mark.parametrize("axis", axis_values, ids=axis_keys)
Repository: modin-project/modin

Issue:
Calling any() with arguments on a groupby object raises error
### Describe the problem
<!-- Describe the problem clearly here. -->
When calling groupby.any() with any arguments passed to any(), the operation will error.

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->

Can happen with:
* df.groupby('Person').any(skipna=True)



Discussion:

Resolving patch:
diff --git a/modin/pandas/groupby.py b/modin/pandas/groupby.py
--- a/modin/pandas/groupby.py
+++ b/modin/pandas/groupby.py
@@ -149,8 +149,10 @@ def sem(self, ddof=1):
     def mean(self, *args, **kwargs):
         return self._apply_agg_function(lambda df: df.mean(*args, **kwargs))
 
-    def any(self):
-        return self._groupby_reduce(lambda df: df.any(), None, numeric_only=False)
+    def any(self, **kwargs):
+        return self._groupby_reduce(
+            lambda df: df.any(**kwargs), None, numeric_only=False
+        )
 
     @property
     def plot(self):  # pragma: no cover


Test patch:
Repository: modin-project/modin

Issue:
pyarrow is a dependency but is not in `install_requires`

### Describe the problem
<!-- Describe the problem clearly here. -->
The source comes from this file: https://github.com/modin-project/modin/blob/master/modin/experimental/engines/pyarrow_on_ray/io.py#L4-L5

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:
This has resurfaced. There is some experimental code which relies on `pyarrow`

Resolving patch:
diff --git a/modin/backends/pyarrow/query_compiler.py b/modin/backends/pyarrow/query_compiler.py
--- a/modin/backends/pyarrow/query_compiler.py
+++ b/modin/backends/pyarrow/query_compiler.py
@@ -1,10 +1,14 @@
 from modin.backends.pandas.query_compiler import PandasQueryCompiler
-import pyarrow as pa
 import pandas
 from pandas.core.computation.expr import Expr
 from pandas.core.computation.scope import Scope
 from pandas.core.computation.ops import UnaryOp, BinOp, Term, MathCall, Constant
 
+from modin import __partition_format__
+
+if __partition_format__ == "Pyarrow":
+    import pyarrow as pa
+
 
 class FakeSeries:
     def __init__(self, dtype):


Test patch:
Repository: modin-project/modin

Issue:
concat is repartitioning data, even if there is an alignment in the axes
This is affecting the performance of `concat` where the operation should be effectively trivial (simply concatenating the partitions). 

The link to the code in question is here: https://github.com/modin-project/modin/blob/2a203a6c909a7f8e70a7ea3a7ae396422fdcfc6c/modin/engines/base/frame/data.py#L1076-L1080


Discussion:

Resolving patch:
diff --git a/modin/engines/base/frame/data.py b/modin/engines/base/frame/data.py
--- a/modin/engines/base/frame/data.py
+++ b/modin/engines/base/frame/data.py
@@ -1036,11 +1036,37 @@ def _concat(self, axis, others, how, sort):
         Returns:
             A new dataframe.
         """
-        # TODO Update to no longer force repartition
-        # Requires pruning of the partitions after they have been changed
-        left_parts, right_parts, joined_index = self._copartition(
-            axis ^ 1, others, how, sort, force_repartition=True
-        )
+        # Fast path for equivalent columns and partitioning
+        if (
+            axis == 0
+            and all(o.columns.equals(self.columns) for o in others)
+            and all(o._column_widths == self._column_widths for o in others)
+        ):
+            joined_index = self.columns
+            left_parts = self._partitions
+            right_parts = [o._partitions for o in others]
+            new_lengths = self._row_lengths + [
+                l for o in others for l in o._row_lengths
+            ]
+            new_widths = self._column_widths
+        elif (
+            axis == 1
+            and all(o.index.equals(self.index) for o in others)
+            and all(o._row_lengths == self._row_lengths for o in others)
+        ):
+            joined_index = self.index
+            left_parts = self._partitions
+            right_parts = [o._partitions for o in others]
+            new_lengths = self._row_lengths
+            new_widths = self._column_widths + [
+                l for o in others for l in o._column_widths
+            ]
+        else:
+            left_parts, right_parts, joined_index = self._copartition(
+                axis ^ 1, others, how, sort, force_repartition=True
+            )
+            new_lengths = None
+            new_widths = None
         new_partitions = self._frame_mgr_cls.concat(axis, left_parts, right_parts)
         if axis == 0:
             new_index = self.index.append([other.index for other in others])
@@ -1048,7 +1074,9 @@ def _concat(self, axis, others, how, sort):
         else:
             new_columns = self.columns.append([other.columns for other in others])
             new_index = joined_index
-        return self.__constructor__(new_partitions, new_index, new_columns)
+        return self.__constructor__(
+            new_partitions, new_index, new_columns, new_lengths, new_widths
+        )
 
     def groupby_reduce(
         self, axis, by, map_func, reduce_func, new_index=None, new_columns=None


Test patch:
Repository: modin-project/modin

Issue:
__repr__ blocking on dtype computation if they are not 
The `head` and `tail` calls for `__repr__` are passing the physical `dtype` which forces computation to compute them, but `repr` does not need `dtypes`.

Relevant code: https://github.com/modin-project/modin/blob/2a203a6c909a7f8e70a7ea3a7ae396422fdcfc6c/modin/engines/base/frame/data.py#L1230-L1324


Discussion:

Resolving patch:
diff --git a/modin/engines/base/frame/data.py b/modin/engines/base/frame/data.py
--- a/modin/engines/base/frame/data.py
+++ b/modin/engines/base/frame/data.py
@@ -1214,7 +1214,7 @@ def head(self, n):
             self.columns,
             new_row_lengths,
             self._column_widths,
-            self.dtypes,
+            self._dtypes,
         )
 
     def tail(self, n):
@@ -1239,7 +1239,7 @@ def tail(self, n):
             self.columns,
             new_row_lengths,
             self._column_widths,
-            self.dtypes,
+            self._dtypes,
         )
 
     def front(self, n):
@@ -1261,7 +1261,7 @@ def front(self, n):
             self.columns[:n],
             self._row_lengths,
             new_col_lengths,
-            self.dtypes[:n],
+            self.dtypes[:n] if self._dtypes is not None else None,
         )
 
     def back(self, n):
@@ -1283,7 +1283,7 @@ def back(self, n):
             self.columns[-n:],
             self._row_lengths,
             new_col_lengths,
-            self.dtypes[n:],
+            self.dtypes[n:] if self._dtypes is not None else None,
         )
 
     # End Head/Tail/Front/Back


Test patch: