internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: modin-project/modin

Issue:
Series results out of order
### Describe the problem
When doing certain functions that return a Series, the values in the Series are returned out of order.

### Source code / logs
```
random_state = np.random.RandomState(seed=42)
data = {"col{}".format(i+1): random_state.randint(-100, 100, size=(17)) for i in range(17)}

modin_df = pd.DataFrame(data)
kwargs = {"axis" : 'columns', "skipna" :True, "numeric_only" : False, "ddof" : 0}
modin_result = modin_df.std(axis='columns')
```
From Modin, we get 
```
0     54.913899
1     56.853664
2     52.637232
3     46.299237
4     44.255236
5     60.790411
6     59.509878
7     59.605450
8     73.900616
9     51.435316
10    48.410585
11    42.955678
12    60.365438
13    56.463480
14    53.407489
15    48.937217
16    62.427370
dtype: float64
```
While in Pandas, we get 
```
0     54.913899
1     56.853664
2     59.605450
3     73.900616
4     53.407489
5     52.637232
6     46.299237
7     51.435316
8     48.410585
9     48.937217
10    44.255236
11    60.790411
12    42.955678
13    60.365438
14    62.427370
15    59.509878
16    56.463480
dtype: float64
```
Notice that index 2 is different between the two series (and differs at more places as well).



Discussion:

Resolving patch:
diff --git a/modin/data_management/data_manager.py b/modin/data_management/data_manager.py
--- a/modin/data_management/data_manager.py
+++ b/modin/data_management/data_manager.py
@@ -1103,8 +1103,13 @@ def full_axis_reduce(self, func, axis):
         Return:
             Pandas series containing the reduced data.
         """
+        # We XOR with axis because if we are doing an operation over the columns
+        # (i.e. along the rows), we want to take the transpose so that the
+        # results from the same parition will be concated together first.
+        # We need this here because if the operations is over the columns,
+        # map_across_full_axis does not transpose the result before returning.
         result = self.data.map_across_full_axis(axis, func).to_pandas(
-            self._is_transposed
+            self._is_transposed ^ axis
         )
         if not axis:
             result.index = self.columns
diff --git a/modin/data_management/partitioning/partition_collections.py b/modin/data_management/partitioning/partition_collections.py
--- a/modin/data_management/partitioning/partition_collections.py
+++ b/modin/data_management/partitioning/partition_collections.py
@@ -425,6 +425,9 @@ def to_pandas(self, is_transposed=False):
                 for part in row
             ):
                 axis = 0
+                # We take the transpose here so that the results from the same
+                # partition will be concated together first before results
+                # from different partitions.
                 retrieved_objects = np.array(retrieved_objects).T
             elif all(
                 isinstance(part, pandas.DataFrame)


Test patch:
Repository: modin-project/modin

Issue:
sort_values not updating inplace and na_position argument is ignored
### Describe the problem
`sort_values` does not update in-place when `update_inplace = True` and `na_position` argument is ignored.

### Source code / logs
#### Not updating inplace
```
data = {
    "col1": [np.NaN, 2, np.NaN, 0],
    "col2": [3, 4, np.NaN, 1],
    "col3": [np.NaN, np.NaN, np.NaN, 5],
    "col4": [6, np.NaN, np.NaN, 10],
}
modin_df = pd.DataFrame(data)
modin_df.sort_values(by=[modin_df.index[0]], axis='columns', ascending=False, na_position='last', inplace=True)
modin_df
```
returns 
```
	col1	col2	col3	col4
0	NaN	3.0	NaN	6.0
1	2.0	4.0	NaN	NaN
2	NaN	NaN	NaN	NaN
3	0.0	1.0	5.0	10.0
```
instead of 
```
	col4	col2	col1	col3
0	6.0	3.0	NaN	NaN
1	NaN	4.0	2.0	NaN
2	NaN	NaN	NaN	NaN
3	10.0	1.0	0.0	5.0
```
#### `na_argument` ignored
```
data = {
    "col1": [np.NaN, 2, np.NaN, 0],
    "col2": [3, 4, np.NaN, 1],
    "col3": [np.NaN, np.NaN, np.NaN, 5],
    "col4": [6, np.NaN, np.NaN, 10],
}
modin_df = pd.DataFrame(data)
modin_result = modin_df.sort_values(by=[modin_df.index[0]], axis='columns', ascending=False, na_position='first')
modin_result
```
returns 
```
	col4	col2	col1	col3
0	6.0	3.0	NaN	NaN
1	NaN	4.0	2.0	NaN
2	NaN	NaN	NaN	NaN
3	10.0	1.0	0.0	5.0
```
instead of 
```
	col1	col3	col4	col2
0	NaN	NaN	6.0	3.0
1	2.0	NaN	NaN	4.0
2	NaN	NaN	NaN	NaN
3	0.0	5.0	10.0	1.0
```


Discussion:

Resolving patch:
diff --git a/modin/pandas/dataframe.py b/modin/pandas/dataframe.py
--- a/modin/pandas/dataframe.py
+++ b/modin/pandas/dataframe.py
@@ -3570,24 +3570,30 @@ def sort_values(
             broadcast_value_dict = {col: self[col] for col in by}
             broadcast_values = pandas.DataFrame(broadcast_value_dict, index=self.index)
             new_index = broadcast_values.sort_values(
-                by=by, axis=axis, ascending=ascending, kind=kind
+                by=by,
+                axis=axis,
+                ascending=ascending,
+                kind=kind,
+                na_position=na_position,
             ).index
-            return self.reindex(index=new_index)
+            return self.reindex(index=new_index, copy=not inplace)
         else:
             broadcast_value_list = [
                 to_pandas(self[row :: len(self.index)]) for row in by
             ]
             index_builder = list(zip(broadcast_value_list, by))
-
             broadcast_values = pandas.concat(
                 [row for row, idx in index_builder], copy=False
             )
             broadcast_values.columns = self.columns
             new_columns = broadcast_values.sort_values(
-                by=by, axis=axis, ascending=ascending, kind=kind
+                by=by,
+                axis=axis,
+                ascending=ascending,
+                kind=kind,
+                na_position=na_position,
             ).columns
-
-            return self.reindex(columns=new_columns)
+            return self.reindex(columns=new_columns, copy=not inplace)
 
     def sortlevel(
         self, level=0, axis=0, ascending=True, inplace=False, sort_remaining=True


Test patch:
Repository: modin-project/modin

Issue:
abs returning wrong dtypes
### Describe the problem
`abs` on an `int` column returns the type as `float64`. Additionally, when taking `abs` of a dataframe with nonnumeric objects, a `RayGetError` is thrown.

### Source code / logs
For the first `abs` issue, the following code reproduces the issue,
```
import modin.pandas as pd

data = {"col{}".format((i-50)%100+1): np.random.randint(-100, 100, size=(100)) for i in range(100)}

modin_df = pd.DataFrame(data)
modin_df.abs().dtypes
```

For the second `abs` issue, the following code reproduces the issue,
```
import modin.pandas as pd

data = {
    "col3": 1.0,
    "col4": np.datetime64("2011-06-15T00:00"),
    "col5": np.array([3] * 4, dtype="int32"),
    "col1": "foo",
    "col2": True,
}

pd.DataFrame(data).abs()
```



Discussion:

Resolving patch:
diff --git a/modin/data_management/data_manager.py b/modin/data_management/data_manager.py
--- a/modin/data_management/data_manager.py
+++ b/modin/data_management/data_manager.py
@@ -1058,10 +1058,7 @@ def map_partitions(self, func, new_dtypes=None):
 
     def abs(self):
         func = self._prepare_method(pandas.DataFrame.abs)
-        new_dtypes = pandas.Series(
-            [np.dtype("float64") for _ in self.columns], index=self.columns
-        )
-        return self.map_partitions(func, new_dtypes=new_dtypes)
+        return self.map_partitions(func, new_dtypes=self.dtypes.copy())
 
     def applymap(self, func):
         remote_func = self._prepare_method(pandas.DataFrame.applymap, func=func)
diff --git a/modin/pandas/dataframe.py b/modin/pandas/dataframe.py
--- a/modin/pandas/dataframe.py
+++ b/modin/pandas/dataframe.py
@@ -418,10 +418,7 @@ def abs(self):
         Returns:
             A new DataFrame with the applied absolute value.
         """
-        for t in self.dtypes:
-            if np.dtype("O") == t:
-                # TODO Give a more accurate error to Pandas
-                raise TypeError("bad operand type for abs():", "str")
+        self._validate_dtypes(numeric_only=True)
 
         return DataFrame(data_manager=self._data_manager.abs())
 


Test patch:
Repository: modin-project/modin

Issue:
__neg__ throwing RayGetError
### Describe the problem
`__neg__` throws `RayGetError` for time dtypes

### Source code / logs
```
import modin.pandas as pd

data = {
    "col3": [
        np.datetime64("2010"),
        np.datetime64("2011"),
        np.datetime64("2011-06-15T00:00"),
        np.datetime64("2009-01-01"),
    ],
    "col4": [
        np.datetime64("2010"),
        np.datetime64("2011"),
        np.datetime64("2011-06-15T00:00"),
        np.datetime64("2009-01-01"),
    ]}

-modin_df
```



Discussion:

Resolving patch:
diff --git a/modin/pandas/dataframe.py b/modin/pandas/dataframe.py
--- a/modin/pandas/dataframe.py
+++ b/modin/pandas/dataframe.py
@@ -8,7 +8,6 @@
 import pandas.core.common as com
 from pandas.core.dtypes.common import (
     _get_dtype_from_object,
-    is_bool_dtype,
     is_list_like,
     is_numeric_dtype,
     is_datetime_or_timedelta_dtype,
@@ -4840,16 +4839,7 @@ def __neg__(self):
         Returns:
             A modified DataFrame where every element is the negation of before
         """
-        for t in self.dtypes:
-            if not (
-                is_bool_dtype(t)
-                or is_numeric_dtype(t)
-                or is_datetime_or_timedelta_dtype(t)
-            ):
-                raise TypeError(
-                    "Unary negative expects numeric dtype, not {}".format(t)
-                )
-
+        self._validate_dtypes(numeric_only=True)
         return DataFrame(query_compiler=self._query_compiler.negative())
 
     def __sizeof__(self):


Test patch:
Repository: modin-project/modin

Issue:
read_csv with header passed from outside doesn't seem to work
<!--
General questions should be asked on the mailing list modin-dev@googlegroups.com.

Before submitting an issue, please fill out the following form.
-->

### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu 16.04
- **Modin installed from (source or binary)**: pypi
- **Modin version**:0.2.2
- **Python version**:3.6
- **Exact command to reproduce**: pd.read_csv('file,sep='\t',index_col=False,names=headerlist)

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->
It appears that read_csv with header passed from outside and index unspecified doesn't work. Get the following error.

ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

The above command works with Pandas.

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:
@aakashganga Thanks for posting this issue! I will look into the issue to reproduce it and get it fixed. Thanks again!
So it looks like this will work as long as you pass a `list` object, but if you try to pass a `modin.pandas.Series` or `modin.pandas.Index` object it will not. 

I will fix this, in the meantime if it is blocking you, you can convert the names to a python `list` and it should work normally. Thanks again for posting the issue!
Got it.

On Thu, Nov 1, 2018 at 1:10 PM Devin Petersohn <notifications@github.com>
wrote:

> So it looks like this will work as long as you pass a list object, but if
> you try to pass a modin.pandas.Series or modin.pandas.Index object it
> will not.
>
> I will fix this, in the meantime if it is blocking you, you can convert
> the names to a python list and it should work normally. Thanks again for
> posting the issue!
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/modin-project/modin/issues/239#issuecomment-435109340>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AEGprjWJWth2jgtndy6UuxMnAytfxpoMks5uqysWgaJpZM4YGuKg>
> .
>


Resolving patch:
diff --git a/modin/pandas/io.py b/modin/pandas/io.py
--- a/modin/pandas/io.py
+++ b/modin/pandas/io.py
@@ -288,7 +288,9 @@ def read_csv(
         kwargs = {
             kw: kwargs[kw]
             for kw in kwargs
-            if kw in defaults and kwargs[kw] != defaults[kw]
+            if kw in defaults
+            and not isinstance(kwargs[kw], type(defaults[kw]))
+            and kwargs[kw] != defaults[kw]
         }
     # This happens on Python2, we will just default to serializing the entire dictionary
     except AttributeError:


Test patch:
Repository: modin-project/modin

Issue:
Attribute error when doing sample on axis=1
### Describe the problem
The following error gets thrown when running `df.sample(axis=1)`
```AttributeError: 'DataFrame' object has no attribute 'column'```

### Source code / logs
```
import modin.pandas as pd
data = {
    "col3": [1, 2, 3, 4],
    "col4": [4, 5, 6, 7],
    "col1": [8.0, 9.4, 10.1, 11.3],
    "col2": ["a", "b", "c", "d"],
}
pd.DataFrame(data).sample(axis=1)
```



Discussion:

Resolving patch:
diff --git a/modin/pandas/dataframe.py b/modin/pandas/dataframe.py
--- a/modin/pandas/dataframe.py
+++ b/modin/pandas/dataframe.py
@@ -3391,14 +3391,12 @@ def sample(
         Returns:
             A new Dataframe
         """
-
         axis = pandas.DataFrame()._get_axis_number(axis) if axis is not None else 0
-
-        if axis == 0:
-            axis_labels = self.index
+        if axis:
+            axis_labels = self.columns
             axis_length = len(axis_labels)
         else:
-            axis_labels = self.column
+            axis_labels = self.index
             axis_length = len(axis_labels)
         if weights is not None:
             # Index of the weights Series should correspond to the index of the
@@ -3492,8 +3490,8 @@ def sample(
             samples = np.random.choice(
                 a=axis_labels, size=n, replace=replace, p=weights
             )
-        if axis == 1:
-            query_compiler = self._query_compiler.getitem_col_array(samples)
+        if axis:
+            query_compiler = self._query_compiler.getitem_column_array(samples)
             return DataFrame(query_compiler=query_compiler)
         else:
             query_compiler = self._query_compiler.getitem_row_array(samples)


Test patch:
Repository: modin-project/modin

Issue:
to_pandas issue with python backend
### Describe the problem
`to_pandas` does not work when some rows are empty

### Source code / logs
```
import os
os.environ['MODIN_DEBUG'] = '1'

import modin.pandas as pd

frame_data = {"A": [1, 2, 3, 4], "B": [0, 1, 2, 3]}
df = pd.DataFrame(frame_data)
df.drop(["A", "B"], axis="columns")
```



Discussion:

Resolving patch:
diff --git a/modin/data_management/partitioning/partition_collections/base_block_partitions.py b/modin/data_management/partitioning/partition_collections/base_block_partitions.py
--- a/modin/data_management/partitioning/partition_collections/base_block_partitions.py
+++ b/modin/data_management/partitioning/partition_collections/base_block_partitions.py
@@ -450,6 +450,7 @@ def to_pandas(self, is_transposed=False):
             df_rows = [
                 pandas.concat([part for part in row], axis=axis)
                 for row in retrieved_objects
+                if not all(part.empty for part in row)
             ]
             if len(df_rows) == 0:
                 return pandas.DataFrame()


Test patch:
Repository: modin-project/modin

Issue:
from_pandas does not create independent object in python backend
### Describe the problem
When building a modin dataframe from a pandas dataframe in the python backend, any changes to the pandas dataframe is propagated to the modin dataframe. This should not happen.

### Source code / logs
```
import os
os.environ['MODIN_DEBUG'] = '1'

import modin.pandas as pd

frame_data = random_state.randn(10, 4)
df = pandas.DataFrame(frame_data)
df[1][:4] = np.nan
df[3][-4:] = np.nan

modin_df = pd.DataFrame(df)
df.fillna(value=0, inplace=True)
```


Discussion:

Resolving patch:
diff --git a/modin/data_management/partitioning/partition_collections/base_block_partitions.py b/modin/data_management/partitioning/partition_collections/base_block_partitions.py
--- a/modin/data_management/partitioning/partition_collections/base_block_partitions.py
+++ b/modin/data_management/partitioning/partition_collections/base_block_partitions.py
@@ -466,7 +466,7 @@ def from_pandas(cls, df):
         # Each chunk must have a RangeIndex that spans its length and width
         # according to our invariant.
         def chunk_builder(i, j):
-            chunk = df.iloc[i : i + row_chunksize, j : j + col_chunksize]
+            chunk = df.iloc[i : i + row_chunksize, j : j + col_chunksize].copy()
             chunk.index = pandas.RangeIndex(len(chunk.index))
             chunk.columns = pandas.RangeIndex(len(chunk.columns))
             return put_func(chunk)


Test patch:
Repository: modin-project/modin

Issue:
iloc functionality issue
For certain DataFrames (will have more detail on this shortly), `df.iloc[[<index_number>]]` to pull a row from a DataFrame doesn't work. Seems to be working fine for smaller DataFrames (including mixed-type DataFrames), but the issue is present in larger DataFrames.

Use this to reproduce:
```
import modin.pandas as pd
data = {"col{}".format(i+1): random_state.randint(RAND_LOW, RAND_HIGH, size=(100)) for i in range(100)}
df = pd.DataFrame(data)
print(df.iloc[[12]])
```


Discussion:

Resolving patch:
diff --git a/modin/data_management/query_compiler/pandas_query_compiler.py b/modin/data_management/query_compiler/pandas_query_compiler.py
--- a/modin/data_management/query_compiler/pandas_query_compiler.py
+++ b/modin/data_management/query_compiler/pandas_query_compiler.py
@@ -2644,16 +2644,7 @@ def __constructor__(
         columns: pandas.Index,
         dtypes=None,
     ):
-        new_index_map = self.index_map.reindex(index)
-        new_columns_map = self.columns_map.reindex(columns)
-        return type(self)(
-            block_partitions_object,
-            index,
-            columns,
-            dtypes,
-            new_index_map,
-            new_columns_map,
-        )
+        return PandasQueryCompiler(block_partitions_object, index, columns, dtypes)
 
     def _get_data(self) -> BaseBlockPartitions:
         """Perform the map step


Test patch:
Repository: modin-project/modin

Issue:
__repr__ sometimes throws an error

### Describe the problem
<!-- Describe the problem clearly here. -->
`__repr__` is sometimes throwing an error. The code below reproduces the error

### Source code / logs
```python
import modin.pandas as pd 
import numpy as np 
data = {"col{}".format(i+1): np.random.randint(0, 100, size=(100)) for i in range(100)} 
df = pd.DataFrame(data) 
```
```python
df # will not result in error
```
```python
df # will  result in error
```
The second time it is run (and every other time after) an error is thrown.
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:

Resolving patch:
diff --git a/modin/engines/base/block_partitions.py b/modin/engines/base/block_partitions.py
--- a/modin/engines/base/block_partitions.py
+++ b/modin/engines/base/block_partitions.py
@@ -304,9 +304,7 @@ def take(self, axis, n):
             partitions = self.partitions.T
             bin_lengths = self.block_widths
         if n < 0:
-            reversed_bins = bin_lengths
-            reversed_bins.reverse()
-            length_bins = np.cumsum(reversed_bins)
+            length_bins = np.cumsum(bin_lengths[::-1])
             n *= -1
             idx = int(np.digitize(n, length_bins))
             if idx > 0:


Test patch:
Repository: modin-project/modin

Issue:
median returns series where index and values are mismatched
### Describe the problem
On the python backend, `median` returns values that are not matched to the correct index

### Source code / logs
```
import modin.pandas as pd
import pandas
from modin.pandas.test.utils import test_data
data = {
    "col3": [0.0, 1.0, 2.0, 3.0],
    "col4": [4.0, 5.0, 6.0, 7.0],
    "col5": [8.0, 9.0, 10.0, 11.0],
    "col1": [12.0, 13.0, 14.0, 15.0],
    "col2": [0.0, 0.0, 0.0, 0.0],
}
modin_df = pd.DataFrame(data)
pandas_df = pandas.DataFrame(data)
```


Discussion:

Resolving patch:
diff --git a/modin/engines/python/pandas_on_python/axis_partition.py b/modin/engines/python/pandas_on_python/axis_partition.py
--- a/modin/engines/python/pandas_on_python/axis_partition.py
+++ b/modin/engines/python/pandas_on_python/axis_partition.py
@@ -105,7 +105,9 @@ def deploy_python_axis_func(axis, func, num_splits, kwargs, *partitions):
     """
     dataframe = pandas.concat(partitions, axis=axis, copy=False)
     result = func(dataframe, **kwargs)
-    if num_splits != len(partitions) or isinstance(result, pandas.Series):
+    if isinstance(result, pandas.Series):
+        return [result] + [pandas.Series([]) for _ in range(num_splits - 1)]
+    if num_splits != len(partitions):
         lengths = None
     else:
         if axis == 0:


Test patch: