internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: modin-project/modin

Issue:
df.drop() performance
### Describe the problem

Running df.drop([0]) on larger dataframes is much slower than df.iloc[1:], which should be getting the same result

`frame_data = np.random.randint(-100, 100, size=(2**22, 2**8))`
`df = pd.DataFrame(frame_data)`
`df.drop([0])`
`df.iloc[1:]`



Discussion:

Resolving patch:
diff --git a/modin/engines/base/frame/data.py b/modin/engines/base/frame/data.py
--- a/modin/engines/base/frame/data.py
+++ b/modin/engines/base/frame/data.py
@@ -603,7 +603,6 @@ def mask(
             new_row_lengths,
             new_col_widths,
             new_dtypes,
-            validate_axes="all" if new_partitions.size != 0 else False,
         )
         # Check if monotonically increasing, return if it is. Fast track code path for
         # common case to keep it fast.
diff --git a/modin/pandas/base.py b/modin/pandas/base.py
--- a/modin/pandas/base.py
+++ b/modin/pandas/base.py
@@ -786,19 +786,26 @@ def at(self, axis=None):
 
     def at_time(self, time, asof=False, axis=None):
         axis = self._get_axis_number(axis)
-        if axis == 0:
-            return self.iloc[self.index.indexer_at_time(time, asof=asof)]
-        return self.iloc[:, self.columns.indexer_at_time(time, asof=asof)]
+        idx = self.index if axis == 0 else self.columns
+        indexer = pandas.Series(index=idx).at_time(time, asof=asof).index
+        return self.loc[indexer] if axis == 0 else self.loc[:, indexer]
 
     def between_time(
         self, start_time, end_time, include_start=True, include_end=True, axis=None
     ):
         axis = self._get_axis_number(axis)
         idx = self.index if axis == 0 else self.columns
-        indexer = idx.indexer_between_time(
-            start_time, end_time, include_start=include_start, include_end=include_end
+        indexer = (
+            pandas.Series(index=idx)
+            .between_time(
+                start_time,
+                end_time,
+                include_start=include_start,
+                include_end=include_end,
+            )
+            .index
         )
-        return self.iloc[indexer] if axis == 0 else self.iloc[:, indexer]
+        return self.loc[indexer] if axis == 0 else self.loc[:, indexer]
 
     def bfill(self, axis=None, inplace=False, limit=None, downcast=None):
         return self.fillna(
diff --git a/modin/pandas/indexing.py b/modin/pandas/indexing.py
--- a/modin/pandas/indexing.py
+++ b/modin/pandas/indexing.py
@@ -554,6 +554,15 @@ def __getitem__(self, key):
             and all(col_loc[i] in result.columns.levels[i] for i in range(len(col_loc)))
         ):
             result.columns = result.columns.droplevel(list(range(len(col_loc))))
+        # This is done for cases where the index passed in has other state, like a
+        # frequency in the case of DateTimeIndex.
+        if (
+            row_lookup is not None
+            and isinstance(col_loc, slice)
+            and col_loc == slice(None)
+            and isinstance(key, pandas.Index)
+        ):
+            result.index = key
         return result
 
     def __setitem__(self, key, item):


Test patch:
Repository: modin-project/modin

Issue:
Upgrade to Pyarrow 1.0
In #1377 we require `pyarrow<0.17`, but since Pyarrow 1.0 released, we should update to new stable API.


Discussion:
Gandiva appears to be no longer part of the default pyarrow wheel (https://github.com/apache/arrow/pull/7334).

There is an open issue to package it separately: https://issues.apache.org/jira/browse/ARROW-8518
Thanks @itamarst, this will be an issue in updating. Unfortunately that issue is slated for resolution in version 2. We probably will have to make the gandiva import conditional.

Resolving patch:
diff --git a/modin/engines/base/io/column_stores/feather_reader.py b/modin/engines/base/io/column_stores/feather_reader.py
--- a/modin/engines/base/io/column_stores/feather_reader.py
+++ b/modin/engines/base/io/column_stores/feather_reader.py
@@ -32,8 +32,7 @@ def _read(cls, path, columns=None, **kwargs):
             https://arrow.apache.org/docs/python/api.html#feather-format
         """
         if columns is None:
-            from pyarrow.feather import FeatherReader
+            from pyarrow.feather import read_feather
 
-            fr = FeatherReader(path)
-            columns = [fr.get_column_name(i) for i in range(fr.num_columns)]
-        return cls.build_query_compiler(path, columns, use_threads=False)
+            df = read_feather(path)
+        return cls.build_query_compiler(path, df.columns, use_threads=False)
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -38,7 +38,7 @@ def is_pure(self):
 
 
 dask_deps = ["dask>=2.12.0,<=2.19.0", "distributed>=2.12.0,<=2.19.0"]
-ray_deps = ["ray>=1.0.0", "pyarrow<0.17"]
+ray_deps = ["ray>=1.0.0", "pyarrow==1.0"]
 remote_deps = ["rpyc==4.1.5", "cloudpickle==1.4.1", "boto3==1.4.8"]
 
 all_deps = dask_deps + ray_deps + remote_deps


Test patch:
diff --git a/modin/pandas/test/test_io.py b/modin/pandas/test/test_io.py
--- a/modin/pandas/test/test_io.py
+++ b/modin/pandas/test/test_io.py
@@ -40,7 +40,7 @@
     IO_OPS_DATA_DIR,
 )
 
-from modin.config import Engine, Backend
+from modin.config import Engine, Backend, IsExperimental
 
 if Backend.get() == "Pandas":
     import modin.pandas as pd
@@ -495,6 +495,10 @@ def teardown_fwf_file():
             pass
 
 
+@pytest.mark.skipif(
+    IsExperimental.get() and Backend.get() == "Pyarrow",
+    reason="Segmentation fault; see PR #2347 ffor details",
+)
 class TestReadCSV:
     # delimiter tests
     @pytest.mark.parametrize("sep", ["_", ",", ".", "\n"])
Repository: modin-project/modin

Issue:
Fix default to pandas warnings 




Discussion:
```python
import numpy as np
import modin.pandas as pd
#import pandas as pd
df = pd.DataFrame({
    "col1": [ 1,  2,  1,  2,  1,  3,  3,  2,  3],
    "col2": [11, 12, 13, 14, 15, 16, 17, 18, 19],
    "col3": [ 4,  4,  4,  4,  5,  5,  5,  5,  5],
    "col4": [ 1,  1,  1,  1,  1,  1,  1,  1,  1]
})
print(df)
gb0 = df.groupby('col3')
df0 = gb0.agg(sum)
print(df0)
gb0 = df.groupby('col3')
df0 = gb0.agg({'col2': 'max'})
print(df0)
gb1 = df.groupby('col3')
df1 = gb1.agg(new_col=('col2', max))
print(df1)
gb2 = df.groupby(['col1', 'col3'])
df2 = gb2.agg({'col2': 'max'})
print(df2)
gb3 = df.groupby(['col1', 'col3'])
df3 = gb3.agg({'col2': 'max', 'col4': 'sum'})
print(df3)
gb4 = df.groupby(['col1', 'col3'])
df4 = gb4.agg(new_col1=('col2', 'max'), new_col2=('col2', 'min'), new_col3=('col2', 'sum'),
              new_col4=('col4', 'max'), new_col5=('col4', 'min'), new_col6=('col4', 'sum'))
print(df4)
```

Resolving patch:
diff --git a/modin/backends/pandas/query_compiler.py b/modin/backends/pandas/query_compiler.py
--- a/modin/backends/pandas/query_compiler.py
+++ b/modin/backends/pandas/query_compiler.py
@@ -2559,31 +2559,6 @@ def _callable_func(self, func, axis, *args, **kwargs):
         method="size",
     )
 
-    def groupby_dict_agg(self, by, func_dict, groupby_args, agg_args, drop=False):
-        """Apply aggregation functions to a grouped dataframe per-column.
-
-        Parameters
-        ----------
-        by : PandasQueryCompiler
-            The column to group by
-        func_dict : dict of str, callable/string
-            The dictionary mapping of column to function
-        groupby_args : dict
-            The dictionary of keyword arguments for the group by.
-        agg_args : dict
-            The dictionary of keyword arguments for the aggregation functions
-        drop : bool
-            Whether or not to drop the column from the data.
-
-        Returns
-        -------
-        PandasQueryCompiler
-            The result of the per-column aggregations on the grouped dataframe.
-        """
-        return self.default_to_pandas(
-            lambda df: df.groupby(by=by, **groupby_args).agg(func_dict, **agg_args)
-        )
-
     def groupby_agg(
         self,
         by,


Test patch:
Repository: modin-project/modin

Issue:
Conda run doesn't forward exit code
We need to use explicit activation of environment.


Discussion:

Resolving patch:
diff --git a/modin/experimental/cloud/rayscale.py b/modin/experimental/cloud/rayscale.py
--- a/modin/experimental/cloud/rayscale.py
+++ b/modin/experimental/cloud/rayscale.py
@@ -262,6 +262,7 @@ def wrap_cmd(self, cmd: list):
             [
                 "bash",
                 "-ic",
-                subprocess.list2cmdline(["conda", "run", "-n", "modin"] + cmd),
+                # workaround for https://github.com/conda/conda/issues/8385
+                subprocess.list2cmdline(["conda", "activate", "modin", "&&"] + cmd),
             ]
         )


Test patch:
Repository: modin-project/modin

Issue:
Enable Calcite by default in OmniSci backend



Discussion:

Resolving patch:
diff --git a/modin/config/envvars.py b/modin/config/envvars.py
--- a/modin/config/envvars.py
+++ b/modin/config/envvars.py
@@ -196,6 +196,7 @@ class DoUseCalcite(EnvironmentVariable, type=bool):
     """
 
     varname = "MODIN_USE_CALCITE"
+    default = True
 
 
 class TestDatasetSize(EnvironmentVariable, type=str):


Test patch:
Repository: modin-project/modin

Issue:
Fix inserting Series into DataFrame
Behavior of inserting Series into DataFrame is differ from pandas. An example:
```python
import modin.pandas as pd
import pandas
mdf = pd.DataFrame({"a": [1,2,3]})
pdf = pandas.DataFrame({"a": [1,2,3]})
mdf["b"] = pd.Series([4,5,6,7,8])
pdf["b"] = pandas.Series([4,5,6,7,8])
mdf
     a  b
0  1.0  4
1  2.0  5
2  3.0  6
3  NaN  7
4  NaN  8
pdf
   a  b
0  1  4
1  2  5
2  3  6
```


Discussion:

Resolving patch:
diff --git a/modin/backends/base/query_compiler.py b/modin/backends/base/query_compiler.py
--- a/modin/backends/base/query_compiler.py
+++ b/modin/backends/base/query_compiler.py
@@ -137,10 +137,9 @@ def concat(df, axis, other, **kwargs):
             else:
                 if isinstance(other, (list, np.ndarray)) and len(other) == 1:
                     other = other[0]
-                how = kwargs.pop("join", None)
                 ignore_index = kwargs.pop("ignore_index", None)
-                kwargs["how"] = how
-                result = df.join(other, **kwargs)
+                kwargs["how"] = kwargs.pop("join", None)
+                result = df.join(other, rsuffix="r_", **kwargs)
             if ignore_index:
                 if axis == 0:
                     result = result.reset_index(drop=True)
diff --git a/modin/pandas/dataframe.py b/modin/pandas/dataframe.py
--- a/modin/pandas/dataframe.py
+++ b/modin/pandas/dataframe.py
@@ -1972,7 +1972,11 @@ def __setitem__(self, key, value):
                     self._query_compiler = value._query_compiler.copy()
                 else:
                     self._create_or_update_from_compiler(
-                        self._query_compiler.concat(1, value._query_compiler),
+                        self._query_compiler.concat(
+                            1,
+                            value._query_compiler,
+                            join="left",
+                        ),
                         inplace=True,
                     )
                 # Now that the data is appended, we need to update the column name for


Test patch:
diff --git a/modin/pandas/test/dataframe/test_indexing.py b/modin/pandas/test/dataframe/test_indexing.py
--- a/modin/pandas/test/dataframe/test_indexing.py
+++ b/modin/pandas/test/dataframe/test_indexing.py
@@ -1161,6 +1161,13 @@ def test___setitem__(data):
 
     df_equals(modin_df, pandas_df)
 
+    # from issue #2390
+    modin_df = pd.DataFrame({"a": [1, 2, 3]})
+    pandas_df = pandas.DataFrame({"a": [1, 2, 3]})
+    modin_df["b"] = pd.Series([4, 5, 6, 7, 8])
+    pandas_df["b"] = pandas.Series([4, 5, 6, 7, 8])
+    df_equals(modin_df, pandas_df)
+
 
 def test___setitem__with_mismatched_partitions():
     fname = "200kx99.csv"
Repository: modin-project/modin

Issue:
[REFACTOR]: Redundant assignment in modin/pandas/base/query_compiler.py



Discussion:

Resolving patch:
diff --git a/modin/backends/base/query_compiler.py b/modin/backends/base/query_compiler.py
--- a/modin/backends/base/query_compiler.py
+++ b/modin/backends/base/query_compiler.py
@@ -1412,8 +1412,6 @@ def groupby_agg(
                 by = by.columns[0] if drop else by.to_pandas().squeeze()
             elif isinstance(by, type(self)):
                 by = list(by.columns)
-            else:
-                by = by
         else:
             by = by.to_pandas().squeeze() if isinstance(by, type(self)) else by
 


Test patch:
Repository: modin-project/modin

Issue:
Update pandas version to 1.1.5



Discussion:

Resolving patch:
diff --git a/modin/pandas/__init__.py b/modin/pandas/__init__.py
--- a/modin/pandas/__init__.py
+++ b/modin/pandas/__init__.py
@@ -13,7 +13,7 @@
 
 import pandas
 
-__pandas_version__ = "1.1.4"
+__pandas_version__ = "1.1.5"
 
 if pandas.__version__ != __pandas_version__:
     import warnings
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -55,7 +55,7 @@ def is_pure(self):
     url="https://github.com/modin-project/modin",
     long_description=long_description,
     long_description_content_type="text/markdown",
-    install_requires=["pandas==1.1.4", "packaging"],
+    install_requires=["pandas==1.1.5", "packaging"],
     extras_require={
         # can be installed by pip install modin[dask]
         "dask": dask_deps,


Test patch:
Repository: modin-project/modin

Issue:
Handling of space limited Ray Plasma directories
**Is your feature request related to a problem? Please describe.**
Ray Plasma default directory (`/tmp` for Linux) can be space limited and in the case of that directory overflow, Ray fails with very long and noninformative logs (examples are https://github.com/modin-project/modin/issues/1918 and https://github.com/modin-project/modin/issues/2209) . It will be helpful if we detect that cases and inform user or set plasma directory to directory without space limiting.


Discussion:

Resolving patch:
diff --git a/modin/engines/ray/utils.py b/modin/engines/ray/utils.py
--- a/modin/engines/ray/utils.py
+++ b/modin/engines/ray/utils.py
@@ -149,6 +149,20 @@ def initialize_ray(
                 _lru_evict=True,
             )
 
+            global_node = ray.worker._global_node
+            # Check only for head node
+            if global_node.head:
+                import psutil
+                from modin.error_message import ErrorMessage
+
+                ray_session_dir = os.path.dirname(global_node._session_dir)
+                ray_free_space_GB = psutil.disk_usage(ray_session_dir).free // 10 ** 9
+                ErrorMessage.single_warning(
+                    f"Modin Ray engine was started with {ray_free_space_GB} GB free space avaliable, "
+                    "if it is not enough for your application, please set environment variable "
+                    "MODIN_ON_RAY_PLASMA_DIR=/directory/without/space/limiting"
+                )
+
         _move_stdlib_ahead_of_site_packages()
         ray.worker.global_worker.run_function_on_all_workers(
             _move_stdlib_ahead_of_site_packages


Test patch:
Repository: modin-project/modin

Issue:
Add the ability to run microbenchmarks for older versions of Modin
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:
- **Modin version** (`modin.__version__`):
- **Python version**:
- **Code we can use to reproduce**:

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:

Resolving patch:
diff --git a/asv_bench/benchmarks/benchmarks.py b/asv_bench/benchmarks/benchmarks.py
--- a/asv_bench/benchmarks/benchmarks.py
+++ b/asv_bench/benchmarks/benchmarks.py
@@ -21,12 +21,20 @@
 import numpy as np
 import pandas
 
-from modin.config import TestDatasetSize, AsvImplementation
 from .utils import generate_dataframe, RAND_LOW, RAND_HIGH, random_string
 
-ASV_USE_IMPL = AsvImplementation.get()
+try:
+    from modin.config import TestDatasetSize, AsvImplementation
 
-if TestDatasetSize.get() == "Big":
+    ASV_USE_IMPL = AsvImplementation.get()
+    ASV_DATASET_SIZE = TestDatasetSize.get()
+except ImportError:
+    # The same benchmarking code can be run for different versions of Modin, so in
+    # case of an error importing important variables, we'll just use predefined values
+    ASV_USE_IMPL = "modin"
+    ASV_DATASET_SIZE = "Big" if pd.DEFAULT_NPARTITIONS >= 32 else "Small"
+
+if ASV_DATASET_SIZE == "Big":
     BINARY_OP_DATA_SIZE = [
         (5000, 5000, 5000, 5000),
         # the case extremely inefficient


Test patch:
Repository: modin-project/modin

Issue:
docs: fix simple typo, parition -> partition
There is a small typo in modin/engines/dask/pandas_on_dask/frame/partition.py, modin/engines/ray/pandas_on_ray/frame/partition.py.

Should read partition rather than parition.


Discussion:

Resolving patch:
diff --git a/modin/engines/dask/pandas_on_dask/frame/partition.py b/modin/engines/dask/pandas_on_dask/frame/partition.py
--- a/modin/engines/dask/pandas_on_dask/frame/partition.py
+++ b/modin/engines/dask/pandas_on_dask/frame/partition.py
@@ -125,7 +125,7 @@ def to_pandas(self):
 
     def to_numpy(self, **kwargs):
         """
-        Convert the object stored in this parition to a NumPy array.
+        Convert the object stored in this partition to a NumPy array.
 
         Returns
         -------
diff --git a/modin/engines/ray/pandas_on_ray/frame/partition.py b/modin/engines/ray/pandas_on_ray/frame/partition.py
--- a/modin/engines/ray/pandas_on_ray/frame/partition.py
+++ b/modin/engines/ray/pandas_on_ray/frame/partition.py
@@ -95,7 +95,7 @@ def to_pandas(self):
 
     def to_numpy(self, **kwargs):
         """
-        Convert the object stored in this parition to a NumPy array.
+        Convert the object stored in this partition to a NumPy array.
 
         Returns
         -------


Test patch:
Repository: modin-project/modin

Issue:
Some asv benchmarks fall off by timeout
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:
- **Modin version** (`modin.__version__`): 477c5f689d08c851feb669ab6a81b4983db0cc57
- **Python version**:
- **Code we can use to reproduce**: `cd asv_bench && asv run --show-stderr --launch-method=spawn -b TimeMultiColumnGroupby -b TimeMerge`

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->

`TimeMerge`, `TimeMultiColumnGroupby.time_groupby_agg_mean` run longer than `60s`(default asv timeout) or exceed memory limits.

For now, we can just reduce the size of the data. Further, optimization is needed.

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:

Resolving patch:
diff --git a/asv_bench/benchmarks/benchmarks.py b/asv_bench/benchmarks/benchmarks.py
--- a/asv_bench/benchmarks/benchmarks.py
+++ b/asv_bench/benchmarks/benchmarks.py
@@ -75,7 +75,16 @@ def setup(self, data_size, ncols=1):
 
 class TimeMultiColumnGroupby(BaseTimeGroupBy):
     param_names = ["data_size", "ncols"]
-    params = [UNARY_OP_DATA_SIZE, [6]]
+    params = [
+        [
+            (5000, 5000),
+            (10_000, 10),
+            # TODO: after optimization try to use UNARY_OP_DATA_SIZE here
+        ]
+        if ASV_DATASET_SIZE == "Big"
+        else UNARY_OP_DATA_SIZE,
+        [6],
+    ]
 
     def time_groupby_agg_quan(self, data_size, ncols):
         execute(self.df.groupby(by=self.groupby_columns).agg("quantile"))
@@ -150,7 +159,13 @@ def time_join(self, data_size, how, sort):
 class TimeMerge:
     param_names = ["data_size", "how", "sort"]
     params = [
-        BINARY_OP_DATA_SIZE,
+        [
+            (5000, 5000, 5000, 5000),
+            (125_000, 15, 100_000, 10),
+            # TODO: after optimization try to use BINARY_OP_DATA_SIZE here
+        ]
+        if ASV_DATASET_SIZE == "Big"
+        else BINARY_OP_DATA_SIZE,
         ["left", "inner"],
         [False],
     ]


Test patch:
Repository: modin-project/modin

Issue:
Up python version for test jobs
### Describe the problem
<!-- Describe the problem clearly here. -->

pandas 1.2.0 supports Python 3.7.1 and higher. In order to avoid dependencies conflict for our jobs we need to up python version.



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -63,5 +63,5 @@ def is_pure(self):
         "remote": remote_deps,
         "all": all_deps,
     },
-    python_requires=">=3.6.1",
+    python_requires=">=3.7.1",
 )


Test patch:
Repository: modin-project/modin

Issue:
Add documentation page for modin.experimental.xgboost
### Describe the problem
<!-- Describe the problem clearly here. -->

Need to add documentation page for `modin.experimental.xgboost`.



Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -10,7 +10,7 @@
 import modin
 
 project = u"Modin"
-copyright = u"2018-2020, Modin"
+copyright = u"2018-2021, Modin"
 author = u"Modin contributors"
 
 # The short X.Y version


Test patch:
Repository: modin-project/modin

Issue:
Change name of files for testing conda/pip environment
### Describe the problem
<!-- Describe the problem clearly here. -->

Change names of environment files for `conda/pip`: `requirements.txt`->`test_requires.txt`, `environment.yml`->`test_environment.yml` as this envs are needed for development/testing but not for necessary installation by users.




Discussion:

Resolving patch:
diff --git a/ci/teamcity/build-docker.py b/ci/teamcity/build-docker.py
--- a/ci/teamcity/build-docker.py
+++ b/ci/teamcity/build-docker.py
@@ -15,7 +15,7 @@ def execute_command(cmd):
         "(cd ../.. && git archive -o ci/teamcity/modin.tar $(cat ci/teamcity/git-rev))"
     )
     base_image = "ray-project/deploy"
-    requirements = "requirements.txt"
+    requirements = "requirements-dev.txt"
     execute_command(
         "docker build -f Dockerfile.modin-base --build-arg BASE_IMAGE={} -t modin-project/modin-base .".format(
             base_image


Test patch: