internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: modin-project/modin

Issue:
Defer index materialization for MapReduce implemented groupby
This line is currently executed in the main process and triggers the materialization of groupby result:
https://github.com/modin-project/modin/blob/67efa96e14e7cd2339c68f90e81703f1fa755b3e/modin/core/dataframe/algebra/groupby.py#L402-L404
We can move this logic to a kernel function being executed by remote workers.


Discussion:

Resolving patch:
diff --git a/modin/core/dataframe/algebra/groupby.py b/modin/core/dataframe/algebra/groupby.py
--- a/modin/core/dataframe/algebra/groupby.py
+++ b/modin/core/dataframe/algebra/groupby.py
@@ -272,7 +272,10 @@ def reduce(
                 inplace=True,
             )
         # Result could not always be a frame, so wrapping it into DataFrame
-        return pandas.DataFrame(result)
+        result = pandas.DataFrame(result)
+        if result.index.name == MODIN_UNNAMED_SERIES_LABEL:
+            result.index.name = None
+        return result
 
     @classmethod
     def caller(
@@ -400,8 +403,6 @@ def caller(
         )
 
         result = query_compiler.__constructor__(new_modin_frame)
-        if result.index.name == MODIN_UNNAMED_SERIES_LABEL:
-            result.index.name = None
         return result
 
     @classmethod
diff --git a/modin/core/storage_formats/pandas/query_compiler.py b/modin/core/storage_formats/pandas/query_compiler.py
--- a/modin/core/storage_formats/pandas/query_compiler.py
+++ b/modin/core/storage_formats/pandas/query_compiler.py
@@ -3215,6 +3215,9 @@ def compute_groupby(df, drop=False, partition_idx=0):
 
         # that means that exception in `compute_groupby` was raised
         # in every partition, so we also should raise it
+        # TODO: we should be able to drop this logic with pandas 2.0 as it removes `numeric_only=None`
+        # parameter for groupby thus making the behavior of processing of non-numeric columns more
+        # predictable (we can decide whether to raise an exception before actually executing groupby)
         if len(result.columns) == 0 and len(self.columns) != 0:
             # determening type of raised exception by applying `aggfunc`
             # to empty DataFrame


Test patch:
Repository: modin-project/modin

Issue:
REFACTOR: Remove inplace kwarg from query compiler clip() arguments
As the docs say, 

> This parameter serves the compatibility purpose.

https://github.com/modin-project/modin/blob/67efa96e14e7cd2339c68f90e81703f1fa755b3e/modin/core/storage_formats/base/query_compiler.py#LL797C13-L797C61

This param shouldn't go to the query compiler at all. API layer handles the `inplace` parameter: https://github.com/modin-project/modin/blob/67efa96e14e7cd2339c68f90e81703f1fa755b3e/modin/pandas/base.py#L1089


Discussion:
> This parameter serves the compatibility purpose.

I no longer see this comment in the pandas docs. Looks like we should support `inplace=True`. @mvashishtha?
https://pandas.pydata.org/pandas-docs/version/1.5.3/reference/api/pandas.DataFrame.clip.html?highlight=clip#pandas.DataFrame.clip
> This parameter serves the compatibility purpose.

@anmyachev I think I meant that this was in the documentation of the query compiler method. What I meant was that we should handle this at the API layer and the query compiler methods should not accept an `inplace` parameter.

Resolving patch:
diff --git a/modin/core/storage_formats/base/query_compiler.py b/modin/core/storage_formats/base/query_compiler.py
--- a/modin/core/storage_formats/base/query_compiler.py
+++ b/modin/core/storage_formats/base/query_compiler.py
@@ -793,8 +793,6 @@ def clip(self, lower, upper, **kwargs):  # noqa: PR02
         lower : float or list-like
         upper : float or list-like
         axis : {0, 1}
-        inplace : {False}
-            This parameter serves the compatibility purpose. Always has to be False.
         **kwargs : dict
             Serves the compatibility purpose. Does not affect the result.
 
diff --git a/modin/pandas/base.py b/modin/pandas/base.py
--- a/modin/pandas/base.py
+++ b/modin/pandas/base.py
@@ -1097,7 +1097,7 @@ def clip(
         # FIXME: Judging by pandas docs `*args` and `**kwargs` serves only compatibility
         # purpose and does not affect the result, we shouldn't pass them to the query compiler.
         new_query_compiler = self._query_compiler.clip(
-            lower=lower, upper=upper, axis=axis, inplace=inplace, *args, **kwargs
+            lower=lower, upper=upper, axis=axis, *args, **kwargs
         )
         return self._create_or_update_from_compiler(new_query_compiler, inplace)
 


Test patch:
Repository: modin-project/modin

Issue:
Incorrect use of dtypes cache for `concat` operation



Discussion:

Resolving patch:
diff --git a/modin/core/dataframe/pandas/dataframe/dataframe.py b/modin/core/dataframe/pandas/dataframe/dataframe.py
--- a/modin/core/dataframe/pandas/dataframe/dataframe.py
+++ b/modin/core/dataframe/pandas/dataframe/dataframe.py
@@ -3269,8 +3269,9 @@ def _compute_new_widths():
         if axis == Axis.ROW_WISE:
             new_index = self.index.append([other.index for other in others])
             new_columns = joined_index
-            all_dtypes = [frame.copy_dtypes_cache() for frame in [self] + others]
-            if all(dtypes is not None for dtypes in all_dtypes):
+            frames = [self] + others
+            if all(frame.has_materialized_dtypes for frame in frames):
+                all_dtypes = [frame.dtypes for frame in frames]
                 new_dtypes = pandas.concat(all_dtypes, axis=1)
                 # 'nan' value will be placed in a row if a column doesn't exist in all frames;
                 # this value is np.float64 type so we need an explicit conversion


Test patch:
Repository: modin-project/modin

Issue:
Looks like `apply_license_header.py` is unused



Discussion:

Resolving patch:
diff --git a/modin/apply_license_header.py b/modin/apply_license_header.py
deleted file mode 100644
--- a/modin/apply_license_header.py
+++ /dev/null
@@ -1,43 +0,0 @@
-# Licensed to Modin Development Team under one or more contributor license agreements.
-# See the NOTICE file distributed with this work for additional information regarding
-# copyright ownership.  The Modin Development Team licenses this file to you under the
-# Apache License, Version 2.0 (the "License"); you may not use this file except in
-# compliance with the License.  You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software distributed under
-# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
-# ANY KIND, either express or implied. See the License for the specific language
-# governing permissions and limitations under the License.
-
-import os
-from os.path import dirname, abspath
-
-
-# This is the python file root directory (modin/modin)
-rootdir = dirname(abspath(__file__))
-exclude_files = ["_version.py"]
-
-with open("{}{}".format(dirname(rootdir), "/LICENSE_HEADER"), "r") as f:
-    # Lines to check each line individually
-    header_lines = f.readlines()
-
-for subdir, dirs, files in os.walk(rootdir):
-    for file in files:
-        filepath = os.path.join(subdir, file)
-        if file.endswith(".py") and file not in exclude_files:
-            with open(filepath, "r+") as f:
-                # Lines for line by line comparison
-                py_file_lines = f.readlines()
-                if len(py_file_lines):
-                    first_non_comment_idx = next(
-                        i
-                        for i in range(len(py_file_lines))
-                        if not py_file_lines[i].startswith("# ")
-                    )
-                    py_file_lines = py_file_lines[first_non_comment_idx:]
-                new_contents = "".join(header_lines + py_file_lines)
-                f.seek(0)
-                f.write(new_contents)
-                f.truncate()


Test patch:
Repository: modin-project/modin

Issue:
add more `# pragma: no cover` for io functions



Discussion:

Resolving patch:
diff --git a/modin/core/execution/dask/implementations/pandas_on_dask/io/io.py b/modin/core/execution/dask/implementations/pandas_on_dask/io/io.py
--- a/modin/core/execution/dask/implementations/pandas_on_dask/io/io.py
+++ b/modin/core/execution/dask/implementations/pandas_on_dask/io/io.py
@@ -117,7 +117,7 @@ def to_parquet(cls, qc, **kwargs):
         fs, url = fsspec.core.url_to_fs(output_path, client_kwargs=client_kwargs)
         fs.mkdirs(url, exist_ok=True)
 
-        def func(df, **kw):
+        def func(df, **kw):  # pragma: no cover
             """
             Dump a chunk of rows as parquet, then save them to target maintaining order.
 
diff --git a/modin/core/execution/ray/implementations/pandas_on_ray/io/io.py b/modin/core/execution/ray/implementations/pandas_on_ray/io/io.py
--- a/modin/core/execution/ray/implementations/pandas_on_ray/io/io.py
+++ b/modin/core/execution/ray/implementations/pandas_on_ray/io/io.py
@@ -169,7 +169,7 @@ def to_csv(cls, qc, **kwargs):
 
         signals = SignalActor.remote(len(qc._modin_frame._partitions) + 1)
 
-        def func(df, **kw):
+        def func(df, **kw):  # pragma: no cover
             """
             Dump a chunk of rows as csv, then save them to target maintaining order.
 
@@ -286,7 +286,7 @@ def to_parquet(cls, qc, **kwargs):
         fs, url = fsspec.core.url_to_fs(output_path, client_kwargs=client_kwargs)
         fs.mkdirs(url, exist_ok=True)
 
-        def func(df, **kw):
+        def func(df, **kw):  # pragma: no cover
             """
             Dump a chunk of rows as parquet, then save them to target maintaining order.
 
diff --git a/modin/core/execution/unidist/implementations/pandas_on_unidist/io/io.py b/modin/core/execution/unidist/implementations/pandas_on_unidist/io/io.py
--- a/modin/core/execution/unidist/implementations/pandas_on_unidist/io/io.py
+++ b/modin/core/execution/unidist/implementations/pandas_on_unidist/io/io.py
@@ -168,7 +168,7 @@ def to_csv(cls, qc, **kwargs):
 
         signals = SignalActor.remote(len(qc._modin_frame._partitions) + 1)
 
-        def func(df, **kw):
+        def func(df, **kw):  # pragma: no cover
             """
             Dump a chunk of rows as csv, then save them to target maintaining order.
 
@@ -285,7 +285,7 @@ def to_parquet(cls, qc, **kwargs):
         fs, url = fsspec.core.url_to_fs(output_path, client_kwargs=client_kwargs)
         fs.mkdirs(url, exist_ok=True)
 
-        def func(df, **kw):
+        def func(df, **kw):  # pragma: no cover
             """
             Dump a chunk of rows as parquet, then save them to target maintaining order.
 


Test patch:
Repository: modin-project/modin

Issue:
Implement `Series.dt` through `CachedAccessor` as it done for `Series.sparse`



Discussion:

Resolving patch:
diff --git a/modin/pandas/series.py b/modin/pandas/series.py
--- a/modin/pandas/series.py
+++ b/modin/pandas/series.py
@@ -40,7 +40,7 @@
 from .iterator import PartitionIterator
 from .utils import from_pandas, is_scalar, _doc_binary_op, cast_function_modin2pandas
 from .accessor import CachedAccessor, SparseAccessor
-from .series_utils import CategoryMethods, StringMethods
+from .series_utils import CategoryMethods, StringMethods, DatetimeProperties
 
 
 if TYPE_CHECKING:
@@ -1832,6 +1832,7 @@ def sort_values(
     cat = CachedAccessor("cat", CategoryMethods)
     sparse = CachedAccessor("sparse", SparseAccessor)
     str = CachedAccessor("str", StringMethods)
+    dt = CachedAccessor("dt", DatetimeProperties)
 
     def squeeze(self, axis=None):  # noqa: PR01, RT01, D200
         """
@@ -2162,15 +2163,6 @@ def axes(self):  # noqa: RT01, D200
         """
         return [self.index]
 
-    @property
-    def dt(self):  # noqa: RT01, D200
-        """
-        Accessor object for datetimelike properties of the Series values.
-        """
-        from .series_utils import DatetimeProperties
-
-        return DatetimeProperties(self)
-
     @property
     def dtype(self):  # noqa: RT01, D200
         """
diff --git a/modin/pandas/series_utils.py b/modin/pandas/series_utils.py
--- a/modin/pandas/series_utils.py
+++ b/modin/pandas/series_utils.py
@@ -507,9 +507,9 @@ def _default_to_pandas(self, op, *args, **kwargs):
 
 @_inherit_docstrings(pandas.core.indexes.accessors.CombinedDatetimelikeProperties)
 class DatetimeProperties(ClassLogger):
-    def __init__(self, series):
-        self._series = series
-        self._query_compiler = series._query_compiler
+    def __init__(self, data):
+        self._series = data
+        self._query_compiler = data._query_compiler
 
     @pandas.util.cache_readonly
     def _Series(self):  # noqa: GL08


Test patch:
diff --git a/modin/pandas/test/test_api.py b/modin/pandas/test/test_api.py
--- a/modin/pandas/test/test_api.py
+++ b/modin/pandas/test/test_api.py
@@ -253,7 +253,7 @@ def test_series_str_api_equality():
 
 
 def test_series_dt_api_equality():
-    modin_dir = [obj for obj in dir(pd.Series().dt) if obj[0] != "_"]
+    modin_dir = [obj for obj in dir(pd.Series.dt) if obj[0] != "_"]
     pandas_dir = [obj for obj in dir(pandas.Series.dt) if obj[0] != "_"]
 
     missing_from_modin = set(pandas_dir) - set(modin_dir)
Repository: modin-project/modin

Issue:
Implement `Series.cat` through `CachedAccessor` as it done for `Series.sparse`



Discussion:

Resolving patch:
diff --git a/modin/pandas/series.py b/modin/pandas/series.py
--- a/modin/pandas/series.py
+++ b/modin/pandas/series.py
@@ -39,7 +39,7 @@
 from .iterator import PartitionIterator
 from .utils import from_pandas, is_scalar, _doc_binary_op, cast_function_modin2pandas
 from .accessor import CachedAccessor, SparseAccessor
-from .series_utils import StringMethods
+from .series_utils import CategoryMethods, StringMethods
 
 
 if TYPE_CHECKING:
@@ -1836,6 +1836,7 @@ def sort_values(
             result._query_compiler, inplace=inplace
         )
 
+    cat = CachedAccessor("cat", CategoryMethods)
     sparse = CachedAccessor("sparse", SparseAccessor)
     str = CachedAccessor("str", StringMethods)
 
@@ -2168,15 +2169,6 @@ def axes(self):  # noqa: RT01, D200
         """
         return [self.index]
 
-    @property
-    def cat(self):  # noqa: RT01, D200
-        """
-        Accessor object for categorical properties of the Series values.
-        """
-        from .series_utils import CategoryMethods
-
-        return CategoryMethods(self)
-
     @property
     def dt(self):  # noqa: RT01, D200
         """
diff --git a/modin/pandas/series_utils.py b/modin/pandas/series_utils.py
--- a/modin/pandas/series_utils.py
+++ b/modin/pandas/series_utils.py
@@ -37,9 +37,9 @@
 
 @_inherit_docstrings(pandas.core.arrays.categorical.CategoricalAccessor)
 class CategoryMethods(ClassLogger):
-    def __init__(self, series):
-        self._series = series
-        self._query_compiler = series._query_compiler
+    def __init__(self, data):
+        self._series = data
+        self._query_compiler = data._query_compiler
 
     @pandas.util.cache_readonly
     def _Series(self):  # noqa: GL08


Test patch:
diff --git a/modin/pandas/test/test_api.py b/modin/pandas/test/test_api.py
--- a/modin/pandas/test/test_api.py
+++ b/modin/pandas/test/test_api.py
@@ -266,7 +266,7 @@ def test_series_dt_api_equality():
 
 
 def test_series_cat_api_equality():
-    modin_dir = [obj for obj in dir(pd.Series().cat) if obj[0] != "_"]
+    modin_dir = [obj for obj in dir(pd.Series.cat) if obj[0] != "_"]
     pandas_dir = [obj for obj in dir(pandas.Series.cat) if obj[0] != "_"]
 
     missing_from_modin = set(pandas_dir) - set(modin_dir)
Repository: modin-project/modin

Issue:
`pd.read_feather(file)` actually reads the file twice!
There's a really strange logic in our Feather dispatcher introduced a long time ago that reads the whole Feather file just to extract its column names:
https://github.com/modin-project/modin/blob/632d724adb013d6f4d18aa2a4f2faa9325d324e8/modin/core/io/column_stores/feather_dispatcher.py#L55-L64

Note that `.build_query_compiler(..)` launch parsing kernels that would also be reading the same parquet file but in a distributed way.

We should be using more smarter way to extract feather's metadata: https://issues.apache.org/jira/browse/ARROW-10344


Discussion:

Resolving patch:
diff --git a/modin/core/io/column_stores/feather_dispatcher.py b/modin/core/io/column_stores/feather_dispatcher.py
--- a/modin/core/io/column_stores/feather_dispatcher.py
+++ b/modin/core/io/column_stores/feather_dispatcher.py
@@ -52,13 +52,15 @@ def _read(cls, path, columns=None, **kwargs):
             import_optional_dependency(
                 "pyarrow", "pyarrow is required to read feather files."
             )
-            from pyarrow.feather import read_feather
+            from pyarrow import ipc
 
             with OpenFile(
                 path,
                 **(kwargs.get("storage_options", None) or {}),
             ) as file:
-                df = read_feather(file)
-            # pyarrow.feather.read_feather doesn't support columns as pandas.Index
-            columns = list(df.columns)
+                # Opens the file to extract its metadata
+                reader = ipc.open_file(file)
+            # TODO: pyarrow's schema contains much more metadata than just column names, it also
+            # has dtypes and index information that we could use when building a dataframe
+            columns = reader.schema.names
         return cls.build_query_compiler(path, columns, use_threads=False)


Test patch:
Repository: modin-project/modin

Issue:
BUG: don't use join() when indicator is truthy
porting https://github.com/ponder-org/modin-public/commit/2dc824701e8fb7deaa9635bf50077c7adfac6c60


Discussion:

Resolving patch:
diff --git a/modin/pandas/dataframe.py b/modin/pandas/dataframe.py
--- a/modin/pandas/dataframe.py
+++ b/modin/pandas/dataframe.py
@@ -1379,7 +1379,9 @@ def merge(
                 f"Can only merge Series or DataFrame objects, a {type(right)} was passed"
             )
 
-        if left_index and right_index:
+        # If we are joining on the index and we are using
+        # default parameters we can map this to a join
+        if left_index and right_index and not indicator:
             return self.join(
                 right, how=how, lsuffix=suffixes[0], rsuffix=suffixes[1], sort=sort
             )


Test patch:
Repository: modin-project/modin

Issue:
REFACTOR: check for _to_datetime
this is upstreaming:
- https://github.com/ponder-org/modin-public/commit/f2523abf13f3c36c65dabb2d1f0a7cf79496e36d 
- https://github.com/ponder-org/modin-public/commit/096de38707d5d541c5f021bd10597d634bac81ab

This way, we can accommodate modin.pandas index objects that are not instances of `pandas.Index`. These objects just have to implement `_to_datetime`


Discussion:

Resolving patch:
diff --git a/modin/pandas/general.py b/modin/pandas/general.py
--- a/modin/pandas/general.py
+++ b/modin/pandas/general.py
@@ -505,7 +505,7 @@ def to_datetime(
     """
     Convert argument to datetime.
     """
-    if not isinstance(arg, (DataFrame, Series)):
+    if not hasattr(arg, "_to_datetime"):
         return pandas.to_datetime(
             arg,
             errors=errors,


Test patch:
Repository: modin-project/modin

Issue:
BUG: truncate should raise  ValueError when before is after after
upstreaming https://github.com/ponder-org/modin-public/commit/577a6e14ab0e7f1a52462d8cfa2f4147aace661c


Discussion:

Resolving patch:
diff --git a/modin/pandas/base.py b/modin/pandas/base.py
--- a/modin/pandas/base.py
+++ b/modin/pandas/base.py
@@ -3421,6 +3421,10 @@ def truncate(
             and not self.axes[axis].is_monotonic_decreasing
         ):
             raise ValueError("truncate requires a sorted index")
+
+        if before is not None and after is not None and before > after:
+            raise ValueError(f"Truncate: {after} must be after {before}")
+
         s = slice(*self.axes[axis].slice_locs(before, after))
         slice_obj = s if axis == 0 else (slice(None), s)
         return self.iloc[slice_obj]


Test patch:
diff --git a/modin/pandas/test/dataframe/test_default.py b/modin/pandas/test/dataframe/test_default.py
--- a/modin/pandas/test/dataframe/test_default.py
+++ b/modin/pandas/test/dataframe/test_default.py
@@ -1076,6 +1076,12 @@ def test_truncate(data):
         df_equals(modin_result, pandas_result)
 
 
+def test_truncate_before_greater_than_after():
+    df = pd.DataFrame([[1, 2, 3]])
+    with pytest.raises(ValueError, match="Truncate: 1 must be after 2"):
+        df.truncate(before=2, after=1)
+
+
 def test_tshift():
     idx = pd.date_range("1/1/2012", periods=5, freq="M")
     data = np.random.randint(0, 100, size=(len(idx), 4))
Repository: modin-project/modin

Issue:
BUG: series.reindex() fill value is None instead of np.nan
have to port https://github.com/ponder-org/modin-public/commit/5f9aceab8d48455df5d5a7e68c69e619b27a0793


Discussion:

Resolving patch:
diff --git a/modin/pandas/series.py b/modin/pandas/series.py
--- a/modin/pandas/series.py
+++ b/modin/pandas/series.py
@@ -1551,7 +1551,7 @@ def reindex(self, *args, **kwargs):
         copy = kwargs.pop("copy", True)
         limit = kwargs.pop("limit", None)
         tolerance = kwargs.pop("tolerance", None)
-        fill_value = kwargs.pop("fill_value", None)
+        fill_value = kwargs.pop("fill_value", np.nan)
         if kwargs:
             raise TypeError(
                 "reindex() got an unexpected keyword "


Test patch: