internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: modin-project/modin

Issue:
HDK: `modin.numpy.array.sum` failed when using Modin NumPy experimental API
```bash
  File "/_work/modin/modin/numpy/arr.py", line 2053, in sum
    result = target._query_compiler.sum(axis=apply_axis, skipna=False)
  File "/_work/modin/modin/experimental/core/storage_formats/hdk/query_compiler.py", line 99, in method_wrapper
    return method(self, *args, **kwargs)
  File "/_work/modin/modin/logging/logger_decorator.py", line 129, in run_and_log
    return obj(*args, **kwargs)
  File "/_work/modin/modin/experimental/core/storage_formats/hdk/query_compiler.py", line 402, in sum
    min_count = kwargs.pop("min_count")
KeyError: 'min_count'
```


Discussion:

Resolving patch:
diff --git a/modin/experimental/core/storage_formats/hdk/query_compiler.py b/modin/experimental/core/storage_formats/hdk/query_compiler.py
--- a/modin/experimental/core/storage_formats/hdk/query_compiler.py
+++ b/modin/experimental/core/storage_formats/hdk/query_compiler.py
@@ -399,7 +399,7 @@ def min(self, **kwargs):
         return self._agg("min", **kwargs)
 
     def sum(self, **kwargs):
-        min_count = kwargs.pop("min_count")
+        min_count = kwargs.pop("min_count", 0)
         if min_count != 0:
             raise NotImplementedError(
                 f"HDK's sum does not support such set of parameters: min_count={min_count}."


Test patch:
Repository: modin-project/modin

Issue:
Avoid label synchronization for `dot` operation



Discussion:

Resolving patch:
diff --git a/modin/core/storage_formats/pandas/query_compiler.py b/modin/core/storage_formats/pandas/query_compiler.py
--- a/modin/core/storage_formats/pandas/query_compiler.py
+++ b/modin/core/storage_formats/pandas/query_compiler.py
@@ -2231,14 +2231,6 @@ def dot(self, other, squeeze_self=None, squeeze_other=None):
                 else other.to_pandas()
             )
 
-        def map_func(df, other=other, squeeze_self=squeeze_self):  # pragma: no cover
-            """Compute matrix multiplication of the passed frames."""
-            result = df.squeeze(axis=1).dot(other) if squeeze_self else df.dot(other)
-            if is_list_like(result):
-                return pandas.DataFrame(result)
-            else:
-                return pandas.DataFrame([result])
-
         num_cols = other.shape[1] if len(other.shape) > 1 else 1
         if len(self.columns) == 1:
             new_index = (
@@ -2255,8 +2247,38 @@ def map_func(df, other=other, squeeze_self=squeeze_self):  # pragma: no cover
             new_columns = [MODIN_UNNAMED_SERIES_LABEL] if num_cols == 1 else None
             axis = 1
 
+        # If either new index or new columns are supposed to be a single-dimensional,
+        # then we use a special labeling for them. Besides setting the new labels as
+        # a metadata to the resulted frame, we also want to set them inside the kernel,
+        # so actual partitions would be labeled accordingly (there's a 'sync_label'
+        # parameter that can do the same, but doing it manually is faster)
+        align_index = isinstance(new_index, list) and new_index == [
+            MODIN_UNNAMED_SERIES_LABEL
+        ]
+        align_columns = new_columns == [MODIN_UNNAMED_SERIES_LABEL]
+
+        def map_func(df, other=other, squeeze_self=squeeze_self):  # pragma: no cover
+            """Compute matrix multiplication of the passed frames."""
+            result = df.squeeze(axis=1).dot(other) if squeeze_self else df.dot(other)
+
+            if is_list_like(result):
+                res = pandas.DataFrame(result)
+            else:
+                res = pandas.DataFrame([result])
+
+            # manual aligning with external index to avoid `sync_labels` overhead
+            if align_columns:
+                res.columns = [MODIN_UNNAMED_SERIES_LABEL]
+            if align_index:
+                res.index = [MODIN_UNNAMED_SERIES_LABEL]
+            return res
+
         new_modin_frame = self._modin_frame.apply_full_axis(
-            axis, map_func, new_index=new_index, new_columns=new_columns
+            axis,
+            map_func,
+            new_index=new_index,
+            new_columns=new_columns,
+            sync_labels=False,
         )
         return self.__constructor__(new_modin_frame)
 


Test patch:
Repository: modin-project/modin

Issue:
BUG: modin.experimental.sql.hdk.query is not part of modin package
### Modin version checks

- [X] I have checked that this issue has not already been reported.

- [X] I have confirmed this bug exists on the latest released version of Modin.

- [ ] I have confirmed this bug exists on the main branch of Modin. (In order to do this you can follow [this guide](https://modin.readthedocs.io/en/stable/getting_started/installation.html#installing-from-the-github-master-branch).)


### Reproducible Example

```python
python -c "from modin.experimental.sql.hdk.query import hdk_query"
```


### Issue Description

The file modin.experimental.sql.hdk.query is not part of modin package, which can be checked here https://pypi.org/project/modin/0.24.1/#files
That's a problem for those who want to use this functionality in HDK, for instance, H2O benchmark.

It's a problem with package content, not with modin master branch! So import works with modin package cloned from git.

### Expected Behavior

This import should work

### Error Logs

<details>

```
ModuleNotFoundError: No module named 'modin.experimental.sql.hdk'
```

</details>


### Installed Versions

<details>
(modin3) ekrivov@r01u05wfp:~/large/db-benchmark$ python -c "import modin.pandas as pd; pd.show_versions()"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/localdisk/ekrivov/miniconda/envs/modin3/lib/python3.10/site-packages/modin/utils.py", line 790, in show_versions
    modin_deps = _get_modin_deps_info()
  File "/localdisk/ekrivov/miniconda/envs/modin3/lib/python3.10/site-packages/modin/utils.py", line 746, in _get_modin_deps_info
    result[pkg_name] = pkg.__version__ + (
AttributeError: module 'dask' has no attribute '__version__'
</details>



Discussion:

Resolving patch:
diff --git a/modin/experimental/sql/hdk/__init__.py b/modin/experimental/sql/hdk/__init__.py
new file mode 100644
--- /dev/null
+++ b/modin/experimental/sql/hdk/__init__.py
@@ -0,0 +1,14 @@
+# Licensed to Modin Development Team under one or more contributor license agreements.
+# See the NOTICE file distributed with this work for additional information regarding
+# copyright ownership.  The Modin Development Team licenses this file to you under the
+# Apache License, Version 2.0 (the "License"); you may not use this file except in
+# compliance with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software distributed under
+# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
+# ANY KIND, either express or implied. See the License for the specific language
+# governing permissions and limitations under the License.
+
+"""Implementation of HDK SQL functionality."""


Test patch:
Repository: modin-project/modin

Issue:
PERF: value_counts(): Eliminate redundant sorting.
The value_counts() implementation calls groupby() without the sort argument, which defaults to True - https://github.com/modin-project/modin/blob/master/modin/pandas/base.py#L3633 . This sorting is redundant here.


Discussion:

Resolving patch:
diff --git a/modin/pandas/base.py b/modin/pandas/base.py
--- a/modin/pandas/base.py
+++ b/modin/pandas/base.py
@@ -3630,7 +3630,15 @@ def value_counts(
     ):
         if subset is None:
             subset = self._query_compiler.columns
-        counted_values = self.groupby(by=subset, dropna=dropna, observed=True).size()
+        with warnings.catch_warnings():
+            warnings.filterwarnings(
+                "ignore",
+                message=".*groupby keys will be sorted anyway.*",
+                category=UserWarning,
+            )
+            counted_values = self.groupby(
+                by=subset, dropna=dropna, observed=True, sort=False
+            ).size()
         if sort:
             counted_values.sort_values(ascending=ascending, inplace=True)
         if normalize:


Test patch:
Repository: modin-project/modin

Issue:
AttributeError: 'PandasQueryCompiler' object has no attribute 'between'
https://github.com/modin-project/modin/blob/ce54013f2116fcc57e7a6a09cfdd3532beb24d45/modin/pandas/series.py#L722


Discussion:

Resolving patch:
diff --git a/modin/pandas/series.py b/modin/pandas/series.py
--- a/modin/pandas/series.py
+++ b/modin/pandas/series.py
@@ -718,9 +718,9 @@ def between(self, left, right, inclusive: str = "both"):  # noqa: PR01, RT01, D2
         """
         Return boolean Series equivalent to left <= series <= right.
         """
-        return self.__constructor__(
-            query_compiler=self._query_compiler.between(left, right, inclusive)
-        )
+        # 'pandas.Series.between()' only uses public Series' API,
+        # so passing a Modin Series there is safe
+        return pandas.Series.between(self, left, right, inclusive)
 
     def combine(self, other, func, fill_value=None):  # noqa: PR01, RT01, D200
         """


Test patch:
diff --git a/modin/pandas/test/test_series.py b/modin/pandas/test/test_series.py
--- a/modin/pandas/test/test_series.py
+++ b/modin/pandas/test/test_series.py
@@ -1198,13 +1198,14 @@ def test_array(data):
     eval_general(modin_series, pandas_series, lambda df: df.array)
 
 
-@pytest.mark.xfail(reason="Using pandas Series.")
 @pytest.mark.parametrize("data", test_data_values, ids=test_data_keys)
 def test_between(data):
-    modin_series = create_test_series(data)
+    modin_series, pandas_series = create_test_series(data)
 
-    with pytest.raises(NotImplementedError):
-        modin_series.between(None, None)
+    df_equals(
+        modin_series.between(1, 4),
+        pandas_series.between(1, 4),
+    )
 
 
 def test_between_time():
Repository: modin-project/modin

Issue:
PERF: Do not convert columns dtypes if the new dtypes are the same
The PandasDataframe.astype() method always performs the dtypes conversion, even if the new dtypes are the same. It results in unnecessary computation and remote calls, in case of remote backends.


Discussion:

Resolving patch:
diff --git a/modin/core/dataframe/pandas/dataframe/dataframe.py b/modin/core/dataframe/pandas/dataframe/dataframe.py
--- a/modin/core/dataframe/pandas/dataframe/dataframe.py
+++ b/modin/core/dataframe/pandas/dataframe/dataframe.py
@@ -25,7 +25,7 @@
 import pandas
 from pandas._libs.lib import no_default
 from pandas.api.types import is_object_dtype
-from pandas.core.dtypes.common import is_list_like, is_numeric_dtype
+from pandas.core.dtypes.common import is_dtype_equal, is_list_like, is_numeric_dtype
 from pandas.core.indexes.api import Index, RangeIndex
 
 from modin.config import Engine, IsRayCluster, NPartitions
@@ -1486,21 +1486,18 @@ def astype(self, col_dtypes, errors: str = "raise"):
         BaseDataFrame
             Dataframe with updated dtypes.
         """
-        columns = col_dtypes.keys()
-        # Create Series for the updated dtypes
-        new_dtypes = self.dtypes.copy()
+        new_dtypes = None
+        self_dtypes = self.dtypes
         # When casting to "category" we have to make up the whole axis partition
         # to get the properly encoded table of categories. Every block partition
         # will store the encoded table. That can lead to higher memory footprint.
         # TODO: Revisit if this hurts users.
         use_full_axis_cast = False
         has_categorical_cast = False
-        for i, column in enumerate(columns):
-            dtype = col_dtypes[column]
-            if (
-                not isinstance(dtype, type(self.dtypes[column]))
-                or dtype != self.dtypes[column]
-            ):
+        for column, dtype in col_dtypes.items():
+            if not is_dtype_equal(dtype, self_dtypes[column]):
+                if new_dtypes is None:
+                    new_dtypes = self_dtypes.copy()
                 # Update the new dtype series to the proper pandas dtype
                 try:
                     new_dtype = np.dtype(dtype)
@@ -1525,6 +1522,9 @@ def astype(self, col_dtypes, errors: str = "raise"):
                 else:
                     new_dtypes[column] = new_dtype
 
+        if new_dtypes is None:
+            return self.copy()
+
         def astype_builder(df):
             """Compute new partition frame with dtypes updated."""
             # TODO(https://github.com/modin-project/modin/issues/6266): Remove this


Test patch:
Repository: modin-project/modin

Issue:
Use `copy=False` for internal usage of `set_axis`
To reduce memory allocation.


Discussion:

Resolving patch:
diff --git a/modin/core/dataframe/pandas/dataframe/dataframe.py b/modin/core/dataframe/pandas/dataframe/dataframe.py
--- a/modin/core/dataframe/pandas/dataframe/dataframe.py
+++ b/modin/core/dataframe/pandas/dataframe/dataframe.py
@@ -808,7 +808,12 @@ def _propagate_index_objs(self, axis=None):
         if axis is None:
 
             def apply_idx_objs(df, idx, cols):
-                return df.set_axis(idx, axis="index").set_axis(cols, axis="columns")
+                # We should make at least one copy to avoid the data modification problem
+                # that may arise when sharing buffers from distributed storage
+                # (zero-copy pickling).
+                return df.set_axis(idx, axis="index").set_axis(
+                    cols, axis="columns", copy=False
+                )
 
             self._partitions = np.array(
                 [
diff --git a/modin/pandas/groupby.py b/modin/pandas/groupby.py
--- a/modin/pandas/groupby.py
+++ b/modin/pandas/groupby.py
@@ -1990,7 +1990,9 @@ def aggregate(self, func=None, *args, engine=None, engine_kwargs=None, **kwargs)
             # because there is no need to identify which original column's aggregation
             # the new column represents. alternatively we could give the query compiler
             # a hint that it's for a series, not a dataframe.
-            return result.set_axis(labels=self._try_get_str_func(func), axis=1)
+            return result.set_axis(
+                labels=self._try_get_str_func(func), axis=1, copy=False
+            )
         else:
             return super().aggregate(
                 func, *args, engine=engine, engine_kwargs=engine_kwargs, **kwargs


Test patch:
Repository: modin-project/modin

Issue:
Avoid one extra `copy()` call for `Series.reset_index`



Discussion:

Resolving patch:
diff --git a/modin/pandas/series.py b/modin/pandas/series.py
--- a/modin/pandas/series.py
+++ b/modin/pandas/series.py
@@ -1600,7 +1600,9 @@ def reset_index(
             obj.name = name
             from .dataframe import DataFrame
 
-            return DataFrame(obj).reset_index(
+            # Here `query_compiler` is passed instead of `obj` to avoid unnecessary `copy()`
+            # inside `DataFrame` constructor
+            return DataFrame(query_compiler=obj._query_compiler).reset_index(
                 level=level,
                 drop=drop,
                 inplace=inplace,


Test patch:
Repository: modin-project/modin

Issue:
Do not iterate over the result of the `Series.tolist` function.



Discussion:

Resolving patch:
diff --git a/modin/core/storage_formats/base/query_compiler.py b/modin/core/storage_formats/base/query_compiler.py
--- a/modin/core/storage_formats/base/query_compiler.py
+++ b/modin/core/storage_formats/base/query_compiler.py
@@ -169,6 +169,9 @@ def default_to_pandas(self, pandas_op, *args, **kwargs):
             warnings.filterwarnings("ignore", category=FutureWarning)
             result = pandas_op(try_cast_to_pandas(self), *args, **kwargs)
         if isinstance(result, (tuple, list)):
+            if "Series.tolist" in pandas_op.__name__:
+                # fast path: no need to iterate over the result from `tolist` function
+                return result
             return [self.__wrap_in_qc(obj) for obj in result]
         return self.__wrap_in_qc(result)
 


Test patch:
Repository: modin-project/modin

Issue:
Update Modin on cluster documentation



Discussion:

Resolving patch:
diff --git a/modin/core/execution/ray/common/utils.py b/modin/core/execution/ray/common/utils.py
--- a/modin/core/execution/ray/common/utils.py
+++ b/modin/core/execution/ray/common/utils.py
@@ -50,6 +50,8 @@
 _RAY_IGNORE_UNHANDLED_ERRORS_VAR = "RAY_IGNORE_UNHANDLED_ERRORS"
 
 ObjectIDType = ray.ObjectRef
+# TODO: Minimum version of Ray - 1.13
+# `if` branch can be deleted
 if version.parse(ray.__version__) >= version.parse("1.2.0"):
     from ray.util.client.common import ClientObjectRef
 


Test patch:
Repository: modin-project/modin

Issue:
Specify `navigation_with_keys=True` to fix docs build
```sphinx.errors.SphinxWarning: The default value for `navigation_with_keys` will change to `False` in the next release. If you wish to preserve the old behavior for your site, set `navigation_with_keys=True` in the `html_theme_options` dict in your `conf.py` file.Be aware that `navigation_with_keys = True` has negative accessibility implications:https://github.com/pydata/pydata-sphinx-theme/issues/1492```


Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -178,6 +178,7 @@ def noop_decorator(*args, **kwargs):
             "icon": "fas fa-envelope-square",
         },
     ],
+    "navigation_with_keys": True,
 }
 
 # Custom sidebar templates, must be a dictionary that maps document names


Test patch:
Repository: modin-project/modin

Issue:
Use `sync_labels=False` for `rank` function



Discussion:

Resolving patch:
diff --git a/modin/core/storage_formats/pandas/query_compiler.py b/modin/core/storage_formats/pandas/query_compiler.py
--- a/modin/core/storage_formats/pandas/query_compiler.py
+++ b/modin/core/storage_formats/pandas/query_compiler.py
@@ -2535,6 +2535,7 @@ def rank(self, **kwargs):
             if not numeric_only
             else None,
             dtypes=np.float64,
+            sync_labels=False,
         )
         return self.__constructor__(new_modin_frame)
 


Test patch:
Repository: modin-project/modin

Issue:
Revert creating an additional copy in `astype` function



Discussion:

Resolving patch:
diff --git a/modin/core/dataframe/pandas/dataframe/dataframe.py b/modin/core/dataframe/pandas/dataframe/dataframe.py
--- a/modin/core/dataframe/pandas/dataframe/dataframe.py
+++ b/modin/core/dataframe/pandas/dataframe/dataframe.py
@@ -28,7 +28,7 @@
 from pandas.core.dtypes.common import is_dtype_equal, is_list_like, is_numeric_dtype
 from pandas.core.indexes.api import Index, RangeIndex
 
-from modin.config import Engine, IsRayCluster, NPartitions
+from modin.config import IsRayCluster, NPartitions
 from modin.core.dataframe.base.dataframe.dataframe import ModinDataframe
 from modin.core.dataframe.base.dataframe.utils import Axis, JoinType
 from modin.core.dataframe.pandas.dataframe.utils import (
@@ -1498,7 +1498,6 @@ def astype(self, col_dtypes, errors: str = "raise"):
         # will store the encoded table. That can lead to higher memory footprint.
         # TODO: Revisit if this hurts users.
         use_full_axis_cast = False
-        has_categorical_cast = False
         for column, dtype in col_dtypes.items():
             if not is_dtype_equal(dtype, self_dtypes[column]):
                 if new_dtypes is None:
@@ -1523,7 +1522,7 @@ def astype(self, col_dtypes, errors: str = "raise"):
                             columns=[column]
                         )[column],
                     )
-                    use_full_axis_cast = has_categorical_cast = True
+                    use_full_axis_cast = True
                 else:
                     new_dtypes[column] = new_dtype
 
@@ -1532,14 +1531,7 @@ def astype(self, col_dtypes, errors: str = "raise"):
 
         def astype_builder(df):
             """Compute new partition frame with dtypes updated."""
-            # TODO(https://github.com/modin-project/modin/issues/6266): Remove this
-            # copy, which is a workaround for https://github.com/pandas-dev/pandas/issues/53658
-            df_for_astype = (
-                df.copy(deep=True)
-                if Engine.get() == "Ray" and has_categorical_cast
-                else df
-            )
-            return df_for_astype.astype(
+            return df.astype(
                 {k: v for k, v in col_dtypes.items() if k in df}, errors=errors
             )
 


Test patch:
Repository: modin-project/modin

Issue:
PERF: Use lazy_map_partitions() for dtypes conversion

The code below is ~6 times faster if `lazy_map_partitions()` is used by `astype()`.

```python
import timeit
import modin.pandas as pd
import ray

ray.init()

df = pd.DataFrame({i: range(1000) for i in range(100)})
def test():
    for c in df.columns:
        df[c] = df[c].astype(float)
    df.shape
time = timeit.timeit(test, number=1)
print(f"Time: {time}")
```


Discussion:

Resolving patch:
diff --git a/modin/core/dataframe/pandas/dataframe/dataframe.py b/modin/core/dataframe/pandas/dataframe/dataframe.py
--- a/modin/core/dataframe/pandas/dataframe/dataframe.py
+++ b/modin/core/dataframe/pandas/dataframe/dataframe.py
@@ -1548,7 +1548,7 @@ def astype_builder(df):
                 0, self._partitions, astype_builder, keep_partitioning=True
             )
         else:
-            new_frame = self._partition_mgr_cls.map_partitions(
+            new_frame = self._partition_mgr_cls.lazy_map_partitions(
                 self._partitions, astype_builder
             )
         return self.__constructor__(


Test patch:
Repository: modin-project/modin

Issue:
PERF: Use cached dtypes in fillna when possible.



Discussion:

Resolving patch:
diff --git a/modin/core/storage_formats/pandas/query_compiler.py b/modin/core/storage_formats/pandas/query_compiler.py
--- a/modin/core/storage_formats/pandas/query_compiler.py
+++ b/modin/core/storage_formats/pandas/query_compiler.py
@@ -2420,6 +2420,7 @@ def fillna(self, **kwargs):
         method = kwargs.get("method", None)
         limit = kwargs.get("limit", None)
         full_axis = method is not None or limit is not None
+        new_dtypes = None
         if isinstance(value, BaseQueryCompiler):
             if squeeze_self:
                 # Self is a Series type object
@@ -2487,7 +2488,25 @@ def fillna(df):
                     }
                     return df.fillna(value=func_dict, **kwargs)
 
+                if self._modin_frame.has_materialized_dtypes:
+                    dtypes = self._modin_frame.dtypes
+                    value_dtypes = pandas.DataFrame(
+                        {k: [v] for (k, v) in value.items()}
+                    ).dtypes
+                    if all(
+                        find_common_type([dtypes[col], dtype]) == dtypes[col]
+                        for (col, dtype) in value_dtypes.items()
+                        if col in dtypes
+                    ):
+                        new_dtypes = dtypes
+
         else:
+            if self._modin_frame.has_materialized_dtypes:
+                dtype = pandas.Series(value).dtype
+                if all(
+                    find_common_type([t, dtype]) == t for t in self._modin_frame.dtypes
+                ):
+                    new_dtypes = self._modin_frame.dtypes
 
             def fillna(df):
                 return df.fillna(value=value, **kwargs)
@@ -2495,7 +2514,7 @@ def fillna(df):
         if full_axis:
             new_modin_frame = self._modin_frame.fold(axis, fillna)
         else:
-            new_modin_frame = self._modin_frame.map(fillna)
+            new_modin_frame = self._modin_frame.map(fillna, dtypes=new_dtypes)
         return self.__constructor__(new_modin_frame)
 
     def quantile_for_list_of_values(self, **kwargs):


Test patch: