internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: modin-project/modin

Issue:
BUG: HDK: Binary operations on strings are not supported
### Modin version checks

- [X] I have checked that this issue has not already been reported.

- [X] I have confirmed this bug exists on the latest released version of Modin.

- [X] I have confirmed this bug exists on the main branch of Modin. (In order to do this you can follow [this guide](https://modin.readthedocs.io/en/stable/getting_started/installation.html#installing-from-the-github-master-branch).)


### Reproducible Example

```python
import modin.pandas as pd
df = pd.DataFrame({"a": ["a"]})
print(df + "_sfx")
print("pref_" + df)
print(df * 10)
```


### Issue Description

Binary operations on strings are not supported by HDK.

### Expected Behavior

```
       a
0  a_sfx
        a
0  pref_a
            a
0  aaaaaaaaaa
```

### Error Logs

<details>

```python-traceback

modin/experimental/core/execution/native/implementations/hdk_on_native/expr.py", line 59, in _get_common_dtype
    raise TypeError(f"Cannot perform operation on types: {lhs_dtype}, {rhs_dtype}")
TypeError: Cannot perform operation on types: object, <U0

```

</details>


### Installed Versions

<details>

Replace this line with the output of pd.show_versions()

</details>



Discussion:

Resolving patch:
diff --git a/modin/experimental/core/execution/native/implementations/hdk_on_native/expr.py b/modin/experimental/core/execution/native/implementations/hdk_on_native/expr.py
--- a/modin/experimental/core/execution/native/implementations/hdk_on_native/expr.py
+++ b/modin/experimental/core/execution/native/implementations/hdk_on_native/expr.py
@@ -56,7 +56,9 @@ def _get_common_dtype(lhs_dtype, rhs_dtype):
         return get_dtype(float)
     if is_integer_dtype(lhs_dtype) and is_integer_dtype(rhs_dtype):
         return get_dtype(int)
-    raise TypeError(f"Cannot perform operation on types: {lhs_dtype}, {rhs_dtype}")
+    raise NotImplementedError(
+        f"Cannot perform operation on types: {lhs_dtype}, {rhs_dtype}"
+    )
 
 
 _aggs_preserving_numeric_type = {"sum", "min", "max"}


Test patch:
diff --git a/modin/experimental/core/execution/native/implementations/hdk_on_native/test/test_dataframe.py b/modin/experimental/core/execution/native/implementations/hdk_on_native/test/test_dataframe.py
--- a/modin/experimental/core/execution/native/implementations/hdk_on_native/test/test_dataframe.py
+++ b/modin/experimental/core/execution/native/implementations/hdk_on_native/test/test_dataframe.py
@@ -1765,6 +1765,21 @@ def filter_or(df, **kwargs):
         run_and_compare(filter_and, data=self.cmp_data)
         run_and_compare(filter_or, data=self.cmp_data)
 
+    def test_string_bin_op(self):
+        def test_bin_op(df, op_name, op_arg, **kwargs):
+            return getattr(df, op_name)(op_arg)
+
+        bin_ops = {
+            "__add__": "_sfx",
+            "__radd__": "pref_",
+            "__mul__": 10,
+        }
+
+        for op, arg in bin_ops.items():
+            run_and_compare(
+                test_bin_op, data={"a": ["a"]}, op_name=op, op_arg=arg, force_lazy=False
+            )
+
 
 class TestDateTime:
     datetime_data = {
Repository: modin-project/modin

Issue:
Filter out empty bins in advance when doing range-based reshuffling
The pivot values for bins are known in the main process, meaning that we can filter out obviously empty bins (that results in empty partitions) in advance.
https://github.com/modin-project/modin/blob/8fbe04ff64a1c1658d9ed8317dd64a48b8e61ee0/modin/core/dataframe/pandas/partitioning/partition_manager.py#L1549


Discussion:

Resolving patch:
diff --git a/modin/core/dataframe/pandas/partitioning/partition_manager.py b/modin/core/dataframe/pandas/partitioning/partition_manager.py
--- a/modin/core/dataframe/pandas/partitioning/partition_manager.py
+++ b/modin/core/dataframe/pandas/partitioning/partition_manager.py
@@ -1580,7 +1580,7 @@ def shuffle_partitions(
         samples = [partition.apply(sample_func) for partition in masked_partitions]
         # Get each sample to pass in to the pivot function
         samples = cls.get_objects_from_partitions(samples)
-        pivots = shuffle_functions.pivot_function(samples)
+        pivots = np.unique(shuffle_functions.pivot_function(samples))
         # Convert our list of block partitions to row partitions. We need to create full-axis
         # row partitions since we need to send the whole partition to the split step as otherwise
         # we wouldn't know how to split the block partitions that don't contain the shuffling key.


Test patch:
Repository: modin-project/modin

Issue:
Do not build row-partitions for a final function during reshuffling
There's absolutely no need to wrap partitions from `split_row_partitions` to a `row_partition_class`.
https://github.com/modin-project/modin/blob/8fbe04ff64a1c1658d9ed8317dd64a48b8e61ee0/modin/core/dataframe/pandas/partitioning/partition_manager.py#L1567-L1578
There seems to be no effect for this extra wrapping besides of performance degradation caused by force materialization in `column_partition_cls.list_of_blocks`:
https://github.com/modin-project/modin/blob/8fbe04ff64a1c1658d9ed8317dd64a48b8e61ee0/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/virtual_partition.py#L120-L128



Discussion:

Resolving patch:
diff --git a/modin/core/dataframe/pandas/partitioning/partition_manager.py b/modin/core/dataframe/pandas/partitioning/partition_manager.py
--- a/modin/core/dataframe/pandas/partitioning/partition_manager.py
+++ b/modin/core/dataframe/pandas/partitioning/partition_manager.py
@@ -1598,10 +1598,6 @@ def shuffle_partitions(
             ]
         ).T
         # We need to convert every partition that came from the splits into a full-axis column partition.
-        col_partitioning_func = np.vectorize(
-            lambda partition: cls._row_partition_class(partition)
-        )
-        split_row_partitions = col_partitioning_func(split_row_partitions)
         new_partitions = [
             [
                 cls._column_partitions_class(row_partition, full_axis=False).apply(


Test patch:
Repository: modin-project/modin

Issue:
TEST: add ASV configs for Dask and Unidist
These configs allow us to use ASV in the following scenarios (for example):
```bash
asv continuous main-branch custom-branch --launch-method=spawn -b TimeMergeDefault \
   --no-only-changed -a repeat=5 --show-stderr --config asv.conf.unidist.json
```

```bash
asv continuous main-branch custom-branch --launch-method=spawn -b TimeMergeDefault \
   --no-only-changed -a repeat=5 --show-stderr --config asv.conf.dask.json
```


Discussion:

Resolving patch:
diff --git a/asv_bench/benchmarks/utils/compatibility.py b/asv_bench/benchmarks/utils/compatibility.py
--- a/asv_bench/benchmarks/utils/compatibility.py
+++ b/asv_bench/benchmarks/utils/compatibility.py
@@ -45,5 +45,5 @@
 
 assert ASV_USE_IMPL in ("modin", "pandas")
 assert ASV_DATASET_SIZE in ("big", "small")
-assert ASV_USE_ENGINE in ("ray", "dask", "python", "native")
+assert ASV_USE_ENGINE in ("ray", "dask", "python", "native", "unidist")
 assert ASV_USE_STORAGE_FORMAT in ("pandas", "hdk", "pyarrow")


Test patch:
Repository: modin-project/modin

Issue:
IndexError when concatenating dict of series along columns
Consider following code example:

```python
df = pd.concat(
    {
        s: pd.Series(index=range(3))
        for s in ["a", "b"]
    },
    axis="columns",
)
```

This works just fine on pandas and produces a dataframe with columns `["a", "b"]` and index `[0, 1, 2]` containing all `nan`. Yet the same code fails for modin on Ray with `IndexError: list index out of range` on line `modin/pandas/general.py:458`:

https://github.com/modin-project/modin/blob/8d3db2b4a7fa79716796b33f7cb3673ed729b652/modin/pandas/general.py#L458

I'm running Python 3.9.6, Ray 2.3.0 and Modin 0.19.0 on a Windows machine.


Discussion:

Resolving patch:
diff --git a/modin/pandas/general.py b/modin/pandas/general.py
--- a/modin/pandas/general.py
+++ b/modin/pandas/general.py
@@ -430,6 +430,8 @@ def concat(
         else DataFrame(obj)._query_compiler
         for obj in list_of_objs
     ]
+    if keys is None and isinstance(objs, dict):
+        keys = list(objs.keys())
     if keys is not None:
         if all_series:
             new_idx = keys
@@ -453,10 +455,6 @@ def concat(
                 old_name = _determine_name(list_of_objs, axis)
                 if old_name is not None:
                     new_idx.names = [None] + old_name
-    elif isinstance(objs, dict):
-        new_idx = pandas.concat(
-            {k: pandas.Series(index=obj.axes[axis]) for k, obj in objs.items()}
-        ).index
     else:
         new_idx = None
 


Test patch:
diff --git a/modin/pandas/test/test_concat.py b/modin/pandas/test/test_concat.py
--- a/modin/pandas/test/test_concat.py
+++ b/modin/pandas/test/test_concat.py
@@ -176,6 +176,15 @@ def test_concat_series_only():
     )
 
 
+def test_concat_5776():
+    modin_data = {key: pd.Series(index=range(3)) for key in ["a", "b"]}
+    pandas_data = {key: pandas.Series(index=range(3)) for key in ["a", "b"]}
+    df_equals(
+        pd.concat(modin_data, axis="columns"),
+        pandas.concat(pandas_data, axis="columns"),
+    )
+
+
 def test_concat_with_empty_frame():
     modin_empty_df = pd.DataFrame()
     pandas_empty_df = pandas.DataFrame()
Repository: modin-project/modin

Issue:
Enable test_binary test on the HDK engine
### Modin version checks

- [X] I have checked that this issue has not already been reported.

- [X] I have confirmed this bug exists on the latest released version of Modin.

- [X] I have confirmed this bug exists on the main branch of Modin. (In order to do this you can follow [this guide](https://modin.readthedocs.io/en/stable/getting_started/installation.html#installing-from-the-github-master-branch).)


### Reproducible Example

```python
Run test_binary on the HDK engine.
```


### Issue Description

There are 18 failures, when the test is running on the HDK engine.

### Expected Behavior

All these failures should be fixed and the test added to CI.

### Error Logs

<details>

```python-traceback

Replace this line with the error backtrace (if applicable).

```

</details>


### Installed Versions

<details>

Replace this line with the output of pd.show_versions()

</details>



Discussion:

Resolving patch:
diff --git a/modin/experimental/core/execution/native/implementations/hdk_on_native/dataframe/dataframe.py b/modin/experimental/core/execution/native/implementations/hdk_on_native/dataframe/dataframe.py
--- a/modin/experimental/core/execution/native/implementations/hdk_on_native/dataframe/dataframe.py
+++ b/modin/experimental/core/execution/native/implementations/hdk_on_native/dataframe/dataframe.py
@@ -1458,6 +1458,8 @@ def bin_op(self, other, op_name, **kwargs):
                 force_execution_mode=self._force_execution_mode,
             )
         elif isinstance(other, list):
+            if kwargs.get("axis", 1) == 0:
+                raise NotImplementedError(f"{op_name} on rows")
             if len(other) != len(self.columns):
                 raise ValueError(
                     f"length must be {len(self.columns)}: given {len(other)}"


Test patch:
diff --git a/modin/pandas/test/dataframe/test_binary.py b/modin/pandas/test/dataframe/test_binary.py
--- a/modin/pandas/test/dataframe/test_binary.py
+++ b/modin/pandas/test/dataframe/test_binary.py
@@ -30,7 +30,7 @@
     CustomIntegerForAddition,
     NonCommutativeMultiplyInteger,
 )
-from modin.config import Engine, NPartitions
+from modin.config import Engine, NPartitions, StorageFormat
 from modin.test.test_utils import warns_that_defaulting_to_pandas
 
 NPartitions.put(4)
@@ -163,9 +163,17 @@ def test_math_alias(math_op, alias):
 @pytest.mark.parametrize("op", ["eq", "ge", "gt", "le", "lt", "ne"])
 @pytest.mark.parametrize("data", test_data_values, ids=test_data_keys)
 def test_comparison(data, op, other):
+    def operation(df):
+        df = getattr(df, op)(df if other == "as_left" else other)
+        if other == "as_left" and StorageFormat.get() == "Hdk":
+            # In case of comparison with a DataFrame, HDK returns
+            # a DataFrame with sorted columns.
+            df = df.sort_index(axis=1)
+        return df
+
     eval_general(
         *create_test_dfs(data),
-        lambda df: getattr(df, op)(df if other == "as_left" else other),
+        operation=operation,
     )
Repository: modin-project/modin

Issue:
Remove `Engine.subscribe(_update_engine)` in DataFrame/Series constructors
After [this](https://github.com/modin-project/modin/pull/4228) pull request, the engine will be initialized when the following methods are called: `from_pandas` and `from_non_pandas` (They call `FactoryDispatcher.from_pandas/from_non_pandas` functions).


Discussion:

Resolving patch:
diff --git a/modin/pandas/dataframe.py b/modin/pandas/dataframe.py
--- a/modin/pandas/dataframe.py
+++ b/modin/pandas/dataframe.py
@@ -52,14 +52,13 @@
     MODIN_UNNAMED_SERIES_LABEL,
     try_cast_to_pandas,
 )
-from modin.config import Engine, IsExperimental, PersistentPickle
+from modin.config import IsExperimental, PersistentPickle
 from .utils import (
     from_pandas,
     from_non_pandas,
     broadcast_item,
     SET_DATAFRAME_ATTRIBUTE_WARNING,
 )
-from . import _update_engine
 from .iterator import PartitionIterator
 from .series import Series
 from .base import BasePandasDataset, _ATTRS_NO_LOOKUP
@@ -129,7 +128,6 @@ def __init__(
         # Siblings are other dataframes that share the same query compiler. We
         # use this list to update inplace when there is a shallow copy.
         self._siblings = []
-        Engine.subscribe(_update_engine)
         if isinstance(data, (DataFrame, Series)):
             self._query_compiler = data._query_compiler.copy()
             if index is not None and any(i not in data.index for i in index):
diff --git a/modin/pandas/series.py b/modin/pandas/series.py
--- a/modin/pandas/series.py
+++ b/modin/pandas/series.py
@@ -32,7 +32,6 @@
 from modin.utils import (
     _inherit_docstrings,
     to_pandas,
-    Engine,
     MODIN_UNNAMED_SERIES_LABEL,
 )
 from modin.config import IsExperimental, PersistentPickle
@@ -40,7 +39,6 @@
 from .iterator import PartitionIterator
 from .utils import from_pandas, is_scalar, _doc_binary_op
 from .accessor import CachedAccessor, SparseAccessor
-from . import _update_engine
 
 
 if TYPE_CHECKING:
@@ -98,7 +96,6 @@ def __init__(
         # Siblings are other dataframes that share the same query compiler. We
         # use this list to update inplace when there is a shallow copy.
         self._siblings = []
-        Engine.subscribe(_update_engine)
         if isinstance(data, type(self)):
             query_compiler = data._query_compiler.copy()
             if index is not None:


Test patch:
Repository: modin-project/modin

Issue:
build_docs is failing after 5853
This check has not been run in #5853.


Discussion:

Resolving patch:
diff --git a/modin/experimental/pandas/io.py b/modin/experimental/pandas/io.py
--- a/modin/experimental/pandas/io.py
+++ b/modin/experimental/pandas/io.py
@@ -114,17 +114,17 @@ def read_custom_text(
     nrows: Optional[int] = None,
     is_quoting=True,
 ):
-    """
+    r"""
     Load custom text data from file.
 
     Parameters
     ----------
     filepath_or_buffer : str
         File path where the custom text data will be loaded from.
-    columns : list or callable(file-like object, **kwargs) -> list
+    columns : list or callable(file-like object, \*\*kwargs) -> list
         Column names of list type or callable that create column names from opened file
         and passed `kwargs`.
-    custom_parser : callable(file-like object, **kwargs) -> pandas.DataFrame
+    custom_parser : callable(file-like object, \*\*kwargs) -> pandas.DataFrame
         Function that takes as input a part of the `filepath_or_buffer` file loaded into
         memory in file-like object form.
     compression : {'infer', 'gzip', 'bz2', 'zip', 'xz', None}, default: 'infer'


Test patch:
Repository: modin-project/modin

Issue:
REFACTOR: Use default implementations for dt methods from the base query compiler
Initially found in https://github.com/modin-project/modin/pull/5848#discussion_r1147794258.


Discussion:

Resolving patch:
diff --git a/modin/core/storage_formats/pandas/query_compiler.py b/modin/core/storage_formats/pandas/query_compiler.py
--- a/modin/core/storage_formats/pandas/query_compiler.py
+++ b/modin/core/storage_formats/pandas/query_compiler.py
@@ -792,21 +792,6 @@ def is_series_like(self):
     # END Transpose
 
     # TreeReduce operations
-
-    def is_monotonic_decreasing(self):
-        def is_monotonic_decreasing(df):
-            """Check whether data in the passed frame is monotonic decreasing."""
-            return pandas.DataFrame([df.squeeze(axis=1).is_monotonic_decreasing])
-
-        return self.default_to_pandas(is_monotonic_decreasing)
-
-    def is_monotonic_increasing(self):
-        def is_monotonic_increasing(df):
-            """Check whether data in the passed frame is monotonic increasing."""
-            return pandas.DataFrame([df.squeeze(axis=1).is_monotonic_increasing])
-
-        return self.default_to_pandas(is_monotonic_increasing)
-
     count = TreeReduce.register(pandas.DataFrame.count, pandas.DataFrame.sum)
     sum = TreeReduce.register(pandas.DataFrame.sum)
     prod = TreeReduce.register(pandas.DataFrame.prod)
@@ -1649,19 +1634,6 @@ def searchsorted(df):
     dt_is_leap_year = Map.register(_dt_prop_map("is_leap_year"), dtypes=np.bool_)
     dt_daysinmonth = Map.register(_dt_prop_map("daysinmonth"), dtypes=np.int64)
     dt_days_in_month = Map.register(_dt_prop_map("days_in_month"), dtypes=np.int64)
-
-    def dt_tz(self):
-        def datetime_tz(df):
-            return pandas.DataFrame([df.squeeze(axis=1).dt.tz])
-
-        return self.default_to_pandas(datetime_tz)
-
-    def dt_freq(self):
-        def datetime_freq(df):
-            return pandas.DataFrame([df.squeeze(axis=1).dt.freq])
-
-        return self.default_to_pandas(datetime_freq)
-
     dt_asfreq = Map.register(_dt_func_map("asfreq"))
     dt_to_period = Map.register(_dt_func_map("to_period"))
     dt_to_pydatetime = Map.register(_dt_func_map("to_pydatetime"), dtypes=np.object_)


Test patch:
Repository: modin-project/modin

Issue:
Add read_fwf parallel implementation for Dask



Discussion:

Resolving patch:
diff --git a/modin/core/execution/dask/implementations/pandas_on_dask/io/io.py b/modin/core/execution/dask/implementations/pandas_on_dask/io/io.py
--- a/modin/core/execution/dask/implementations/pandas_on_dask/io/io.py
+++ b/modin/core/execution/dask/implementations/pandas_on_dask/io/io.py
@@ -27,6 +27,7 @@
 )
 from modin.core.io import (
     CSVDispatcher,
+    FWFDispatcher,
     JSONDispatcher,
     ParquetDispatcher,
     FeatherDispatcher,
@@ -35,6 +36,7 @@
 )
 from modin.core.storage_formats.pandas.parsers import (
     PandasCSVParser,
+    PandasFWFParser,
     PandasJSONParser,
     PandasParquetParser,
     PandasFeatherParser,
@@ -56,6 +58,7 @@ class PandasOnDaskIO(BaseIO):
     )
 
     read_csv = type("", (DaskWrapper, PandasCSVParser, CSVDispatcher), build_args).read
+    read_fwf = type("", (DaskWrapper, PandasFWFParser, FWFDispatcher), build_args).read
     read_json = type(
         "", (DaskWrapper, PandasJSONParser, JSONDispatcher), build_args
     ).read


Test patch:
Repository: modin-project/modin

Issue:
HDK: remove unused parameters from `run_exec_plan`



Discussion:

Resolving patch:
diff --git a/modin/experimental/core/execution/native/implementations/hdk_on_native/dataframe/dataframe.py b/modin/experimental/core/execution/native/implementations/hdk_on_native/dataframe/dataframe.py
--- a/modin/experimental/core/execution/native/implementations/hdk_on_native/dataframe/dataframe.py
+++ b/modin/experimental/core/execution/native/implementations/hdk_on_native/dataframe/dataframe.py
@@ -1931,7 +1931,7 @@ def _execute(self):
                 raise RuntimeError("forced arrow execution failed")
 
             new_partitions = self._partition_mgr_cls.run_exec_plan(
-                self._op, self._index_cols, self._dtypes, self._table_cols
+                self._op, self._table_cols
             )
         self._partitions = new_partitions
         self._op = FrameNode(self)
diff --git a/modin/experimental/core/execution/native/implementations/hdk_on_native/partitioning/partition_manager.py b/modin/experimental/core/execution/native/implementations/hdk_on_native/partitioning/partition_manager.py
--- a/modin/experimental/core/execution/native/implementations/hdk_on_native/partitioning/partition_manager.py
+++ b/modin/experimental/core/execution/native/implementations/hdk_on_native/partitioning/partition_manager.py
@@ -238,7 +238,7 @@ def is_supported_dtype(dtype):
         )
 
     @classmethod
-    def run_exec_plan(cls, plan, index_cols, dtypes, columns):
+    def run_exec_plan(cls, plan, columns):
         """
         Run execution plan in HDK storage format to materialize frame.
 
@@ -246,10 +246,6 @@ def run_exec_plan(cls, plan, index_cols, dtypes, columns):
         ----------
         plan : DFAlgNode
             A root of an execution plan tree.
-        index_cols : list of str
-            A list of index columns.
-        dtypes : pandas.Index
-            Column data types.
         columns : list of str
             A frame column names.
 


Test patch:
diff --git a/modin/experimental/core/execution/native/implementations/hdk_on_native/test/utils.py b/modin/experimental/core/execution/native/implementations/hdk_on_native/test/utils.py
--- a/modin/experimental/core/execution/native/implementations/hdk_on_native/test/utils.py
+++ b/modin/experimental/core/execution/native/implementations/hdk_on_native/test/utils.py
@@ -198,8 +198,6 @@ def export_frames(self):
             # as it has a chance of running via pyarrow bypassing HDK
             new_partitions = modin_frame._partition_mgr_cls.run_exec_plan(
                 modin_frame._op,
-                modin_frame._index_cols,
-                modin_frame._dtypes,
                 modin_frame._table_cols,
             )
             modin_frame._partitions = new_partitions
Repository: modin-project/modin

Issue:
HDK: remove `_dtypes_for_cols` internal function as unused



Discussion:

Resolving patch:
diff --git a/modin/experimental/core/execution/native/implementations/hdk_on_native/dataframe/dataframe.py b/modin/experimental/core/execution/native/implementations/hdk_on_native/dataframe/dataframe.py
--- a/modin/experimental/core/execution/native/implementations/hdk_on_native/dataframe/dataframe.py
+++ b/modin/experimental/core/execution/native/implementations/hdk_on_native/dataframe/dataframe.py
@@ -387,38 +387,6 @@ def _has_arrow_table(self):
             return False
         return all(p.arrow_table is not None for p in self._partitions.flatten())
 
-    def _dtypes_for_cols(self, new_index, new_columns):
-        """
-        Return dtypes index for a specified set of index and data columns.
-
-        Parameters
-        ----------
-        new_index : pandas.Index or list
-            Index columns.
-        new_columns : pandas.Index or list
-            Data Columns.
-
-        Returns
-        -------
-        pandas.Index
-        """
-        if new_index is not None:
-            if isinstance(self._dtypes, MultiIndex):
-                new_index = [
-                    (col, *([""] * (self._dtypes.nlevels - 1))) for col in new_index
-                ]
-            res = self._dtypes[
-                new_index
-                + (
-                    new_columns
-                    if isinstance(new_columns, list)
-                    else new_columns.to_list()
-                )
-            ]
-        else:
-            res = self._dtypes[new_columns]
-        return res
-
     def _dtypes_for_exprs(self, exprs):
         """
         Return dtypes for expressions.


Test patch: