internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: modin-project/modin

Issue:
BUG: Series.copy() with empty series loses dtype
### Modin version checks

- [X] I have checked that this issue has not already been reported.

- [X] I have confirmed this bug exists on the latest released version of Modin.

- [X] I have confirmed this bug exists on the main branch of Modin. (In order to do this you can follow [this guide](https://modin.readthedocs.io/en/stable/getting_started/installation.html#installing-from-the-github-master-branch).)


### Reproducible Example

```python
import modin.pandas as pd

ser =  pd.Series(range(3))

res = ser[:0].copy()
assert res.dtype == ser.dtype  # <- nope!
```


### Issue Description

dtype should be preserved

### Expected Behavior

NA

### Error Logs

<details>

```python-traceback

Replace this line with the error backtrace (if applicable).

```

</details>


### Installed Versions

<details>

Replace this line with the output of pd.show_versions()

</details>



Discussion:
Our current implementation defaults to pandas on empty dataframes/series.

So this bug boils down to `ser[:0]._to_pandas()` always producing `object` dtypes, which in turn is caused by Modin DataFrame object producing an empty pandas DataFrame without specifying any dtypes (and internal query compiler in that case seems to have wrong dtypes, too).

Resolving patch:
diff --git a/modin/core/dataframe/pandas/dataframe/dataframe.py b/modin/core/dataframe/pandas/dataframe/dataframe.py
--- a/modin/core/dataframe/pandas/dataframe/dataframe.py
+++ b/modin/core/dataframe/pandas/dataframe/dataframe.py
@@ -3725,6 +3725,8 @@ def to_pandas(self):
         df = self._partition_mgr_cls.to_pandas(self._partitions)
         if df.empty:
             df = pandas.DataFrame(columns=self.columns, index=self.index)
+            if len(df.columns) and self.has_materialized_dtypes:
+                df = df.astype(self.dtypes)
         else:
             for axis, has_external_index in enumerate(
                 ["has_materialized_index", "has_materialized_columns"]


Test patch:
diff --git a/modin/pandas/test/dataframe/test_map_metadata.py b/modin/pandas/test/dataframe/test_map_metadata.py
--- a/modin/pandas/test/dataframe/test_map_metadata.py
+++ b/modin/pandas/test/dataframe/test_map_metadata.py
@@ -320,6 +320,12 @@ def test_copy(data):
     df_equals(modin_df, modin_df_cp)
 
 
+def test_copy_empty_dataframe():
+    df = pd.DataFrame(range(3))
+    res = df[:0].copy()
+    assert res.dtypes.equals(df.dtypes)
+
+
 @pytest.mark.parametrize("data", test_data_values, ids=test_data_keys)
 def test_dtypes(data):
     modin_df = pd.DataFrame(data)
diff --git a/modin/pandas/test/test_series.py b/modin/pandas/test/test_series.py
--- a/modin/pandas/test/test_series.py
+++ b/modin/pandas/test/test_series.py
@@ -1409,6 +1409,12 @@ def test_copy(data):
     df_equals(modin_series.copy(), pandas_series.copy())
 
 
+def test_copy_empty_series():
+    ser = pd.Series(range(3))
+    res = ser[:0].copy()
+    assert res.dtype == ser.dtype
+
+
 @pytest.mark.parametrize("data", test_data_values, ids=test_data_keys)
 def test_corr(data):
     modin_series, pandas_series = create_test_series(data)
Repository: modin-project/modin

Issue:
Preserve dtypes cache for `_repartition`



Discussion:

Resolving patch:
diff --git a/modin/core/storage_formats/base/query_compiler.py b/modin/core/storage_formats/base/query_compiler.py
--- a/modin/core/storage_formats/base/query_compiler.py
+++ b/modin/core/storage_formats/base/query_compiler.py
@@ -6593,6 +6593,7 @@ def repartition(self, axis=None):
                     lambda df: df,
                     new_index=self._modin_frame.copy_index_cache(),
                     new_columns=self._modin_frame.copy_columns_cache(),
+                    dtypes=self._modin_frame.copy_dtypes_cache(),
                     keep_partitioning=False,
                     sync_labels=False,
                 )


Test patch:
Repository: modin-project/modin

Issue:
Execute bitwise NOT (~) operations on HDK
Currently HDK does not support the bitwise NOT operation (see https://github.com/intel-ai/hdk/issues/542) and this operation is forced for arrow execution. This behavior should be changed once the operation is implemented in HDK.



Discussion:

Resolving patch:
diff --git a/modin/experimental/core/execution/native/implementations/hdk_on_native/expr.py b/modin/experimental/core/execution/native/implementations/hdk_on_native/expr.py
--- a/modin/experimental/core/execution/native/implementations/hdk_on_native/expr.py
+++ b/modin/experimental/core/execution/native/implementations/hdk_on_native/expr.py
@@ -485,7 +485,7 @@ def invert(self) -> "OpExpr":
         OpExpr
             The resulting bitwise inverse expression.
         """
-        return OpExpr("~", [self], self._dtype)
+        return OpExpr("BIT_NOT", [self], self._dtype)
 
     def _cmp_op(self, other, op_name):
         """
@@ -983,7 +983,7 @@ class OpExpr(BaseExpr):
         "POWER": lambda self: self._fold_arithm("__pow__"),
         "/": lambda self: self._fold_arithm("__truediv__"),
         "//": lambda self: self._fold_arithm("__floordiv__"),
-        "~": lambda self: self._fold_invert(),
+        "BIT_NOT": lambda self: self._fold_invert(),
         "CAST": lambda self: self._fold_literal("cast", self._dtype),
         "IS NULL": lambda self: self._fold_literal("is_null"),
         "IS NOT NULL": lambda self: self._fold_literal("is_not_null"),
@@ -996,7 +996,7 @@ class OpExpr(BaseExpr):
         "POWER": lambda self, table: self._pc("power", table),
         "/": lambda self, table: self._pc("divide", table),
         "//": lambda self, table: self._pc("divide", table),
-        "~": lambda self, table: self._invert(table),
+        "BIT_NOT": lambda self, table: self._invert(table),
         "CAST": lambda self, table: self._col(table).cast(to_arrow_type(self._dtype)),
         "IS NULL": lambda self, table: self._col(table).is_null(nan_is_null=True),
         "IS NOT NULL": lambda self, table: pc.invert(
@@ -1004,7 +1004,7 @@ class OpExpr(BaseExpr):
         ),
     }
 
-    _UNSUPPORTED_HDK_OPS = {"~"}
+    _UNSUPPORTED_HDK_OPS = {}
 
     def __init__(self, op, operands, dtype):
         self.op = op


Test patch:
Repository: modin-project/modin

Issue:
We should set  `__factory` to `None` in case of any problems that may arise during initialization.
In order not to hide the original error.


Discussion:

Resolving patch:
diff --git a/modin/core/execution/dispatching/factories/dispatcher.py b/modin/core/execution/dispatching/factories/dispatcher.py
--- a/modin/core/execution/dispatching/factories/dispatcher.py
+++ b/modin/core/execution/dispatching/factories/dispatcher.py
@@ -165,6 +165,9 @@ def _update_factory(cls, _):
                 raise ModuleNotFoundError(
                     f"Make sure all required packages are installed: {str(err)}"
                 ) from err
+            except BaseException:
+                cls.__factory = None
+                raise
 
     @classmethod
     @_inherit_docstrings(factories.BaseFactory._from_pandas)


Test patch:
Repository: modin-project/modin

Issue:
Add support for modin's numpy array in modin's `dataframe.insert()`
Currently, `df.insert()` treats modin's numpy array as a scalar:
```python
import modin.config as cfg
import modin.pandas as pd

cfg.ExperimentalNumPyAPI.put(False)

df = pd.DataFrame({"a": [1, 2, 3]})
df["b"] = df["a"].to_numpy()
print(df)
#    a  b
# 0  1  1
# 1  2  2
# 2  3  3

cfg.ExperimentalNumPyAPI.put(True)
df["c"] = df["a"].to_numpy()
print(df)
#    a  b                 c
# 0  1  1  array([1, 2, 3])
# 1  2  2  array([1, 2, 3])
# 2  3  3  array([1, 2, 3])
```



Discussion:

Resolving patch:
diff --git a/modin/pandas/dataframe.py b/modin/pandas/dataframe.py
--- a/modin/pandas/dataframe.py
+++ b/modin/pandas/dataframe.py
@@ -991,9 +991,11 @@ def insert(
         """
         Insert column into ``DataFrame`` at specified location.
         """
+        from modin.numpy import array
+
         if (
             isinstance(value, (DataFrame, pandas.DataFrame))
-            or isinstance(value, np.ndarray)
+            or isinstance(value, (array, np.ndarray))
             and len(value.shape) > 1
         ):
             if value.shape[1] != 1:
@@ -1042,7 +1044,7 @@ def insert(
                 )
             elif loc < 0:
                 raise ValueError("unbounded slice")
-            if isinstance(value, Series):
+            if isinstance(value, (Series, array)):
                 value = value._query_compiler
             new_query_compiler = self._query_compiler.insert(loc, column, value)
 


Test patch:
diff --git a/modin/pandas/test/dataframe/test_map_metadata.py b/modin/pandas/test/dataframe/test_map_metadata.py
--- a/modin/pandas/test/dataframe/test_map_metadata.py
+++ b/modin/pandas/test/dataframe/test_map_metadata.py
@@ -1385,6 +1385,19 @@ def comparator(df1, df2):
         )
 
 
+def test_insert_modin_array():
+    from modin.numpy import array
+
+    data = {"col1": [1, 2, 3], "col2": [2, 3, 4]}
+    modin_df1, modin_df2 = pd.DataFrame(data), pd.DataFrame(data)
+    np_value = np.array([7, 7, 7])
+    md_np_value = array(np_value)
+
+    modin_df1.insert(1, "new_col", np_value)
+    modin_df2.insert(1, "new_col", md_np_value)
+    df_equals(modin_df1, modin_df2)
+
+
 @pytest.mark.parametrize("data", test_data_values, ids=test_data_keys)
 def test_ndim(data):
     modin_df = pd.DataFrame(data)
Repository: modin-project/modin

Issue:
TypeError: read_feather() got an unexpected keyword argument 'types_mapper'
`read_feather` does not work with pyarrow<11.0


Discussion:

Resolving patch:
diff --git a/modin/core/storage_formats/pandas/parsers.py b/modin/core/storage_formats/pandas/parsers.py
--- a/modin/core/storage_formats/pandas/parsers.py
+++ b/modin/core/storage_formats/pandas/parsers.py
@@ -891,7 +891,14 @@ def parse(fname, **kwargs):
             fname,
             **(kwargs.pop("storage_options", None) or {}),
         ) as file:
-            df = feather.read_feather(file, **kwargs, **to_pandas_kwargs)
+            # The implementation is as close as possible to the one in pandas.
+            # For reference see `read_feather` in pandas/io/feather_format.py.
+            if not to_pandas_kwargs:
+                df = feather.read_feather(file, **kwargs)
+            else:
+                # `read_feather` doesn't accept `types_mapper` if pyarrow<11.0
+                pa_table = feather.read_table(file, **kwargs)
+                df = pa_table.to_pandas(**to_pandas_kwargs)
         # Append the length of the index here to build it externally
         return _split_result_for_readers(0, num_splits, df) + [len(df.index), df.dtypes]
 


Test patch:
Repository: modin-project/modin

Issue:
`modin.array.__getitem__` works very inefficient when the key is modin.array and its size is > 50% of all rows



Discussion:

Resolving patch:
diff --git a/modin/core/dataframe/pandas/dataframe/dataframe.py b/modin/core/dataframe/pandas/dataframe/dataframe.py
--- a/modin/core/dataframe/pandas/dataframe/dataframe.py
+++ b/modin/core/dataframe/pandas/dataframe/dataframe.py
@@ -990,6 +990,22 @@ def _take_2d_positional(
         if col_positions is None and row_positions is None:
             return self.copy()
 
+        if col_positions is None and row_positions is not None:
+            # Check if the optimization that first takes part of the data using the mask
+            # operation so that later less data is concatenated into a whole column is useful.
+            # In the case when only a small portion of the data is discarded, the overhead of the
+            # engine (for putting data in and out of storage) can exceed the resulting speedup.
+            all_rows = None
+            if self.has_materialized_index:
+                all_rows = len(self.index)
+            elif self._row_lengths_cache:
+                all_rows = sum(self._row_lengths_cache)
+            if all_rows:
+                if len(row_positions) > 0.9 * all_rows:
+                    return self._reorder_labels(
+                        row_positions=row_positions, col_positions=col_positions
+                    )
+
         sorted_row_positions = sorted_col_positions = None
 
         if row_positions is not None:


Test patch:
Repository: modin-project/modin

Issue:
Preserve dtypes cache for `reindex`



Discussion:

Resolving patch:
diff --git a/modin/core/storage_formats/pandas/query_compiler.py b/modin/core/storage_formats/pandas/query_compiler.py
--- a/modin/core/storage_formats/pandas/query_compiler.py
+++ b/modin/core/storage_formats/pandas/query_compiler.py
@@ -640,13 +640,33 @@ def map_func(left, right=right_pandas, kwargs=kwargs):  # pragma: no cover
 
     # Reindex/reset_index (may shuffle data)
     def reindex(self, axis, labels, **kwargs):
-        new_index, _ = (self.index, None) if axis else self.index.reindex(labels)
+        new_index, indexer = (self.index, None) if axis else self.index.reindex(labels)
         new_columns, _ = self.columns.reindex(labels) if axis else (self.columns, None)
+        new_dtypes = None
+        if (
+            self._modin_frame.has_materialized_dtypes
+            and kwargs.get("method", None) is None
+        ):
+            # For columns, defining types is easier because we don't have to calculate the common
+            # type, since the entire column is filled. A simple `reindex` covers our needs.
+            # For rows, we can avoid calculating common types if we know that no new strings of
+            # arbitrary type have been added (this information is in `indexer`).
+            dtype = pandas.Index([kwargs.get("fill_value", np.nan)]).dtype
+            if axis == 0:
+                new_dtypes = self.dtypes.copy()
+                # "-1" means that the required labels are missing in the dataframe and the
+                # corresponding rows will be filled with "fill_value" that may change the column type.
+                if indexer is not None and -1 in indexer:
+                    for col, col_dtype in new_dtypes.items():
+                        new_dtypes[col] = find_common_type((col_dtype, dtype))
+            else:
+                new_dtypes = self.dtypes.reindex(labels, fill_value=dtype)
         new_modin_frame = self._modin_frame.apply_full_axis(
             axis,
             lambda df: df.reindex(labels=labels, axis=axis, **kwargs),
             new_index=new_index,
             new_columns=new_columns,
+            dtypes=new_dtypes,
         )
         return self.__constructor__(new_modin_frame)
 


Test patch:
diff --git a/modin/test/storage_formats/pandas/test_internals.py b/modin/test/storage_formats/pandas/test_internals.py
--- a/modin/test/storage_formats/pandas/test_internals.py
+++ b/modin/test/storage_formats/pandas/test_internals.py
@@ -1089,3 +1089,14 @@ def test_setitem_bool_preserve_dtypes():
     # scalar as a col_loc
     df.loc[indexer, "a"] = 2.0
     assert df._query_compiler._modin_frame.has_materialized_dtypes
+
+
+@pytest.mark.parametrize(
+    "kwargs",
+    [dict(axis=0, labels=[]), dict(axis=1, labels=["a"]), dict(axis=1, labels=[])],
+)
+def test_reindex_preserve_dtypes(kwargs):
+    df = pd.DataFrame({"a": [1, 1, 2, 2], "b": [3, 4, 5, 6]})
+
+    reindexed_df = df.reindex(**kwargs)
+    assert reindexed_df._query_compiler._modin_frame.has_materialized_dtypes
Repository: modin-project/modin

Issue:
Some functions signatures are outdated
For example signature of `modin.pandas.base.BasePandasDataset.apply` function contains unused `broadcast`, `reduce` and `convert_dtype` parameters (`pandas.DataFrame.apply` doesn't uses these parameters too).


Discussion:

Resolving patch:
diff --git a/modin/pandas/base.py b/modin/pandas/base.py
--- a/modin/pandas/base.py
+++ b/modin/pandas/base.py
@@ -854,11 +854,8 @@ def apply(
         self,
         func,
         axis,
-        broadcast,
         raw,
-        reduce,
         result_type,
-        convert_dtype,
         args,
         **kwds,
     ):  # noqa: PR01, RT01, D200
diff --git a/modin/pandas/dataframe.py b/modin/pandas/dataframe.py
--- a/modin/pandas/dataframe.py
+++ b/modin/pandas/dataframe.py
@@ -394,11 +394,8 @@ def apply(
         query_compiler = super(DataFrame, self).apply(
             func,
             axis=axis,
-            broadcast=None,
             raw=raw,
-            reduce=None,
             result_type=result_type,
-            convert_dtype=None,
             args=args,
             **kwargs,
         )
diff --git a/modin/pandas/series.py b/modin/pandas/series.py
--- a/modin/pandas/series.py
+++ b/modin/pandas/series.py
@@ -604,11 +604,8 @@ def apply(
             result = super(Series, self).apply(
                 func,
                 axis=0,
-                broadcast=None,
                 raw=False,
-                reduce=None,
                 result_type=None,
-                convert_dtype=convert_dtype,
                 args=args,
                 **kwargs,
             )


Test patch:
Repository: modin-project/modin

Issue:
Update copyright year



Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -62,7 +62,7 @@ def noop_decorator(*args, **kwargs):
 export_config_help(configs_file_path)
 
 project = "Modin"
-copyright = "2018-2022, Modin Developers."
+copyright = "2018-2023, Modin Developers."
 author = "Modin contributors"
 
 # The short X.Y version


Test patch:
Repository: modin-project/modin

Issue:
autodoc: failed to import class 'xgboost.DMatrix'
Found out in https://github.com/modin-project/modin/actions/runs/5761620564/job/15619967154?pr=6423


Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -25,7 +25,7 @@ def noop_decorator(*args, **kwargs):
 ray.remote = noop_decorator
 
 # fake modules if they're missing
-for mod_name in ("cudf", "cupy", "pyarrow.gandiva", "pyhdk", "pyhdk.hdk"):
+for mod_name in ("cudf", "cupy", "pyarrow.gandiva", "pyhdk", "pyhdk.hdk", "xgboost"):
     try:
         __import__(mod_name)
     except ImportError:
@@ -49,6 +49,8 @@ def noop_decorator(*args, **kwargs):
 if not hasattr(sys.modules["pyhdk"], "__version__"):
     # Show all known pyhdk config options in documentation
     sys.modules["pyhdk"].__version__ = "999"
+if not hasattr(sys.modules["xgboost"], "Booster"):
+    sys.modules["xgboost"].Booster = type("Booster", (object,), {})
 
 sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
 import modin


Test patch:
Repository: modin-project/modin

Issue:
Remove `Patcher` class
https://github.com/modin-project/modin/blob/f24ba6f562759c5807f7f6af1db96899d02a00c8/modin/conftest.py#L102

This class was needed for the Modin in the cloud experimental feature.


Discussion:

Resolving patch:
diff --git a/modin/conftest.py b/modin/conftest.py
--- a/modin/conftest.py
+++ b/modin/conftest.py
@@ -99,34 +99,6 @@ def pytest_addoption(parser):
     )
 
 
-class Patcher:
-    def __init__(self, conn, *pairs):
-        self.pairs = pairs
-        self.originals = None
-        self.conn = conn
-
-    def __wrap(self, func):
-        def wrapper(*a, **kw):
-            return func(
-                *(tuple(self.conn.obtain(x) for x in a)),
-                **({k: self.conn.obtain(v) for k, v in kw.items()}),
-            )
-
-        return func, wrapper
-
-    def __enter__(self):
-        self.originals = []
-        for module, attrname in self.pairs:
-            orig, wrapped = self.__wrap(getattr(module, attrname))
-            self.originals.append((module, attrname, orig))
-            setattr(module, attrname, wrapped)
-        return self
-
-    def __exit__(self, *a, **kw):
-        for module, attrname, orig in self.originals:
-            setattr(module, attrname, orig)
-
-
 def set_experimental_env(mode):
     IsExperimental.put(mode == "experimental")
 


Test patch:
Repository: modin-project/modin

Issue:
HDK: Performance degradation after #6444 
### Modin version checks

- [X] I have checked that this issue has not already been reported.

- [X] I have confirmed this bug exists on the latest released version of Modin.

- [X] I have confirmed this bug exists on the main branch of Modin. (In order to do this you can follow [this guide](https://modin.readthedocs.io/en/stable/getting_started/installation.html#installing-from-the-github-master-branch).)


### Reproducible Example

Run H2O benchmark. Q3 is 10x slower after #6444



### Issue Description

The issue is caused by this https://github.com/modin-project/modin/pull/6444/files#diff-7972fd5dddcdf656c95eae334851ace130660d9aff5751510b11e6b62fd82187R906 call to `is_bool_dtype()`. In case of a categorical dtype, it checks all the categories.

### Expected Behavior

Do not call is_bool_dtype() for categorical.

### Error Logs

<details>

```python-traceback

Replace this line with the error backtrace (if applicable).

```

</details>


### Installed Versions

<details>

Replace this line with the output of pd.show_versions()

</details>



Discussion:

Resolving patch:
diff --git a/modin/experimental/core/execution/native/implementations/hdk_on_native/calcite_builder.py b/modin/experimental/core/execution/native/implementations/hdk_on_native/calcite_builder.py
--- a/modin/experimental/core/execution/native/implementations/hdk_on_native/calcite_builder.py
+++ b/modin/experimental/core/execution/native/implementations/hdk_on_native/calcite_builder.py
@@ -45,7 +45,7 @@
 )
 
 from collections import abc
-from pandas.core.dtypes.common import get_dtype, is_bool_dtype
+from pandas.core.dtypes.common import get_dtype, is_categorical_dtype, is_bool_dtype
 
 
 class CalciteBuilder:
@@ -906,7 +906,10 @@ def _process_groupby(self, op):
             bool_cols := {
                 c: cast_agg[agg_exprs[c].agg]
                 for c, t in frame.dtypes.items()
-                if is_bool_dtype(t) and agg_exprs[c].agg in cast_agg
+                # Do not call is_bool_dtype() for categorical since it checks all the categories
+                if not is_categorical_dtype(t)
+                and is_bool_dtype(t)
+                and agg_exprs[c].agg in cast_agg
             }
         ):
             trans = self._input_ctx()._maybe_copy_and_translate_expr


Test patch: