internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: modin-project/modin

Issue:
DOCS: Recommend GitHub issues over feature_requests@modin.org
The "not implemented error message recommends feature_requests@modin.org but does not mention GitHub: https://github.com/modin-project/modin/blob/0f70e828ed127229f7160075d2ad06a6b16b1002/modin/error_message.py#L29

We should point users to GitHub, with the email as a backup in case they can't post an issue on GitHub.


Discussion:

Resolving patch:
diff --git a/modin/error_message.py b/modin/error_message.py
--- a/modin/error_message.py
+++ b/modin/error_message.py
@@ -25,8 +25,9 @@ def not_implemented(cls, message=""):
             message = "This functionality is not yet available in Modin."
         raise NotImplementedError(
             f"{message}\n"
-            + "To request implementation, send an email to "
-            + "feature_requests@modin.org"
+            + "To request implementation, file an issue at "
+            + "https://github.com/modin-project/modin/issues or, if that's "
+            + "not possible, send an email to feature_requests@modin.org."
         )
 
     @classmethod


Test patch:
Repository: modin-project/modin

Issue:
Logging mode leaves a hanging thread in interactive python session
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: MacOS Monterey
- **Modin version** (`modin.__version__`): latest source
- **Python version**: 3.9.12
- **Code we can use to reproduce**:

In ipython, run 

```python
from modin.config import LogMode
import modin.pandas as pd

LogMode.enable()
df = pd.DataFrame([[1]])
```

and try to exit. You have to control-c or control-d twice.

This bug came up in the discussion on #4372 but it wasn't fixed in that PR.

The problematic thread is started [here](https://github.com/modin-project/modin/blob/49fc2cf3733f20ac6cf8a7c61e42ef7aa5cf4b03/modin/logging/config.py#L127).



Discussion:

Resolving patch:
diff --git a/modin/logging/config.py b/modin/logging/config.py
--- a/modin/logging/config.py
+++ b/modin/logging/config.py
@@ -126,7 +126,9 @@ def configure_logging():
 
     if LogMode.get() != "enable_api_only":
         mem_sleep = LogMemoryInterval.get()
-        mem = threading.Thread(target=memory_thread, args=[logger, mem_sleep])
+        mem = threading.Thread(
+            target=memory_thread, args=[logger, mem_sleep], daemon=True
+        )
         mem.start()
 
     __LOGGER_CONFIGURED__ = True


Test patch:
Repository: modin-project/modin

Issue:
doc_checker does not show `pydocstyle` stderr
### Describe the problem
There could be issues when `pydocstyle` would error out without saying a word to `stdout` (in my case I had accidentally installed some ancient `pydocstyle` which didn't know `numpy` convention).
In that case `doc_checker.py` just says

```
ERROR:
ERROR:INVALID DOCUMENTATION FOUND
```

and does not give any pointers to what went wrong.



Discussion:

Resolving patch:
diff --git a/scripts/doc_checker.py b/scripts/doc_checker.py
--- a/scripts/doc_checker.py
+++ b/scripts/doc_checker.py
@@ -491,6 +491,7 @@ def pydocstyle_validate(
     if result.returncode:
         logging.info(f"PYDOCSTYLE OUTPUT FOR {path}")
         logging.error(result.stdout)
+        logging.error(result.stderr)
     return True if result.returncode == 0 else False
 
 


Test patch:
Repository: modin-project/modin

Issue:
BUG: We need to pin ray < 1.13.0 until we fix deserialization race condition bug #4564.
Installing modin on any system may cause serialization race conditions like in #4564.


Discussion:

Resolving patch:
diff --git a/modin/config/envvars.py b/modin/config/envvars.py
--- a/modin/config/envvars.py
+++ b/modin/config/envvars.py
@@ -81,7 +81,11 @@ def _get_default(cls):
         -------
         str
         """
-        from modin.utils import MIN_RAY_VERSION, MIN_DASK_VERSION
+        from modin.utils import (
+            MIN_RAY_VERSION,
+            MAX_RAY_VERSION_EXCLUSIVE,
+            MIN_DASK_VERSION,
+        )
 
         if IsDebug.get():
             return "Python"
@@ -91,9 +95,14 @@ def _get_default(cls):
         except ImportError:
             pass
         else:
-            if version.parse(ray.__version__) < MIN_RAY_VERSION:
+            if (
+                version.parse(ray.__version__) < MIN_RAY_VERSION
+                or version.parse(ray.__version__) >= MAX_RAY_VERSION_EXCLUSIVE
+            ):
                 raise ImportError(
-                    f"Please `pip install modin[ray]` to install compatible Ray version (>={MIN_RAY_VERSION})."
+                    "Please `pip install modin[ray]` to install compatible Ray "
+                    + "version "
+                    + f"(>={MIN_RAY_VERSION},<{MAX_RAY_VERSION_EXCLUSIVE})."
                 )
             return "Ray"
         try:
diff --git a/modin/utils.py b/modin/utils.py
--- a/modin/utils.py
+++ b/modin/utils.py
@@ -36,6 +36,9 @@
 from modin._version import get_versions
 
 MIN_RAY_VERSION = version.parse("1.4.0")
+# TODO(https://github.com/modin-project/modin/issues/4564):
+# once ray can go past 1.13.0, remove this constant and stop checking it.
+MAX_RAY_VERSION_EXCLUSIVE = version.parse("1.13.0")
 MIN_DASK_VERSION = version.parse("2.22.0")
 
 PANDAS_API_URL_TEMPLATE = f"https://pandas.pydata.org/pandas-docs/version/{pandas.__version__}/reference/api/{{}}.html"
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -6,7 +6,9 @@
 
 dask_deps = ["dask>=2.22.0,<2022.2.0", "distributed>=2.22.0,<2022.2.0"]
 # TODO: remove redis dependency when resolving GH#4398
-ray_deps = ["ray[default]>=1.4.0", "pyarrow>=4.0.1", "redis>=3.5.0,<4.0.0"]
+# Pin ray to < 1.13.0 to work around GH#4564
+# TODO(https://github.com/modin-project/modin/issues/4564): let ray go past 1.13.0.
+ray_deps = ["ray[default]>=1.4.0,<1.13.0", "pyarrow>=4.0.1", "redis>=3.5.0,<4.0.0"]
 remote_deps = ["rpyc==4.1.5", "cloudpickle", "boto3"]
 spreadsheet_deps = ["modin-spreadsheet>=0.1.0"]
 sql_deps = ["dfsql>=0.4.2", "pyparsing<=2.4.7"]


Test patch:
Repository: modin-project/modin

Issue:
Modin custom logging layer is not inherited
### System information
- **Modin version** (`modin.__version__`): 81648e711f7525d87dbc58f53e67a2fd22650ae7

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
It's possible to override default logging layer (`PANDAS-API`) when constructing a class, like
```python
class Foo(ClassLogger, modin_layer="CUSTOM"):
    def method1(self):
        pass
```
but then it won't get inherited, i.e.
```python
class Bar(Foo):
    def method2(self):
        pass
```
would get default logging layer again, which feels wrong.



Discussion:

Resolving patch:
diff --git a/modin/logging/class_logger.py b/modin/logging/class_logger.py
--- a/modin/logging/class_logger.py
+++ b/modin/logging/class_logger.py
@@ -31,10 +31,12 @@ class ClassLogger:
     This mixin must go first in class bases declaration to have the desired effect.
     """
 
+    _modin_logging_layer = "PANDAS-API"
+
     @classmethod
     def __init_subclass__(
         cls,
-        modin_layer: str = "PANDAS-API",
+        modin_layer: Optional[str] = None,
         class_name: Optional[str] = None,
         log_level: Optional[str] = "info",
         **kwargs,
@@ -53,5 +55,7 @@ def __init_subclass__(
             The log level (INFO, DEBUG, WARNING, etc.).
         **kwargs : dict
         """
+        modin_layer = modin_layer or cls._modin_logging_layer
         super().__init_subclass__(**kwargs)
         enable_logging(modin_layer, class_name, log_level)(cls)
+        cls._modin_logging_layer = modin_layer


Test patch:
diff --git a/modin/test/test_logging.py b/modin/test/test_logging.py
--- a/modin/test/test_logging.py
+++ b/modin/test/test_logging.py
@@ -144,6 +144,6 @@ def method2(self):
         "STOP::CUSTOM::Foo.method1",
         "START::CUSTOM::Foo.method1",
         "STOP::CUSTOM::Foo.method1",
-        "START::PANDAS-API::Bar.method2",
-        "STOP::PANDAS-API::Bar.method2",
+        "START::CUSTOM::Bar.method2",
+        "STOP::CUSTOM::Bar.method2",
     ]
Repository: modin-project/modin

Issue:
Facilitate pdb usage in Modin-in-the-Cloud
**Is your feature request related to a problem? Please describe.**
Currently it's impossible to use `pdb` when debugging some failure on remote side of the RPyC tunnel, even if tests are running locally, because `pdb` requires stdin/stdout/stderr to be user-controllable, and current implementation of `LocalConnection` hides those either in `subprocess.PIPE` or in `subprocess.DEVNULL`.


Discussion:

Resolving patch:
diff --git a/modin/conftest.py b/modin/conftest.py
--- a/modin/conftest.py
+++ b/modin/conftest.py
@@ -129,6 +129,18 @@ def simulate_cloud(request):
     if mode == "off":
         yield
         return
+    if (
+        request.config.getoption("usepdb")
+        and request.config.getoption("capture") != "no"
+    ):
+        with request.config.pluginmanager.getplugin(
+            "capturemanager"
+        ).global_and_fixture_disabled():
+            sys.stderr.write(
+                "WARNING! You're running tests in simulate-cloud mode. "
+                + "To enable pdb in remote side please disable output capturing "
+                + "by passing '-s' or '--capture=no' to pytest command line\n"
+            )
 
     if mode not in ("normal", "experimental"):
         raise ValueError(f"Unsupported --simulate-cloud mode: {mode}")
diff --git a/modin/experimental/cloud/local_cluster.py b/modin/experimental/cloud/local_cluster.py
--- a/modin/experimental/cloud/local_cluster.py
+++ b/modin/experimental/cloud/local_cluster.py
@@ -22,6 +22,10 @@
 from .tracing.tracing_connection import TracingWrappingConnection
 from modin.config import DoTraceRpyc
 
+_IS_PYTEST_DEBUG = (
+    any(arg.endswith("pytest") for arg in sys.argv) and "--pdb" in sys.argv
+)
+
 
 class LocalWrappingConnection(
     TracingWrappingConnection if DoTraceRpyc.get() else WrappingConnection
@@ -57,9 +61,9 @@ def _run(self, sshcmd: list, cmd: list, capture_out: bool = True):
             redirect.write(f"Running: {cmd}\n")
         return subprocess.Popen(
             cmd,
-            stdin=subprocess.DEVNULL,
-            stdout=redirect,
-            stderr=redirect,
+            stdin=None if _IS_PYTEST_DEBUG and not capture_out else subprocess.DEVNULL,
+            stdout=None if _IS_PYTEST_DEBUG and not capture_out else redirect,
+            stderr=None if _IS_PYTEST_DEBUG and not capture_out else redirect,
         )
 
     @staticmethod


Test patch:
Repository: modin-project/modin

Issue:
CI downloads protobuf 4.21.1 and ray 1.12.1, which are incompatible.
See [this comment](https://github.com/ray-project/ray/issues/25638#issuecomment-1154422416). We should pin protobuf<4 to fix this bug.


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -8,7 +8,14 @@
 # TODO: remove redis dependency when resolving GH#4398
 # Pin ray to < 1.13.0 to work around GH#4564
 # TODO(https://github.com/modin-project/modin/issues/4564): let ray go past 1.13.0.
-ray_deps = ["ray[default]>=1.4.0,<1.13.0", "pyarrow>=4.0.1", "redis>=3.5.0,<4.0.0"]
+# Following https://github.com/ray-project/ray/pull/25648, pin protobuf < 4,
+# because versions >= 4.0.0 are incompatible with ray<1.13.0.
+ray_deps = [
+    "ray[default]>=1.4.0,<1.13.0",
+    "pyarrow>=4.0.1",
+    "redis>=3.5.0,<4.0.0",
+    "protobuf<4.0.0",
+]
 remote_deps = ["rpyc==4.1.5", "cloudpickle", "boto3"]
 spreadsheet_deps = ["modin-spreadsheet>=0.1.0"]
 sql_deps = ["dfsql>=0.4.2", "pyparsing<=2.4.7"]


Test patch:
Repository: modin-project/modin

Issue:
Bump supported pandas version to 1.4.3
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. What kind of performance improvements would you like to see with this new API?



Discussion:

Resolving patch:
diff --git a/modin/pandas/__init__.py b/modin/pandas/__init__.py
--- a/modin/pandas/__init__.py
+++ b/modin/pandas/__init__.py
@@ -14,7 +14,7 @@
 import pandas
 import warnings
 
-__pandas_version__ = "1.4.2"
+__pandas_version__ = "1.4.3"
 
 if pandas.__version__ != __pandas_version__:
     warnings.warn(
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -32,7 +32,7 @@
     url="https://github.com/modin-project/modin",
     long_description=long_description,
     long_description_content_type="text/markdown",
-    install_requires=["pandas==1.4.2", "packaging", "numpy>=1.18.5", "fsspec", "psutil"],
+    install_requires=["pandas==1.4.3", "packaging", "numpy>=1.18.5", "fsspec", "psutil"],
     extras_require={
         # can be installed by pip install modin[dask]
         "dask": dask_deps,


Test patch:
Repository: modin-project/modin

Issue:
Some pictures are not visible in documentation when dark mode is used
### Describe the problem
<!-- Describe the problem clearly here. -->

All our .svg pictures in documentation assume that documentation page background is white. In the result some picture elements are hidden in dark mode (https://modin.readthedocs.io/en/stable/development/architecture.html#data-transformation)



Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -124,11 +124,14 @@ def noop_decorator(*args, **kwargs):
 
 html_logo = "img/MODIN_ver2.png"
 
+html_context = {"default_mode": "light"}
+
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the
 # documentation.
 #
 html_theme_options = {
+    "navbar_end": ["navbar-icon-links"],
     "sidebarwidth": 270,
     "collapse_navigation": False,
     "navigation_depth": 4,


Test patch:
Repository: modin-project/modin

Issue:
PERF: Avoid deepcopy in Series arithmetic operations
xref #4689 

In cases such as `df[0] - df[1]`, we have `df[0].index._id is df[1].index._id`, which means the `.equals` check in the alignment step is cheep.  Making a deep copy resets the two `_id` objects, making the alignment much more expensive.


Discussion:

Resolving patch:
diff --git a/modin/pandas/series.py b/modin/pandas/series.py
--- a/modin/pandas/series.py
+++ b/modin/pandas/series.py
@@ -2393,8 +2393,9 @@ def _prepare_inter_op(self, other):
             Prepared `other`.
         """
         if isinstance(other, Series):
-            new_self = self.copy()
-            new_other = other.copy()
+            # NB: deep=False is important for performance bc it retains obj.index._id
+            new_self = self.copy(deep=False)
+            new_other = other.copy(deep=False)
             if self.name == other.name:
                 new_self.name = new_other.name = self.name
             else:


Test patch:
Repository: modin-project/modin

Issue:
PERF: avoid to_pandas/default_to_pandas
in `Series.__array_priority__` and CategoricalMethods


Discussion:

Resolving patch:
diff --git a/modin/pandas/series.py b/modin/pandas/series.py
--- a/modin/pandas/series.py
+++ b/modin/pandas/series.py
@@ -70,6 +70,7 @@ class Series(SeriesCompat, BasePandasDataset):
     """
 
     _pandas_class = pandas.Series
+    __array_priority__ = pandas.Series.__array_priority__
 
     def __init__(
         self,
@@ -186,34 +187,6 @@ def __array__(self, dtype=None):  # noqa: PR01, RT01, D200
         """
         return super(Series, self).__array__(dtype).flatten()
 
-    @property
-    def __array_priority__(self):  # pragma: no cover
-        """
-        Return pandas `__array_priority__` Series internal parameter.
-
-        Returns
-        -------
-        int
-            Internal pandas parameter ``__array_priority__`` used during interaction with NumPy.
-        """
-        return self._to_pandas().__array_priority__
-
-    # FIXME: __bytes__ was removed in newer pandas versions, so Modin
-    # can remove it too.
-    def __bytes__(self):
-        """
-        Return bytes representation of the Series.
-
-        Returns
-        -------
-        bytes
-
-        Notes
-        -----
-        Method is deprecated.
-        """
-        return self._default_to_pandas(pandas.Series.__bytes__)
-
     def __contains__(self, key):
         """
         Check if `key` in the `Series.index`.
diff --git a/modin/pandas/series_utils.py b/modin/pandas/series_utils.py
--- a/modin/pandas/series_utils.py
+++ b/modin/pandas/series_utils.py
@@ -40,7 +40,7 @@ def __init__(self, series):
 
     @property
     def categories(self):
-        return self._series._default_to_pandas(pandas.Series.cat).categories
+        return self._series.dtype.categories
 
     @categories.setter
     def categories(self, categories):
@@ -51,7 +51,7 @@ def set_categories(series, categories):
 
     @property
     def ordered(self):
-        return self._series._default_to_pandas(pandas.Series.cat).ordered
+        return self._series.dtype.ordered
 
     @property
     def codes(self):


Test patch:
Repository: modin-project/modin

Issue:
Add Modin ClassLogger to PandasDataframePartitionManager
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. What kind of performance improvements would you like to see with this new API?




Discussion:

Resolving patch:
diff --git a/modin/core/dataframe/pandas/partitioning/partition_manager.py b/modin/core/dataframe/pandas/partitioning/partition_manager.py
--- a/modin/core/dataframe/pandas/partitioning/partition_manager.py
+++ b/modin/core/dataframe/pandas/partitioning/partition_manager.py
@@ -28,6 +28,7 @@
 from modin.core.storage_formats.pandas.utils import compute_chunksize
 from modin.core.dataframe.pandas.utils import concatenate
 from modin.config import NPartitions, ProgressBar, BenchmarkMode, Engine, StorageFormat
+from modin.logging import ClassLogger
 
 import os
 
@@ -78,7 +79,7 @@ def wait(cls, *args, **kwargs):
     return func
 
 
-class PandasDataframePartitionManager(ABC):
+class PandasDataframePartitionManager(ClassLogger, ABC):
     """
     Base class for managing the dataframe data layout and operators across the distribution of partitions.
 


Test patch:
Repository: modin-project/modin

Issue:
REFACTOR: remove `_build_treereduce_func` call from `_compute_dtypes`
Place:

https://github.com/modin-project/modin/blob/a28e319d9756c9383afa9efd67e32eb04e6b848c/modin/core/dataframe/pandas/dataframe/dataframe.py#L312

`_build_treereduce_func` will be called in `tree_reduce` further.


Discussion:

Resolving patch:
diff --git a/modin/core/dataframe/pandas/dataframe/dataframe.py b/modin/core/dataframe/pandas/dataframe/dataframe.py
--- a/modin/core/dataframe/pandas/dataframe/dataframe.py
+++ b/modin/core/dataframe/pandas/dataframe/dataframe.py
@@ -309,11 +309,13 @@ def _compute_dtypes(self):
         def dtype_builder(df):
             return df.apply(lambda col: find_common_type(col.values), axis=0)
 
-        map_func = self._build_treereduce_func(0, lambda df: df.dtypes)
-        reduce_func = self._build_treereduce_func(0, dtype_builder)
         # For now we will use a pandas Series for the dtypes.
         if len(self.columns) > 0:
-            dtypes = self.tree_reduce(0, map_func, reduce_func).to_pandas().iloc[0]
+            dtypes = (
+                self.tree_reduce(0, lambda df: df.dtypes, dtype_builder)
+                .to_pandas()
+                .iloc[0]
+            )
         else:
             dtypes = pandas.Series([])
         # reset name to None because we use "__reduced__" internally


Test patch:
Repository: modin-project/modin

Issue:
PERF: don't use `copy=True` parameter for `concat` calls inside `to_pandas`
Places:
https://github.com/modin-project/modin/blob/a28e319d9756c9383afa9efd67e32eb04e6b848c/modin/core/dataframe/pandas/partitioning/partition_manager.py#L657

and

https://github.com/modin-project/modin/blob/a28e319d9756c9383afa9efd67e32eb04e6b848c/modin/core/dataframe/pandas/utils.py#L48


Discussion:

Resolving patch:
diff --git a/modin/core/dataframe/pandas/partitioning/partition_manager.py b/modin/core/dataframe/pandas/partitioning/partition_manager.py
--- a/modin/core/dataframe/pandas/partitioning/partition_manager.py
+++ b/modin/core/dataframe/pandas/partitioning/partition_manager.py
@@ -755,7 +755,7 @@ def is_part_empty(part):
             )
 
         df_rows = [
-            pandas.concat([part for part in row], axis=axis)
+            pandas.concat([part for part in row], axis=axis, copy=False)
             for row in retrieved_objects
             if not all(is_part_empty(part) for part in row)
         ]
diff --git a/modin/core/dataframe/pandas/utils.py b/modin/core/dataframe/pandas/utils.py
--- a/modin/core/dataframe/pandas/utils.py
+++ b/modin/core/dataframe/pandas/utils.py
@@ -43,4 +43,5 @@ def concatenate(dfs):
             df.isetitem(
                 i, pandas.Categorical(df.iloc[:, i], categories=union.categories)
             )
-    return pandas.concat(dfs)
+    # `ValueError: buffer source array is read-only` if copy==False
+    return pandas.concat(dfs, copy=True)


Test patch:
Repository: modin-project/modin

Issue:
PERF: we probably should compute `lengths` and `widths` in put method of Dask partition like Ray do
Place: https://github.com/modin-project/modin/blob/a28e319d9756c9383afa9efd67e32eb04e6b848c/modin/core/execution/dask/implementations/pandas_on_dask/partitioning/partition.py#L231

Ray method as a reference: https://github.com/modin-project/modin/blob/a28e319d9756c9383afa9efd67e32eb04e6b848c/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/partition.py#L252


Discussion:

Resolving patch:
diff --git a/modin/core/execution/dask/implementations/pandas_on_dask/partitioning/partition.py b/modin/core/execution/dask/implementations/pandas_on_dask/partitioning/partition.py
--- a/modin/core/execution/dask/implementations/pandas_on_dask/partitioning/partition.py
+++ b/modin/core/execution/dask/implementations/pandas_on_dask/partitioning/partition.py
@@ -228,7 +228,7 @@ def put(cls, obj):
         PandasOnDaskDataframePartition
             A new ``PandasOnDaskDataframePartition`` object.
         """
-        return cls(DaskWrapper.put(obj, hash=False))
+        return cls(DaskWrapper.put(obj, hash=False), len(obj.index), len(obj.columns))
 
     @classmethod
     def preprocess_func(cls, func):


Test patch: