internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: modin-project/modin

Issue:
Modin Categorical column categories are not correct.
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: MacOS Version 11.6
- **Modin version** (`modin.__version__`): 0.13.2
- **Python version**: 3.8.11
- **Code we can use to reproduce**:

```
import modin.pandas as pd
from modin.pandas import Categorical
 
d = {'a': [1, 2, 3], 'b': [4, 5, 6]} 
df = pd.DataFrame(data=d)
df['a'] = Categorical(df['a'], ordered=True)
df['a'].dtype
```

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->

The categories for Modin only include the first category and are not a complete list when compared to pandas. The output of `CategoricalDtype(categories=[1], ordered=True)` in the code above in Modin vs. `CategoricalDtype(categories=[1, 2, 3], ordered=True)` in pandas.

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:

Resolving patch:
diff --git a/modin/core/storage_formats/pandas/parsers.py b/modin/core/storage_formats/pandas/parsers.py
--- a/modin/core/storage_formats/pandas/parsers.py
+++ b/modin/core/storage_formats/pandas/parsers.py
@@ -123,13 +123,7 @@ def find_common_type_cat(types):
     """
     if all(isinstance(t, pandas.CategoricalDtype) for t in types):
         if all(t.ordered for t in types):
-            categories = np.unique([c for t in types for c in t.categories])[0]
-            if categories.ndim != 0:
-                categories = np.sort(categories)
-            else:
-                # Cannot use np.sort() on array with ndim == 0
-                # tolist() is called on scalar value and returns an int
-                categories = [categories.tolist()]
+            categories = np.sort(np.unique([c for t in types for c in t.categories]))
             return pandas.CategoricalDtype(
                 categories,
                 ordered=True,


Test patch:
diff --git a/modin/pandas/test/test_series.py b/modin/pandas/test/test_series.py
--- a/modin/pandas/test/test_series.py
+++ b/modin/pandas/test/test_series.py
@@ -2139,6 +2139,10 @@ def test_set_ordered_categorical_column():
     pdf["a"] = pandas.Categorical(pdf["a"], ordered=True)
     df_equals(mdf, pdf)
 
+    modin_categories = mdf["a"].dtype
+    pandas_categories = pdf["a"].dtype
+    assert modin_categories == pandas_categories
+
 
 @pytest.mark.parametrize("data", test_data_values, ids=test_data_keys)
 def test_lt(data):
Repository: modin-project/modin

Issue:
df.set_index("inexistent_col") sets the index instead of raising an exception
Modin

```python
import modin.pandas as pd

df = pd.DataFrame({'month': [1, 4, 7, 10],
    'year': [2012, 2014, 2013, 2014],
    'sale': [55, 40, 84, 31]})

df.set_index("inexistent_col")
      month  year  sale
sale
55        1  2012    55
40        4  2014    40
84        7  2013    84
31       10  2014    31
```

pandas

```python
import pandas

df = pandas.DataFrame({'month': [1, 4, 7, 10],
    'year': [2012, 2014, 2013, 2014],
    'sale': [55, 40, 84, 31]})

df.set_index("inexistent_col")
KeyError: "None of ['inexistent_col'] are in the columns"
```


Discussion:

Resolving patch:
diff --git a/modin/pandas/dataframe.py b/modin/pandas/dataframe.py
--- a/modin/pandas/dataframe.py
+++ b/modin/pandas/dataframe.py
@@ -1949,6 +1949,25 @@ def set_index(
                 return frame
             else:
                 return
+
+        missing = []
+        for col in keys:
+            # everything else gets tried as a key;
+            # see https://github.com/pandas-dev/pandas/issues/24969
+            try:
+                found = col in self.columns
+            except TypeError as err:
+                raise TypeError(
+                    'The parameter "keys" may be a column key, one-dimensional '
+                    + "array, or a list containing only valid column keys and "
+                    + f"one-dimensional arrays. Received column of type {type(col)}"
+                ) from err
+            else:
+                if not found:
+                    missing.append(col)
+        if missing:
+            raise KeyError(f"None of {missing} are in the columns")
+
         new_query_compiler = self._query_compiler.set_index_from_columns(
             keys, drop=drop, append=append
         )


Test patch:
diff --git a/modin/pandas/test/dataframe/test_indexing.py b/modin/pandas/test/dataframe/test_indexing.py
--- a/modin/pandas/test/dataframe/test_indexing.py
+++ b/modin/pandas/test/dataframe/test_indexing.py
@@ -283,6 +283,9 @@ def test_set_index(data):
     pandas_result = pandas_df.set_index([pandas_df.index, pandas_df.columns[0]])
     df_equals(modin_result, pandas_result)
 
+    # test for the case from https://github.com/modin-project/modin/issues/4308
+    eval_general(modin_df, pandas_df, lambda df: df.set_index("inexistent_col"))
+
 
 @pytest.mark.gpu
 @pytest.mark.parametrize("data", test_data_values, ids=test_data_keys)
Repository: modin-project/modin

Issue:
REFACTOR: remove `to_pandas`, `to_numpy` function duplication in `QueryCompiler` hierarchy



Discussion:

Resolving patch:
diff --git a/modin/core/storage_formats/pyarrow/query_compiler.py b/modin/core/storage_formats/pyarrow/query_compiler.py
--- a/modin/core/storage_formats/pyarrow/query_compiler.py
+++ b/modin/core/storage_formats/pyarrow/query_compiler.py
@@ -288,9 +288,3 @@ def arrow_index_extraction(table, axis):
             old_blocks=old_blocks,
         )
         return index_obj[new_indices] if compute_diff else new_indices
-
-    def to_pandas(self):
-        return self._modin_frame.to_pandas()
-
-    def to_numpy(self, **kwargs):
-        return self._modin_frame.to_numpy(**kwargs)


Test patch:
Repository: modin-project/modin

Issue:
Upgrade pandas to 1.4.2



Discussion:

Resolving patch:
diff --git a/modin/pandas/__init__.py b/modin/pandas/__init__.py
--- a/modin/pandas/__init__.py
+++ b/modin/pandas/__init__.py
@@ -14,7 +14,7 @@
 import pandas
 import warnings
 
-__pandas_version__ = "1.4.1"
+__pandas_version__ = "1.4.2"
 
 if pandas.__version__ != __pandas_version__:
     warnings.warn(
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -22,7 +22,7 @@
     url="https://github.com/modin-project/modin",
     long_description=long_description,
     long_description_content_type="text/markdown",
-    install_requires=["pandas==1.4.1", "packaging", "numpy>=1.18.5", "fsspec"],
+    install_requires=["pandas==1.4.2", "packaging", "numpy>=1.18.5", "fsspec"],
     extras_require={
         # can be installed by pip install modin[dask]
         "dask": dask_deps,


Test patch:
Repository: modin-project/modin

Issue:
Update Ray usage to be in line with new Ray upgrades and requirements
E.g. Ray removed Redis from required dependencies and no longer starts Redis by default.


Discussion:
One of the failing jobs is https://github.com/modin-project/modin/runs/6035629871?check_suite_focus=true.
@modin-project/modin-core, @modin-project/modin-contributors, could anyone take a look at this please?

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -5,7 +5,8 @@
     long_description = fh.read()
 
 dask_deps = ["dask>=2.22.0,<2022.2.0", "distributed>=2.22.0,<2022.2.0"]
-ray_deps = ["ray[default]>=1.4.0", "pyarrow>=4.0.1"]
+# TODO: remove redis dependency when resolving GH#4398
+ray_deps = ["ray[default]>=1.4.0", "pyarrow>=4.0.1", "redis>=3.5.0,<4.0.0"]
 remote_deps = ["rpyc==4.1.5", "cloudpickle", "boto3"]
 spreadsheet_deps = ["modin-spreadsheet>=0.1.0"]
 sql_deps = ["dfsql>=0.4.2", "pyparsing<=2.4.7"]


Test patch:
Repository: modin-project/modin

Issue:
Modin `eval_insert` util function doesn't actually check results of `insert` operation
If parameter `__inplace__=True` is not passed into `eval_general`, then two `None` values will always be compared.

Link: https://github.com/modin-project/modin/blob/0ab04e104a60419c464fc1833a13f7f2b92e7983/modin/pandas/test/dataframe/test_map_metadata.py#L64


Discussion:

Resolving patch:
diff --git a/modin/pandas/dataframe.py b/modin/pandas/dataframe.py
--- a/modin/pandas/dataframe.py
+++ b/modin/pandas/dataframe.py
@@ -1161,7 +1161,11 @@ def insert(self, loc, column, value, allow_duplicates=False):  # noqa: PR01, D20
                 and not isinstance(value, (pandas.Series, Series))
                 and len(value) != len(self.index)
             ):
-                raise ValueError("Length of values does not match length of index")
+                raise ValueError(
+                    "Length of values ({}) does not match length of index ({})".format(
+                        len(value), len(self.index)
+                    )
+                )
             if not allow_duplicates and column in self.columns:
                 raise ValueError(f"cannot insert {column}, already exists")
             if not -len(self.columns) <= loc <= len(self.columns):


Test patch:
diff --git a/modin/pandas/test/dataframe/test_map_metadata.py b/modin/pandas/test/dataframe/test_map_metadata.py
--- a/modin/pandas/test/dataframe/test_map_metadata.py
+++ b/modin/pandas/test/dataframe/test_map_metadata.py
@@ -71,6 +71,7 @@ def eval_insert(modin_df, pandas_df, **kwargs):
         modin_df,
         pandas_df,
         operation=lambda df, **kwargs: df.insert(**kwargs),
+        __inplace__=True,
         **_kwargs,
     )
 
@@ -1049,7 +1050,6 @@ def comparator(df1, df2):
             loc=0,
             col=f"test_col{idx}",
             value=value,
-            __inplace__=True,
             comparator=lambda df1, df2: comparator(df1, df2),
         )
Repository: modin-project/modin

Issue:
Missing `f` prefix on f-strings
Some strings looks like they're meant to be f-strings but are missing the `f` prefix meaning variable interpolation won't happen.

https://github.com/modin-project/modin/blob/0ab04e104a60419c464fc1833a13f7f2b92e7983/asv_bench/benchmarks/benchmarks.py#L692
https://github.com/modin-project/modin/blob/0ab04e104a60419c464fc1833a13f7f2b92e7983/modin/core/dataframe/pandas/dataframe/dataframe.py#L326
https://github.com/modin-project/modin/blob/0ab04e104a60419c464fc1833a13f7f2b92e7983/modin/config/envvars.py#L96
https://github.com/modin-project/modin/blob/0ab04e104a60419c464fc1833a13f7f2b92e7983/modin/config/envvars.py#L111

I found this issue automatically. I'm a bot. Beep Boop 🦊. See other issues I found in your repo [here](https://codereview.doctor/modin-project/modin)


Discussion:

Resolving patch:
diff --git a/asv_bench/benchmarks/benchmarks.py b/asv_bench/benchmarks/benchmarks.py
--- a/asv_bench/benchmarks/benchmarks.py
+++ b/asv_bench/benchmarks/benchmarks.py
@@ -689,7 +689,7 @@ def setup(self, shape, dtype, astype_ncolumns):
         elif astype_ncolumns == "one":
             self.astype_arg = {"col1": dtype}
         else:
-            raise ValueError("astype_ncolumns: {astype_ncolumns} isn't supported")
+            raise ValueError(f"astype_ncolumns: {astype_ncolumns} isn't supported")
 
     def time_astype(self, shape, dtype, astype_ncolumns):
         execute(self.df.astype(self.astype_arg))
diff --git a/modin/config/envvars.py b/modin/config/envvars.py
--- a/modin/config/envvars.py
+++ b/modin/config/envvars.py
@@ -93,7 +93,7 @@ def _get_default(cls):
         else:
             if version.parse(ray.__version__) < MIN_RAY_VERSION:
                 raise ImportError(
-                    "Please `pip install modin[ray]` to install compatible Ray version (>={MIN_RAY_VERSION})."
+                    f"Please `pip install modin[ray]` to install compatible Ray version (>={MIN_RAY_VERSION})."
                 )
             return "Ray"
         try:
@@ -108,7 +108,7 @@ def _get_default(cls):
                 or version.parse(distributed.__version__) < MIN_DASK_VERSION
             ):
                 raise ImportError(
-                    "Please `pip install modin[dask]` to install compatible Dask version (>={MIN_DASK_VERSION})."
+                    f"Please `pip install modin[dask]` to install compatible Dask version (>={MIN_DASK_VERSION})."
                 )
             return "Dask"
         try:
diff --git a/modin/core/dataframe/pandas/dataframe/dataframe.py b/modin/core/dataframe/pandas/dataframe/dataframe.py
--- a/modin/core/dataframe/pandas/dataframe/dataframe.py
+++ b/modin/core/dataframe/pandas/dataframe/dataframe.py
@@ -323,7 +323,7 @@ def _validate_set_axis(self, new_labels, old_labels):
         if old_len != new_len:
             raise ValueError(
                 f"Length mismatch: Expected axis has {old_len} elements, "
-                + "new values have {new_len} elements"
+                + f"new values have {new_len} elements"
             )
         return new_labels
 


Test patch:
Repository: modin-project/modin

Issue:
Modin `DataFrameGroupBy.pct_change()` is missing all parameters
### System information
ray 1.9.2
modin 0.12.0
pandas 1.3.4

### Describe the problem
In Modin, calling `DataFrameGroupBy.pct_change()` is missing all parameters.

To Reproduce in Modin:

```
import modin.pandas as pd
df = pd.DataFrame({'a':[1,2]})
df.groupby('a').pct_change(periods=1)
```

The Call to pct_change produces the following error message:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: pct_change() got an unexpected keyword argument 'periods'
```

Expected result is:

```
Empty DataFrame
Columns: []
Index: [0, 1]
```

Upon using the inspect library, it’s clear that all parameters are missing from this API

```
>>> import modin.pandas as pd
>>> from inspect import signature
>>>>signature(pd.DataFrame({'a':[1,2]}).groupby('a').pct_change)
<Signature ()>
```

Signature for Pandas is:

```
>>> import pandas as pd
>>> from inspect import signature
>>> signature(pd.DataFrame({'a':[1,2]}).groupby('a').pct_change)
<Signature (periods=1, fill_method='pad', limit=None, freq=None, axis=0)>
```


Discussion:
@c3-andrewfitch thank you for reporting this bug! I can reproduce it at Modin commit 5825856a4ee19ff0dd5d1beaf6090bf601d0b411. I'll start working on the fix.

Resolving patch:
diff --git a/modin/pandas/groupby.py b/modin/pandas/groupby.py
--- a/modin/pandas/groupby.py
+++ b/modin/pandas/groupby.py
@@ -293,8 +293,16 @@ def indices(self):
         self._indices_cache = self._compute_index_grouped(numerical=True)
         return self._indices_cache
 
-    def pct_change(self):
-        return self._default_to_pandas(lambda df: df.pct_change())
+    def pct_change(self, periods=1, fill_method="ffill", limit=None, freq=None, axis=0):
+        return self._default_to_pandas(
+            lambda df: df.pct_change(
+                periods=periods,
+                fill_method=fill_method,
+                limit=limit,
+                freq=freq,
+                axis=axis,
+            )
+        )
 
     def filter(self, func, dropna=True, *args, **kwargs):
         return self._default_to_pandas(


Test patch:
diff --git a/modin/pandas/test/test_groupby.py b/modin/pandas/test/test_groupby.py
--- a/modin/pandas/test/test_groupby.py
+++ b/modin/pandas/test/test_groupby.py
@@ -362,7 +362,9 @@ def maybe_get_columns(df, by):
     eval_general(
         modin_groupby,
         pandas_groupby,
-        lambda df: df.pct_change(),
+        lambda df: df.pct_change(
+            periods=2, fill_method="pad", limit=1, freq=None, axis=1
+        ),
         modin_df_almost_equals_pandas,
         is_default=True,
     )
Repository: modin-project/modin

Issue:
Add support of MultiIndex in `reindex` method
Currently the method defaults to pandas.

https://github.com/modin-project/modin/blob/e073c4f71575cc837b60de4e463d9717633170ca/modin/pandas/base.py#L2228


Discussion:

Resolving patch:
diff --git a/modin/core/storage_formats/pandas/query_compiler.py b/modin/core/storage_formats/pandas/query_compiler.py
--- a/modin/core/storage_formats/pandas/query_compiler.py
+++ b/modin/core/storage_formats/pandas/query_compiler.py
@@ -528,8 +528,8 @@ def map_func(left, right=right, kwargs=kwargs):
 
     # Reindex/reset_index (may shuffle data)
     def reindex(self, axis, labels, **kwargs):
-        new_index = self.index if axis else labels
-        new_columns = labels if axis else self.columns
+        new_index, _ = (self.index, None) if axis else self.index.reindex(labels)
+        new_columns, _ = self.columns.reindex(labels) if axis else (self.columns, None)
         new_modin_frame = self._modin_frame.apply_full_axis(
             axis,
             lambda df: df.reindex(labels=labels, axis=axis, **kwargs),
diff --git a/modin/pandas/base.py b/modin/pandas/base.py
--- a/modin/pandas/base.py
+++ b/modin/pandas/base.py
@@ -2202,24 +2202,9 @@ def reindex(
         """
         Conform `BasePandasDataset` to new index with optional filling logic.
         """
-        if (
-            kwargs.get("level") is not None
-            or (index is not None and self._query_compiler.has_multiindex())
-            or (columns is not None and self._query_compiler.has_multiindex(axis=1))
-        ):
-            if index is not None:
-                kwargs["index"] = index
-            if columns is not None:
-                kwargs["columns"] = columns
-            return self._default_to_pandas("reindex", copy=copy, **kwargs)
-
         new_query_compiler = None
         if index is not None:
-            if not isinstance(index, pandas.Index):
-                index = self._copy_index_metadata(
-                    source=self.index, destination=self._ensure_index(index, axis=0)
-                )
-            if not index.equals(self.index):
+            if not isinstance(index, pandas.Index) or not index.equals(self.index):
                 new_query_compiler = self._query_compiler.reindex(
                     axis=0, labels=index, **kwargs
                 )
@@ -2227,11 +2212,7 @@ def reindex(
             new_query_compiler = self._query_compiler
         final_query_compiler = None
         if columns is not None:
-            if not isinstance(columns, pandas.Index):
-                columns = self._copy_index_metadata(
-                    source=self.columns, destination=self._ensure_index(columns, axis=1)
-                )
-            if not columns.equals(self.columns):
+            if not isinstance(index, pandas.Index) or not columns.equals(self.columns):
                 final_query_compiler = new_query_compiler.reindex(
                     axis=1, labels=columns, **kwargs
                 )


Test patch:
Repository: modin-project/modin

Issue:
Benchmark Mode blocks on computation so all computation occurs on one node
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:
- **Modin version** (`modin.__version__`):
- **Python version**:
- **Code we can use to reproduce**:

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->
Since Benchmark Mode calls `wait` on each partition sequentially, computation that can be done in parallel ends up happening sequentially on a single node. We need to instead collect all the OID's of all partitions, and wait on all of them together - that way each of the tasks is kicked off in parallel, and computation happens in parallel.
### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:
I just want to clarify that the bug applies only when we have a function like a transpose or a reindex that adds a remote function to each partition's call queue, but doesn't trigger the function's execution. In that case, benchmark mode will [trigger](https://github.com/modin-project/modin/blob/c17dde71ae8114723a13279e36bf5532dcba328a/modin/core/dataframe/pandas/partitioning/partition_manager.py#L66) all of the executions serially instead of in parallel.

Resolving patch:
diff --git a/modin/core/dataframe/pandas/partitioning/partition_manager.py b/modin/core/dataframe/pandas/partitioning/partition_manager.py
--- a/modin/core/dataframe/pandas/partitioning/partition_manager.py
+++ b/modin/core/dataframe/pandas/partitioning/partition_manager.py
@@ -60,9 +60,18 @@ def wait(*args, **kwargs):
                 partitions = result[0]
             else:
                 partitions = result
+            # When partitions have a deferred call queue, calling
+            # partition.wait() on each partition serially will serially kick
+            # off each deferred computation and wait for each partition to
+            # finish before kicking off the next one. Instead, we want to
+            # serially kick off all the deferred computations so that they can
+            # all run asynchronously, then wait on all the results.
+            [part.drain_call_queue() for part in partitions.flatten()]
             # need to go through all the values of the map iterator
             # since `wait` does not return anything, we need to explicitly add
             # the return `True` value from the lambda
+            # TODO(https://github.com/modin-project/modin/issues/4491): Wait
+            # for all the partitions in parallel.
             all(map(lambda partition: partition.wait() or True, partitions.flatten()))
             return result
 


Test patch:
Repository: modin-project/modin

Issue:
Replace bug_reports@modin.org with a link to create a new issue on GitHub
- **Modin version**: cf0eb393daa41abdd8cdf32b52ceee938cdcbe1a

### Describe the problem
There are a few links to bug_reports@modin.org in the source, e.g. [here](https://github.com/modin-project/modin/blob/6d419655bba85b3b0f2b827624788f62d7c729cd/modin/error_message.py#L62). We should replace those with a link to the GitHub issues and a message saying to search the existing GitHub issues and file a new one if the bug hasn't already been reported.




Discussion:

Resolving patch:
diff --git a/modin/core/io/text/excel_dispatcher.py b/modin/core/io/text/excel_dispatcher.py
--- a/modin/core/io/text/excel_dispatcher.py
+++ b/modin/core/io/text/excel_dispatcher.py
@@ -73,8 +73,10 @@ def _read(cls, io, **kwargs):
             return cls.single_worker_read(io, **kwargs)
 
         warnings.warn(
-            "Parallel `read_excel` is a new feature! Please email "
-            + "bug_reports@modin.org if you run into any problems."
+            "Parallel `read_excel` is a new feature! If you run into any "
+            + "problems, please visit https://github.com/modin-project/modin/issues. "
+            + "If you find a new issue and can't file it on GitHub, please "
+            + "email bug_reports@modin.org."
         )
 
         # NOTE: ExcelReader() in read-only mode does not close file handle by itself
diff --git a/modin/error_message.py b/modin/error_message.py
--- a/modin/error_message.py
+++ b/modin/error_message.py
@@ -60,8 +60,10 @@ def catch_bugs_and_request_email(cls, failure_condition, extra_log=""):
         if failure_condition:
             raise Exception(
                 "Internal Error. "
-                + "Please email bug_reports@modin.org with the traceback and command that"
-                + f" caused this error.\n{extra_log}"
+                + "Please visit https://github.com/modin-project/modin/issues "
+                + "to file an issue with the traceback and the command that "
+                + "caused this error. If you can't file a GitHub issue, "
+                + f"please email bug_reports@modin.org.\n{extra_log}"
             )
 
     @classmethod


Test patch: