internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: modin-project/modin

Issue:
``Series.pop()`` doesn`t work correctly at MODIN_STORAGE_FORMAT=omnisci
When I run:
```
import modin.pandas as pd
from modin.pandas.test.utils import test_data_values
import modin.config as cfg
cfg.StorageFormat.put('omnisci')

data = test_data_values[0]
modin_series = pd.Series(data[next(iter(data.keys()))])
for key in modin_series.keys():
    modin_series.pop(key)
```
I have error:
```
AssertionError: Only column drop is supported
```


Discussion:
@modin-project/modin-omnisci  could someone please check if this still happens?

Resolving patch:
diff --git a/modin/experimental/core/storage_formats/hdk/query_compiler.py b/modin/experimental/core/storage_formats/hdk/query_compiler.py
--- a/modin/experimental/core/storage_formats/hdk/query_compiler.py
+++ b/modin/experimental/core/storage_formats/hdk/query_compiler.py
@@ -556,7 +556,8 @@ def concat(self, axis, other, **kwargs):
         return self.__constructor__(new_modin_frame)
 
     def drop(self, index=None, columns=None):
-        assert index is None, "Only column drop is supported"
+        if index is not None:
+            raise NotImplementedError("Row drop")
         return self.__constructor__(
             self._modin_frame.take_2d_labels_or_positional(
                 row_labels=index, col_labels=self.columns.drop(columns)


Test patch:
diff --git a/modin/experimental/core/execution/native/implementations/hdk_on_native/test/test_dataframe.py b/modin/experimental/core/execution/native/implementations/hdk_on_native/test/test_dataframe.py
--- a/modin/experimental/core/execution/native/implementations/hdk_on_native/test/test_dataframe.py
+++ b/modin/experimental/core/execution/native/implementations/hdk_on_native/test/test_dataframe.py
@@ -2145,6 +2145,28 @@ def drop_rename_concat(df1, df2, lib, **kwargs):
             force_arrow_execute=True,
         )
 
+    def test_drop_row(self):
+        def drop_row(df, **kwargs):
+            return df.drop(labels=1)
+
+        run_and_compare(
+            drop_row,
+            data=self.data1,
+            force_lazy=False,
+        )
+
+    def test_series_pop(self):
+        def pop(df, **kwargs):
+            col = df["a"]
+            col.pop(0)
+            return col
+
+        run_and_compare(
+            pop,
+            data=self.data1,
+            force_lazy=False,
+        )
+
     def test_empty_transform(self):
         def apply(df, **kwargs):
             return df + 1
Repository: modin-project/modin

Issue:
PERF: `concatenate` on wide DataFrames has poor performance
Our current implementation of `concatenate` (used in `to_pandas()` to combine row partitions) tries to iterate over all columns to find which ones are categorical types. This approach results in very poor performance for wide DataFrames:

```python
In [1]: import modin.pandas as pd

In [2]: import numpy as np

In [3]: df = pd.DataFrame(np.random.randint(0,100, size=(1,100000)))
UserWarning: Ray execution environment not yet initialized. Initializing...
To remove this warning, run the following python code before doing dataframe operations:

    import ray
    ray.init(runtime_env={'env_vars': {'__MODIN_AUTOIMPORT_PANDAS__': '1'}})

UserWarning: Distributing <class 'numpy.ndarray'> object. This may take some time.

In [4]: %time df._to_pandas()
CPU times: user 1min 47s, sys: 984 ms, total: 1min 48s
Wall time: 1min 48s
Out[4]: 
   0      1      2      3      4      5      6      7      8      9      10     11     12     ...  99987  99988  99989  99990  99991  99992  99993  99994  99995  99996  99997  99998  99999
0     38     89     98     43     25     34     63     32     32     71     34     82     15  ...     90     40      0     41     12      6     24      1     45     11     74     17     28

[1 rows x 100000 columns]
```
A cleaner solution proposed by @mvashishtha would use `select_dtypes` to get columns that are the category dtype and then iterate over those. 


Discussion:
@pyrito thanks for filing this. While we're working on this, we should investigate whether we are really casting correctly to the union categorical type: https://github.com/mvashishtha/modin/commit/46cce7c021eea2e4abd20e2ca4dcd18cecc0fcb3#r80443299

Resolving patch:
diff --git a/modin/core/dataframe/pandas/utils.py b/modin/core/dataframe/pandas/utils.py
--- a/modin/core/dataframe/pandas/utils.py
+++ b/modin/core/dataframe/pandas/utils.py
@@ -36,9 +36,7 @@ def concatenate(dfs):
     """
     for df in dfs:
         assert df.columns.equals(dfs[0].columns)
-    for i in range(len(dfs[0].columns)):
-        if dfs[0].dtypes.iloc[i].name != "category":
-            continue
+    for i in dfs[0].columns.get_indexer_for(dfs[0].select_dtypes("category").columns):
         columns = [df.iloc[:, i] for df in dfs]
         union = union_categoricals(columns)
         for df in dfs:


Test patch:
Repository: modin-project/modin

Issue:
PERF: `_compute_axis_labels_and_lengths` does extra work if the index is already known
The function returns indices from remote functions (on the basis of which lengths are calculated), basically the time spent on their deserialization / serialization is much more than just using numbers describing lengths and widths.

In some case we able to avoid this.


Discussion:

Resolving patch:
diff --git a/modin/core/dataframe/pandas/dataframe/dataframe.py b/modin/core/dataframe/pandas/dataframe/dataframe.py
--- a/modin/core/dataframe/pandas/dataframe/dataframe.py
+++ b/modin/core/dataframe/pandas/dataframe/dataframe.py
@@ -246,13 +246,16 @@ def _row_lengths(self):
             A list of row partitions lengths.
         """
         if self._row_lengths_cache is None:
-            if len(self._partitions) > 0:
-                (
-                    index,
-                    self._row_lengths_cache,
-                ) = self._compute_axis_labels_and_lengths(0)
-                if self._index_cache is None:
-                    self._index_cache = index
+            if len(self._partitions.T) > 0:
+                row_parts = self._partitions.T[0]
+                if self._index_cache is not None:
+                    # do not do extra work to get an index that is already known
+                    self._row_lengths_cache = [part.length() for part in row_parts]
+                else:
+                    (
+                        self._index_cache,
+                        self._row_lengths_cache,
+                    ) = self._compute_axis_labels_and_lengths(0)
             else:
                 self._row_lengths_cache = []
         return self._row_lengths_cache
@@ -269,12 +272,15 @@ def _column_widths(self):
         """
         if self._column_widths_cache is None:
             if len(self._partitions) > 0:
-                (
-                    columns,
-                    self._column_widths_cache,
-                ) = self._compute_axis_labels_and_lengths(1)
-                if self._columns_cache is None:
-                    self._columns_cache = columns
+                col_parts = self._partitions[0]
+                if self._columns_cache is not None:
+                    # do not do extra work to get columns that is already known
+                    self._column_widths_cache = [part.width() for part in col_parts]
+                else:
+                    (
+                        self._columns_cache,
+                        self._column_widths_cache,
+                    ) = self._compute_axis_labels_and_lengths(1)
             else:
                 self._column_widths_cache = []
         return self._column_widths_cache


Test patch:
Repository: modin-project/modin

Issue:
Rewrite Pandas version mismatch UserWarning
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. What kind of performance improvements would you like to see with this new API?
I'd like to reformat the pandas version mismatch warning. I think it can be kind of confusing, since it seems to indicate Modin's version as the pandas version that is supported. E.g.:
```python
UserWarning: The pandas version installed 1.4.3 does not match the supported pandas version in Modin 1.4.2. This may cause undesired side effects!
```
This seems to imply that Modin is currently version 1.4.2


Discussion:
Maybe adding braces and some rephrasing would work?
```
UserWarning: The pandas version installed (1.4.3) does not match the pandas version Modin supports (1.4.2). This may cause undesired side effects!
```
> UserWarning: The pandas version installed (1.4.3) does not match the pandas version Modin supports (1.4.2). This may cause undesired side effects!

This LGTM.
This seems like something that would be "low effort, high impact", so we should just get it merged ASAP.

Resolving patch:
diff --git a/modin/pandas/__init__.py b/modin/pandas/__init__.py
--- a/modin/pandas/__init__.py
+++ b/modin/pandas/__init__.py
@@ -21,8 +21,9 @@
 
     if pandas.__version__ != __pandas_version__:
         warnings.warn(
-            f"The pandas version installed {pandas.__version__} does not match the supported pandas version in"
-            + f" Modin {__pandas_version__} compatibility mode. This may cause undesired side effects!"
+            f"The pandas version installed ({pandas.__version__}) does not match the pandas version"
+            + f" Modin supports ({__pandas_version__}) in Python 3.6 legacy compatibility mode."
+            + " This may cause undesired side effects!"
         )
     else:
         warnings.warn(
@@ -33,8 +34,8 @@
 
     if pandas.__version__ != __pandas_version__:
         warnings.warn(
-            f"The pandas version installed {pandas.__version__} does not match the supported pandas version in"
-            + f" Modin {__pandas_version__}. This may cause undesired side effects!"
+            f"The pandas version installed ({pandas.__version__}) does not match the pandas version"
+            + f" Modin supports ({__pandas_version__}). This may cause undesired side effects!"
         )
 
 with warnings.catch_warnings():


Test patch:
Repository: modin-project/modin

Issue:
REFACTOR: rewrite base implementations of a partition' `width/length`
These functions can be reused for `pandas_on_python` execution.


Discussion:

Resolving patch:
diff --git a/modin/core/dataframe/pandas/partitioning/partition.py b/modin/core/dataframe/pandas/partitioning/partition.py
--- a/modin/core/dataframe/pandas/partitioning/partition.py
+++ b/modin/core/dataframe/pandas/partitioning/partition.py
@@ -298,10 +298,7 @@ def length(self):
             The length of the object.
         """
         if self._length_cache is None:
-            cls = type(self)
-            func = cls._length_extraction_fn()
-            preprocessed_func = cls.preprocess_func(func)
-            self._length_cache = self.apply(preprocessed_func)
+            self._length_cache = self.apply(self._length_extraction_fn()).get()
         return self._length_cache
 
     def width(self):
@@ -314,10 +311,7 @@ def width(self):
             The width of the object.
         """
         if self._width_cache is None:
-            cls = type(self)
-            func = cls._width_extraction_fn()
-            preprocessed_func = cls.preprocess_func(func)
-            self._width_cache = self.apply(preprocessed_func)
+            self._width_cache = self.apply(self._width_extraction_fn()).get()
         return self._width_cache
 
     @classmethod
diff --git a/modin/core/execution/python/implementations/pandas_on_python/partitioning/partition.py b/modin/core/execution/python/implementations/pandas_on_python/partitioning/partition.py
--- a/modin/core/execution/python/implementations/pandas_on_python/partitioning/partition.py
+++ b/modin/core/execution/python/implementations/pandas_on_python/partitioning/partition.py
@@ -63,7 +63,7 @@ def get(self):
         Since this object is a simple wrapper, just return the copy of data.
         """
         self.drain_call_queue()
-        return self._data.copy()
+        return self._data.copy() if hasattr(self._data, "copy") else self._data
 
     def apply(self, func, *args, **kwargs):
         """
@@ -195,29 +195,3 @@ def preprocess_func(cls, func):
         `func` will be returned.
         """
         return func
-
-    def length(self):
-        """
-        Get the length of the object wrapped by this partition.
-
-        Returns
-        -------
-        int
-            The length of the object.
-        """
-        if self._length_cache is None:
-            self._length_cache = self.apply(self._length_extraction_fn())._data
-        return self._length_cache
-
-    def width(self):
-        """
-        Get the width of the object wrapped by the partition.
-
-        Returns
-        -------
-        int
-            The width of the object.
-        """
-        if self._width_cache is None:
-            self._width_cache = self.apply(self._width_extraction_fn())._data
-        return self._width_cache


Test patch:
Repository: modin-project/modin

Issue:
PERF: doesn't need to call `apply` in virtual partition' `drain_call_queue` if `call_queue` is empty



Discussion:

Resolving patch:
diff --git a/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/virtual_partition.py b/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/virtual_partition.py
--- a/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/virtual_partition.py
+++ b/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/virtual_partition.py
@@ -423,6 +423,10 @@ def drain_call_queue(self, num_splits=None):
         num_splits : int, default: None
             The number of times to split the result object.
         """
+        if len(self.call_queue) == 0:
+            # this implicitly calls `drain_call_queue`
+            _ = self.list_of_blocks
+            return
         drained = super(PandasOnRayDataframeVirtualPartition, self).apply(
             _DRAIN, num_splits=num_splits, call_queue=self.call_queue
         )


Test patch:
Repository: modin-project/modin

Issue:
REFACTOR: use `finalize` method instead of list comprehension + drain_call_queue



Discussion:

Resolving patch:
diff --git a/modin/core/dataframe/pandas/partitioning/partition_manager.py b/modin/core/dataframe/pandas/partitioning/partition_manager.py
--- a/modin/core/dataframe/pandas/partitioning/partition_manager.py
+++ b/modin/core/dataframe/pandas/partitioning/partition_manager.py
@@ -67,7 +67,7 @@ def wait(cls, *args, **kwargs):
             # finish before kicking off the next one. Instead, we want to
             # serially kick off all the deferred computations so that they can
             # all run asynchronously, then wait on all the results.
-            [part.drain_call_queue() for part in partitions.flatten()]
+            cls.finalize(partitions)
             # The partition manager invokes the relevant .wait() method under
             # the hood, which should wait in parallel for all computations to finish
             # (We can't just add a `cls` argument to this `wait` function, since doing so
@@ -1316,7 +1316,6 @@ def get_right_block(right_partitions, row_idx, col_idx):
         )
 
     @classmethod
-    @wait_computations_if_benchmark_mode
     def finalize(cls, partitions):
         """
         Perform all deferred calls on partitions.


Test patch:
Repository: modin-project/modin

Issue:
PERF: `reset_index` shouldn't trigger index materialization in case when `drop==True` and `level==None`
RangeIndex can be built on the basis of `_row_lengths`. Materializing lengths is much cheaper than materializing indices.


Discussion:

Resolving patch:
diff --git a/modin/core/storage_formats/pandas/query_compiler.py b/modin/core/storage_formats/pandas/query_compiler.py
--- a/modin/core/storage_formats/pandas/query_compiler.py
+++ b/modin/core/storage_formats/pandas/query_compiler.py
@@ -550,7 +550,7 @@ def reset_index(self, **kwargs):
                     if len(level) < self.index.nlevels
                     else pandas.RangeIndex(len(self.index))
                 )
-        else:
+        elif not drop:
             uniq_sorted_level = list(range(self.index.nlevels))
 
         if not drop:
@@ -627,7 +627,8 @@ def reset_index(self, **kwargs):
         else:
             new_self = self.copy()
             new_self.index = (
-                pandas.RangeIndex(len(new_self.index))
+                # Cheaper to compute row lengths than index
+                pandas.RangeIndex(sum(new_self._modin_frame._row_lengths))
                 if new_index is None
                 else new_index
             )


Test patch:
Repository: modin-project/modin

Issue:
PERF: don't use `_compute_axis_labels_and_lengths` for computing `_row_lengths/_column_widths`
It will be much faster here to call `length/width` functions on partitions.

I can think of an arbitrarily large multi-index that needs to be serialized/deserialized instead of serializing/deserializing just ints.


Discussion:

Resolving patch:
diff --git a/modin/core/dataframe/pandas/dataframe/dataframe.py b/modin/core/dataframe/pandas/dataframe/dataframe.py
--- a/modin/core/dataframe/pandas/dataframe/dataframe.py
+++ b/modin/core/dataframe/pandas/dataframe/dataframe.py
@@ -248,14 +248,7 @@ def row_lengths(self):
         if self._row_lengths_cache is None:
             if len(self._partitions.T) > 0:
                 row_parts = self._partitions.T[0]
-                if self._index_cache is not None:
-                    # do not do extra work to get an index that is already known
-                    self._row_lengths_cache = [part.length() for part in row_parts]
-                else:
-                    (
-                        self._index_cache,
-                        self._row_lengths_cache,
-                    ) = self._compute_axis_labels_and_lengths(0)
+                self._row_lengths_cache = [part.length() for part in row_parts]
             else:
                 self._row_lengths_cache = []
         return self._row_lengths_cache
@@ -273,14 +266,7 @@ def column_widths(self):
         if self._column_widths_cache is None:
             if len(self._partitions) > 0:
                 col_parts = self._partitions[0]
-                if self._columns_cache is not None:
-                    # do not do extra work to get columns that is already known
-                    self._column_widths_cache = [part.width() for part in col_parts]
-                else:
-                    (
-                        self._columns_cache,
-                        self._column_widths_cache,
-                    ) = self._compute_axis_labels_and_lengths(1)
+                self._column_widths_cache = [part.width() for part in col_parts]
             else:
                 self._column_widths_cache = []
         return self._column_widths_cache


Test patch:
Repository: modin-project/modin

Issue:
PERF: use cache for widths/lengths/index/columns if possible
This improves asynchronous execution.


Discussion:

Resolving patch:
diff --git a/modin/core/dataframe/pandas/dataframe/dataframe.py b/modin/core/dataframe/pandas/dataframe/dataframe.py
--- a/modin/core/dataframe/pandas/dataframe/dataframe.py
+++ b/modin/core/dataframe/pandas/dataframe/dataframe.py
@@ -1172,10 +1172,10 @@ def astype_builder(df):
         )
         return self.__constructor__(
             new_frame,
-            self.index,
-            self.columns,
-            self.row_lengths,
-            self.column_widths,
+            self._index_cache,
+            self._columns_cache,
+            self._row_lengths_cache,
+            self._column_widths_cache,
             new_dtypes,
         )
 
@@ -1689,8 +1689,8 @@ def map(self, func: Callable, dtypes: Optional[str] = None) -> "PandasDataframe"
             )
         return self.__constructor__(
             new_partitions,
-            self.axes[0],
-            self.axes[1],
+            self._index_cache,
+            self._columns_cache,
             self._row_lengths_cache,
             self._column_widths_cache,
             dtypes=dtypes,
@@ -1757,10 +1757,10 @@ def fold(self, axis, func):
         )
         return self.__constructor__(
             new_partitions,
-            self.index,
-            self.columns,
-            self.row_lengths,
-            self.column_widths,
+            self._index_cache,
+            self._columns_cache,
+            self._row_lengths_cache,
+            self._column_widths_cache,
         )
 
     def infer_objects(self) -> "PandasDataframe":
@@ -1802,10 +1802,10 @@ def infer_types(self, col_labels: List[str]) -> "PandasDataframe":
         new_dtypes[col_labels] = new_cols_dtypes
         return self.__constructor__(
             self._partitions,
-            self.index,
-            self.columns,
-            self.row_lengths,
-            self.column_widths,
+            self._index_cache,
+            self._columns_cache,
+            self._row_lengths_cache,
+            self._column_widths_cache,
             new_dtypes,
         )
 
@@ -1910,8 +1910,8 @@ def map_fn(df):
             new_parts,
             new_index,
             new_cols,
-            self.row_lengths,
-            self.column_widths,
+            self._row_lengths_cache,
+            self._column_widths_cache,
             new_dtypes,
         )
 
@@ -3105,10 +3105,10 @@ def transpose(self):
             new_dtypes = None
         return self.__constructor__(
             new_partitions,
-            self.columns,
-            self.index,
-            self.column_widths,
-            self.row_lengths,
+            self._columns_cache,
+            self._index_cache,
+            self._column_widths_cache,
+            self._row_lengths_cache,
             dtypes=new_dtypes,
         )
 


Test patch:
Repository: modin-project/modin

Issue:
BUG: `execute` function in ASV utils failed if len(partitions) == 0
Log:
```bash
  execute(IMPL.merge(self.left.iloc[:0], self.right, sort=sort))
File "/home/runner/work/modin/modin/asv_bench/benchmarks/utils/common.py", line 499, in execute
 if hasattr(partitions[0], "wait"):
IndexError: index 0 is out of bounds for axis 0 with size 0
asv: benchmark failed (exit status 1)
```


Discussion:

Resolving patch:
diff --git a/asv_bench/benchmarks/utils/common.py b/asv_bench/benchmarks/utils/common.py
--- a/asv_bench/benchmarks/utils/common.py
+++ b/asv_bench/benchmarks/utils/common.py
@@ -496,7 +496,7 @@ def execute(
             df._query_compiler._modin_frame._execute()
             return
         partitions = df._query_compiler._modin_frame._partitions.flatten()
-        if hasattr(partitions[0], "wait"):
+        if len(partitions) > 0 and hasattr(partitions[0], "wait"):
             all(map(lambda partition: partition.wait(), partitions))
             return
 


Test patch:
Repository: modin-project/modin

Issue:
BENCH: update `TimeConcat` benchmark with new parameter `ignore_index`



Discussion:

Resolving patch:
diff --git a/asv_bench/benchmarks/benchmarks.py b/asv_bench/benchmarks/benchmarks.py
--- a/asv_bench/benchmarks/benchmarks.py
+++ b/asv_bench/benchmarks/benchmarks.py
@@ -149,19 +149,24 @@ def time_merge(self, shapes, how, sort):
 
 
 class TimeConcat:
-    param_names = ["shapes", "how", "axis"]
+    param_names = ["shapes", "how", "axis", "ignore_index"]
     params = [
         get_benchmark_shapes("TimeConcat"),
         ["inner"],
         [0, 1],
+        [True, False],
     ]
 
-    def setup(self, shapes, how, axis):
+    def setup(self, shapes, how, axis, ignore_index):
         self.df1 = generate_dataframe("int", *shapes[0], RAND_LOW, RAND_HIGH)
         self.df2 = generate_dataframe("int", *shapes[1], RAND_LOW, RAND_HIGH)
 
-    def time_concat(self, shapes, how, axis):
-        execute(IMPL.concat([self.df1, self.df2], axis=axis, join=how))
+    def time_concat(self, shapes, how, axis, ignore_index):
+        execute(
+            IMPL.concat(
+                [self.df1, self.df2], axis=axis, join=how, ignore_index=ignore_index
+            )
+        )
 
 
 class TimeAppend:


Test patch: