internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: modin-project/modin

Issue:
Improve OmniSci "quick start" usability
Right now to activate OmniSci you'd have to set both `Backend` to `Omnisci` and `Engine` to `native`.
I think we should try to set `Engine` to `native` if it's unset by user while `Backend` is set to `Omnisci`.


Discussion:
Note that after #3537 is merged `Backend` will be replaced to `StorageFormat`.

Resolving patch:
diff --git a/modin/pandas/__init__.py b/modin/pandas/__init__.py
--- a/modin/pandas/__init__.py
+++ b/modin/pandas/__init__.py
@@ -105,8 +105,22 @@
 def _update_engine(publisher: Parameter):
     global dask_client
     from modin.config import StorageFormat, CpuCount
+    from modin.config.envvars import IsExperimental
+    from modin.config.pubsub import ValueSource
 
-    if publisher.get() == "Ray":
+    if (
+        StorageFormat.get() == "Omnisci"
+        and publisher.get_value_source() == ValueSource.DEFAULT
+    ):
+        publisher.put("Native")
+        IsExperimental.put(True)
+    elif (
+        publisher.get() == "Native"
+        and StorageFormat.get_value_source() == ValueSource.DEFAULT
+    ):
+        StorageFormat.put("Omnisci")
+        IsExperimental.put(True)
+    elif publisher.get() == "Ray":
         if _is_first_update.get("Ray", True):
             from modin.core.execution.ray.common.utils import initialize_ray
 


Test patch:
Repository: modin-project/modin

Issue:
Change setup.py to be pure Python
At one point, we needed to have different distributions for Windows and Unix/Linux. Now, we no longer need that, so it would be good if we could remove the distribution specific logic in the setup.py and in the future create only one wheel (since Modin is pure python).


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -1,42 +1,9 @@
 from setuptools import setup, find_packages
 import versioneer
-import os
-from setuptools.dist import Distribution
-
-try:
-    from wheel.bdist_wheel import bdist_wheel
-
-    HAS_WHEEL = True
-except ImportError:
-    HAS_WHEEL = False
 
 with open("README.md", "r", encoding="utf-8") as fh:
     long_description = fh.read()
 
-if HAS_WHEEL:
-
-    class ModinWheel(bdist_wheel):
-        def finalize_options(self):
-            bdist_wheel.finalize_options(self)
-            self.root_is_pure = False
-
-        def get_tag(self):
-            _, _, plat = bdist_wheel.get_tag(self)
-            py = "py3"
-            abi = "none"
-            return py, abi, plat
-
-
-class ModinDistribution(Distribution):
-    def __init__(self, *attrs):
-        Distribution.__init__(self, *attrs)
-        if HAS_WHEEL:
-            self.cmdclass["bdist_wheel"] = ModinWheel
-
-    def is_pure(self):
-        return False
-
-
 dask_deps = ["dask>=2.22.0", "distributed>=2.22.0"]
 ray_deps = ["ray[default]>=1.4.0", "pyarrow>=1.0"]
 remote_deps = ["rpyc==4.1.5", "cloudpickle", "boto3"]
@@ -44,15 +11,10 @@ def is_pure(self):
 sql_deps = ["dfsql>=0.4.2", "pyparsing<=2.4.7"]
 all_deps = dask_deps + ray_deps + remote_deps + spreadsheet_deps
 
-# dfsql does not support Windows yet
-if os.name != 'nt':
-    all_deps += sql_deps
-
 setup(
     name="modin",
     version=versioneer.get_version(),
     cmdclass=versioneer.get_cmdclass(),
-    distclass=ModinDistribution,
     description="Modin: Make your pandas code run faster by changing one line of code.",
     packages=find_packages(),
     include_package_data=True,


Test patch:
Repository: modin-project/modin

Issue:
Ray tasks are restarted when workers died, this behavior is not always desirable
### System information
- **Modin version** (`modin.__version__`): 8acad95ed2dea2c7671d6dc3518d4b0b12c8b682
- **Link to Ray docs**: https://docs.ray.io/en/latest/fault-tolerance.html#tasks

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->

An example of undesirable behavior is when tasks write to disk. In this case, duplication of data may occur. This behavior can be observed for `to_csv` function on Ray engine.

For such cases, we should use `max_retries=0` parameter of `ray.remote` function.


Discussion:
What is the reason why a worker dies?
I think overload, we tell Modin to use 4 cores, although there are only 2 cores on the github machines.

Resolving patch:
diff --git a/modin/core/execution/ray/generic/io/io.py b/modin/core/execution/ray/generic/io/io.py
--- a/modin/core/execution/ray/generic/io/io.py
+++ b/modin/core/execution/ray/generic/io/io.py
@@ -191,6 +191,7 @@ def func(df, **kw):
             keep_partitioning=True,
             lengths=None,
             enumerate_partitions=True,
+            max_retries=0,
         )
 
         # pending completion
diff --git a/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/axis_partition.py b/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/axis_partition.py
--- a/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/axis_partition.py
+++ b/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/axis_partition.py
@@ -76,8 +76,10 @@ def deploy_axis_func(
             A list of ``pandas.DataFrame``-s.
         """
         lengths = kwargs.get("_lengths", None)
+        max_retries = kwargs.pop("max_retries", None)
         return deploy_ray_func.options(
-            num_returns=(num_splits if lengths is None else len(lengths)) * 4
+            num_returns=(num_splits if lengths is None else len(lengths)) * 4,
+            **({"max_retries": max_retries} if max_retries is not None else {}),
         ).remote(
             PandasDataframeAxisPartition.deploy_axis_func,
             axis,


Test patch:
Repository: modin-project/modin

Issue:
REFACTOR: remove compression check from ` _read_csv_check_support` function
This [check](https://github.com/modin-project/modin/blob/6178a5764489e05aa85b4bf47a6fecac2c89c53e/modin/core/io/text/csv_dispatcher.py#L205) is used to identify situations where Modin does not support any compression format in order to use the pandas code in these cases. However we can rely on `infer_compression` [function](https://github.com/modin-project/modin/blob/6178a5764489e05aa85b4bf47a6fecac2c89c53e/modin/core/io/text/csv_dispatcher.py#L75) from the pandas(called before checks), which will throw an exception if the compression algorithm is not supported.


Discussion:

Resolving patch:
diff --git a/modin/core/io/text/csv_dispatcher.py b/modin/core/io/text/csv_dispatcher.py
--- a/modin/core/io/text/csv_dispatcher.py
+++ b/modin/core/io/text/csv_dispatcher.py
@@ -21,7 +21,6 @@
 import pandas
 from pandas.core.dtypes.common import is_list_like
 from csv import QUOTE_NONE, Dialect
-import sys
 import io
 from typing import Union, Sequence, Callable, Dict, Tuple
 from pandas._typing import FilePathOrBuffer
@@ -97,9 +96,7 @@ def _read(cls, filepath_or_buffer, **kwargs):
             skiprows_md, int
         )
 
-        use_modin_impl = cls._read_csv_check_support(
-            filepath_or_buffer, kwargs, compression_infered
-        )
+        use_modin_impl = cls._read_csv_check_support(filepath_or_buffer, kwargs)
         if not use_modin_impl:
             return cls.single_worker_read(filepath_or_buffer, **kwargs)
 
@@ -177,7 +174,6 @@ def _read_csv_check_support(
         cls,
         filepath_or_buffer: FilePathOrBuffer,
         read_csv_kwargs: ReadCsvKwargsType,
-        compression_infered: str,
     ) -> bool:
         """
         Check if passed parameters are supported by current `read_csv` implementation.
@@ -188,8 +184,6 @@ def _read_csv_check_support(
             `filepath_or_buffer` parameter of read_csv function.
         read_csv_kwargs : dict
             Parameters of read_csv function.
-        compression_infered : str
-            Inferred `compression` parameter of read_csv function.
 
         Returns
         -------
@@ -202,19 +196,8 @@ def _read_csv_check_support(
         elif not cls.pathlib_or_pypath(filepath_or_buffer):
             return False
 
-        if compression_infered is not None:
-            use_modin_impl = compression_infered in ["gzip", "bz2", "xz"] or (
-                compression_infered == "zip"
-                # need python3.7 to .seek and .tell ZipExtFile
-                and sys.version_info[0] == 3
-                and sys.version_info[1] >= 7
-            )
-            if not use_modin_impl:
-                return False
-
         if read_csv_kwargs.get("chunksize") is not None:
             return False
-
         return True
 
     @classmethod


Test patch:
Repository: modin-project/modin

Issue:
add groupby benchmark for OmniSci where `agg` called with dict parameter
Interesting case when Modin on OmniSci doesn't defaulting to pandas.


Discussion:

Resolving patch:
diff --git a/asv_bench/benchmarks/omnisci/benchmarks.py b/asv_bench/benchmarks/omnisci/benchmarks.py
--- a/asv_bench/benchmarks/omnisci/benchmarks.py
+++ b/asv_bench/benchmarks/omnisci/benchmarks.py
@@ -416,6 +416,9 @@ def setup(self, shape, ngroups=5, groupby_ncols=1):
             groupby_ncols,
             count_groups=ngroups,
         )
+        # correct while we use 'col*' like name for non-groupby columns
+        # and 'groupby_col*' like name for groupby columns
+        self.non_groupby_columns = self.df.columns[:-groupby_ncols]
         trigger_import(self.df)
 
 
@@ -449,3 +452,10 @@ def time_groupby_agg_mean(self, *args, **kwargs):
 
     def time_groupby_agg_nunique(self, *args, **kwargs):
         execute(self.df.groupby(by=self.groupby_columns).agg("nunique"))
+
+    def time_groupby_agg_mean_dict(self, *args, **kwargs):
+        execute(
+            self.df.groupby(by=self.groupby_columns).agg(
+                {col: "mean" for col in self.non_groupby_columns}
+            )
+        )


Test patch:
Repository: modin-project/modin

Issue:
Change readthedocs theme to use the same on that pandas uses
https://github.com/pandas-dev/pandas/blob/master/doc/source/conf.py#L218


Discussion:
happy to work on this issue :)

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -12,12 +12,15 @@
 import types
 
 import ray
+
 # stub ray.remote to be a no-op so it doesn't shadow docstrings
 def noop_decorator(*args, **kwargs):
     if len(args) == 1 and len(kwargs) == 0 and callable(args[0]):
         # This is the case where the decorator is just @ray.remote without parameters.
         return args[0]
     return lambda cls_or_func: cls_or_func
+
+
 ray.remote = noop_decorator
 
 # fake modules if they're missing
@@ -107,13 +110,15 @@ def noop_decorator(*args, **kwargs):
 # -- Options for HTML output -------------------------------------------------
 
 # Maps git branches to Sphinx themes
-default_html_theme = "sphinx_rtd_theme"
+default_html_theme = "pydata_sphinx_theme"
 current_branch = "nature"
 
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #
-html_theme = "sphinx_rtd_theme"
+html_theme = "pydata_sphinx_theme"
+
+html_favicon = "img/MODIN_ver2.ico"
 
 html_logo = "img/MODIN_ver2.png"
 
@@ -125,6 +130,35 @@ def noop_decorator(*args, **kwargs):
     "sidebarwidth": 270,
     "collapse_navigation": False,
     "navigation_depth": 4,
+    "show_toc_level": 2,
+    "github_url": "https://github.com/modin-project/modin",
+    "icon_links": [
+        {
+            "name": "PyPI",
+            "url": "https://pypi.org/project/modin",
+            "icon": "fab fa-python",
+        },
+        {
+            "name": "conda-forge",
+            "url": "https://anaconda.org/conda-forge/modin",
+            "icon": "fas fa-circle-notch",
+        },
+        {
+            "name": "Join the Slack",
+            "url": "https://modin.org/slack.html",
+            "icon": "fab fa-slack",
+        },
+        {
+            "name": "Discourse",
+            "url": "https://discuss.modin.org/",
+            "icon": "fab fa-discourse",
+        },
+        {
+            "name": "Mailing List",
+            "url": "https://groups.google.com/forum/#!forum/modin-dev",
+            "icon": "fas fa-envelope-square",
+        },
+    ],
 }
 
 # Custom sidebar templates, must be a dictionary that maps document names
@@ -135,8 +169,8 @@ def noop_decorator(*args, **kwargs):
 # default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
 # 'searchbox.html']``.
 #
-html_sidebars = {
-    "**": ["globaltoc.html", "relations.html", "sourcelink.html", "searchbox.html"]
-}
+# The default pydata_sphinx_theme sidebar templates are
+# sidebar-nav-bs.html and search-field.html.
+html_sidebars = {}
 
 issues_github_path = "modin-project/modin"


Test patch:
Repository: modin-project/modin

Issue:
Transpose operator inevitably triggers lazy `dtypes` metadata computation.
Doing `df.T` triggers `dtypes` property access and so its computation, even if it has been delayed by setting `modin_frame._dtypes = None`:
https://github.com/modin-project/modin/blob/a80cb3bcb5fe0294398c96e3c4fa0b32b688db48/modin/core/dataframe/pandas/dataframe/dataframe.py#L2269-L2272
Data types computation may take up to several seconds for wide frames, we should avoid it when possible. 

BTW `df.T` is a part of the reduction operations flow ([columnarization at `Series` constructor](https://github.com/modin-project/modin/blob/f2133da30ba40786fcd421396b92f29bd121083c/modin/pandas/series.py#L160)), so it's generally slowing down every reduction method.


Discussion:

Resolving patch:
diff --git a/modin/core/dataframe/pandas/dataframe/dataframe.py b/modin/core/dataframe/pandas/dataframe/dataframe.py
--- a/modin/core/dataframe/pandas/dataframe/dataframe.py
+++ b/modin/core/dataframe/pandas/dataframe/dataframe.py
@@ -2295,10 +2295,13 @@ def transpose(self):
         new_partitions = self._partition_mgr_cls.lazy_map_partitions(
             self._partitions, lambda df: df.T
         ).T
-        new_dtypes = pandas.Series(
-            np.full(len(self.index), find_common_type(self.dtypes.values)),
-            index=self.index,
-        )
+        if self._dtypes is not None:
+            new_dtypes = pandas.Series(
+                np.full(len(self.index), find_common_type(self.dtypes.values)),
+                index=self.index,
+            )
+        else:
+            new_dtypes = None
         return self.__constructor__(
             new_partitions,
             self.columns,


Test patch:
Repository: modin-project/modin

Issue:
Remove providing of `address` for `ray.init` in case local cluster is used
<!--
General questions should be asked on the mailing list modin-dev@googlegroups.com.

Before submitting an issue, please fill out the following form.
-->

### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: all
- **Modin installed from (source or binary)**: source
- **Modin version**: 72a3e51aeeae3465a1a3076d962aef95af689dde
- **Python version**: 3.8.12
- **Exact command to reproduce**:

```python
import modin.pandas as pd
from modin.config import RayRedisAddress
RayRedisAddress.put("127.0.0.1:8080")

df = pd.DataFrame([])
print(df)
```
<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->
Current set of `ray.init` kwrags is incompatible in a case of local cluster using. Need to exclude providing of parameter `address` for Ray local cluster case
https://github.com/modin-project/modin/blob/72a3e51aeeae3465a1a3076d962aef95af689dde/modin/core/execution/ray/common/utils.py#L165-L173

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->

```bash
Traceback (most recent call last):
  File "test4.py", line 5, in <module>
    df = pd.DataFrame([])
  File "/localdisk/aprutsko/modin/modin/pandas/dataframe.py", line 109, in __init__
    Engine.subscribe(_update_engine)
  File "/localdisk/aprutsko/modin/modin/config/pubsub.py", line 213, in subscribe
    callback(cls)
  File "/localdisk/aprutsko/modin/modin/pandas/__init__.py", line 127, in _update_engine
    initialize_ray()
  File "/localdisk/aprutsko/modin/modin/core/execution/ray/common/utils.py", line 175, in initialize_ray
    ray.init(**ray_init_kwargs)
  File "/localdisk/aprutsko/miniconda3/envs/modin/lib/python3.8/site-packages/ray/_private/client_mode_hook.py", line 62, in wrapper
    return func(*args, **kwargs)
  File "/localdisk/aprutsko/miniconda3/envs/modin/lib/python3.8/site-packages/ray/worker.py", line 801, in init
    raise ValueError(
ValueError: When connecting to an existing cluster, num_cpus and num_gpus must not be provided.
```



Discussion:

Resolving patch:
diff --git a/modin/core/execution/ray/common/utils.py b/modin/core/execution/ray/common/utils.py
--- a/modin/core/execution/ray/common/utils.py
+++ b/modin/core/execution/ray/common/utils.py
@@ -168,7 +168,6 @@ def initialize_ray(
                 "include_dashboard": False,
                 "ignore_reinit_error": True,
                 "object_store_memory": object_store_memory,
-                "address": redis_address,
                 "_redis_password": redis_password,
                 "_memory": object_store_memory,
             }


Test patch:
Repository: modin-project/modin

Issue:
Adding support for getArrowTable() method of OmnisciServer
Added selection of OmniSciDb's DBEngine::getArrowTable() function. This function is supposed to increase performance of SQL queries.  Modifications has been done to `/modin/experimental/core/execution/native/implementations/omnisci_on_native/partitioning/partition_manager.py`, function `run_exec_plan`.

In the new version, table is retrieved by `getArrowTable()' method.  If the method is not present, it falls back to already existing `getArrowRecordBatch()` method.


Discussion:

Resolving patch:
diff --git a/modin/experimental/core/execution/native/implementations/omnisci_on_native/partitioning/partition_manager.py b/modin/experimental/core/execution/native/implementations/omnisci_on_native/partitioning/partition_manager.py
--- a/modin/experimental/core/execution/native/implementations/omnisci_on_native/partitioning/partition_manager.py
+++ b/modin/experimental/core/execution/native/implementations/omnisci_on_native/partitioning/partition_manager.py
@@ -251,9 +251,13 @@ def run_exec_plan(cls, plan, index_cols, dtypes, columns):
 
         curs = omniSession.executeRA(cmd_prefix + calcite_json)
         assert curs
-        rb = curs.getArrowRecordBatch()
-        assert rb is not None
-        at = pyarrow.Table.from_batches([rb])
+        if hasattr(curs, "getArrowTable"):
+            at = curs.getArrowTable()
+        else:
+            rb = curs.getArrowRecordBatch()
+            assert rb is not None
+            at = pyarrow.Table.from_batches([rb])
+        assert at is not None
 
         res = np.empty((1, 1), dtype=np.dtype(object))
         # workaround for https://github.com/modin-project/modin/issues/1851


Test patch:
Repository: modin-project/modin

Issue:
Improve documentation for `experimental pandas on Ray` execution



Discussion:

Resolving patch:
diff --git a/modin/experimental/core/execution/ray/implementations/pandas_on_ray/io/io.py b/modin/experimental/core/execution/ray/implementations/pandas_on_ray/io/io.py
--- a/modin/experimental/core/execution/ray/implementations/pandas_on_ray/io/io.py
+++ b/modin/experimental/core/execution/ray/implementations/pandas_on_ray/io/io.py
@@ -123,6 +123,10 @@ def read_sql(
         """
         Read SQL query or database table into a DataFrame.
 
+        The function extended with `Spark-like parameters <https://spark.apache.org/docs/2.0.0/api/R/read.jdbc.html>`_
+        such as ``partition_column``, ``lower_bound`` and ``upper_bound``. With these
+        parameters, the user will be able to specify how to partition the imported data.
+
         Parameters
         ----------
         sql : str or SQLAlchemy Selectable (select or text object)
diff --git a/modin/experimental/pandas/io.py b/modin/experimental/pandas/io.py
--- a/modin/experimental/pandas/io.py
+++ b/modin/experimental/pandas/io.py
@@ -231,10 +231,9 @@ def read_pickle_distributed(
     """
     Load pickled pandas object from files.
 
-    In experimental mode, we can use `*` in the filename. The files must contain
-    parts of one dataframe, which can be obtained, for example, by
-    `to_pickle_distributed` function.
-    Note: the number of partitions is equal to the number of input files.
+    This experimental feature provides parallel reading from multiple pickle files which are
+    defined by glob pattern. The files must contain parts of one dataframe, which can be
+    obtained, for example, by `to_pickle_distributed` function.
 
     Parameters
     ----------
@@ -256,6 +255,10 @@ def read_pickle_distributed(
     Returns
     -------
     unpickled : same type as object stored in file
+
+    Notes
+    -----
+    The number of partitions is equal to the number of input files.
     """
     Engine.subscribe(_update_engine)
     assert IsExperimental.get(), "This only works in experimental mode"
@@ -273,8 +276,8 @@ def to_pickle_distributed(
     """
     Pickle (serialize) object to file.
 
-    If `*` in the filename all partitions are written to their own separate file,
-    otherwise default pandas implementation is used.
+    This experimental feature provides parallel writing into multiple pickle files which are
+    defined by glob pattern, otherwise (without glob pattern) default pandas implementation is used.
 
     Parameters
     ----------


Test patch:
Repository: modin-project/modin

Issue:
REFACTOR: FactoryDispatcher imported twice in modin\experimental\pandas\io.py
Code duplication, need to remove second `import` statement.


Discussion:

Resolving patch:
diff --git a/modin/experimental/pandas/io.py b/modin/experimental/pandas/io.py
--- a/modin/experimental/pandas/io.py
+++ b/modin/experimental/pandas/io.py
@@ -303,8 +303,6 @@ def to_pickle_distributed(
         this argument with a non-fsspec URL. See the fsspec and backend storage
         implementation docs for the set of allowed keys and values.
     """
-    from modin.core.execution.dispatching.factories.dispatcher import FactoryDispatcher
-
     obj = self
     Engine.subscribe(_update_engine)
     if isinstance(self, DataFrame):


Test patch:
Repository: modin-project/modin

Issue:
Update pandas to 1.3.5



Discussion:

Resolving patch:
diff --git a/modin/pandas/__init__.py b/modin/pandas/__init__.py
--- a/modin/pandas/__init__.py
+++ b/modin/pandas/__init__.py
@@ -13,7 +13,7 @@
 
 import pandas
 
-__pandas_version__ = "1.3.4"
+__pandas_version__ = "1.3.5"
 
 if pandas.__version__ != __pandas_version__:
     import warnings
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -22,7 +22,7 @@
     url="https://github.com/modin-project/modin",
     long_description=long_description,
     long_description_content_type="text/markdown",
-    install_requires=["pandas==1.3.4", "packaging", "numpy>=1.16.5", "fsspec"],
+    install_requires=["pandas==1.3.5", "packaging", "numpy>=1.16.5", "fsspec"],
     extras_require={
         # can be installed by pip install modin[dask]
         "dask": dask_deps,


Test patch: