internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: modin-project/modin

Issue:
Fix passing redis password to ray.init
Currently, we pass generated redis password to `ray.init()` that is not exactly correct at all times - this breaks the ability to connect to an already spawned Ray cluster in ordinary, non-cloud workflow.

We need to add an environment variable that is responsible for setting redis password. The value from that variable could be passed to `ray.init()` instead of a generated password.


Discussion:

Resolving patch:
diff --git a/modin/config/envvars.py b/modin/config/envvars.py
--- a/modin/config/envvars.py
+++ b/modin/config/envvars.py
@@ -16,6 +16,7 @@
 from textwrap import dedent
 import warnings
 from packaging import version
+import secrets
 
 from .pubsub import Parameter, _TYPE_PARAMS, ExactStr, ValueSource
 
@@ -121,6 +122,15 @@ class RayRedisAddress(EnvironmentVariable, type=ExactStr):
     varname = "MODIN_REDIS_ADDRESS"
 
 
+class RayRedisPassword(EnvironmentVariable, type=ExactStr):
+    """
+    What password to use for connecting to Redis
+    """
+
+    varname = "MODIN_REDIS_PASSWORD"
+    default = secrets.token_hex(32)
+
+
 class CpuCount(EnvironmentVariable, type=int):
     """
     How many CPU cores to use when initialization of the Modin engine.
diff --git a/modin/engines/ray/utils.py b/modin/engines/ray/utils.py
--- a/modin/engines/ray/utils.py
+++ b/modin/engines/ray/utils.py
@@ -21,6 +21,7 @@
     Backend,
     IsRayCluster,
     RayRedisAddress,
+    RayRedisPassword,
     CpuCount,
     GpuCount,
     Memory,
@@ -132,16 +133,14 @@ def initialize_ray(
         If not specified, ``modin.config.RayRedisAddress`` is used.
     override_redis_password : str, optional
         What password to use when connecting to Redis.
-        If not specified, a new random one is generated.
+        If not specified, ``modin.config.RayRedisPassword`` is used.
     """
     import ray
 
     if not ray.is_initialized() or override_is_cluster:
-        import secrets
-
         cluster = override_is_cluster or IsRayCluster.get()
         redis_address = override_redis_address or RayRedisAddress.get()
-        redis_password = override_redis_password or secrets.token_hex(32)
+        redis_password = override_redis_password or RayRedisPassword.get()
 
         if cluster:
             # We only start ray in a cluster setting for the head node.


Test patch:
Repository: modin-project/modin

Issue:
[Ray] Work around for Ray's race condition with import thread
We already do this when we initialize a Ray environment, but we need to handle it when ray is already initialized. This line of code needs to be moved to resolve it. 

https://github.com/modin-project/modin/blob/master/modin/engines/ray/utils.py#L184

The issue ray-project/ray#7879 tracks the progress, but after some discussions with the Ray team it seems unlikely that it will be fixed soon due to a lack of consistent reproducer.


Discussion:

Resolving patch:
diff --git a/modin/engines/ray/utils.py b/modin/engines/ray/utils.py
--- a/modin/engines/ray/utils.py
+++ b/modin/engines/ray/utils.py
@@ -220,9 +220,6 @@ def initialize_ray(
         ray.worker.global_worker.run_function_on_all_workers(
             _move_stdlib_ahead_of_site_packages
         )
-
-        ray.worker.global_worker.run_function_on_all_workers(_import_pandas)
-
         if Backend.get() == "Cudf":
             from modin.engines.ray.cudf_on_ray.frame.gpu_manager import GPUManager
             from modin.engines.ray.cudf_on_ray.frame.partition_manager import (
@@ -233,7 +230,7 @@ def initialize_ray(
             if not GPU_MANAGERS:
                 for i in range(GpuCount.get()):
                     GPU_MANAGERS.append(GPUManager.remote(i))
-
+    ray.worker.global_worker.run_function_on_all_workers(_import_pandas)
     num_cpus = int(ray.cluster_resources()["CPU"])
     num_gpus = int(ray.cluster_resources().get("GPU", 0))
     if Backend.get() == "Cudf":


Test patch:
Repository: modin-project/modin

Issue:
Add API section for `modin.pandas.series` in the documentation.



Discussion:

Resolving patch:
diff --git a/modin/pandas/series.py b/modin/pandas/series.py
--- a/modin/pandas/series.py
+++ b/modin/pandas/series.py
@@ -87,7 +87,7 @@ class Series(BasePandasDataset):
     """
     Modin distributed representation of `pandas.Series`.
 
-    Internally passed data is divided into partitions in order to parallelize
+    Internally, the data can be divided into partitions in order to parallelize
     computations and utilize the user's hardware as much as possible.
 
     Inherit common for DataFrames and Series functionality from the


Test patch:
Repository: modin-project/modin

Issue:
Fix consistency in connecting to Ray cluster and local cluster
We work around a couple of things that users have seen in terms of using Ray and local python environments. However, we are not consistent with how we handle these for existing clusters. #3128 was a start toward this, but we also have a case where the `typing` library from pypi can fall before the Python builtin library causing errors.

We need to move these lines outside of the if block to handle them the same between both existing clusters and local clusters: 

https://github.com/modin-project/modin/blob/8165051b1bafb1b6dc3cc6069449637c60071f66/modin/engines/ray/utils.py#L218-L221

cc @YarShev 


Discussion:

Resolving patch:
diff --git a/modin/engines/ray/utils.py b/modin/engines/ray/utils.py
--- a/modin/engines/ray/utils.py
+++ b/modin/engines/ray/utils.py
@@ -156,10 +156,6 @@ def initialize_ray(
             }
             ray.init(**ray_init_kwargs)
 
-        _move_stdlib_ahead_of_site_packages()
-        ray.worker.global_worker.run_function_on_all_workers(
-            _move_stdlib_ahead_of_site_packages
-        )
         if Backend.get() == "Cudf":
             from modin.engines.ray.cudf_on_ray.frame.gpu_manager import GPUManager
             from modin.engines.ray.cudf_on_ray.frame.partition_manager import (
@@ -170,6 +166,10 @@ def initialize_ray(
             if not GPU_MANAGERS:
                 for i in range(GpuCount.get()):
                     GPU_MANAGERS.append(GPUManager.remote(i))
+    _move_stdlib_ahead_of_site_packages()
+    ray.worker.global_worker.run_function_on_all_workers(
+        _move_stdlib_ahead_of_site_packages
+    )
     ray.worker.global_worker.run_function_on_all_workers(_import_pandas)
     num_cpus = int(ray.cluster_resources()["CPU"])
     num_gpus = int(ray.cluster_resources().get("GPU", 0))


Test patch:
Repository: modin-project/modin

Issue:
Fix inconsistencies in the `modin.config` docs
Comments https://github.com/modin-project/modin/pull/3129#discussion_r655444346 and https://github.com/modin-project/modin/pull/3129#discussion_r655445196 should be addressed.



Discussion:

Resolving patch:
diff --git a/modin/config/__main__.py b/modin/config/__main__.py
--- a/modin/config/__main__.py
+++ b/modin/config/__main__.py
@@ -56,7 +56,9 @@ def export_config_help(filename: str):
         if isinstance(obj, type) and issubclass(obj, Parameter) and not obj.is_abstract:
             data = {
                 "Config Name": obj.__name__,
-                "Env. Variable Name": obj.varname,
+                "Env. Variable Name": getattr(
+                    obj, "varname", "not backed by environment"
+                ),
                 "Default Value": obj._get_default()
                 if obj.__name__ != "RayRedisPassword"
                 else "random string",
@@ -71,7 +73,7 @@ def export_config_help(filename: str):
 if __name__ == "__main__":
     parser = argparse.ArgumentParser()
     parser.add_argument(
-        "-export_path",
+        "--export-path",
         dest="export_path",
         type=str,
         required=False,


Test patch:
Repository: modin-project/modin

Issue:
Disable OmniSci' watchdog by default
In cases when watchdog raises an exception we can't do better than default to pandas, which anyway gives a worse performance in comparison with OmniSci implementation (even when watchdog consider this operation as "too slow")


Discussion:

Resolving patch:
diff --git a/modin/experimental/engines/omnisci_on_ray/frame/omnisci_worker.py b/modin/experimental/engines/omnisci_on_ray/frame/omnisci_worker.py
--- a/modin/experimental/engines/omnisci_on_ray/frame/omnisci_worker.py
+++ b/modin/experimental/engines/omnisci_on_ray/frame/omnisci_worker.py
@@ -44,6 +44,7 @@ def start_server(cls):
                 enable_columnar_output=1,
                 enable_lazy_fetch=0,
                 null_div_by_zero=1,
+                enable_watchdog=0,
             )
 
     @classmethod


Test patch:
Repository: modin-project/modin

Issue:
Remove Dask pin in setup.py
https://github.com/conda-forge/modin-feedstock/pull/17


Discussion:

Resolving patch:
diff --git a/modin/config/envvars.py b/modin/config/envvars.py
--- a/modin/config/envvars.py
+++ b/modin/config/envvars.py
@@ -102,9 +102,9 @@ def _get_default(cls):
             raise ImportError(
                 "Please `pip install modin[ray]` or `modin[dask]` to install an engine"
             )
-        if version.parse(dask.__version__) < version.parse("2.1.0") or version.parse(
+        if version.parse(dask.__version__) < version.parse("2.22.0") or version.parse(
             distributed.__version__
-        ) < version.parse("2.3.2"):
+        ) < version.parse("2.22.0"):
             raise ImportError(
                 "Please `pip install modin[dask]` to install compatible Dask version."
             )
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -37,9 +37,9 @@ def is_pure(self):
         return False
 
 
-dask_deps = ["dask>=2.12.0,<=2.19.0", "distributed>=2.12.0,<=2.19.0"]
+dask_deps = ["dask>=2.22.0", "distributed>=2.22.0"]
 ray_deps = ["ray[default]>=1.4.0", "pyarrow>=1.0"]
-remote_deps = ["rpyc==4.1.5", "cloudpickle==1.4.1", "boto3==1.4.8"]
+remote_deps = ["rpyc==4.1.5", "cloudpickle", "boto3==1.4.8"]
 spreadsheet_deps = ["modin-spreadsheet>=0.1.0"]
 sql_deps = ["dfsql>=0.2.6"]
 all_deps = dask_deps + ray_deps + remote_deps + spreadsheet_deps


Test patch:
Repository: modin-project/modin

Issue:
Update install command for Ray to remove `FutureWarning`
<!--
General questions should be asked on the mailing list modin-dev@googlegroups.com.

Before submitting an issue, please fill out the following form.
-->

### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: all
- **Modin version**: 0.10.0
- **Python version**: 3.8.10
- **Exact command to reproduce**:

```
pip install -e .[ray]
```
```python
import modin.pandas as pd
df = pd.DataFrame([0])
```

Output:
```
FutureWarning: Not all Ray CLI dependencies were found. In Ray 1.4+, the Ray CLI, autoscaler, and dashboard will only be usable via `pip install 'ray[default]'`. Please update your install command.
```

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->

`FutureWarning` is raised when functions of installed Modin is used.


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -38,7 +38,7 @@ def is_pure(self):
 
 
 dask_deps = ["dask>=2.12.0,<=2.19.0", "distributed>=2.12.0,<=2.19.0"]
-ray_deps = ["ray>=1.4.0", "pyarrow>=1.0"]
+ray_deps = ["ray[default]>=1.4.0", "pyarrow>=1.0"]
 remote_deps = ["rpyc==4.1.5", "cloudpickle==1.4.1", "boto3==1.4.8"]
 spreadsheet_deps = ["modin-spreadsheet>=0.1.0"]
 sql_deps = ["dfsql>=0.2.6"]


Test patch:
Repository: modin-project/modin

Issue:
Documentation broken link
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Any
- **Modin version**: 6c23a23f3b4d2d83d57219ec8a0c94f6932f259a

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->
It was found, that after switching to pandas 1.3.0, broken links to pandas docs occured.

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->
GitHub Action logs:
```
doc_urls = ['https://pandas.pydata.org/pandas-docs/version/1.3.0/reference/api/pandas.DataFrame.abs.html', 'https://pandas.pydata....aggregate.html', 'https://pandas.pydata.org/pandas-docs/version/1.3.0/reference/api/pandas.DataFrame.align.html', ...]

    def test_all_urls_exist(doc_urls):
        broken = []
    
        def _test_url(url):
            try:
                with urlopen(url):
                    pass
            except HTTPError:
                broken.append(url)
    
        with ThreadPoolExecutor(32) as pool:
            pool.map(_test_url, doc_urls)
    
>       assert not broken, "Invalid URLs detected"
E       AssertionError: Invalid URLs detected
E       assert not ['https://pandas.pydata.org/pandas-docs/version/1.3.0/reference/api/pandas.read_json.html']
```



Discussion:

Resolving patch:
diff --git a/modin/engines/base/io/io.py b/modin/engines/base/io/io.py
--- a/modin/engines/base/io/io.py
+++ b/modin/engines/base/io/io.py
@@ -266,7 +266,7 @@ def _read(cls, **kwargs):
         return pd_obj
 
     @classmethod
-    @_inherit_docstrings(pandas.read_json, apilink="pandas.read_json")
+    @_inherit_docstrings(pandas.read_json, apilink="pandas.io.json.read_json")
     @doc(
         _doc_default_io_method,
         summary="Convert a JSON string to query compiler",


Test patch:
Repository: modin-project/modin

Issue:
[ASV] add groupby multicolumn case for omnisci backend
An important case for that we want to track performance.


Discussion:

Resolving patch:
diff --git a/asv_bench/benchmarks/omnisci/benchmarks.py b/asv_bench/benchmarks/omnisci/benchmarks.py
--- a/asv_bench/benchmarks/omnisci/benchmarks.py
+++ b/asv_bench/benchmarks/omnisci/benchmarks.py
@@ -407,6 +407,7 @@ def setup(self, shape, ngroups=5, groupby_ncols=1):
             groupby_ncols,
             count_groups=ngroups,
         )
+        trigger_import(self.df)
 
 
 class TimeGroupByDefaultAggregations(BaseTimeGroupBy):
@@ -421,3 +422,21 @@ def time_groupby_count(self, *args, **kwargs):
 
     def time_groupby_sum(self, *args, **kwargs):
         execute(self.df.groupby(by=self.groupby_columns).sum())
+
+
+class TimeGroupByMultiColumn(BaseTimeGroupBy):
+    param_names = ["shape", "ngroups", "groupby_ncols"]
+    params = [
+        UNARY_OP_DATA_SIZE[ASV_DATASET_SIZE],
+        GROUPBY_NGROUPS[ASV_DATASET_SIZE],
+        [6],
+    ]
+
+    def time_groupby_sum(self, *args, **kwargs):
+        execute(self.df.groupby(by=self.groupby_columns).sum())
+
+    def time_groupby_agg_mean(self, *args, **kwargs):
+        execute(self.df.groupby(by=self.groupby_columns).agg("mean"))
+
+    def time_groupby_agg_nunique(self, *args, **kwargs):
+        execute(self.df.groupby(by=self.groupby_columns).agg("nunique"))


Test patch:
Repository: modin-project/modin

Issue:
ASV: OmniSci fillna benchmark should use column names instead of indexes
It was found that the expected behavior of the `fillna` function for `pandas=1.3.0` can be obtained using only the column names, not the indixes.

Problem place:
https://github.com/modin-project/modin/blob/9b94cb27bd7ea0b28ce6e6a163c056c3fa38d609/asv_bench/benchmarks/omnisci/benchmarks.py#L253


Discussion:

Resolving patch:
diff --git a/asv_bench/benchmarks/omnisci/benchmarks.py b/asv_bench/benchmarks/omnisci/benchmarks.py
--- a/asv_bench/benchmarks/omnisci/benchmarks.py
+++ b/asv_bench/benchmarks/omnisci/benchmarks.py
@@ -238,7 +238,7 @@ def setup(self, value_type, shape, limit):
         self.df = pd.DataFrame(np.nan, index=pd.RangeIndex(shape[0]), columns=columns)
         trigger_import(self.df)
 
-        value = self.create_fillna_value(value_type, shape)
+        value = self.create_fillna_value(value_type, columns)
         limit = int(limit * shape[0]) if limit else None
         self.kw = {"value": value, "limit": limit}
 
@@ -246,11 +246,11 @@ def time_fillna(self, value_type, shape, limit):
         execute(self.df.fillna(**self.kw))
 
     @staticmethod
-    def create_fillna_value(value_type: str, shape: tuple):
+    def create_fillna_value(value_type: str, columns: list):
         if value_type == "scalar":
             value = 18.19
         elif value_type == "dict":
-            value = {k: k * 1.23 for k in range(shape[0])}
+            value = {k: i * 1.23 for i, k in enumerate(columns)}
         else:
             assert False
         return value


Test patch:
Repository: modin-project/modin

Issue:
pin version of `mindsdb_sql` used by `dfsql`
### Describe the problem
<!-- Describe the problem clearly here. -->
CI is broken starting with update `mindsdb-sql` to [0.0.13](https://pypi.org/project/mindsdb-sql/0.0.13/ ). This is dependency of `dfsql` package, that's used in our ci:
https://github.com/modin-project/modin/blob/9b94cb27bd7ea0b28ce6e6a163c056c3fa38d609/.github/workflows/ci.yml#L472




Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -41,7 +41,7 @@ def is_pure(self):
 ray_deps = ["ray[default]>=1.4.0", "pyarrow>=1.0"]
 remote_deps = ["rpyc==4.1.5", "cloudpickle", "boto3==1.4.8"]
 spreadsheet_deps = ["modin-spreadsheet>=0.1.0"]
-sql_deps = ["dfsql>=0.2.6"]
+sql_deps = ["dfsql>=0.2.6", "mindsdb_sql<=0.0.12"]
 all_deps = dask_deps + ray_deps + remote_deps + spreadsheet_deps
 
 # dfsql does not support Windows yet


Test patch:
Repository: modin-project/modin

Issue:
Update modin version that used in docker omnisci containers
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. What kind of performance improvements would you like to see with this new API?

At the time of using Modin version: e7812858a3b651156b0cded4b9c9eeb5428f1faa, `modin==0.8.2.1` is installed in docker image from https://github.com/modin-project/modin/blob/b58663bd8f2c8802bb45d7bafa2cc65ae988f959/examples/docker/plasticc-on-omnisci/plasticc-omnisci.dockerfile.

Seems like `intel/label/modin` channel contains an outdated version of the Modin. May need to be changed to `intel/label/modintest` that used in https://github.com/modin-project/modin/blob/b58663bd8f2c8802bb45d7bafa2cc65ae988f959/requirements/env_omnisci.yml



Discussion:

Resolving patch:
diff --git a/examples/docker/census-on-omnisci/census-omnisci.py b/examples/docker/census-on-omnisci/census-omnisci.py
--- a/examples/docker/census-on-omnisci/census-omnisci.py
+++ b/examples/docker/census-on-omnisci/census-omnisci.py
@@ -18,9 +18,9 @@
 from modin.experimental.engines.omnisci_on_ray.frame.omnisci_worker import OmnisciServer
 
 from sklearn import config_context
-import daal4py.sklearn as sklearn
+import sklearnex
 
-sklearn.patch_sklearn()
+sklearnex.patch_sklearn()
 from sklearn.model_selection import train_test_split
 import sklearn.linear_model as lm
 import numpy as np
diff --git a/examples/docker/plasticc-on-omnisci/plasticc-omnisci.py b/examples/docker/plasticc-on-omnisci/plasticc-omnisci.py
--- a/examples/docker/plasticc-on-omnisci/plasticc-omnisci.py
+++ b/examples/docker/plasticc-on-omnisci/plasticc-omnisci.py
@@ -22,9 +22,9 @@
 import numpy as np
 import xgboost as xgb
 
-import daal4py.sklearn as sklearn
+import sklearnex
 
-sklearn.patch_sklearn()
+sklearnex.patch_sklearn()
 from sklearn.model_selection import train_test_split
 from sklearn.preprocessing import LabelEncoder
 


Test patch:
Repository: modin-project/modin

Issue:
Fix warnings during build documentation
[Build log](https://readthedocs.org/projects/modin/builds/14404303/) of documentation contains the next warnings:

```
/home/docs/checkouts/readthedocs.org/user_builds/modin/checkouts/latest/docs/flow/modin/backends/index.rst:4: WARNING: toctree contains reference to nonexisting document 'flow/modin/backends/cudf/index'
looking for now-outdated files... /home/docs/checkouts/readthedocs.org/user_builds/modin/checkouts/latest/docs/flow/modin/experimental/engines/omnisci_on_ray/frame/index.rst:32: WARNING: toctree contains reference to nonexisting document 'flow/modin/experimental/engines/omnisci_on_ray/frame/omnisci_engine'
none found
...
/home/docs/checkouts/readthedocs.org/user_builds/modin/checkouts/latest/docs/flow/modin/experimental/engines/omnisci_on_ray/frame/index.rst: WARNING: document isn't included in any toctree
done
```


Discussion:
Related to #3099.

Resolving patch:
diff --git a/modin/experimental/engines/omnisci_on_ray/frame/df_algebra.py b/modin/experimental/engines/omnisci_on_ray/frame/df_algebra.py
--- a/modin/experimental/engines/omnisci_on_ray/frame/df_algebra.py
+++ b/modin/experimental/engines/omnisci_on_ray/frame/df_algebra.py
@@ -24,7 +24,7 @@ class TransformMapper:
 
     This class is used to map column references to expressions used
     for their computation. This mapper is used to fold expressions
-    from multiple `TransformNode`s into a single expression.
+    from multiple ``TransformNode``-s into a single expression.
 
     Parameters
     ----------
@@ -549,7 +549,7 @@ def __init__(self, base, exprs, fold=True):
 
     def fold(self):
         """
-        Fold two ``TransformNode``s.
+        Fold two ``TransformNode``-s.
 
         If base of this node is another `TransformNode`, then translate all
         expressions in `expr` to its base.
diff --git a/modin/spreadsheet/general.py b/modin/spreadsheet/general.py
--- a/modin/spreadsheet/general.py
+++ b/modin/spreadsheet/general.py
@@ -110,7 +110,7 @@ def from_dataframe(
 
     The first group of options are SlickGrid "grid options" which are
     described in the `SlickGrid documentation
-    <https://github.com/mleibman/SlickGrid/wiki/Grid-Options>`_.
+    <https://github.com/mleibman/SlickGrid/wiki/Grid-Options>`__.
 
     The second group of option are options that were added specifically
     for modin-spreadsheet and therefore are not documented in the SlickGrid documentation.
@@ -148,7 +148,7 @@ def from_dataframe(
 
     The first group of options are SlickGrid "column options" which are
     described in the `SlickGrid documentation
-    <https://github.com/mleibman/SlickGrid/wiki/Column-Options>`_.
+    <https://github.com/mleibman/SlickGrid/wiki/Column-Options>`__.
 
     The ``editable`` option was added specifically for modin-spreadsheet and therefore is
     not documented in the SlickGrid documentation.  This option specifies


Test patch: