internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: modin-project/modin

Issue:
kurt function with level parameter should have it's own implementation
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu 18.04
- **Modin version** (`modin.__version__`): 0.7.3+200.g8e7a568
- **Python version**: 3.8.3
- **Code we can use to reproduce**:

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->
Pandas DataFrame.apply("<func_name>") function calls <func_name> of the DataFrame, so Modin DataFrame should have own implementation of kurt function

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:

Resolving patch:
diff --git a/modin/pandas/base.py b/modin/pandas/base.py
--- a/modin/pandas/base.py
+++ b/modin/pandas/base.py
@@ -1473,13 +1473,16 @@ def kurt(self, axis=None, skipna=None, level=None, numeric_only=None, **kwargs):
         axis = self._get_axis_number(axis)
         if level is not None:
             func_kwargs = {
-                "axis": axis,
                 "skipna": skipna,
                 "level": level,
                 "numeric_only": numeric_only,
             }
 
-            return self.apply("kurt", **func_kwargs)
+            return self.__constructor__(
+                query_compiler=self._query_compiler._apply_text_func_elementwise(
+                    "kurt", axis, **func_kwargs
+                )
+            )
 
         if numeric_only:
             self._validate_dtypes(numeric_only=True)


Test patch:
Repository: modin-project/modin

Issue:
Dask Error: distributed.core - ERROR - addresses should be strings or tuples, got None
The default to pandas test has been failing recently on Dask with Windows. I have searched and the errors thrown do not appear to be known issues at the moment.

Link to test failure: https://github.com/modin-project/modin/runs/862907959?check_suite_focus=true


Discussion:
```
Traceback (most recent call last):
  File "C:\hostedtoolcache\windows\Python\3.7.8\x64\lib\site-packages\distributed\utils.py", line 656, in log_errors
    yield
  File "C:\hostedtoolcache\windows\Python\3.7.8\x64\lib\site-packages\distributed\scheduler.py", line 2208, in remove_worker
    address = self.coerce_address(address)
  File "C:\hostedtoolcache\windows\Python\3.7.8\x64\lib\site-packages\distributed\scheduler.py", line 4946, in coerce_address
    raise TypeError("addresses should be strings or tuples, got %r" % (addr,))
TypeError: addresses should be strings or tuples, got None
distributed.core - ERROR - addresses should be strings or tuples, got None
Traceback (most recent call last):
  File "C:\hostedtoolcache\windows\Python\3.7.8\x64\lib\site-packages\distributed\core.py", line 513, in handle_comm
    result = await result

```

Resolving patch:
diff --git a/modin/pandas/__init__.py b/modin/pandas/__init__.py
--- a/modin/pandas/__init__.py
+++ b/modin/pandas/__init__.py
@@ -262,13 +262,15 @@ def _update_engine(publisher: Publisher):
 
             warnings.warn("The Dask Engine for Modin is experimental.")
 
-        try:
-            dask_client = get_client()
-        except ValueError:
-            from distributed import Client
+            try:
+                dask_client = get_client()
+            except ValueError:
+                from distributed import Client
 
-            num_cpus = os.environ.get("MODIN_CPUS", None) or multiprocessing.cpu_count()
-            dask_client = Client(n_workers=int(num_cpus))
+                num_cpus = (
+                    os.environ.get("MODIN_CPUS", None) or multiprocessing.cpu_count()
+                )
+                dask_client = Client(n_workers=int(num_cpus))
 
     elif publisher.get() != "Python":
         raise ImportError("Unrecognized execution engine: {}.".format(publisher.get()))


Test patch:
Repository: modin-project/modin

Issue:
ClusterError class should implement its own version of __str__ method
<!--
General questions should be asked on the mailing list modin-dev@googlegroups.com.

Before submitting an issue, please fill out the following form.
-->

### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:
- **Modin installed from (source or binary)**:
- **Modin version**:
- **Python version**:
- **Exact command to reproduce**:

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->
`ClusterError` includes the `cause` field that is not printed. This makes it difficult to understand the problems that cause exceptions.

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:

Resolving patch:
diff --git a/modin/experimental/cloud/base.py b/modin/experimental/cloud/base.py
--- a/modin/experimental/cloud/base.py
+++ b/modin/experimental/cloud/base.py
@@ -26,6 +26,11 @@ def __init__(self, *args, cause: BaseException = None, traceback: str = None, **
         self.traceback = traceback
         super().__init__(*args, **kw)
 
+    def __str__(self):
+        if self.clause:
+            return f"clause: {self.cause}\n{super()}"
+        return str(super())
+
 
 class CannotSpawnCluster(ClusterError):
     """


Test patch:
Repository: modin-project/modin

Issue:
dataframe repr broken if unlimited pandas printing is enabled
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Docker container: `python:3.8.3-slim`, uname output: `Linux bf69e4362f35 4.19.76-linuxkit #1 SMP Tue May 26 11:42:35 UTC 2020 x86_64 GNU/Linux`
- **Modin version** (`modin.__version__`): initially encountered on 0.7.4, but confirmed on current SHA `37b7d2c4c5b3ac04408f362b3989de30dd3df1a2`
- **Python version**: 3.8.3
- **Code we can use to reproduce**: see in source code


### Describe the problem

I'm only just starting to switch from pandas to modin, so this could be an obvious case. but in case it's not, I figured I'd report this.

if you set_option on max_rows/max_columns to `None` (unlimited), you encounter a few issues. I think in my case they all stem from this snippet here.

### Source code / logs
```
Python 3.8.3 (default, Jun  9 2020, 17:49:41) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import modin.pandas as pandas
>>> pandas.DataFrame()
UserWarning: Distributing <class 'NoneType'> object. This may take some time.
Empty DataFrame
Columns: []
Index: []
>>> pandas.set_option("display.max_rows", None)
>>> pandas.DataFrame()
UserWarning: Distributing <class 'NoneType'> object. This may take some time.
Empty DataFrame
Columns: []
Index: []
>>> pandas.set_option("display.max_columns", None)
>>> pandas.DataFrame()
UserWarning: Distributing <class 'NoneType'> object. This may take some time.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/.cache/pypoetry/virtualenvs/onepassword-etl-NcjVOHQt-py3.8/src/modin/modin/pandas/dataframe.py", line 151, in __repr__
    col_counter += len(str(self.columns[i])) + 1
  File "/root/.cache/pypoetry/virtualenvs/onepassword-etl-NcjVOHQt-py3.8/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3930, in __getitem__
    return getitem(key)
IndexError: index 0 is out of bounds for axis 0 with size 0
>>> ```



Discussion:
Hi @orenmazor thanks for reporting this!

I'll flag the fix for the next release, but it is happening within the next week so it may fall to the release after that. Thanks again for reporting!
This turned out to be a one line fix 🎉 . 

Resolving patch:
diff --git a/modin/pandas/dataframe.py b/modin/pandas/dataframe.py
--- a/modin/pandas/dataframe.py
+++ b/modin/pandas/dataframe.py
@@ -145,6 +145,7 @@ def __repr__(self):
             "display.expand_frame_repr"
         ):
             width, _ = console.get_console_size()
+            width = min(width, len(self.columns))
             col_counter = 0
             i = 0
             while col_counter < width:


Test patch:
Repository: modin-project/modin

Issue:
Add notes about using MODIN_SOCKS_PROXY variable
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. What kind of performance improvements would you like to see with this new API?




Discussion:

Resolving patch:
diff --git a/modin/experimental/cloud/cluster.py b/modin/experimental/cloud/cluster.py
--- a/modin/experimental/cloud/cluster.py
+++ b/modin/experimental/cloud/cluster.py
@@ -252,6 +252,14 @@ def create(
         The object that knows how to destroy the cluster and how to activate it as remote context.
         Note that by default spawning and destroying of the cluster happens in the background,
         as it's usually a rather lengthy process.
+
+    Notes
+    -----
+    Cluster computation actually can work when proxies are required to access the cloud.
+    You should set normal "http_proxy"/"https_proxy" variables for HTTP/HTTPS proxies and
+    set "MODIN_SOCKS_PROXY" variable for SOCKS proxy before calling the function.
+
+    Using SOCKS proxy requires Ray newer than 0.8.6, which might need to be installed manually.
     """
     if not isinstance(provider, Provider):
         provider = Provider(


Test patch:
Repository: modin-project/modin

Issue:
add runner of taxi benchmark as example



Discussion:

Resolving patch:
diff --git a/examples/cluster/taxi-runner.py b/examples/cluster/taxi-runner.py
new file mode 100644
--- /dev/null
+++ b/examples/cluster/taxi-runner.py
@@ -0,0 +1,52 @@
+# Licensed to Modin Development Team under one or more contributor license agreements.
+# See the NOTICE file distributed with this work for additional information regarding
+# copyright ownership.  The Modin Development Team licenses this file to you under the
+# Apache License, Version 2.0 (the "License"); you may not use this file except in
+# compliance with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software distributed under
+# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
+# ANY KIND, either express or implied. See the License for the specific language
+# governing permissions and limitations under the License.
+
+
+# pip install git+https://github.com/intel-go/ibis.git@develop
+
+# NOTE: expects https://github.com/intel-go/omniscripts/ checked out and in PYTHONPATH
+
+# the following import turns on experimental mode in Modin,
+# including enabling running things in remote cloud
+import modin.experimental.pandas as pd  # noqa: F401
+from modin.experimental.cloud import create_cluster, get_connection
+
+from taxi import run_benchmark as run_benchmark
+from taxi.taxibench_pandas_ibis import etl_pandas
+
+test_cluster = create_cluster(
+    "aws",
+    "aws_credentials",
+    cluster_name="rayscale-test",
+    region="eu-north-1",
+    zone="eu-north-1b",
+    image="ami-00e1e82d7d4ca80d3",
+)
+with test_cluster:
+    conn = get_connection()
+    np = conn.modules["numpy"]
+    etl_pandas.__globals__["np"] = np
+
+    parameters = {
+        "data_file": "https://modin-datasets.s3.amazonaws.com/trips_data.csv",
+        # "data_file": "s3://modin-datasets/trips_data.csv",
+        "dfiles_num": 1,
+        "validation": False,
+        "no_ibis": True,
+        "no_pandas": False,
+        "pandas_mode": "Modin_on_ray",
+        "ray_tmpdir": "/tmp",
+        "ray_memory": 1024 * 1024 * 1024,
+    }
+
+    run_benchmark(parameters)


Test patch:
Repository: modin-project/modin

Issue:
Fix AttributeError in ClusterError class
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:
- **Modin version** (`modin.__version__`):
- **Python version**:
- **Code we can use to reproduce**:

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->

>>> test = CannotDestroyCluster("Cannot destroy cluster")
>>> print(test)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "\modin\modin\experimental\cloud\base.py", line 30, in __str__
    if self.clause:
AttributeError: 'CannotDestroyCluster' object has no attribute 'clause'



Discussion:

Resolving patch:
diff --git a/modin/experimental/cloud/base.py b/modin/experimental/cloud/base.py
--- a/modin/experimental/cloud/base.py
+++ b/modin/experimental/cloud/base.py
@@ -27,9 +27,9 @@ def __init__(self, *args, cause: BaseException = None, traceback: str = None, **
         super().__init__(*args, **kw)
 
     def __str__(self):
-        if self.clause:
-            return f"clause: {self.cause}\n{super()}"
-        return str(super())
+        if self.cause:
+            return f"cause: {self.cause}\n{super().__str__()}"
+        return super().__str__()
 
 
 class CannotSpawnCluster(ClusterError):


Test patch:
Repository: modin-project/modin

Issue:
test_internals fails if modin.pandas is imported before it runs
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:
- **Modin version** (`modin.__version__`):
- **Python version**:
- **Code we can use to reproduce**:

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
Doing `import modin.pandas` adds another callback to `execution_engine` which fails if a value it gets is an unknown value.
We put some strange values there during unit tests, though.

We need to fix the tests so they no longer fail.

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:

Resolving patch:
diff --git a/modin/pandas/__init__.py b/modin/pandas/__init__.py
--- a/modin/pandas/__init__.py
+++ b/modin/pandas/__init__.py
@@ -97,6 +97,9 @@
 
 _is_first_update = {}
 dask_client = None
+_NOINIT_ENGINES = {
+    "Python"
+}  # engines that don't require initialization, useful for unit tests
 
 
 def _update_engine(publisher: Publisher):
@@ -157,7 +160,7 @@ def init_remote_ray():
 
         num_cpus = remote_ray.cluster_resources()["CPU"]
 
-    elif publisher.get() != "Python":
+    elif publisher.get() not in _NOINIT_ENGINES:
         raise ImportError("Unrecognized execution engine: {}.".format(publisher.get()))
 
     _is_first_update[publisher.get()] = False


Test patch:
diff --git a/modin/data_management/test/test_dispatcher.py b/modin/data_management/test/test_dispatcher.py
--- a/modin/data_management/test/test_dispatcher.py
+++ b/modin/data_management/test/test_dispatcher.py
@@ -18,6 +18,8 @@
 from modin.data_management.dispatcher import EngineDispatcher, FactoryNotFoundError
 from modin.data_management import factories
 
+import modin.pandas as pd
+
 
 class PandasOnTestFactory(factories.BaseFactory):
     """
@@ -63,6 +65,9 @@ def prepare(cls):
 factories.TestOnPythonFactory = TestOnPythonFactory
 factories.FooOnBarFactory = FooOnBarFactory
 
+# register them as known "no init" engines for modin.pandas
+pd._NOINIT_ENGINES |= {"Test", "Bar"}
+
 
 def test_default_engine():
     assert issubclass(EngineDispatcher.get_engine(), factories.BaseFactory)
Repository: modin-project/modin

Issue:
add runner of h2o benchmark as example



Discussion:

Resolving patch:
diff --git a/examples/cluster/h2o-runner.py b/examples/cluster/h2o-runner.py
new file mode 100644
--- /dev/null
+++ b/examples/cluster/h2o-runner.py
@@ -0,0 +1,50 @@
+# Licensed to Modin Development Team under one or more contributor license agreements.
+# See the NOTICE file distributed with this work for additional information regarding
+# copyright ownership.  The Modin Development Team licenses this file to you under the
+# Apache License, Version 2.0 (the "License"); you may not use this file except in
+# compliance with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software distributed under
+# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
+# ANY KIND, either express or implied. See the License for the specific language
+# governing permissions and limitations under the License.
+
+
+# pip install git+https://github.com/intel-go/ibis.git@develop
+
+# NOTE: expects https://github.com/intel-go/omniscripts/ checked out and in PYTHONPATH
+
+# the following import turns on experimental mode in Modin,
+# including enabling running things in remote cloud
+import modin.experimental.pandas as pd  # noqa: F401
+from modin.experimental.cloud import create_cluster
+
+from h2o import run_benchmark
+
+test_cluster = create_cluster(
+    "aws",
+    "aws_credentials",
+    cluster_name="rayscale-test",
+    region="eu-north-1",
+    zone="eu-north-1b",
+    image="ami-00e1e82d7d4ca80d3",
+)
+with test_cluster:
+    parameters = {
+        "no_pandas": False,
+        "pandas_mode": "Modin_on_ray",
+        "ray_tmpdir": "/tmp",
+        "ray_memory": 1024 * 1024 * 1024,
+        "extended_functionality": False,
+    }
+
+    # G1... - for groupby queries; J1... - for join queries;
+    # Additional required files inside h2o-data folder:
+    # - J1_1e6_1e0_0_0.csv
+    # - J1_1e6_1e3_0_0.csv
+    # - J1_1e6_1e6_0_0.csv
+    for data_file in ["G1_5e5_1e2_0_0.csv", "J1_1e6_NA_0_0.csv"]:
+        parameters["data_file"] = f"https://modin-datasets.s3.amazonaws.com/h2o/{data_file}"
+        run_benchmark(parameters)


Test patch:
Repository: modin-project/modin

Issue:
'sort_index' do not consider 'axis` parameter in case of MultiIndex
### Describe the problem
<!-- Describe the problem clearly here. -->
In a current `sort_index` implementation `axis` parameter do not consider in case of MultiIndex
https://github.com/modin-project/modin/blob/79be7d6f4640ec0fed5e67595606840f781fbe32/modin/backends/pandas/query_compiler.py#L1559-L1568



Discussion:

Resolving patch:
diff --git a/modin/backends/pandas/query_compiler.py b/modin/backends/pandas/query_compiler.py
--- a/modin/backends/pandas/query_compiler.py
+++ b/modin/backends/pandas/query_compiler.py
@@ -1562,6 +1562,7 @@ def sort_index(self, **kwargs):
         ):
             return self.default_to_pandas(
                 pandas.DataFrame.sort_index,
+                axis=axis,
                 level=level,
                 sort_remaining=sort_remaining,
                 **kwargs


Test patch:
diff --git a/modin/pandas/test/test_dataframe.py b/modin/pandas/test/test_dataframe.py
--- a/modin/pandas/test/test_dataframe.py
+++ b/modin/pandas/test/test_dataframe.py
@@ -5970,11 +5970,19 @@ def test_sort_index(self, data, axis, ascending, na_position, sort_remaining):
         pandas_df.index = pandas.MultiIndex.from_tuples(
             [(i // 10, i // 5, i) for i in range(len(pandas_df))]
         )
+        modin_df.columns = pd.MultiIndex.from_tuples(
+            [(i // 10, i // 5, i) for i in range(len(modin_df.columns))]
+        )
+        pandas_df.columns = pd.MultiIndex.from_tuples(
+            [(i // 10, i // 5, i) for i in range(len(pandas_df.columns))]
+        )
 
         with pytest.warns(UserWarning):
             df_equals(modin_df.sort_index(level=0), pandas_df.sort_index(level=0))
         with pytest.warns(UserWarning):
             df_equals(modin_df.sort_index(axis=0), pandas_df.sort_index(axis=0))
+        with pytest.warns(UserWarning):
+            df_equals(modin_df.sort_index(axis=1), pandas_df.sort_index(axis=1))
 
     @pytest.mark.parametrize("data", test_data_values, ids=test_data_keys)
     @pytest.mark.parametrize("axis", axis_values, ids=axis_keys)
Repository: modin-project/modin

Issue:
Use cloudpickle library for experimental.cloud features
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. What kind of performance improvements would you like to see with this new API?

For some cases we need to do pickling for lambdas.


Discussion:

Resolving patch:
diff --git a/modin/experimental/cloud/rpyc_proxy.py b/modin/experimental/cloud/rpyc_proxy.py
--- a/modin/experimental/cloud/rpyc_proxy.py
+++ b/modin/experimental/cloud/rpyc_proxy.py
@@ -16,7 +16,7 @@
 import os
 
 import rpyc
-from rpyc.lib.compat import pickle
+import cloudpickle as pickle
 from rpyc.lib import get_methods
 
 from rpyc.core import netref, AsyncResult, consts


Test patch:
Repository: modin-project/modin

Issue:
Update setup.py dependencies by libraries used in experimental features.



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -10,7 +10,7 @@
 except ImportError:
     HAS_WHEEL = False
 
-with open("README.md", "r") as fh:
+with open("README.md", "r", encoding="utf-8") as fh:
     long_description = fh.read()
 
 if HAS_WHEEL:
@@ -39,8 +39,9 @@ def is_pure(self):
 
 dask_deps = ["dask>=2.12.0", "distributed>=2.12.0"]
 ray_deps = ["ray==0.8.6", "pyarrow<0.17"]
+remote_deps = ["rpyc", "cloudpickle", "boto3"]
 
-all_deps = dask_deps + ray_deps
+all_deps = dask_deps + ray_deps + remote_deps
 
 setup(
     name="modin",
@@ -49,6 +50,7 @@ def is_pure(self):
     distclass=ModinDistribution,
     description="Modin: Make your pandas code run faster by changing one line of code.",
     packages=find_packages(),
+    include_package_data=True,
     license="Apache 2",
     url="https://github.com/modin-project/modin",
     long_description=long_description,
@@ -58,6 +60,7 @@ def is_pure(self):
         # can be installed by pip install modin[dask]
         "dask": dask_deps,
         "ray": ray_deps,
+        "remote": remote_deps,
         "all": all_deps,
     },
     python_requires=">=3.6.1",


Test patch:
Repository: modin-project/modin

Issue:
[REFACTOR] Avoid unconditional index access in DataFrameGroupBy ctor
Currently, each groupby statement works as a barrier for lazy execution due to unconditional index access in DataFrameGroupBy ctor.


Discussion:

Resolving patch:
diff --git a/modin/pandas/groupby.py b/modin/pandas/groupby.py
--- a/modin/pandas/groupby.py
+++ b/modin/pandas/groupby.py
@@ -49,7 +49,6 @@ def __init__(
         self._idx_name = idx_name
         self._df = df
         self._query_compiler = self._df._query_compiler
-        self._index = self._query_compiler.index
         self._columns = self._query_compiler.columns
         self._by = by
         self._drop = drop
@@ -83,6 +82,10 @@ def __init__(
         }
         self._kwargs.update(kwargs)
 
+    @property
+    def _index(self):
+        return self._query_compiler.index
+
     @property
     def _sort(self):
         return self._kwargs.get("sort")


Test patch: