internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: modin-project/modin

Issue:
Assignment to DataFrame[column][row] does not work
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:

Ubuntu 19.10

- **Modin version** (`modin.__version__`):

0.7.3+15.gc09e61f.dirty

- **Python version**:

Python 3.7.5

- **Code we can use to reproduce**:

```python
import pandas as pd
import modin.pandas as mpd

pdf = pd.DataFrame([[1, 2], [4, 5], [7, 8]],
                   index=[0, 1, 2],
                   columns=['index', 'shield'])
mdf = mpd.DataFrame([[1, 2], [4, 5], [7, 8]],
                    index=[0, 1, 2],
                    columns=['index', 'shield'])

print(pdf['index'][0])
print(mdf['index'][0])

pdf['index'][0] = 1000
mdf['index'][0] = 1000

print(pdf['index'][0])
print(mdf['index'][0])
```

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->

When assignment is done to `DataFrame[column][row]` in Modin it has no effect. I think assignment is done to a copy of `DataFrame[column]` while the original contents of `DataFrame[column]` is not changed. It is possible to write assignment like this: `col = DataFrame[column]; col[row] = value; DataFrame[column] = col` and this works, but plain assignment to `DataFrame[column][row]` doesn't work as expected.

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:
@gshimansky This is likely because the `__setitem__` in the `Series` is not propagating back to the `_parent` field. `Series` should have a `_parent` if it came from a `DataFrame`. 

Resolving patch:
diff --git a/modin/pandas/series.py b/modin/pandas/series.py
--- a/modin/pandas/series.py
+++ b/modin/pandas/series.py
@@ -323,6 +323,9 @@ def __setitem__(self, key, value):
         self._create_or_update_from_compiler(
             self._query_compiler.setitem(1, key, value), inplace=True
         )
+        # Propagate changes back to parent so that column in dataframe had the same contents
+        if self._parent is not None:
+            self._parent[self.name] = self
 
     def __sub__(self, right):
         return self.sub(right)


Test patch:
diff --git a/modin/pandas/test/test_dataframe.py b/modin/pandas/test/test_dataframe.py
--- a/modin/pandas/test/test_dataframe.py
+++ b/modin/pandas/test/test_dataframe.py
@@ -5111,6 +5111,11 @@ def test___setitem__(self, data):
 
         df_equals(modin_df, pandas_df)
 
+        modin_df[modin_df.columns[0]][modin_df.index[0]] = 12345
+        pandas_df[pandas_df.columns[0]][pandas_df.index[0]] = 12345
+
+        df_equals(modin_df, pandas_df)
+
     def test_setitem_on_empty_df(self):
         columns = ["id", "max_speed", "health"]
         modin_df = pd.DataFrame(columns=columns)
Repository: modin-project/modin

Issue:
Update Ray to 0.8.5
Ray 0.8.5 was released, we should test and update.


Discussion:

Resolving patch:
diff --git a/modin/__init__.py b/modin/__init__.py
--- a/modin/__init__.py
+++ b/modin/__init__.py
@@ -54,7 +54,7 @@ def get_execution_engine():
                 except ImportError:
                     pass
                 else:
-                    if version.parse(ray.__version__) != version.parse("0.8.4"):
+                    if version.parse(ray.__version__) != version.parse("0.8.5"):
                         raise ImportError(
                             "Please `pip install modin[ray]` to install compatible Ray version."
                         )
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -38,7 +38,7 @@ def is_pure(self):
 
 
 dask_deps = ["dask>=2.1.0", "distributed>=2.3.2"]
-ray_deps = ["ray==0.8.4", "pyarrow<0.17"]
+ray_deps = ["ray==0.8.5", "pyarrow<0.17"]
 if "SETUP_PLAT_NAME" in os.environ:
     if "win" in os.environ["SETUP_PLAT_NAME"]:
         all_deps = dask_deps


Test patch:
Repository: modin-project/modin

Issue:
Truncated PR comments don't have a header
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:

Ubuntu 16.04

- **Modin version** (`modin.__version__`):

0.7.3+78.g2d74813

- **Python version**:

Python 3.7.7

- **Code we can use to reproduce**:

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->

When comment is too long, usually because of failed tests, comment script cuts off its beginning which contains `<details>` and `<summary>` tags. This makes comment text very long inside of PR.

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->

Truncated PR comments don't have a header
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:

Ubuntu 16.04

- **Modin version** (`modin.__version__`):

0.7.3+78.g2d74813

- **Python version**:

Python 3.7.7

- **Code we can use to reproduce**:

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->

When comment is too long, usually because of failed tests, comment script cuts off its beginning which contains `<details>` and `<summary>` tags. This makes comment text very long inside of PR.

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:

Resolving patch:
diff --git a/ci/teamcity/comment_on_pr.py b/ci/teamcity/comment_on_pr.py
--- a/ci/teamcity/comment_on_pr.py
+++ b/ci/teamcity/comment_on_pr.py
@@ -40,8 +40,6 @@
 split_by_second = "--------------------------------------------------------------------------------------"
 
 for out in pytest_outputs:
-    full_comment += "<details><summary>Tests Logs</summary>\n"
-    full_comment += "\n\n```\n"
     content = open(out, "r").read()
     full_comment += "".join(
         "".join(
@@ -54,6 +52,7 @@
     )
     if len(full_comment) > 65_000:
         full_comment = full_comment[-65_000:] + "\n\n<b>Remaining output truncated<b>\n\n"
+    full_comment = "<details><summary>Tests Logs</summary>\n\n\n```\n" + full_comment
     full_comment += "\n```\n\n</details>\n"
 if "FAILURES" not in full_comment:
     header += '<h3 align="center">Tests PASSed</h3>\n\n'


Test patch:
Repository: modin-project/modin

Issue:
Update python version requirements to 3.6.1+
Modin requires pandas 1.0.3, which minimum python version is 3.6.1: [link](https://github.com/pandas-dev/pandas/blob/master/setup.py#L774)

But at Modin [setup.py](https://github.com/modin-project/modin/blob/master/setup.py#L68) required version is 3.5. We should to get rid of that inconsistency by updating it to 3.6.1


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -65,5 +65,5 @@ def is_pure(self):
         "ray": ray_deps,
         "all": all_deps,
     },
-    python_requires=">=3.5",
+    python_requires=">=3.6.1",
 )


Test patch:
Repository: modin-project/modin

Issue:
[Ray] `redis_address` parameter of `ray.init` function deprecated
Deprecated since `ray==0.8.4` 

See: https://github.com/ray-project/ray/blob/89aa47012a2e77c7ddcf5b82e1dcdb2fd1d0bed1/python/ray/worker.py#L575

Need to replace `redis_address` with `address` in all place in modin.


Discussion:

Resolving patch:
diff --git a/modin/pandas/__init__.py b/modin/pandas/__init__.py
--- a/modin/pandas/__init__.py
+++ b/modin/pandas/__init__.py
@@ -159,7 +159,7 @@ def initialize_ray():
                 num_cpus=int(num_cpus),
                 include_webui=False,
                 ignore_reinit_error=True,
-                redis_address=redis_address,
+                address=redis_address,
                 redis_password=redis_password,
                 logging_level=100,
             )
@@ -193,7 +193,7 @@ def initialize_ray():
                 ignore_reinit_error=True,
                 plasma_directory=plasma_directory,
                 object_store_memory=object_store_memory,
-                redis_address=redis_address,
+                address=redis_address,
                 redis_password=redis_password,
                 logging_level=100,
                 memory=object_store_memory,


Test patch:
diff --git a/stress_tests/test_kaggle_ipynb.py b/stress_tests/test_kaggle_ipynb.py
--- a/stress_tests/test_kaggle_ipynb.py
+++ b/stress_tests/test_kaggle_ipynb.py
@@ -6,7 +6,7 @@
 import pytest
 
 # import ray
-# ray.init(redis_address="localhost:6379")
+# ray.init(address="localhost:6379")
 
 import modin.pandas as pd
Repository: modin-project/modin

Issue:
TypeError: object of type 'slice' has no len()
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 18.04)**:
- **Modin version** (`modin.__version__`): 0.7.3
- **Python version**: 3.7
- **Code we can use to reproduce**: The code is available on [google collab](https://colab.research.google.com/drive/1n48GuNeeXr1HpbHAQsqIQQgsn1x3VIcM?usp=sharing)

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->
Indexing multicolumn index dataframe generates an exception.

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->
```
TypeError                                 Traceback (most recent call last)
<ipython-input-2-ad6eab4f4683> in <module>()
      4              np.array(['one', 'two', 'one', 'two', 'one', 'two', 'one', 'two'])]
      5 df = pd.DataFrame(np.random.randn(8, 4), index=arrays).T
----> 6 df.loc[df.index>2,:]

/usr/local/lib/python3.6/dist-packages/modin/pandas/indexing.py in __getitem__(self, key)
    249             hasattr(result, "columns")
    250             and isinstance(result.columns, pandas.MultiIndex)
--> 251             and all(col_loc[i] in result.columns.levels[i] for i in range(len(col_loc)))
    252         ):
    253             result.columns = result.columns.droplevel(list(range(len(col_loc))))

TypeError: object of type 'slice' has no len()
```



Discussion:
Hi @jcrichard thanks for the report!

I can reproduce the issue on current master.

Resolving patch:
diff --git a/modin/pandas/indexing.py b/modin/pandas/indexing.py
--- a/modin/pandas/indexing.py
+++ b/modin/pandas/indexing.py
@@ -247,6 +247,7 @@ def __getitem__(self, key):
                 result.index = result.index.droplevel(list(range(len(row_loc))))
         if (
             hasattr(result, "columns")
+            and not isinstance(col_loc, slice)
             and isinstance(result.columns, pandas.MultiIndex)
             and all(col_loc[i] in result.columns.levels[i] for i in range(len(col_loc)))
         ):


Test patch:
diff --git a/modin/pandas/test/test_dataframe.py b/modin/pandas/test/test_dataframe.py
--- a/modin/pandas/test/test_dataframe.py
+++ b/modin/pandas/test/test_dataframe.py
@@ -4496,6 +4496,14 @@ def test_loc_multi_index(self):
             pandas_df.loc["bar", ("col1", "col2")],
         )
 
+        # From issue #1456
+        transposed_modin = modin_df.T
+        transposed_pandas = pandas_df.T
+        df_equals(
+            transposed_modin.loc[transposed_modin.index[:-2], :],
+            transposed_pandas.loc[transposed_pandas.index[:-2], :],
+        )
+
     @pytest.mark.parametrize("data", test_data_values, ids=test_data_keys)
     def test_pop(self, request, data):
         modin_df = pd.DataFrame(data)
Repository: modin-project/modin

Issue:
DataFrame and Series `mean` performance
`mean` is written as a `ReductionFunction`, but can be rewritten as a `MapReduceFunction` for performance and scalability.


Discussion:

Resolving patch:
diff --git a/modin/backends/pandas/query_compiler.py b/modin/backends/pandas/query_compiler.py
--- a/modin/backends/pandas/query_compiler.py
+++ b/modin/backends/pandas/query_compiler.py
@@ -497,6 +497,17 @@ def is_monotonic_decreasing(self):
         lambda x, *args, **kwargs: pandas.DataFrame.sum(x),
         axis=0,
     )
+    mean = MapReduceFunction.register(
+        lambda df, **kwargs: df.apply(
+            lambda x: (x.sum(skipna=kwargs.get("skipna", True)), x.count()),
+            axis=kwargs.get("axis", 0),
+        ),
+        lambda df, **kwargs: df.apply(
+            lambda x: x.apply(lambda d: d[0]).sum(skipna=kwargs.get("skipna", True))
+            / x.apply(lambda d: d[1]).sum(skipna=kwargs.get("skipna", True)),
+            axis=kwargs.get("axis", 0),
+        ),
+    )
 
     # END MapReduce operations
 
@@ -513,7 +524,6 @@ def is_monotonic_decreasing(self):
     var = ReductionFunction.register(pandas.DataFrame.var)
     sum_min_count = ReductionFunction.register(pandas.DataFrame.sum)
     prod_min_count = ReductionFunction.register(pandas.DataFrame.prod)
-    mean = ReductionFunction.register(pandas.DataFrame.mean)
     quantile_for_single_value = ReductionFunction.register(pandas.DataFrame.quantile)
     mad = ReductionFunction.register(pandas.DataFrame.mad)
     to_datetime = ReductionFunction.register(
@@ -1163,6 +1173,7 @@ def dropna(self, **kwargs):
         Return:
             a new QueryCompiler
         """
+
         return self.__constructor__(
             self._modin_frame.filter_full_axis(
                 kwargs.get("axis", 0) ^ 1,


Test patch:
Repository: modin-project/modin

Issue:
Upgrade to pandas 1.0.5
https://github.com/pandas-dev/pandas/releases/tag/v1.0.5


Discussion:

Resolving patch:
diff --git a/modin/pandas/__init__.py b/modin/pandas/__init__.py
--- a/modin/pandas/__init__.py
+++ b/modin/pandas/__init__.py
@@ -13,7 +13,7 @@
 
 import pandas
 
-__pandas_version__ = "1.0.3"
+__pandas_version__ = "1.0.5"
 
 if pandas.__version__ != __pandas_version__:
     import warnings
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -58,7 +58,7 @@ def is_pure(self):
     url="https://github.com/modin-project/modin",
     long_description=long_description,
     long_description_content_type="text/markdown",
-    install_requires=["pandas==1.0.3", "packaging"],
+    install_requires=["pandas==1.0.5", "packaging"],
     extras_require={
         # can be installed by pip install modin[dask]
         "dask": dask_deps,


Test patch:
diff --git a/requirements/df_test_requires.txt b/requirements/df_test_requires.txt
--- a/requirements/df_test_requires.txt
+++ b/requirements/df_test_requires.txt
@@ -1,4 +1,4 @@
-pandas==1.0.3
+pandas==1.0.5
 numpy
 psutil==5.6.6
 pytest
diff --git a/requirements/windows_test_requires.txt b/requirements/windows_test_requires.txt
--- a/requirements/windows_test_requires.txt
+++ b/requirements/windows_test_requires.txt
@@ -1,4 +1,4 @@
-pandas==1.0.3
+pandas==1.0.5
 numpy
 pyarrow<0.17
 dask[complete]>=2.1.0
Repository: modin-project/modin

Issue:
`num_cpus` parameter shouldn't be passed in `ray.init`
### Describe the problem
<!-- Describe the problem clearly here. -->

In the case when `modin` connects to the cluster it should not pass `num_cpus` parameter (see [ray exception message](https://github.com/ray-project/ray/blob/46962f5db11d23be59c8a0abcf84d4281456cef4/python/ray/worker.py#L728))

I don't have any logs at the moment :( 


Discussion:

Resolving patch:
diff --git a/modin/pandas/__init__.py b/modin/pandas/__init__.py
--- a/modin/pandas/__init__.py
+++ b/modin/pandas/__init__.py
@@ -156,7 +156,6 @@ def initialize_ray():
         if cluster == "True" and redis_address is not None:
             # We only start ray in a cluster setting for the head node.
             ray.init(
-                num_cpus=int(num_cpus),
                 include_webui=False,
                 ignore_reinit_error=True,
                 address=redis_address,


Test patch:
Repository: modin-project/modin

Issue:
Upgrade to ray 0.8.6
This release also adds experimental support for windows.

We can try to use this feature and give ray's developers feedback.



Discussion:
I have tested it, and it appears to work fine in a clean environment.
So we can update this dependency and make `ray` as the default engine for windows. Right?

Resolving patch:
diff --git a/modin/__init__.py b/modin/__init__.py
--- a/modin/__init__.py
+++ b/modin/__init__.py
@@ -55,7 +55,7 @@ def get_execution_engine():
                 except ImportError:
                     pass
                 else:
-                    if version.parse(ray.__version__) != version.parse("0.8.5"):
+                    if version.parse(ray.__version__) != version.parse("0.8.6"):
                         raise ImportError(
                             "Please `pip install modin[ray]` to install compatible Ray version."
                         )
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -38,7 +38,7 @@ def is_pure(self):
 
 
 dask_deps = ["dask>=2.1.0", "distributed>=2.3.2"]
-ray_deps = ["ray==0.8.5", "pyarrow<0.17"]
+ray_deps = ["ray==0.8.6", "pyarrow<0.17"]
 if "SETUP_PLAT_NAME" in os.environ:
     if "win" in os.environ["SETUP_PLAT_NAME"]:
         all_deps = dask_deps


Test patch:
Repository: modin-project/modin

Issue:
Support Ray on Windows
Related to #1664.

Ray 0.8.6 has "alpha" support for Windows. It works, but has not had extensive testing yet. It'd be great to be able to us Modin + Ray on Windows.



Discussion:
Feel free to close if this is a duplicate.

Resolving patch:
diff --git a/modin/__init__.py b/modin/__init__.py
--- a/modin/__init__.py
+++ b/modin/__init__.py
@@ -12,7 +12,6 @@
 # governing permissions and limitations under the License.
 
 import os
-import sys
 import warnings
 from packaging import version
 import collections
@@ -48,27 +47,24 @@ def get_execution_engine():
         if "MODIN_DEBUG" in os.environ:
             return "Python"
         else:
-            if sys.platform != "win32":
-                try:
-                    import ray
-
-                except ImportError:
-                    pass
-                else:
-                    if version.parse(ray.__version__) != version.parse("0.8.6"):
-                        raise ImportError(
-                            "Please `pip install modin[ray]` to install compatible Ray version."
-                        )
-                    return "Ray"
+            try:
+                import ray
+
+            except ImportError:
+                pass
+            else:
+                if version.parse(ray.__version__) != version.parse("0.8.6"):
+                    raise ImportError(
+                        "Please `pip install modin[ray]` to install compatible Ray version."
+                    )
+                return "Ray"
             try:
                 import dask
                 import distributed
 
             except ImportError:
                 raise ImportError(
-                    "Please `pip install {}modin[dask]` to install an engine".format(
-                        "modin[ray]` or `" if sys.platform != "win32" else ""
-                    )
+                    "Please `pip install modin[ray]` or `modin[dask]` to install an engine"
                 )
             else:
                 if version.parse(dask.__version__) < version.parse(
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -39,13 +39,8 @@ def is_pure(self):
 
 dask_deps = ["dask>=2.1.0", "distributed>=2.3.2"]
 ray_deps = ["ray==0.8.6", "pyarrow<0.17"]
-if "SETUP_PLAT_NAME" in os.environ:
-    if "win" in os.environ["SETUP_PLAT_NAME"]:
-        all_deps = dask_deps
-    else:
-        all_deps = dask_deps + ray_deps
-else:
-    all_deps = dask_deps if os.name == "nt" else dask_deps + ray_deps
+
+all_deps = dask_deps + ray_deps
 
 setup(
     name="modin",


Test patch:
Repository: modin-project/modin

Issue:
autocorr function implementation



Discussion:

Resolving patch:
diff --git a/modin/pandas/series.py b/modin/pandas/series.py
--- a/modin/pandas/series.py
+++ b/modin/pandas/series.py
@@ -546,7 +546,29 @@ def array(self):
         return self._default_to_pandas(pandas.Series.array)
 
     def autocorr(self, lag=1):
-        return self._default_to_pandas(pandas.Series.autocorr, lag=lag)
+        """
+        Compute the lag-N autocorrelation.
+
+        This method computes the Pearson correlation between
+        the Series and its shifted self.
+
+        Parameters
+        ----------
+        lag : int, default 1
+            Number of lags to apply before performing autocorrelation.
+
+        Returns
+        -------
+        float
+            The Pearson correlation between self and self.shift(lag).
+
+        Notes
+        -----
+        If the Pearson correlation is not well defined return 'NaN'.
+
+        Autocorrelation floating point precision may slightly differ from pandas.
+        """
+        return self.corr(self.shift(lag))
 
     def between(self, left, right, inclusive=True):
         return self._default_to_pandas(


Test patch:
diff --git a/modin/pandas/test/test_series.py b/modin/pandas/test/test_series.py
--- a/modin/pandas/test/test_series.py
+++ b/modin/pandas/test/test_series.py
@@ -880,10 +880,12 @@ def test_at_time():
 
 
 @pytest.mark.parametrize("data", test_data_values, ids=test_data_keys)
-def test_autocorr(data):
-    modin_series, _ = create_test_series(data)  # noqa: F841
-    with pytest.warns(UserWarning):
-        modin_series.autocorr()
+@pytest.mark.parametrize("lag", [1, 2, 3])
+def test_autocorr(data, lag):
+    modin_series, pandas_series = create_test_series(data)
+    modin_result = modin_series.autocorr(lag=lag)
+    pandas_result = pandas_series.autocorr(lag=lag)
+    df_equals(modin_result, pandas_result)
 
 
 @pytest.mark.parametrize("data", test_data_values, ids=test_data_keys)