internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: modin-project/modin

Issue:
flake8 update causes lint failures in master

### Describe the problem
<!-- Describe the problem clearly here. -->
We need to fix 2 new lint errors exposed after an update to flake8

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->
```
modin/pandas/dataframe.py:201:38: F632 use ==/!= to compare str, bytes, and int literals
modin/pandas/test/test_dataframe.py:1266:22: F632 use ==/!= to compare str, bytes, and int literals
```


Discussion:

Resolving patch:
diff --git a/modin/pandas/dataframe.py b/modin/pandas/dataframe.py
--- a/modin/pandas/dataframe.py
+++ b/modin/pandas/dataframe.py
@@ -198,7 +198,7 @@ def _validate_eval_query(self, expr, **kwargs):
             expr: The expression to evaluate. This string cannot contain any
                 Python statements, only Python expressions.
         """
-        if isinstance(expr, str) and expr is "":
+        if isinstance(expr, str) and expr == "":
             raise ValueError("expr cannot be an empty string")
 
         if isinstance(expr, str) and "@" in expr:


Test patch:
diff --git a/modin/pandas/test/test_dataframe.py b/modin/pandas/test/test_dataframe.py
--- a/modin/pandas/test/test_dataframe.py
+++ b/modin/pandas/test/test_dataframe.py
@@ -1263,7 +1263,7 @@ def test_cumsum(request, data, axis, skipna):
     # pandas exhibits weird behavior for this case
     # Remove this case when we can pull the error messages from backend
     if name_contains(request.node.name, ["datetime_timedelta_data"]) and (
-        axis == 0 or axis is "rows"
+        axis == 0 or axis == "rows"
     ):
         with pytest.raises(TypeError):
             modin_df.cumsum(axis=axis, skipna=skipna)
Repository: modin-project/modin

Issue:
Repo does not work in BinderHub because we request too much memory

### Describe the problem
<!-- Describe the problem clearly here. -->
In Modin, we request a minimum of 1GB of memory, rounding. In Docker/Kubernetes/other memory limited areas, this is an issue. It is resolved by setting `plasma_directory` in these environments.

### Source code / logs
https://mybinder.org/v2/gh/modin-project/modin/master
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:
After looking into it, we already have checks in place so it should be possible to request less than 1GB of memory. I will debug this a bit.

Resolving patch:
diff --git a/modin/pandas/__init__.py b/modin/pandas/__init__.py
--- a/modin/pandas/__init__.py
+++ b/modin/pandas/__init__.py
@@ -109,20 +109,13 @@ def initialize_ray():
             object_store_memory = os.environ["MODIN_MEMORY"]
         # In case anything failed above, we can still improve the memory for Modin.
         if object_store_memory is None:
-            # We try to give a little more memory to Modin, but if we can't use psutil,
-            # we'll just use Ray's default
-            try:
-                from psutil import virtual_memory
-
-                # Round down to the nearest Gigabyte.
-                object_store_memory = int(
-                    0.6 * virtual_memory().total // 10 ** 9 * 10 ** 9
-                )
-                # If the memory pool is smaller than 2GB, just use the default in ray.
-                if object_store_memory == 0:
-                    object_store_memory = None
-            except ImportError:
-                pass
+            # Round down to the nearest Gigabyte.
+            object_store_memory = int(
+                0.6 * ray.utils.get_system_memory() // 10 ** 9 * 10 ** 9
+            )
+            # If the memory pool is smaller than 2GB, just use the default in ray.
+            if object_store_memory == 0:
+                object_store_memory = None
         ray.init(
             redirect_output=True,
             include_webui=False,


Test patch:
Repository: modin-project/modin

Issue:
Modin not working on latest ray master
With

```python
import ray
import modin.pandas as pd
frame_data = [1, 2, 3, 4, 5, 6, 7, 8]
frame = pd.DataFrame(frame_data)
assert frame.sum().squeeze() == sum(frame_data)
```

we get the following error:

```
Traceback (most recent call last):
  File "python/ray/test/test_modin.py", line 16, in <module>
    test_modin_import()
  File "python/ray/test/test_modin.py", line 9, in test_modin_import
    import modin.pandas as pd
  File "/home/travis/.local/lib/python3.6/site-packages/ray-0.7.0.dev0-py3.6-linux-x86_64.egg/ray/modin/modin/pandas/__init__.py", line 134, in <module>
    initialize_ray()
  File "/home/travis/.local/lib/python3.6/site-packages/ray-0.7.0.dev0-py3.6-linux-x86_64.egg/ray/modin/modin/pandas/__init__.py", line 126, in initialize_ray
    object_store_memory=object_store_memory,
  File "/home/travis/.local/lib/python3.6/site-packages/ray-0.7.0.dev0-py3.6-linux-x86_64.egg/ray/worker.py", line 1427, in init
    _internal_config=_internal_config,
  File "/home/travis/.local/lib/python3.6/site-packages/ray-0.7.0.dev0-py3.6-linux-x86_64.egg/ray/parameter.py", line 145, in __init__
    self._check_usage()
  File "/home/travis/.local/lib/python3.6/site-packages/ray-0.7.0.dev0-py3.6-linux-x86_64.egg/ray/parameter.py", line 189, in _check_usage
    "The redirect_worker_output argument is deprecated. To "
DeprecationWarning: The redirect_worker_output argument is deprecated. To control logging to the driver, use the 'log_to_driver' argument to 'ray.init()'
```

see also https://github.com/ray-project/ray/pull/4069 and https://github.com/ray-project/ray/commit/5f717518914d303126ed1bac3752150c0284fc3e.


Discussion:

Resolving patch:
diff --git a/modin/pandas/__init__.py b/modin/pandas/__init__.py
--- a/modin/pandas/__init__.py
+++ b/modin/pandas/__init__.py
@@ -118,9 +118,7 @@ def initialize_ray():
             if object_store_memory == 0:
                 object_store_memory = None
         ray.init(
-            redirect_output=True,
             include_webui=False,
-            redirect_worker_output=True,
             ignore_reinit_error=True,
             plasma_directory=plasma_directory,
             object_store_memory=object_store_memory,


Test patch:
Repository: modin-project/modin

Issue:
Numpy 1.16 support for future read_hdf
Due to this issue (https://github.com/PyTables/PyTables/issues/717) it seems that, at least on my host machine, the latest version of numpy is needed to store and play with large datasets using hdf5. Naturally, I would love to use modin (ray) for these purposes and but realized that modin runs with numpy<=1.15.

I downloaded the source of Ray from github to test to see if numpy 1.15+ was supported and it seems that tests were failing for numpy 1.16.1. I was curious if modin planned to support higher versions of numpy in the near term as would be required to interplay with py tables.


Discussion:
Hi @adithyabsk, thanks for posting this!

We do plan to update this dependency in the near-term, but there are some minor changes needed internally to Modin. Ideally this will happen early-mid March. We have other robustness issues higher on the backlog, so for now I will add this to the [Help Wanted](https://github.com/modin-project/modin/labels/Help%20Wanted%20%3Aglobe_with_meridians%3A) queue until we are ready to start it in case anyone is interested in picking this up.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -15,9 +15,9 @@
     url="https://github.com/modin-project/modin",
     long_description=long_description,
     long_description_content_type="text/markdown",
-    install_requires=["pandas==0.24.1", "ray==0.6.2", "numpy<=1.15.0", "typing"],
+    install_requires=["pandas==0.24.1", "ray==0.6.2", "typing"],
     extras_require={
         # can be installed by pip install modin[dask]
-        "dask": ["dask==1.0.0", "distributed==1.25.0"],
+        "dask": ["dask==1.1.0", "distributed==1.25.0"],
     },
 )


Test patch:
Repository: modin-project/modin

Issue:
`read_table` with default `sep=False` fails

### Describe the problem
<!-- Describe the problem clearly here. -->
Asof pandas 0.24, `read_table` uses `sep=False` as the parameter defualt. This is not working in Modin on the master.

### Source code / logs

```python
import modin.pandas as pd
pd.read_table("table.txt")
```
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:

Resolving patch:
diff --git a/modin/pandas/io.py b/modin/pandas/io.py
--- a/modin/pandas/io.py
+++ b/modin/pandas/io.py
@@ -94,6 +94,8 @@ def parser_func(
         float_precision=None,
     ):
         _, _, _, kwargs = inspect.getargvalues(inspect.currentframe())
+        if not kwargs.get("sep", sep):
+            kwargs["sep"] = "\t"
         return _read(**kwargs)
 
     return parser_func


Test patch:
diff --git a/modin/pandas/test/test_io.py b/modin/pandas/test/test_io.py
--- a/modin/pandas/test/test_io.py
+++ b/modin/pandas/test/test_io.py
@@ -113,7 +113,7 @@ def teardown_parquet_file():
 
 
 @pytest.fixture
-def make_csv_file():
+def make_csv_file(delimiter=","):
     """Pytest fixture factory that makes temp csv files for testing.
 
     Yields:
@@ -125,7 +125,7 @@ def _make_csv_file(
         filename=TEST_CSV_FILENAME,
         row_size=SMALL_ROW_SIZE,
         force=False,
-        delimiter=",",
+        delimiter=delimiter,
         encoding=None,
     ):
         if os.path.exists(filename) and not force:
@@ -528,6 +528,21 @@ def test_from_csv(make_csv_file):
         assert modin_df_equals_pandas(modin_df, pandas_df)
 
 
+def test_from_table(make_csv_file):
+    make_csv_file(delimiter="\t")
+
+    pandas_df = pandas.read_table(TEST_CSV_FILENAME)
+    modin_df = pd.read_table(TEST_CSV_FILENAME)
+
+    assert modin_df_equals_pandas(modin_df, pandas_df)
+
+    if not PY2:
+        pandas_df = pandas.read_table(Path(TEST_CSV_FILENAME))
+        modin_df = pd.read_table(Path(TEST_CSV_FILENAME))
+
+        assert modin_df_equals_pandas(modin_df, pandas_df)
+
+
 class FakeS3FS:
     def exists(self, path):
         return "s3://bucket/path.csv" == path
Repository: modin-project/modin

Issue:
Single column DataFrames incorrectly name index `__reduced__` on MapReduce operations

### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: OSX
- **Modin installed from (source or binary)**: source
- **Modin version**: source
- **Python version**: 3.6
- **Exact command to reproduce**: `pd.DataFrame({"col1":[1,2,3,4]}).sum()`

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
Output is
```
__reduced__    10
Name: col1, dtype: int64
```
Correct output is
```
col1    10
dtype: int64
```
<!-- Describe the problem clearly here. -->

### Source code / logs
```python
import modin.pandas as pd
pd.DataFrame({"col1":[1,2,3,4]}).sum()
```
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:

Resolving patch:
diff --git a/modin/pandas/series.py b/modin/pandas/series.py
--- a/modin/pandas/series.py
+++ b/modin/pandas/series.py
@@ -49,7 +49,9 @@ def __init__(
                     )
                 )
             )._query_compiler
-        if len(query_compiler.columns) != 1:
+        if len(query_compiler.columns) != 1 or (
+            len(query_compiler.index) == 1 and query_compiler.index[0] == "__reduced__"
+        ):
             query_compiler = query_compiler.transpose()
         self._query_compiler = query_compiler
 


Test patch:
diff --git a/modin/pandas/test/test_dataframe.py b/modin/pandas/test/test_dataframe.py
--- a/modin/pandas/test/test_dataframe.py
+++ b/modin/pandas/test/test_dataframe.py
@@ -4415,6 +4415,13 @@ def test_sum(self, request, data, axis, skipna, numeric_only, min_count):
             )
             df_equals(modin_result, pandas_result)
 
+    @pytest.mark.parametrize("data", test_data_values, ids=test_data_keys)
+    def test_sum_single_column(self, data):
+        modin_df = pd.DataFrame(data).iloc[:, [0]]
+        pandas_df = pandas.DataFrame(data).iloc[:, [0]]
+        df_equals(modin_df.sum(), pandas_df.sum())
+        df_equals(modin_df.sum(axis=1), pandas_df.sum(axis=1))
+
     def test_swapaxes(self):
         data = test_data_values[0]
         with pytest.warns(UserWarning):
Repository: modin-project/modin

Issue:
drop after transpose does not transpose internal data before computing drop
<!--
General questions should be asked on the mailing list modin-dev@googlegroups.com.

Before submitting an issue, please fill out the following form.
-->

### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: WSL
- **Modin installed from (source or binary)**: source
- **Modin version**: source
- **Python version**: 3.6
- **Exact command to reproduce**: `df.T.drop(columns=[...])`

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
Transpose is not resolved correctly before performing a drop
<!-- Describe the problem clearly here. -->

### Source code / logs
```python
import modin.pandas as pd
df = pd.DataFrame({"col1":[1,2,3,4], "col2":list("ABCD")}).T.drop(columns=[1,2])
print(df)
```
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:

Resolving patch:
diff --git a/modin/backends/pandas/query_compiler.py b/modin/backends/pandas/query_compiler.py
--- a/modin/backends/pandas/query_compiler.py
+++ b/modin/backends/pandas/query_compiler.py
@@ -2012,6 +2012,8 @@ def drop(self, index=None, columns=None):
         Returns:
             A new QueryCompiler.
         """
+        if self._is_transposed:
+            return self.transpose().drop(index=columns, columns=index).transpose()
         if index is None:
             new_data = self.data
             new_index = self.index


Test patch:
diff --git a/modin/pandas/test/test_dataframe.py b/modin/pandas/test/test_dataframe.py
--- a/modin/pandas/test/test_dataframe.py
+++ b/modin/pandas/test/test_dataframe.py
@@ -1618,6 +1618,22 @@ def test_drop_api_equivalence(self):
         with pytest.raises(ValueError):
             modin_df.drop(axis=1)
 
+    @pytest.mark.parametrize("data", test_data_values, ids=test_data_keys)
+    def test_drop_transpose(self, data):
+        modin_df = pd.DataFrame(data)
+        pandas_df = pandas.DataFrame(data)
+        modin_result = modin_df.T.drop(columns=[0, 1, 2])
+        pandas_result = pandas_df.T.drop(columns=[0, 1, 2])
+        df_equals(modin_result, pandas_result)
+
+        modin_result = modin_df.T.drop(index=["col3", "col1"])
+        pandas_result = pandas_df.T.drop(index=["col3", "col1"])
+        df_equals(modin_result, pandas_result)
+
+        modin_result = modin_df.T.drop(columns=[0, 1, 2], index=["col3", "col1"])
+        pandas_result = pandas_df.T.drop(columns=[0, 1, 2], index=["col3", "col1"])
+        df_equals(modin_result, pandas_result)
+
     def test_droplevel(self):
         df = (
             pd.DataFrame([[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]])
Repository: modin-project/modin

Issue:
QueryCompilerView does not correctly extract dtypes
<!--
General questions should be asked on the mailing list modin-dev@googlegroups.com.

Before submitting an issue, please fill out the following form.
-->

### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: OSX
- **Modin installed from (source or binary)**: source
- **Modin version**: source
- **Python version**: 3.6
- **Exact command to reproduce**: 

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
This is a tough one to reproduce with little code, but with #522, we changed the way dtypes were extracted, but that wasn't propagated to QueryCompilerView
<!-- Describe the problem clearly here. -->

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:

Resolving patch:
diff --git a/modin/backends/pandas/query_compiler.py b/modin/backends/pandas/query_compiler.py
--- a/modin/backends/pandas/query_compiler.py
+++ b/modin/backends/pandas/query_compiler.py
@@ -2523,24 +2523,6 @@ def __constructor__(self):
         """Return parent object when getting the constructor."""
         return PandasQueryCompiler
 
-    _dtype_cache = None
-
-    def _get_dtype(self):
-        """Override the parent on this to avoid getting the wrong dtypes"""
-        if self._dtype_cache is None:
-            map_func = self._prepare_method(lambda df: df.dtypes)
-
-            def dtype_builder(df):
-                return df.apply(lambda row: find_common_type(row.values), axis=0)
-
-            self._dtype_cache = self._full_reduce(0, map_func, dtype_builder)
-            self._dtype_cache.index = self.columns
-        elif not self._dtype_cache.index.equals(self.columns):
-            self._dtype_cache = self._dtype_cache.reindex(self.columns)
-        return self._dtype_cache
-
-    dtypes = property(_get_dtype)
-
     def _get_data(self) -> BaseFrameManager:
         """Perform the map step
 


Test patch:
Repository: modin-project/modin

Issue:
Doing a binary operation (e.g. truediv) after transpose throws ValueError

### Describe the problem
<!-- Describe the problem clearly here. -->
Doing a binary operation (e.g. truediv) after transpose throws ValueError.

This is a problem for `Series` which uses `transpose` very frequently.

### Source code / logs

`percent = df.isnull().sum() / df.isnull().count()`
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:

Resolving patch:
diff --git a/modin/backends/pandas/query_compiler.py b/modin/backends/pandas/query_compiler.py
--- a/modin/backends/pandas/query_compiler.py
+++ b/modin/backends/pandas/query_compiler.py
@@ -526,7 +526,7 @@ def _inter_manager_operations(self, other, how_to_join, func):
             New DataManager with new data and index.
         """
         reindexed_self, reindexed_other_list, joined_index = self.copartition(
-            0, other, how_to_join, False
+            0, other, how_to_join, sort=False
         )
         # unwrap list returned by `copartition`.
         reindexed_other = reindexed_other_list[0]
@@ -569,6 +569,18 @@ def _inter_df_op_handler(self, func, other, **kwargs):
         axis = kwargs.get("axis", 0)
         axis = pandas.DataFrame()._get_axis_number(axis) if axis is not None else 0
         if isinstance(other, type(self)):
+            # If this QueryCompiler is transposed, copartition can sometimes fail to
+            # properly co-locate the data. It does not fail if other is transposed, so
+            # if this object is transposed, we will transpose both and do the operation,
+            # then transpose at the end.
+            if self._is_transposed:
+                return (
+                    self.transpose()
+                    ._inter_manager_operations(
+                        other.transpose(), "outer", lambda x, y: func(x, y, **kwargs)
+                    )
+                    .transpose()
+                )
             return self._inter_manager_operations(
                 other, "outer", lambda x, y: func(x, y, **kwargs)
             )


Test patch:
Repository: modin-project/modin

Issue:
Operations resulting in tranposed Series sometimes return DataFrame

### Describe the problem
<!-- Describe the problem clearly here. -->
Operations that use `inplace` will return a `DataFrame` if the operation results in a transposed QueryCompiler

### Source code / logs

`percent = (df_train.isnull().sum()/df_train.isnull().count()).sort_values(ascending=False, inplace=False)`

Can be fixed at the same time as #677 
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:

Resolving patch:
diff --git a/modin/backends/pandas/query_compiler.py b/modin/backends/pandas/query_compiler.py
--- a/modin/backends/pandas/query_compiler.py
+++ b/modin/backends/pandas/query_compiler.py
@@ -1722,6 +1722,8 @@ def sort_index(self, **kwargs):
             DataManager containing the data sorted by columns or indices.
         """
         axis = kwargs.pop("axis", 0)
+        if self._is_transposed:
+            return self.transpose().sort_index(axis=axis ^ 1, **kwargs).transpose()
         index = self.columns if axis else self.index
 
         # sort_index can have ascending be None and behaves as if it is False.
diff --git a/modin/pandas/series.py b/modin/pandas/series.py
--- a/modin/pandas/series.py
+++ b/modin/pandas/series.py
@@ -89,7 +89,9 @@ def _create_or_update_from_compiler(self, new_query_compiler, inplace=False):
             isinstance(new_query_compiler, type(self._query_compiler))
             or type(new_query_compiler) in self._query_compiler.__class__.__bases__
         ), "Invalid Query Compiler object: {}".format(type(new_query_compiler))
-        if not inplace and len(new_query_compiler.columns) == 1:
+        if not inplace and (
+            len(new_query_compiler.columns) == 1 or len(new_query_compiler.index) == 1
+        ):
             return Series(query_compiler=new_query_compiler)
         elif not inplace:
             # This can happen with things like `reset_index` where we can add columns.


Test patch: