Repository: modin-project/modin

Issue:
DataFrame.drop_duplicates raises an exception
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:

Ubuntu 19.10

- **Modin version** (`modin.__version__`):

0.7.1+2.g4f36f23

- **Python version**:

Python 3.7.5

- **Code we can use to reproduce**:

```python
#import pandas as pd
import ray
ray.init(huge_pages=False, plasma_directory="/localdisk/gashiman/plasma", memory=1024*1024*1024*200, object_store_memory=1024*1024*1024*200)
import modin.pandas as pd

df = pd.DataFrame([["one", 1, 10],
                   ["two", 4, 20],
                   ["three", 7, 30]],
                  index=[1, 2, 3],
                  columns=['name', 'max_speed', 'health'])
print(df)

df1 = df.drop_duplicates("name")
print(df1)
```

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->

On Pandas I get this output

```
    name  max_speed  health
1    one          1      10
2    two          4      20
3  three          7      30
    name  max_speed  health
1    one          1      10
2    two          4      20
3  three          7      30
```

while on Modin I get an exception:

```
2020-02-27 10:22:13,644 INFO resource_spec.py:216 -- Starting Ray with 200.0 GiB memory available for workers and up to 200.0 GiB for objects. You can adjust these settings with ray.init(memory=<bytes>, object_store_memory=<bytes>).
2020-02-27 10:22:13,899 WARNING services.py:1365 -- WARNING: object_store_memory is not verified when plasma_directory is set.
UserWarning: Distributing <class 'list'> object. This may take some time.
    name  max_speed  health
1    one          1      10
2    two          4      20
3  three          7      30
Traceback (most recent call last):
  File "drop_duplicates_test.py", line 13, in <module>
    df1 = df.drop_duplicates("name")
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/dataframe.py", line 208, in drop_duplicates
    subset=subset, keep=keep, inplace=inplace
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/base.py", line 1120, in drop_duplicates
    duplicates = self.duplicated(keep=keep, subset=kwargs.get("subset"))
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/dataframe.py", line 242, in duplicated
    if len(df.columns) > 1:
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/series.py", line 227, in __getattr__
    raise e
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/series.py", line 223, in __getattr__
    return object.__getattribute__(self, key)
AttributeError: 'Series' object has no attribute 'columns'
```

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->

DataFrame.drop_duplicates raises an exception
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:

Ubuntu 19.10

- **Modin version** (`modin.__version__`):

0.7.1+2.g4f36f23

- **Python version**:

Python 3.7.5

- **Code we can use to reproduce**:

```python
#import pandas as pd
import ray
ray.init(huge_pages=False, plasma_directory="/localdisk/gashiman/plasma", memory=1024*1024*1024*200, object_store_memory=1024*1024*1024*200)
import modin.pandas as pd

df = pd.DataFrame([["one", 1, 10],
                   ["two", 4, 20],
                   ["three", 7, 30]],
                  index=[1, 2, 3],
                  columns=['name', 'max_speed', 'health'])
print(df)

df1 = df.drop_duplicates("name")
print(df1)
```

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->

On Pandas I get this output

```
    name  max_speed  health
1    one          1      10
2    two          4      20
3  three          7      30
    name  max_speed  health
1    one          1      10
2    two          4      20
3  three          7      30
```

while on Modin I get an exception:

```
2020-02-27 10:22:13,644 INFO resource_spec.py:216 -- Starting Ray with 200.0 GiB memory available for workers and up to 200.0 GiB for objects. You can adjust these settings with ray.init(memory=<bytes>, object_store_memory=<bytes>).
2020-02-27 10:22:13,899 WARNING services.py:1365 -- WARNING: object_store_memory is not verified when plasma_directory is set.
UserWarning: Distributing <class 'list'> object. This may take some time.
    name  max_speed  health
1    one          1      10
2    two          4      20
3  three          7      30
Traceback (most recent call last):
  File "drop_duplicates_test.py", line 13, in <module>
    df1 = df.drop_duplicates("name")
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/dataframe.py", line 208, in drop_duplicates
    subset=subset, keep=keep, inplace=inplace
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/base.py", line 1120, in drop_duplicates
    duplicates = self.duplicated(keep=keep, subset=kwargs.get("subset"))
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/dataframe.py", line 242, in duplicated
    if len(df.columns) > 1:
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/series.py", line 227, in __getattr__
    raise e
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/series.py", line 223, in __getattr__
    return object.__getattribute__(self, key)
AttributeError: 'Series' object has no attribute 'columns'
```

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:
Thanks @gshimansky this bug exists when `subset` is not a list. Our incorrect assumption was that the values passed in would be a list. This is happening in `duplicated`.

As a temporary workaround, you should be able to pass `subset=["name"]` and it should work normally.

For the fix, in the case of a single label we should just convert a single label to a list of labels with one value.
Well, with `["name"]` I get a different exception:
```python
if __name__ == "__main__":
    import modin.pandas as pd
    df = pd.DataFrame([["one", 1, 10],
                       ["two", 4, 20],
                       ["three", 7, 30]],
                      index=[1, 2, 3],
                      columns=['name', 'max_speed', 'health'])
    print(df)

    df1 = df.drop_duplicates(subset=["name"])
    print(df1)
```

```
UserWarning: Distributing <class 'list'> object. This may take some time.
    name  max_speed  health
1    one          1      10
2    two          4      20
3  three          7      30
UserWarning: User-defined function verification is still under development in Modin. The function provided is not verified.
Traceback (most recent call last):
  File "drop_duplicates_test.py", line 14, in <module>
    df1 = df.drop_duplicates(subset=["name"])
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/dataframe.py", line 208, in drop_duplicates
    subset=subset, keep=keep, inplace=inplace
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/base.py", line 1120, in drop_duplicates
    duplicates = self.duplicated(keep=keep, subset=kwargs.get("subset"))
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/dataframe.py", line 246, in duplicated
    duplicates = hashed.apply(lambda s: s.duplicated(keep=keep)).squeeze(axis=1)
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/dataframe.py", line 362, in apply
    if axis == 0 and result.name == self.index[0]:
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/base.py", line 3403, in __nonzero__
    self.__class__.__name__
ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().
```
BTW this workaround fails because of a lucky name collision between my reproducer and Modin code. If I rename column to anything other than `"name"`, it works :)
```python
if __name__ == "__main__":
    import modin.pandas as pd
    df = pd.DataFrame([["one", 1, 10],
                       ["two", 4, 20],
                       ["three", 7, 30]],
                      index=[1, 2, 3],
                      columns=['name1', 'max_speed', 'health'])
    print(df)

    df1 = df.drop_duplicates(subset=["name1"])
    print(df1)
```
The problem is with this code which doesn't allow me to have a column with name `"name"` in my dataframe:
```python
            if hasattr(result, "name"):
                if axis == 0 and result.name == self.index[0]:
                    result.name = None
                elif axis == 1 and result.name == self.columns[0]:
                    result.name = None
```
BTW, Pandas allows not only name of column or a list of columns. It also allows a tuple of columns which currently doesn't work as well. I created a test with all combinations that I could think of:
```python
#import pandas as pd
import ray
ray.init(huge_pages=False, plasma_directory="/localdisk/gashiman/plasma", memory=1024*1024*1024*200, object_store_memory=1024*1024*1024*200)

if __name__ == "__main__":
    import modin.pandas as pd
    #import pandas as pd
    df = pd.DataFrame([["one", 1, 10],
                       ["two", 4, 20],
                       ["two", 5, 20],
                       ["three", 7, 30]],
                      index=[1, 2, 3, 4],
                      columns=['name1', 'max_speed', 'health'])
    print(df)

    def test_func(df, arg):
        print("ARG=", arg)
        df1 = df.drop_duplicates(arg)
        print(df1)

    test_func(df, ["name1"])
    test_func(df, ("name1"))
    test_func(df, "name1")
    test_func(df, ["name1", "health"])
    test_func(df, ("name1", "health"))
    test_func(df, ["name1", "max_speed"])
    test_func(df, ("name1", "max_speed"))
```
and it fails for me on tuple test:
```
2020-03-02 12:57:15,579 INFO resource_spec.py:216 -- Starting Ray with 200.0 GiB memory available for workers and up to 200.0 GiB for objects. You can adjust these settings with ray.init(memory=<bytes>, object_store_memory=<bytes>).
2020-03-02 12:57:15,877 WARNING services.py:1365 -- WARNING: object_store_memory is not verified when plasma_directory is set.
FutureWarning: pandas.core.index is deprecated and will be removed in a future version.  The public classes are available in the top-level namespace.
UserWarning: Distributing <class 'list'> object. This may take some time.
   name1  max_speed  health
1    one          1      10
2    two          4      20
3    two          5      20
4  three          7      30
ARG= ['name1']
UserWarning: User-defined function verification is still under development in Modin. The function provided is not verified.
   name1  max_speed  health
1    one          1      10
2    two          4      20
4  three          7      30
ARG= name1
Traceback (most recent call last):
  File "drop_duplicates_test.py", line 22, in <module>
    test_func(df, ("name1"))
  File "drop_duplicates_test.py", line 18, in test_func
    df1 = df.drop_duplicates(arg)
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/dataframe.py", line 196, in drop_duplicates
    subset=subset, keep=keep, inplace=inplace
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/base.py", line 1098, in drop_duplicates
    duplicates = self.duplicated(keep=keep, subset=kwargs.get("subset"))
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/dataframe.py", line 230, in duplicated
    if len(df.columns) > 1:
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/series.py", line 229, in __getattr__
    raise e
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/series.py", line 225, in __getattr__
    return object.__getattribute__(self, key)
AttributeError: 'Series' object has no attribute 'columns'
```
Thanks @gshimansky this bug exists when `subset` is not a list. Our incorrect assumption was that the values passed in would be a list. This is happening in `duplicated`.

As a temporary workaround, you should be able to pass `subset=["name"]` and it should work normally.

For the fix, in the case of a single label we should just convert a single label to a list of labels with one value.
Well, with `["name"]` I get a different exception:
```python
if __name__ == "__main__":
    import modin.pandas as pd
    df = pd.DataFrame([["one", 1, 10],
                       ["two", 4, 20],
                       ["three", 7, 30]],
                      index=[1, 2, 3],
                      columns=['name', 'max_speed', 'health'])
    print(df)

    df1 = df.drop_duplicates(subset=["name"])
    print(df1)
```

```
UserWarning: Distributing <class 'list'> object. This may take some time.
    name  max_speed  health
1    one          1      10
2    two          4      20
3  three          7      30
UserWarning: User-defined function verification is still under development in Modin. The function provided is not verified.
Traceback (most recent call last):
  File "drop_duplicates_test.py", line 14, in <module>
    df1 = df.drop_duplicates(subset=["name"])
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/dataframe.py", line 208, in drop_duplicates
    subset=subset, keep=keep, inplace=inplace
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/base.py", line 1120, in drop_duplicates
    duplicates = self.duplicated(keep=keep, subset=kwargs.get("subset"))
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/dataframe.py", line 246, in duplicated
    duplicates = hashed.apply(lambda s: s.duplicated(keep=keep)).squeeze(axis=1)
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/dataframe.py", line 362, in apply
    if axis == 0 and result.name == self.index[0]:
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/base.py", line 3403, in __nonzero__
    self.__class__.__name__
ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().
```
BTW this workaround fails because of a lucky name collision between my reproducer and Modin code. If I rename column to anything other than `"name"`, it works :)
```python
if __name__ == "__main__":
    import modin.pandas as pd
    df = pd.DataFrame([["one", 1, 10],
                       ["two", 4, 20],
                       ["three", 7, 30]],
                      index=[1, 2, 3],
                      columns=['name1', 'max_speed', 'health'])
    print(df)

    df1 = df.drop_duplicates(subset=["name1"])
    print(df1)
```
The problem is with this code which doesn't allow me to have a column with name `"name"` in my dataframe:
```python
            if hasattr(result, "name"):
                if axis == 0 and result.name == self.index[0]:
                    result.name = None
                elif axis == 1 and result.name == self.columns[0]:
                    result.name = None
```
BTW, Pandas allows not only name of column or a list of columns. It also allows a tuple of columns which currently doesn't work as well. I created a test with all combinations that I could think of:
```python
#import pandas as pd
import ray
ray.init(huge_pages=False, plasma_directory="/localdisk/gashiman/plasma", memory=1024*1024*1024*200, object_store_memory=1024*1024*1024*200)

if __name__ == "__main__":
    import modin.pandas as pd
    #import pandas as pd
    df = pd.DataFrame([["one", 1, 10],
                       ["two", 4, 20],
                       ["two", 5, 20],
                       ["three", 7, 30]],
                      index=[1, 2, 3, 4],
                      columns=['name1', 'max_speed', 'health'])
    print(df)

    def test_func(df, arg):
        print("ARG=", arg)
        df1 = df.drop_duplicates(arg)
        print(df1)

    test_func(df, ["name1"])
    test_func(df, ("name1"))
    test_func(df, "name1")
    test_func(df, ["name1", "health"])
    test_func(df, ("name1", "health"))
    test_func(df, ["name1", "max_speed"])
    test_func(df, ("name1", "max_speed"))
```
and it fails for me on tuple test:
```
2020-03-02 12:57:15,579 INFO resource_spec.py:216 -- Starting Ray with 200.0 GiB memory available for workers and up to 200.0 GiB for objects. You can adjust these settings with ray.init(memory=<bytes>, object_store_memory=<bytes>).
2020-03-02 12:57:15,877 WARNING services.py:1365 -- WARNING: object_store_memory is not verified when plasma_directory is set.
FutureWarning: pandas.core.index is deprecated and will be removed in a future version.  The public classes are available in the top-level namespace.
UserWarning: Distributing <class 'list'> object. This may take some time.
   name1  max_speed  health
1    one          1      10
2    two          4      20
3    two          5      20
4  three          7      30
ARG= ['name1']
UserWarning: User-defined function verification is still under development in Modin. The function provided is not verified.
   name1  max_speed  health
1    one          1      10
2    two          4      20
4  three          7      30
ARG= name1
Traceback (most recent call last):
  File "drop_duplicates_test.py", line 22, in <module>
    test_func(df, ("name1"))
  File "drop_duplicates_test.py", line 18, in test_func
    df1 = df.drop_duplicates(arg)
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/dataframe.py", line 196, in drop_duplicates
    subset=subset, keep=keep, inplace=inplace
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/base.py", line 1098, in drop_duplicates
    duplicates = self.duplicated(keep=keep, subset=kwargs.get("subset"))
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/dataframe.py", line 230, in duplicated
    if len(df.columns) > 1:
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/series.py", line 229, in __getattr__
    raise e
  File "/nfs/site/proj/scripting_tools/gashiman/modin/modin/pandas/series.py", line 225, in __getattr__
    return object.__getattribute__(self, key)
AttributeError: 'Series' object has no attribute 'columns'
```

Resolving patch:
diff --git a/modin/pandas/base.py b/modin/pandas/base.py
--- a/modin/pandas/base.py
+++ b/modin/pandas/base.py
@@ -1094,8 +1094,14 @@ def drop_duplicates(self, keep="first", inplace=False, **kwargs):
                 deduplicated : DataFrame
         """
         inplace = validate_bool_kwarg(inplace, "inplace")
-        if kwargs.get("subset", None) is not None:
-            duplicates = self.duplicated(keep=keep, subset=kwargs.get("subset"))
+        subset = kwargs.get("subset", None)
+        if subset is not None:
+            if is_list_like(subset):
+                if not isinstance(subset, list):
+                    subset = list(subset)
+            else:
+                subset = [subset]
+            duplicates = self.duplicated(keep=keep, subset=subset)
         else:
             duplicates = self.duplicated(keep=keep)
         indices = duplicates.values.nonzero()[0]
diff --git a/modin/pandas/dataframe.py b/modin/pandas/dataframe.py
--- a/modin/pandas/dataframe.py
+++ b/modin/pandas/dataframe.py
@@ -322,7 +322,7 @@ def apply(self, func, axis=0, raw=False, result_type=None, args=(), **kwds):
             result = getattr(sys.modules[self.__module__], return_type)(
                 query_compiler=query_compiler
             )
-            if hasattr(result, "name"):
+            if isinstance(result, Series):
                 if axis == 0 and result.name == self.index[0]:
                     result.name = None
                 elif axis == 1 and result.name == self.columns[0]:


Test patch:
diff --git a/modin/pandas/test/test_dataframe.py b/modin/pandas/test/test_dataframe.py
--- a/modin/pandas/test/test_dataframe.py
+++ b/modin/pandas/test/test_dataframe.py
@@ -1156,23 +1156,37 @@ def test_droplevel(self):
         "keep", ["last", "first", False], ids=["last", "first", "False"]
     )
     @pytest.mark.parametrize(
-        "subset", [None, ["col1", "col3", "col7"]], ids=["None", "subset"]
+        "subset",
+        [None, "col1", "name", ("col1", "col3"), ["col1", "col3", "col7"]],
+        ids=["None", "string", "name", "tuple", "list"],
     )
     def test_drop_duplicates(self, data, keep, subset):
         modin_df = pd.DataFrame(data)
         pandas_df = pandas.DataFrame(data)
 
-        df_equals(
-            modin_df.drop_duplicates(keep=keep, inplace=False, subset=subset),
-            pandas_df.drop_duplicates(keep=keep, inplace=False, subset=subset),
-        )
-
-        modin_results = modin_df.drop_duplicates(keep=keep, inplace=True, subset=subset)
-        pandas_results = pandas_df.drop_duplicates(
-            keep=keep, inplace=True, subset=subset
-        )
+        try:
+            pandas_df.drop_duplicates(keep=keep, inplace=False, subset=subset)
+        except Exception as e:
+            with pytest.raises(type(e)):
+                modin_df.drop_duplicates(keep=keep, inplace=False, subset=subset)
+        else:
+            df_equals(
+                pandas_df.drop_duplicates(keep=keep, inplace=False, subset=subset),
+                modin_df.drop_duplicates(keep=keep, inplace=False, subset=subset),
+            )
 
-        df_equals(modin_results, pandas_results)
+        try:
+            pandas_results = pandas_df.drop_duplicates(
+                keep=keep, inplace=True, subset=subset
+            )
+        except Exception as e:
+            with pytest.raises(type(e)):
+                modin_df.drop_duplicates(keep=keep, inplace=True, subset=subset)
+        else:
+            modin_results = modin_df.drop_duplicates(
+                keep=keep, inplace=True, subset=subset
+            )
+            df_equals(modin_results, pandas_results)
 
     def test_drop_duplicates_with_missing_index_values(self):
         data = {
diff --git a/modin/pandas/test/utils.py b/modin/pandas/test/utils.py
--- a/modin/pandas/test/utils.py
+++ b/modin/pandas/test/utils.py
@@ -123,6 +123,12 @@
         ]
         for i in range(NCOLS)
     },
+    "has_name_column": {
+        "name": ["one", "two", "two", "three"],
+        "col1": [1, 2, 2, 3],
+        "col3": [10, 20, 20, 3],
+        "col7": [100, 201, 200, 300],
+    },
 }
 
 test_data_with_duplicates_values = list(test_data_with_duplicates.values())