Repository: pandas-dev/pandas
Pull request: BUG: Duplicated returns boolean dataframe
- [x] closes #25184
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
Filtering by exclusion of duplicate rows does not preserve column list for an empty dataframe
#### Code Sample, a copy-pastable example if possible

```python
import pandas as pd

x_df = pd.DataFrame(columns=['a', 'b'])
series = x_df.duplicated(subset=['a'])

list(x_df[~series])

# Expected output on Pandas 0.23.4: ['a', 'b']
# But, Pandas 0.24.1 returns: []
```
#### Problem description

We have been using this approach to remove duplicate rows on a dataframe, where rows are compared by one column only. Everything worked perfectly until we found out that, if the original dataframe is empty, in the result dataframe column list is lost after Pandas upgrade to latest version.

#### Expected Output

We would expect the column list to be preserved.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.15.final.0
python-bits: 64
OS: Linux
OS-release: 4.15.0-44-generic
machine: x86_64
processor: 
byteorder: little
LC_ALL: None
LANG: C.UTF-8
LOCALE: None.None

pandas: 0.24.1
pytest: None
pip: 18.0
setuptools: 40.0.0
Cython: 0.22
numpy: 1.12.1
scipy: None
pyarrow: None
xarray: None
IPython: 4.2.0
sphinx: 1.4.4
patsy: None
dateutil: 2.5.0
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: 2.5.14
xlrd: 1.2.0
xlwt: 1.3.0
xlsxwriter: None
lxml.etree: 3.7.3
bs4: 4.5.3
html5lib: 1.
</details>

Hints from issue discussion:
Just ran into this myself today.

Looks like the root issue may be that duplicated is now returning an empty series of dtype=float64, whereas it used to return an empty series of dtype=bool.

I've never contributed to pandas before, but I _think_ fixing this would be as simple as changing this line to `return Series(dtype=bool)` - https://github.com/pandas-dev/pandas/blob/master/pandas/core/frame.py#L4638
@gnilrets : You are more than welcome to investigate this!  For information about contributing, see here:

https://pandas.pydata.org/pandas-docs/stable/development/contributing.html

cc @jreback
yep looks like a regression

Commit message: BUG: Duplicated returns boolean dataframe

Patch to review:
diff --git a/doc/source/whatsnew/v0.25.0.rst b/doc/source/whatsnew/v0.25.0.rst
index 4032dc20b2e19..af0fca34cac6d 100644
--- a/doc/source/whatsnew/v0.25.0.rst
+++ b/doc/source/whatsnew/v0.25.0.rst
@@ -68,6 +68,8 @@ Performance Improvements
 Bug Fixes
 ~~~~~~~~~
 
+- Bug in using duplicated with empty dataframes (:issue:`25184`)
+
 Categorical
 ^^^^^^^^^^^
 
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
index 19da8ba5c547d..bc521e931e5ae 100644
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -4636,7 +4636,7 @@ def duplicated(self, subset=None, keep='first'):
         from pandas._libs.hashtable import duplicated_int64, _SIZE_HINT_LIMIT
 
         if self.empty:
-            return Series()
+            return Series(dtype=bool)
 
         def f(vals):
             labels, shape = algorithms.factorize(
diff --git a/pandas/tests/frame/test_duplicates.py b/pandas/tests/frame/test_duplicates.py
index f61dbbdb989e4..b2a8cb738614c 100644
--- a/pandas/tests/frame/test_duplicates.py
+++ b/pandas/tests/frame/test_duplicates.py
@@ -182,6 +182,12 @@ def test_drop_duplicates():
         assert df.duplicated(keep=keep).sum() == 0
 
 
+def test_duplicated_on_empty_frame_gives_back_frame():
+    df = DataFrame(columns=['a', 'b'])
+    dupes = df.duplicated('a')
+    tm.assert_frame_equal(df[dupes], df)
+
+
 def test_drop_duplicates_with_duplicate_column_names():
     # GH17836
     df = DataFrame([


Code review comments:
diff_hunk: @@ -182,6 +182,12 @@ def test_drop_duplicates():
         assert df.duplicated(keep=keep).sum() == 0
 
 
+def test_duplicated_on_empty_frame_gives_back_frame():
+    df = DataFrame(columns=['a', 'b'])
+    dupes = df.duplicated('a')
path: pandas/tests/frame/test_duplicates.py
text: @user1:
result = 
expected = 
tm.assert_frame_equal(result, expected)

diff_hunk: @@ -68,6 +68,8 @@ Performance Improvements
 Bug Fixes
 ~~~~~~~~~
 
+- Bug in using duplicated with empty dataframes (:issue:`25184`)
path: doc/source/whatsnew/v0.25.0.rst
text: @user1:
say more like

``:meth:DataFrame.duplicate()`` on empty dataframe was not return a boolean dtyped Series

@user1:
this can go in 0.24.2

diff_hunk: @@ -182,6 +182,12 @@ def test_drop_duplicates():
         assert df.duplicated(keep=keep).sum() == 0
 
 
+def test_duplicated_on_empty_frame_gives_back_frame():
path: pandas/tests/frame/test_duplicates.py
text: @user1:
can you add the issue number as a comment here, rename the test to

test_duplicated_on_empty_frame