Repository: pandas-dev/pandas
Pull request: Fix no raise dup index when using drop with axis=0
- [x] closes #19186
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
BUG: drop on axis with duplicate values doesn't raise when label is absent
#### Code Sample, a copy-pastable example if possible

```
>>> items
             item   name
store1      dress   V
store2       food   A
store2      water   C
```

CASE 1 : axis set to column and given a value which is not available in the structure. Gives ValueError.


```
>>> items.drop('abcd',axis=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/site-packages/pandas/core/generic.py", line 2530, in drop
    obj = obj._drop_axis(labels, axis, level=level, errors=errors)
  File "/usr/local/lib/python3.6/site-packages/pandas/core/generic.py", line 2562, in _drop_axis
    new_axis = axis.drop(labels, errors=errors)
  File "/usr/local/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 3744, in drop
    labels[mask])
ValueError: labels ['abcd'] not contained in axis
```

CASE 2 : axis set to row, and given a value which is not available in structure should behave in the same way as the above case does.
Instead it just prints the structure table. Shouldn’t it be throwing ValueError ? 

```
>>> items.drop('abcd',axis=0)
             item   name
store1      dress   V
store2       food   A
store2      water   C
>>> 
```

#### Problem description

When provided invalid column name(axis=1) it throws "ValueError". Same happens with row as well(axis=0). But when there are duplicate indices in the DataFrame it doesn't.

#### Expected Output


```
>>> items.drop('d',axis=0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/site-packages/pandas/core/generic.py", line 2530, in drop
    obj = obj._drop_axis(labels, axis, level=level, errors=errors)
  File "/usr/local/lib/python3.6/site-packages/pandas/core/generic.py", line 2562, in _drop_axis
    new_axis = axis.drop(labels, errors=errors)
  File "/usr/local/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 3744, in drop
    labels[mask])
ValueError: labels ['d'] not contained in axis
```

#### Output of ``pd.show_versions()``

<details>
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.4.final.0
python-bits: 64
OS: Darwin
OS-release: 17.3.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_AU.UTF-8
LOCALE: en_AU.UTF-8

pandas: 0.22.0
pytest: None
pip: 9.0.1
setuptools: 36.5.0
Cython: 0.27.3
numpy: 1.13.3
scipy: None
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
</details>

Hints from issue discussion:
@vinu2003 Thanks for the report! As you mentioned on the mailing list, this is triggered by the fact there is a duplicate index and is indeed a bug. 
Related to https://github.com/pandas-dev/pandas/issues/18561 (I also mentioned this bug there), but since it is not exactly the same keeping this open as a separate issue.

I reformatted your post a bit, please make sure in the future to use correct markdown syntax (so it 'looks' good, github provides some help if you edit/post a comment)
Reproducible example:

```
In [18]: df = pd.DataFrame({"A": [1, 1, 1], "B": [1, 1, 1]}, index=['a', 'b', 'c'])

In [19]: df.drop("d", axis=0)
...
ValueError: labels ['d'] not contained in axis

In [20]: df = pd.DataFrame({"A": [1, 1, 1], "B": [1, 1, 1]}, index=['a', 'b', 'b'])

In [21]: df.drop("d", axis=0)
Out[21]: 
   A  B
a  1  1
b  1  1
b  1  1
```
I'm going to take a look since I don't see any other PR's for this issue.

Commit message: DOC: Fixed docs

Patch to review:
diff --git a/doc/source/whatsnew/v0.23.0.txt b/doc/source/whatsnew/v0.23.0.txt
index 3a3c2bf0c5ae4..dfbfcc7016219 100644
--- a/doc/source/whatsnew/v0.23.0.txt
+++ b/doc/source/whatsnew/v0.23.0.txt
@@ -453,6 +453,7 @@ Reshaping
 - Bug in :func:`Dataframe.pivot_table` which fails when the ``aggfunc`` arg is of type string.  The behavior is now consistent with other methods like ``agg`` and ``apply`` (:issue:`18713`)
 - Bug in :func:`DataFrame.merge` in which merging using ``Index`` objects as vectors raised an Exception (:issue:`19038`)
 - Bug in :func:`DataFrame.stack`, :func:`DataFrame.unstack`, :func:`Series.unstack` which were not returning subclasses (:issue:`15563`)
+- Bug in :func:`DataFrame.drop`, ``ValueError`` now raises when dropping an ``Index`` that has duplicates (:issue:`19186`)
 -
 
 Numeric
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
index cef1e551f948e..b3e00549ccca0 100644
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -2909,6 +2909,9 @@ def _drop_axis(self, labels, axis, level=None, errors='raise'):
             else:
                 indexer = ~axis.isin(labels)
 
+            if all(indexer) and errors == 'raise':
+                raise ValueError('{} not found in axis'.format(labels))
+
             slicer = [slice(None)] * self.ndim
             slicer[self._get_axis_number(axis_name)] = indexer
 
diff --git a/pandas/tests/frame/test_mutate_columns.py b/pandas/tests/frame/test_mutate_columns.py
index 9acdf2f17d86a..c0706430616e5 100644
--- a/pandas/tests/frame/test_mutate_columns.py
+++ b/pandas/tests/frame/test_mutate_columns.py
@@ -257,3 +257,21 @@ def test_insert_column_bug_4032(self):
         expected = DataFrame([[1.3, 1, 1.1], [2.3, 2, 2.2]],
                              columns=['c', 'a', 'b'])
         assert_frame_equal(result, expected)
+
+    data = [[1, 2, 3], [1, 2, 3]]
+
+    @pytest.mark.parametrize('actual', [
+        DataFrame(data=data, index=['a', 'a']),
+        DataFrame(data=data, index=['a', 'b']),
+        DataFrame(data=data, index=['a', 'b']).set_index([0, 1]),
+        DataFrame(data=data, index=['a', 'a']).set_index([0, 1])
+    ])
+    def test_raise_on_drop_duplicate_index(self, actual):
+
+        # issue 19186
+        level = 0 if isinstance(actual.index, MultiIndex) else None
+        with pytest.raises(ValueError):
+            actual.drop('c', level=level, axis=0)
+        expected_no_err = actual.drop('c', axis=0, level=level,
+                                      errors='ignore')
+        assert_frame_equal(expected_no_err, actual)


Code review comments:
diff_hunk: @@ -2909,6 +2909,9 @@ def _drop_axis(self, labels, axis, level=None, errors='raise'):
             else:
                 indexer = ~axis.isin(labels)
 
+            if indexer.all() and errors == 'raise':
path: pandas/core/generic.py
text: @user1:
Actually, I might have been a little too quick to say switch to `.all()`.  It looks like builtin `all` will be faster for sizable arrays that have a `False` early on.  Leave it as is for now though, unless someone else says to switch it, since `.all()` should have a lower bound for the worst case scenario.

Switching the `errors == 'raise'` check to be first should be more efficient, since that's a cheaper check, and will cause the `and` to terminate before checking `indexer` in cases where this isn't applicable:

```python
In [2]: a = np.repeat([True], 10**5)

In [3]: errors = 'other'

In [4]: %timeit errors == 'raise' and a.all()
58.7 ns ± 10 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

In [5]: %timeit a.all() and errors == 'raise'
6.82 µs ± 856 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
```

@author:
Ok, I'll leave it as is for now and make the other change