API: warning to raise KeyError in the future if not all elements of a list are selected via .loc
State: closed

closes #15747


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Deprecate
description: Functionality to remove in pandas

Comments:
still lots of warnings.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17295?src=pr&el=h1) Report
> Merging [#17295](https://codecov.io/gh/pandas-dev/pandas/pull/17295?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/2e2093e1985b6ae36e2b00edf1bb14197134bc62?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17295/graphs/tree.svg?height=150&width=650&token=eZ4WkYLtcO&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/17295?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17295      +/-   ##
==========================================
- Coverage   91.25%   91.24%   -0.02%     
==========================================
  Files         163      163              
  Lines       49859    49866       +7     
==========================================
  Hits        45499    45499              
- Misses       4360     4367       +7
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.04% <100%> (ø)` | :arrow_up: |
| #single | `40.24% <21.42%> (-0.07%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17295?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexing.py](https://codecov.io/gh/pandas-dev/pandas/pull/17295?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhpbmcucHk=) | `93.97% <100%> (+0.03%)` | :arrow_up: |
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/17295?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `95.04% <100%> (ø)` | :arrow_up: |
| [pandas/io/formats/excel.py](https://codecov.io/gh/pandas-dev/pandas/pull/17295?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvZXhjZWwucHk=) | `96.69% <100%> (+0.04%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17295?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17295?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.73% <0%> (-0.1%)` | :arrow_down: |
| [pandas/core/internals.py](https://codecov.io/gh/pandas-dev/pandas/pull/17295?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzLnB5) | `94.37% <0%> (-0.01%)` | :arrow_down: |
| [pandas/plotting/\_style.py](https://codecov.io/gh/pandas-dev/pandas/pull/17295?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19zdHlsZS5weQ==) | `74.28% <0%> (ø)` | :arrow_up: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/17295?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `96.46% <0%> (+0.11%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17295?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17295?src=pr&el=footer). Last update [2e2093e...44ce09d](https://codecov.io/gh/pandas-dev/pandas/pull/17295?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

ok this is ready to go.

@jorisvandenbossche @TomAugspurger 
cc @toobaz 
Cool!

Notice however that the warning is not being raised for ``MultiIndex``es.

```python
In [2]: pd.DataFrame([[i] for i in range(3)], index=pd.MultiIndex.from_product([[1], [5,6,7]])).loc[[(1,5), (3,7)]]
Out[2]: 
       0
1 5  0.0
3 7  NaN
```

I assume/hope that we will want to change the behavior with partial indexing too (despite the current behaviour being different):
```python
In [3]: pd.DataFrame([[i] for i in range(3)], index=pd.MultiIndex.from_product([[1], [5,6,7]])).loc[[1,3]]
Out[3]: 
     0
1 5  0
  6  1
  7  2
```
So maybe this too should raise a warning (although the whatsnew notes would maybe require some addition). Can go in another PR however.
> So maybe this too should raise a warning (although the whatsnew notes would maybe require some addition). Can go in another PR however.

yeah this doesn't cover MultiIndex cases at all, but agree it should/could. welcome for PR's on top.
One more thought: users might have in their code (I suspect I do) things like

```python
a.loc[list_like].dropna()
```
... in which case the suggested replacement is not ``a.reindex(list_like).dropna()`` but rather ``a.loc[a.index.intersection(list_like)]`` (which avoids a step, and possibly a type casting of ints to floats). Should this second option maybe be mentioned?
@toobaz in your comment https://github.com/pandas-dev/pandas/pull/17295#issuecomment-324018426

yes I think the warning could refer to a new section in the docs (like we do for .ix). where a more extended example could be. (and include this effect as well). let me add.
updated with nicer message and doc section.
I think it is OK, just the two sentences commented above and the "Previously Behavior" sound a bit weird.
ok pushed. @jorisvandenbossche @TomAugspurger if you'd have a look.
> I haven't kept up with whether reindexing into a Index with dupes is supposed to work on not...

Indeed, it can't work in general - assume you had done ``s.reindex(['a', 'b'])``, the expected behavior would be unclear because ``'a'`` is duplicated.

So maybe the ``s.loc[s.index.intersection(keys)]`` option (or does ``s.loc[s.index & keys]`` look better?) should be emphasized for solving this case too. To my eyes, it is actually _the_ idiomatic replacement, because it is hard to conceive that one would use ``s.loc[list_like]`` and be interested in keys _not_ in ``s``...
(Edited my previous comment, which in that form didn't make any sense)
> So maybe the s.loc[s.index.intersection(keys)] option (or does s.loc[s.index & keys] look better?) should be emphasized for solving this case too. To my eyes, it is actually the idiomatic replacement, because it is hard to conceive that one would use s.loc[list_like] and be interested in keys not in s...

changed; that is quite idiomatic
fixed according to @TomAugspurger and @toobaz comments.
ok finally had time to update. @jorisvandenbossche 
@jorisvandenbossche @toobaz comments.
(Will take a look at this tomorrow evening)
updated
@jorisvandenbossche @TomAugspurger 
push to fix all @jorisvandenbossche comments. I deprecated passing 1 or more missing ``.to_excel(..., columns=)`` 
ok pushed
This is really nice, thanks @jreback !