Repository: pandas-dev/pandas
Pull request: ENH: specificy missing labels in loc calls GH34272
- [ ] closes #34272
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
ENH: specify missing keys in KeyError when passing list-like to .loc
I'm a big fan of the strong limitation when passing list-likes to `.loc[]`. However, when the a key is missing and the KeyError is raised, it'd be helpful to know _which_ key/s is/are missing.

I understand this could have performance implications since each key of the list-like object should be searched, but also think the UX would be improved.

At least, the first missing key could be reported, without hindering performance. I'm happy to provide a PR for this if provided with some guidelines (I suppose this affects both Series and DataFrames, as well as maybe some other parts of the code I'm not aware of).

```python
pd.__version__
# '1.0.3'

s = pd.Series({'a': 1, 'b': 2, 'c': 3})
s.loc['d'] # Raises KeyError but specifies 'd' is missing
s.loc[['a', 'b', 'c', 'd']] # Raises KeyError without specifying which key is missing
```

Hints from issue discussion:
Does this needs to be fixed ? Since mentioned here 'https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#deprecate-loc-reindex-listlike'

This says it is deprecated.
Well, I'm not proposing that the behavior be reinstated, just that the KeyError that is thrown be more verbose about what keys are actually missing

> I understand this could have performance implications since each key of the list-like object should be searched, but also think the UX would be improved.
 
> Well, I'm not proposing that the behavior be reinstated, just that the KeyError that is thrown be more verbose about what keys are actually missing

I agree , what I meant that this was purposely deprecated to use .reindex() as per doc.
And as you already mentioned it would require iterating through all the elements , I am also of the view that this was done to increase performance. 
Just adding my understanding. 
I may have expressed myself wrong, sorry if that's the case.

When I've run into this error, I _thought_ all the columns I was wondering were present, but that actually wasn't the case. Many times, I had DataFrames with hundreds of columns were maybe just one was missing from the list-like indexer, generally because of a typo in the original CSV I had read.

Since the error gives no information about what was columns were actually missing in my DataFrame, I'd then have to do something like:

```python
missing_cols = [c for c in my_list if c not in my_df.columns]
```

What I wish is that this behavior of searching the missing columns happened out of the box when raising the KeyError (the missing columns would be listed in the KeyError message). If that were not possible due to performance, at least the first missing column could be mentioned, without hindering performance
I also think this feature would be helpful and a useful enhancement for debugging missing or misspelled columns. 

Since it would only find the missing labels if it has already met the missing columns condition, I don't think this would impact performance in most cases  [pandas/core/indexing.py:1288](https://github.com/pandas-dev/pandas/blob/23d1b02248540df23f168264f3bb0fadb4d438c1/pandas/core/indexing.py#L1288)

I would be happy to try and make the PR.
yep PRs welcome here

Commit message: ENH: Include missing labels in error message for loc GH34272

Fixed rst flake8 issues.

Patch to review:
diff --git a/doc/source/whatsnew/v1.1.0.rst b/doc/source/whatsnew/v1.1.0.rst
index f6ad3a800283d..c143fc5f4c0fa 100644
--- a/doc/source/whatsnew/v1.1.0.rst
+++ b/doc/source/whatsnew/v1.1.0.rst
@@ -13,6 +13,22 @@ including other versions of pandas.
 Enhancements
 ~~~~~~~~~~~~
 
+.. _whatsnew_110.specify_missing_labels:
+
+KeyErrors raised by loc specify missing labels
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Previously, if labels were missing for a loc call, a Key Error was raised stating that this was no longer supported.
+
+Now the error message also includes a list of the missing labels. For example,
+.. code-block:: ipython
+
+   In [1]: s = pd.Series({"a": 1, "b": 2, "c": 3})
+   In [2]: s.loc[["a", "b", "missing_0", "c", "missing_1", "missing_2"]]
+   Passing list-likes to .loc or [] with any missing labels is no longer supported.
+   The following labels were missing: ['missing_0', 'missing_1', 'missing_2']. See
+   https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#deprecate-loc-reindex-listlike
+
+
 .. _whatsnew_110.astype_string:
 
 All dtypes can now be converted to ``StringDtype``
diff --git a/pandas/core/indexing.py b/pandas/core/indexing.py
index 9c8b01003bece..b79b92b52921f 100644
--- a/pandas/core/indexing.py
+++ b/pandas/core/indexing.py
@@ -1283,7 +1283,8 @@ def _validate_read_indexer(
             return
 
         # Count missing values:
-        missing = (indexer < 0).sum()
+        missing_mask = indexer < 0
+        missing = (missing_mask).sum()
 
         if missing:
             if missing == len(indexer):
@@ -1302,10 +1303,12 @@ def _validate_read_indexer(
             # code, so we want to avoid warning & then
             # just raising
             if not ax.is_categorical():
+                not_found = list(key[missing_mask])
                 raise KeyError(
                     "Passing list-likes to .loc or [] with any missing labels "
-                    "is no longer supported, see "
-                    "https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#deprecate-loc-reindex-listlike"  # noqa:E501
+                    "is no longer supported. "
+                    f"The following labels were missing: {not_found}. "
+                    "See https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#deprecate-loc-reindex-listlike"  # noqa:E501
                 )
 
 
diff --git a/pandas/tests/indexing/test_indexing.py b/pandas/tests/indexing/test_indexing.py
index 5c0230e75021c..750eacd385df6 100644
--- a/pandas/tests/indexing/test_indexing.py
+++ b/pandas/tests/indexing/test_indexing.py
@@ -1075,3 +1075,12 @@ def test_setitem_with_bool_mask_and_values_matching_n_trues_in_length():
     result = ser
     expected = pd.Series([None] * 3 + list(range(5)) + [None] * 2).astype("object")
     tm.assert_series_equal(result, expected)
+
+
+def test_missing_labels_inside_loc():
+    # GH34272
+    s = pd.Series({"a": 1, "b": 2, "c": 3})
+    with pytest.raises(KeyError) as e:
+        s.loc[["a", "b", "missing_0", "c", "missing_1", "missing_2"]]
+    missing_labels = ["missing_0", "missing_1", "missing_2"]
+    assert all(missing_label in str(e.value) for missing_label in missing_labels)


Code review comments:
diff_hunk: @@ -1075,3 +1075,12 @@ def test_setitem_with_bool_mask_and_values_matching_n_trues_in_length():
     result = ser
     expected = pd.Series([None] * 3 + list(range(5)) + [None] * 2).astype("object")
     tm.assert_series_equal(result, expected)
+
+
+def test_missing_labels_inside_loc():
+    # GH34272
+    s = pd.Series({"a": 1, "b": 2, "c": 3})
+    with pytest.raises(KeyError) as e:
+        s.loc[["a", "b", "missing_0", "c", "missing_1", "missing_2"]]
+    missing_labels = ["missing_0", "missing_1", "missing_2"]
+    assert all(missing_label in str(e.value) for missing_label in missing_labels)
path: pandas/tests/indexing/test_indexing.py
text: @user1:
Instead you can use the `match` argument in `pytest.raises`

@author:
good point, thanks!