Repository: pandas-dev/pandas
Pull request: Bug in xs raising KeyError for MultiIndex columns with droplevel False and list indexe
- [x] closes #41760
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

cc @simonjayhawkins Should we include in 1.2.5 or push to 1.3?

Problem statement:
BUG: XS for multiple keys in combination with drop_level=False
I believe a bug was introduced in the update of `xs` in pandas 1.2.0. The problem occurs when requesting multiple keys in combination with `drop_level=False`. I use the following code to highlight the problem I face: 

```python
import pandas as pd

d = [[4, 4, 2, 2, 8], [0, 0, 2, 2, 0]]

tuples = [('mammal', 'cat'), ('mammal', 'dog'), ('bird', 'penguin'), ('bird', 'pidgeon'), ('insect', 'spider')]
columns = pd.MultiIndex.from_tuples(tuples)

df = pd.DataFrame(data=d, columns=columns, index=['num_legs', 'num_wings'])
print(df.xs(['mammal', 'bird'], drop_level=False, axis=1))
```

In pandas 1.1.5, the following output is generated (as expected):
```
          mammal        bird        
             cat dog penguin pidgeon
num_legs       4   4       2       2
num_wings      0   0       2       2
```

While in pandas 1.2.4, an exception is raised:

```
Traceback (most recent call last):
  File "C:\python\miniconda3\envs\pandas_12\lib\site-packages\pandas\core\indexes\base.py", line 3080, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas\_libs\index.pyx", line 70, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\index.pyx", line 101, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\hashtable_class_helper.pxi", line 4554, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas\_libs\hashtable_class_helper.pxi", line 4562, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'bird'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "pandas\_libs\index.pyx", line 705, in pandas._libs.index.BaseMultiIndexCodesEngine.get_loc
  File "C:\python\miniconda3\envs\pandas_12\lib\site-packages\pandas\core\indexes\base.py", line 3082, in get_loc
    raise KeyError(key) from err
KeyError: 'bird'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\python\miniconda3\envs\pandas_12\lib\site-packages\pandas\core\indexes\multi.py", line 3036, in _get_loc_level
    return (self._engine.get_loc(key), None)
  File "pandas\_libs\index.pyx", line 708, in pandas._libs.index.BaseMultiIndexCodesEngine.get_loc
KeyError: ('mammal', 'bird')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\jimzw\AppData\Roaming\JetBrains\PyCharmCE2021.1\scratches\scratch_4.py", line 9, in <module>
    print(df.xs(['mammal', 'bird'], drop_level=False, axis=1))
  File "C:\python\miniconda3\envs\pandas_12\lib\site-packages\pandas\core\generic.py", line 3733, in xs
    loc, new_index = index._get_loc_level(
  File "C:\python\miniconda3\envs\pandas_12\lib\site-packages\pandas\core\indexes\multi.py", line 3038, in _get_loc_level
    raise KeyError(key) from e
KeyError: ('mammal', 'bird')
```

Hints from issue discussion:
Hi, thanks for your Report. 

Looks like a bug.

Could you try to narrow your example down a bit? Removing transpose and maybe defining the index explicitly?
> I believe a bug was introduced in the update of `xs` in pandas 1.2.0.

first bad commit: [5c111cb873625a16974f32c4532c7d8c377b543b] BUG: Bug in xs ignored droplevel (#37776)
@phofl I updated the example, hope this is what you were looking for.

Commit message: Merge branch 'master' of https://github.com/pandas-dev/pandas into 41760

 Conflicts:
	doc/source/whatsnew/v1.3.0.rst

Patch to review:
diff --git a/doc/source/whatsnew/v1.3.0.rst b/doc/source/whatsnew/v1.3.0.rst
index 88c69335b39f4..f9d1df7eb5786 100644
--- a/doc/source/whatsnew/v1.3.0.rst
+++ b/doc/source/whatsnew/v1.3.0.rst
@@ -708,7 +708,7 @@ Deprecations
 - Deprecated passing arguments as positional (other than ``filepath_or_buffer``) in :func:`read_csv` (:issue:`41485`)
 - Deprecated passing arguments as positional in :meth:`DataFrame.drop` (other than ``"labels"``) and :meth:`Series.drop` (:issue:`41485`)
 - Deprecated passing arguments as positional (other than ``filepath_or_buffer``) in :func:`read_table` (:issue:`41485`)
-
+- Deprecated passing lists as ``key`` to :meth:`DataFrame.xs` and :meth:`Series.xs` with same length as ``level`` keyword (:issue:`41789`)
 
 .. _whatsnew_130.deprecations.nuisance_columns:
 
@@ -947,6 +947,7 @@ Indexing
 - Bug in :meth:`DataFrame.__setitem__` raising ``TypeError`` when using a str subclass as the column name with a :class:`DatetimeIndex` (:issue:`37366`)
 - Bug in :meth:`PeriodIndex.get_loc` failing to raise ``KeyError`` when given a :class:`Period` with a mismatched ``freq`` (:issue:`41670`)
 - Bug ``.loc.__getitem__`` with a :class:`UInt64Index` and negative-integer keys raising ``OverflowError`` instead of ``KeyError`` in some cases, wrapping around to positive integers in others (:issue:`41777`)
+- Bug in :meth:`DataFrame.xs` and :meth:`Series.xs` accepting list as ``key`` and casting elements to different levels per index in list, now raises ``TypeError`` (:issue:`41760`)
 
 Missing
 ^^^^^^^
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
index 99816237155b3..342b481780d8a 100644
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -3756,6 +3756,18 @@ class   animal   locomotion
         """
         axis = self._get_axis_number(axis)
         labels = self._get_axis(axis)
+
+        if isinstance(key, list):
+            len_levels = 1 if level is None or is_scalar(level) else len(level)
+            if len(key) != len_levels:
+                raise TypeError("Passing lists as key for xs is not allowed.")
+            warnings.warn(
+                "Passing lists as key for xs is deprecated and will be removed in a "
+                "future version. Pass key as a tuple instead.",
+                FutureWarning,
+                stacklevel=2,
+            )
+
         if level is not None:
             if not isinstance(labels, MultiIndex):
                 raise TypeError("Index must be a MultiIndex")
diff --git a/pandas/tests/frame/indexing/test_xs.py b/pandas/tests/frame/indexing/test_xs.py
index 57ea6a63f86e8..17c9365db5261 100644
--- a/pandas/tests/frame/indexing/test_xs.py
+++ b/pandas/tests/frame/indexing/test_xs.py
@@ -106,7 +106,8 @@ def test_xs_keep_level(self):
         expected = df[:1]
         tm.assert_frame_equal(result, expected)
 
-        result = df.xs([2008, "sat"], level=["year", "day"], drop_level=False)
+        with tm.assert_produces_warning(FutureWarning):
+            result = df.xs([2008, "sat"], level=["year", "day"], drop_level=False)
         tm.assert_frame_equal(result, expected)
 
     def test_xs_view(self, using_array_manager):
@@ -187,7 +188,11 @@ def test_xs_with_duplicates(self, key, level, multiindex_dataframe_random_data):
         assert df.index.is_unique is False
         expected = concat([frame.xs("one", level="second")] * 2)
 
-        result = df.xs(key, level=level)
+        if isinstance(key, list):
+            with tm.assert_produces_warning(FutureWarning):
+                result = df.xs(key, level=level)
+        else:
+            result = df.xs(key, level=level)
         tm.assert_frame_equal(result, expected)
 
     def test_xs_missing_values_in_index(self):
@@ -358,3 +363,11 @@ def test_xs_droplevel_false_view(self, using_array_manager):
         df.iloc[0, 0] = 2
         expected = DataFrame({"a": [1]})
         tm.assert_frame_equal(result, expected)
+
+    def test_xs_list_indexer_droplevel_false(self):
+        # GH#41760
+        mi = MultiIndex.from_tuples([("x", "y", "a"), ("x", "z", "b"), ("v", "w", "c")])
+        df = DataFrame([[1, 2, 3], [4, 5, 6]], columns=mi)
+        msg = "Passing lists as key for xs is not allowed."
+        with pytest.raises(TypeError, match=msg):
+            df.xs(["x", "v"], drop_level=False, axis=1)
diff --git a/pandas/tests/indexing/multiindex/test_partial.py b/pandas/tests/indexing/multiindex/test_partial.py
index 932295c28c8cf..1317ced185324 100644
--- a/pandas/tests/indexing/multiindex/test_partial.py
+++ b/pandas/tests/indexing/multiindex/test_partial.py
@@ -69,7 +69,7 @@ def test_xs_partial(
         )
         df = DataFrame(np.random.randn(8, 4), index=index, columns=list("abcd"))
 
-        result = df.xs(["foo", "one"])
+        result = df.xs(("foo", "one"))
         expected = df.loc["foo", "one"]
         tm.assert_frame_equal(result, expected)
 
diff --git a/pandas/tests/series/indexing/test_xs.py b/pandas/tests/series/indexing/test_xs.py
index b6351e970222f..707d19ed6de8f 100644
--- a/pandas/tests/series/indexing/test_xs.py
+++ b/pandas/tests/series/indexing/test_xs.py
@@ -1,4 +1,5 @@
 import numpy as np
+import pytest
 
 from pandas import (
     MultiIndex,
@@ -69,3 +70,14 @@ def test_series_xs_droplevel_false(self):
             ),
         )
         tm.assert_series_equal(result, expected)
+
+    def test_xs_key_as_list(self):
+        # GH#41760
+        mi = MultiIndex.from_tuples([("a", "x")], names=["level1", "level2"])
+        ser = Series([1], index=mi)
+        msg = "Passing lists as key for xs is not allowed."
+        with pytest.raises(TypeError, match=msg):
+            ser.xs(["a", "x"], axis=0, drop_level=False)
+
+        with tm.assert_produces_warning(FutureWarning):
+            ser.xs(["a"], axis=0, drop_level=False)


Code review comments:
diff_hunk: @@ -3756,6 +3756,18 @@ class   animal   locomotion
         """
         axis = self._get_axis_number(axis)
         labels = self._get_axis(axis)
+
+        if isinstance(key, list):
+            len_levels = 1 if level is None or is_scalar(level) else len(level)
+            if len(key) != len_levels:
path: pandas/core/generic.py
text: @user1:
does this break any existing tests?

@author:
One ``test_xs_partial``

Could deprecate this too, if you prefer?

@user1:
yeah let's just deprecate entirely (prob easier to remember too)

@author:
Done