BUG: Fix return type of loc/iloc
State: closed

- [X] closes #60600 
- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [X] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst` file if fixing a bug or adding a new feature.

## Description of Linked Issue
`loc/iloc` inconsistently returns dtype. For example,

```python
>>> import pandas as pd
>>> df = pd.DataFrame([['a', 1., 2.], ['b', 3., 4.]])

>>> df.loc[0, [1, 2]]
1    1.0
2    2.0
Name: 0, dtype: object

>>> df[[1, 2]].loc[0]
1    1.0
2    2.0
Name: 0, dtype: float64

>>> df.loc[[0, 1], 1]
0    1.0
1    3.0
Name: 1, dtype: float64
```

This behaviour seems to happen following the below sequence:
1. For axis=0, `BlockManager.fast_xs()` returns a cross-section of `df`, determining the `dtype` as `object`, since `df.loc[0,:]` is supposed to include `'a'`.
2. For axis=1, `NDFrame._reindex_with_indexers()` returns the result, not additionally inferring the dtype of the result.

## Proposed Solution

Based on the above examples, we can conclude that this issue only apprears where `axis[0]=int` amd `axis[1]=list/slice` - `loc[int/slice]`.
Therefore, I'd like to propose to add the below codes to additionally infer the dtype after the column selection.

```python
    @final
    def _getitem_lowerdim(self, tup: tuple):

...

                # This is an elided recursive call to iloc/loc
                out = getattr(section, self.name)[new_key]
                # Re-interpret dtype of out.values for loc/iloc[int, list/slice].
                # GH60600
                if (
                    i == 0
                    and isinstance(key, int)
                    and isinstance(new_key, (list, slice))
                ):
                    out = out.infer_objects()
                return out
```

Thanks!

Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
I think the way to do it would be by changing getitem_lowerdim to reverse the order in which we do the indexing (i think the loop on L1069).  Would need to try it to see if that breaks anything.  Best guess would be MultiIndex cases (possibly their perf?)

I think a bunch of getitem_lowerdim dates back to when Panel and PanelND existed, so it may be more general than it needs to be.
Hi @jbrockmendel ,
Thanks for the review!

I've tried to reverse the indexing order, but it seems some tests are expecting `_getitem_lowerdim()` to index in a forward direction.
Should I revise these test, or just let the method do the indexing in a forward direction as I did in the first commit?
I'm not sure since I think the error might be confusing by changing the order.
> Should I revise these test, or just let the method do the indexing in a forward direction as I did in the first commit?

is the user-facing behavior in the affected tests changed?
Hi @jbrockmendel ,
I suppose user will see an error that points out `column` not `Index`, when both `Index` and `column` have wrong input since we'll try to get the section based on the given `column` prior to `Index`. 
But I thought this might be kinda consistent behaviour since we do the same thing for `MultiIndex` at `_LocationIndexer._getitem_nested_tuple` as below.

https://github.com/pandas-dev/pandas/blob/5736b9647068d31fdf8673d3528cb64e35060bac/pandas/core/indexing.py#L1152-L1168


Reproducible example:
```python
>>> import pandas as pd
>>> df2 = pd.DataFrame([{"a":1.0, "b":2, "c":3, "d":4}]).set_index(["c", "d"])
>>> df2
       a  b
c d
3 4  1.0  2

>>> r = df2.loc[("X", "Y"), "Z"]
...
KeyError: 'Z'

```
Are you sure that example is right?  It looks like you define `df2` but then access `df`.
My apologies for the confuesion @jbrockmendel , I made a typo. Just fixed the example code.
Thanks I think I understand now.  If the user-facing change is just a different exception message (and the new one still looks quite useful) then this seems reasonable to me
Thanks, @rhshadrach , Here's the result from my side.

<details>

```
| Change   | Before [5fef9793] <main>   | After [3596f764] <fix-loc-dtype>   |   Ratio | Benchmark (Parameter)                                                                                                                                                 |
|----------|----------------------------|------------------------------------|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| +        | 151±3μs                    | 502±8μs                            |    3.32 | indexing_engines.NumericEngineIndexing.time_get_loc_near_middle((<class 'pandas._libs.index.UInt64Engine'>, <class 'numpy.uint64'>), 'monotonic_decr', False, 100000) |
| +        | 190±4μs                    | 224±10μs                           |    1.18 | indexing_engines.MaskedNumericEngineIndexing.time_get_loc((<class 'pandas._libs.index.MaskedInt16Engine'>, 'Int16'), 'monotonic_decr', True, 100000)                  |
| -        | 9.13±0.8ms                 | 8.28±0.06ms                        |    0.91 | indexing_engines.MaskedNumericEngineIndexing.time_get_loc_near_middle((<class 'pandas._libs.index.MaskedUInt8Engine'>, 'UInt8'), 'monotonic_decr', False, 2000000)    |
| -        | 1.85±0.06ms                | 1.63±0.03ms                        |    0.88 | indexing.NumericSeriesIndexing.time_loc_array(<class 'numpy.uint64'>, 'unique_monotonic_inc')                                                                         |
| -        | 7.80±0.2ms                 | 6.89±0.07ms                        |    0.88 | indexing_engines.NumericEngineIndexing.time_get_loc((<class 'pandas._libs.index.UInt64Engine'>, <class 'numpy.uint64'>), 'monotonic_decr', False, 2000000)            |
| -        | 516±20μs                   | 449±20μs                           |    0.87 | indexing.NumericMaskedIndexing.time_get_indexer_dups('UInt64', True)                                                                                                  |
| -        | 627±20μs                   | 547±10μs                           |    0.87 | indexing.NumericSeriesIndexing.time_getitem_list_like(<class 'numpy.uint64'>, 'nonunique_monotonic_inc')                                                              |
| -        | 3.82±0.2ms                 | 3.30±0.2ms                         |    0.87 | indexing.NumericSeriesIndexing.time_loc_array(<class 'numpy.float64'>, 'unique_monotonic_inc')                                                                        |
| -        | 5.66±0.5ms                 | 4.75±0.1ms                         |    0.84 | indexing.NumericSeriesIndexing.time_getitem_lists(<class 'numpy.float64'>, 'unique_monotonic_inc')                                                                    |
| -        | 10.1±0.5ms                 | 8.32±0.3ms                         |    0.83 | indexing.NumericSeriesIndexing.time_getitem_scalar(<class 'numpy.uint64'>, 'nonunique_monotonic_inc')                                                                 |
| -        | 10.0±0.3ms                 | 8.16±0.2ms                         |    0.81 | indexing.NumericSeriesIndexing.time_loc_scalar(<class 'numpy.uint64'>, 'nonunique_monotonic_inc')                                                                     |
| -        | 10.3±0.5ms                 | 8.27±0.2ms                         |    0.8  | indexing.NumericSeriesIndexing.time_loc_slice(<class 'numpy.uint64'>, 'nonunique_monotonic_inc')                                                                      |
| -        | 3.69±0.7ms                 | 2.80±0.1ms                         |    0.76 | indexing.MultiIndexing.time_loc_all_null_slices(False)                                                                                                                |
| -        | 462±9μs                    | 114±1μs                            |    0.25 | indexing_engines.NumericEngineIndexing.time_get_loc((<class 'pandas._libs.index.Int64Engine'>, <class 'numpy.int64'>), 'monotonic_decr', False, 100000)               |

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE DECREASED.

```

</details>
Thanks @sanggon6107 - I think we'll want to see if we can avoid the perf hit in that first row (and 2nd row if possible, but that is more okay). Or at least understand why if we cannot. I can help if desired, should have some time to look into this in the next week.
Thanks for the feedback @rhshadrach !
I also wonder why we see the drop in the performance in the first row whereas it appears that there's an improvement in the last row even though they have similar parameters. I'll do some further investigations from my end and get back to you. Any help would be greatly appreciated as well.
Hi @rhshadrach ,
I've additionally run the asv several times since I was not sure if `get_loc` really is related to `_getitem_lowerdim()` and the results are as follows :

<DETAILS>

```
| Change   | Before [a8113887] <main>   | After [3596f764] <fix-loc-dtype>   |   Ratio | Benchmark (Parameter)                                                                                                                                                   |
|----------|----------------------------|------------------------------------|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| +        | 98.7±30μs                  | 321±20μs                           |    3.25 | indexing_engines.NumericEngineIndexing.time_get_loc((<class 'pandas._libs.index.Int32Engine'>, <class 'numpy.int32'>), 'monotonic_decr', False, 100000)                 |
| -        | 145±2μs                    | 131±2μs                            |    0.91 | indexing.NumericSeriesIndexing.time_loc_slice(<class 'numpy.float64'>, 'nonunique_monotonic_inc')                                                                       |
| -        | 384±7μs                    | 339±7μs                            |    0.88 | indexing.CategoricalIndexIndexing.time_get_indexer_list('monotonic_decr')                                                                                               |
| -        | 1.44±0.03ms                | 1.19±0.02ms                        |    0.83 | indexing.MultiIndexing.time_loc_null_slice_plus_slice(True)                                                                                                             |
| -        | 154±40μs                   | 100±1μs                            |    0.65 | indexing_engines.NumericEngineIndexing.time_get_loc_near_middle((<class 'pandas._libs.index.Int8Engine'>, <class 'numpy.int8'>), 'monotonic_decr', False, 100000)       |
| -        | 359±10μs                   | 133±7μs                            |    0.37 | indexing_engines.NumericEngineIndexing.time_get_loc_near_middle((<class 'pandas._libs.index.Float64Engine'>, <class 'numpy.float64'>), 'monotonic_decr', False, 100000) |

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE DECREASED.
```

</DETAILS>

<DETAILS>

```
| Change   | Before [a8113887] <main>   | After [3596f764] <fix-loc-dtype>   |   Ratio | Benchmark (Parameter)                                                                                                                                       |
|----------|----------------------------|------------------------------------|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------------|
| +        | 189±100μs                  | 331±8μs                            |    1.75 | indexing_engines.NumericEngineIndexing.time_get_loc((<class 'pandas._libs.index.Float32Engine'>, <class 'numpy.float32'>), 'monotonic_decr', False, 100000) |

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE DECREASED.
```

</DETAILS>

<DETAILS>

```
| Change   | Before [a8113887] <main>   | After [3596f764] <fix-loc-dtype>   |   Ratio | Benchmark (Parameter)                                                                                                                                                   |
|----------|----------------------------|------------------------------------|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| +        | 83.3±2μs                   | 327±7μs                            |    3.93 | indexing_engines.NumericEngineIndexing.time_get_loc((<class 'pandas._libs.index.Float64Engine'>, <class 'numpy.float64'>), 'monotonic_decr', False, 100000)             |
| +        | 125±5μs                    | 365±10μs                           |    2.93 | indexing_engines.NumericEngineIndexing.time_get_loc_near_middle((<class 'pandas._libs.index.Float64Engine'>, <class 'numpy.float64'>), 'monotonic_decr', False, 100000) |
| +        | 181±100μs                  | 321±5μs                            |    1.78 | indexing_engines.NumericEngineIndexing.time_get_loc((<class 'pandas._libs.index.UInt32Engine'>, <class 'numpy.uint32'>), 'monotonic_decr', False, 100000)               |
| +        | 217±100μs                  | 341±8μs                            |    1.57 | indexing_engines.NumericEngineIndexing.time_get_loc((<class 'pandas._libs.index.UInt64Engine'>, <class 'numpy.uint64'>), 'monotonic_decr', False, 100000)               |
| -        | 445±9μs                    | 404±6μs                            |    0.91 | indexing.NumericMaskedIndexing.time_get_indexer('UInt64', True)                                                                                                         |
| -        | 11.8±0.4μs                 | 10.7±0.1μs                         |    0.91 | indexing.NumericSeriesIndexing.time_getitem_scalar(<class 'numpy.int64'>, 'nonunique_monotonic_inc')                                                                    |
| -        | 221±20μs                   | 199±2μs                            |    0.9  | indexing.MultiIndexing.time_loc_partial_key_scalar(True)                                                                                                                |
| -        | 771±20μs                   | 691±20μs                           |    0.9  | indexing.NumericMaskedIndexing.time_get_indexer('Float64', True)                                                                                                        |
| -        | 344±10μs                   | 311±4μs                            |    0.9  | indexing.NumericMaskedIndexing.time_get_indexer('Int64', True)                                                                                                          |
| -        | 4.72±0.3ms                 | 4.23±0.04ms                        |    0.9  | indexing_engines.NumericEngineIndexing.time_get_loc((<class 'pandas._libs.index.Int16Engine'>, <class 'numpy.int16'>), 'monotonic_decr', True, 2000000)                 |
| -        | 1.51±0.07ms                | 1.34±0.03ms                        |    0.89 | indexing.MultiIndexing.time_loc_all_slices(True)                                                                                                                        |
| -        | 107±3μs                    | 95.0±2μs                           |    0.89 | indexing_engines.MaskedNumericEngineIndexing.time_get_loc_near_middle((<class 'pandas._libs.index.MaskedUInt64Engine'>, 'UInt64'), 'non_monotonic', True, 2000000)      |
| -        | 139±4μs                    | 123±1μs                            |    0.88 | indexing.MultiIndexing.time_loc_all_scalars(True)                                                                                                                       |
| -        | 5.05±0.2ms                 | 4.42±0.03ms                        |    0.88 | indexing.NumericSeriesIndexing.time_getitem_lists(<class 'numpy.float64'>, 'unique_monotonic_inc')                                                                      |
| -        | 1.19±0.03ms                | 1.05±0.01ms                        |    0.88 | indexing.NumericSeriesIndexing.time_loc_list_like(<class 'numpy.float64'>, 'unique_monotonic_inc')                                                                      |
| -        | 62.0±3μs                   | 54.0±0.8μs                         |    0.87 | indexing.MultiIndexing.time_xs_full_key(False)                                                                                                                          |
| -        | 20.4±1μs                   | 17.5±0.3μs                         |    0.86 | indexing.NumericSeriesIndexing.time_getitem_scalar(<class 'numpy.float64'>, 'nonunique_monotonic_inc')                                                                  |
| -        | 1.78±0.04ms                | 1.51±0.01ms                        |    0.85 | indexing.NonNumericSeriesIndexing.time_getitem_list_like('datetime', 'unique_monotonic_inc')                                                                            |
| -        | 499±70μs                   | 412±10μs                           |    0.83 | indexing.NonNumericSeriesIndexing.time_getitem_list_like('string', 'non_monotonic')                                                                                     |
| -        | 13.1±2ms                   | 10.5±0.1ms                         |    0.8  | indexing_engines.MaskedNumericEngineIndexing.time_get_loc_near_middle((<class 'pandas._libs.index.MaskedFloat64Engine'>, 'Float64'), 'monotonic_decr', False, 2000000)  |
| -        | 11.7±0.3ms                 | 8.95±0.2ms                         |    0.77 | indexing.MultiIndexing.time_loc_multiindex(False)                                                                                                                       |
| -        | 16.7±2ms                   | 11.7±0.1ms                         |    0.7  | indexing.MultiIndexing.time_loc_slice_plus_null_slice(False)                                                                                                            |
| -        | 145±50μs                   | 94.5±1μs                           |    0.65 | indexing_engines.MaskedNumericEngineIndexing.time_get_loc_near_middle((<class 'pandas._libs.index.MaskedInt64Engine'>, 'Int64'), 'non_monotonic', True, 2000000)        |
| -        | 30.8±3μs                   | 18.8±0.6μs                         |    0.61 | indexing.NonNumericSeriesIndexing.time_getitem_pos_slice('string', 'non_monotonic')                                                                                     |
| -        | 31.9±1μs                   | 19.4±0.3μs                         |    0.61 | indexing.NonNumericSeriesIndexing.time_getitem_pos_slice('string', 'nonunique_monotonic_inc')                                                                           |
| -        | 254±100μs                  | 148±2μs                            |    0.58 | indexing_engines.NumericEngineIndexing.time_get_loc_near_middle((<class 'pandas._libs.index.UInt64Engine'>, <class 'numpy.uint64'>), 'monotonic_decr', False, 100000)   |
| -        | 158±80μs                   | 90.6±0.9μs                         |    0.57 | indexing_engines.MaskedNumericEngineIndexing.time_get_loc((<class 'pandas._libs.index.MaskedFloat32Engine'>, 'Float32'), 'non_monotonic', True, 2000000)                |
| -        | 169±80μs                   | 92.1±2μs                           |    0.54 | indexing_engines.MaskedNumericEngineIndexing.time_get_loc_near_middle((<class 'pandas._libs.index.MaskedInt32Engine'>, 'Int32'), 'non_monotonic', True, 2000000)        |
```

</DETAILS>

The result seem to vary everytime even though I've tried in the same commit. I also couldn't find any codes that call `_getitem_lowerdim()` inside `Index.get_loc()`. Did I get something wrong? Can the result vary depending on the PC's internal status or the surrounding environment?
@sanggon6107 - thanks for the information here. It's true, the ASVs can be chaotic. This is where I turn to running any particular benchmark with timeit. E.g.

```python
from asv_bench.benchmarks.indexing_engines import NumericEngineIndexing, _get_numeric_engines

obj = NumericEngineIndexing()
obj.setup((pd._libs.index.UInt64Engine, np.uint64), "monotonic_decr", False, 100000)
%timeit obj.time_get_loc_near_middle((pd._libs.index.UInt64Engine, np.uint64), "monotonic_decr", False, 100000)
# 121 μs ± 891 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)  <-- PR
# 123 μs ± 7.08 μs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)  <-- main
```

I'm good here, just need the CI green.
Thanks, @rhshadrach ! I've learned something new here. The CI is now green.
Of course @rhshadrach , just added the whatsnew.
Thanks @sanggon6107 - note CI failing is not due to the changes here, but will wait on merging until we've fixed the CI outside of this PR.
Thanks @rhshadrach , looks like the CI is back to green now.
Thanks @sanggon6107