Repository: pandas-dev/pandas
Pull request: BUG: partial string indexing with scalar
Closes #27516

cc @jreback if you have thoughts on this approach.

Problem statement:
Partial string indexing returns ndarray rather than Series.
#### Code Sample, a copy-pastable example if possible

```python
import pandas as pd
import numpy as np

print(pd.__version__)

df_pass = pd.DataFrame(index=range(1,1000), columns=['A', 'B', 'C'])
df_pass.loc[:, :] = np.random.uniform(-100, 100, size=(len(df_pass.index), len(df_pass.columns)))
print(df_pass.loc[range(1,500), 'A'].sum(skipna=False)) # everything is fine here

df_fail = pd.DataFrame(index=pd.date_range('01-01-2005', '12-01-2006'), columns=['A', 'B', 'C'])
df_fail .loc[:, :] = np.random.uniform(-100, 100, size=(len(df_fail .index), len(df_fail .columns)))
print(df_fail .loc['2005', 'A'].sum(skipna=False)) # Here the type-error appears
```

Output:
```
0.25.0
-847.9947710494175
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-20-26bdb9fa00f7> in <module>()
     10 df_fail = pd.DataFrame(index=pd.date_range('01-01-2005', '12-01-2006'), columns=['A', 'B', 'C'])
     11 df_fail .loc[:, :] = np.random.uniform(-100, 100, size=(len(df_fail .index), len(df_fail .columns)))
---> 12 print(df_fail .loc['2005', 'A'].sum(skipna=False)) # Here the type-error appears
     13
```

#### Problem description

Before updating from 0.24.0 to 0.25.0 everything worked fine. I can also not see, that there was an API change here. I would expect, that the second sum, works without issues.

#### Expected Output

Output (something like):
```
0.25.0
-847.9947710494175
-451.5691327012012
```

#### Output of ``pd.show_versions()``

```
INSTALLED VERSIONS
------------------
commit           : None
python           : 3.6.5.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
machine          : AMD64
processor        : Intel64 Family 6 Model 78 Stepping 3, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : None
LOCALE           : None.None

pandas           : 0.25.0
numpy            : 1.14.3
pytz             : 2018.4
dateutil         : 2.7.3
pip              : 19.1.1
setuptools       : 39.1.0
Cython           : 0.28.2
pytest           : 3.10.0
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : 1.0.5
lxml.etree       : 4.2.5
html5lib         : 1.0.1
pymysql          : None
psycopg2         : 2.7.6.1 (dt dec pq3 ext lo64)
jinja2           : 2.10
IPython          : 6.4.0
pandas_datareader: None
bs4              : 4.7.1
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : 4.2.5
matplotlib       : 2.2.2
numexpr          : None
odfpy            : None
openpyxl         : 2.6.0
pandas_gbq       : None
pyarrow          : None
pytables         : None
s3fs             : None
scipy            : 1.1.0
sqlalchemy       : 1.2.12
tables           : None
xarray           : None
xlrd             : 1.1.0
xlwt             : None
xlsxwriter       : 1.0.5
```

Hints from issue discussion:
That's an indexing bug. Somehow `.loc` is returning an an ndarray rather than a Series.

```python

In [27]: df = pd.DataFrame({"A": 1}, index=pd.date_range("2000", periods=100))

In [28]: df.loc['2000-01', 'A']
Out[28]:
array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1])
```
https://github.com/pandas-dev/pandas/pull/27110 seems like the most likely candidate (cc @jbrockmendel).

IIUC, we can't treat `('2000-01', 'A')` as a scalar, since it's really shorthand for the expanded indexing. Do you have time to look into this @jbrockmendel?

This may warrant an expedited 0.25.1. WDYT @jreback?

likely more things would show as people actually use the new release

let’s just do a few weeks on this 
At first glance, I don't see how #27110 would cause this since that should affect DatetimeTZBlock but not DatetimeBlock.

There have been some other recent PRs that have tried to simplify core.indexing, maybe something got lost in there.  I'll take a look.
Ah sorry. I was just going  release notes that sounded promising and stopped at that one.
Tracking this down a bit, following Tom's example.

`df.loc.__getitem__` eventually calls `df._get_value('2000-01', 'A')`.  In 0.24.2 `KeyError` is raised by `engine.get_value`.  Now we fall through following that KeyError
Looks like the relevant change was #26298
Thanks @jbrockmendel. I would not have guessed that based on the name. Do you have a fix in mind?
> Do you have a fix in mind?

In `DataFrame._get_value` that PR changed the KeyError behavior to only raise for MultIIndex.  That will need to raise in more cases.  Not yet sure just how tight it will need to be.

Commit message: BUG: partial string indexing with scalar

Closes #27516

Patch to review:
diff --git a/doc/source/whatsnew/v0.25.1.rst b/doc/source/whatsnew/v0.25.1.rst
index c80195af413f7..3097bfa21f9e1 100644
--- a/doc/source/whatsnew/v0.25.1.rst
+++ b/doc/source/whatsnew/v0.25.1.rst
@@ -82,7 +82,7 @@ Interval
 Indexing
 ^^^^^^^^
 
--
+- Bug in partial-string indexing returning a NumPy array rather than a ``Series`` when indexing with a scalar like ``.loc['2015']`` (:issue:`27516`)
 -
 -
 
diff --git a/pandas/core/indexing.py b/pandas/core/indexing.py
index df89dbe6db6dc..c87cb358a1410 100755
--- a/pandas/core/indexing.py
+++ b/pandas/core/indexing.py
@@ -1704,6 +1704,11 @@ def _is_scalar_access(self, key: Tuple):
             if isinstance(ax, MultiIndex):
                 return False
 
+            if isinstance(k, str) and ax.is_all_dates:
+                # partial string indexing, df.loc['2000', 'A']
+                # should not be considered scalar
+                return False
+
             if not ax.is_unique:
                 return False
 
diff --git a/pandas/tests/indexes/datetimes/test_partial_slicing.py b/pandas/tests/indexes/datetimes/test_partial_slicing.py
index 3095bf9657277..5660fa5ffed80 100644
--- a/pandas/tests/indexes/datetimes/test_partial_slicing.py
+++ b/pandas/tests/indexes/datetimes/test_partial_slicing.py
@@ -468,3 +468,14 @@ def test_getitem_with_datestring_with_UTC_offset(self, start, end):
         with pytest.raises(ValueError, match="The index must be timezone"):
             df = df.tz_localize(None)
             df[start:end]
+
+    def test_slice_reduce_to_series(self):
+        # GH 27516
+        df = pd.DataFrame(
+            {"A": range(24)}, index=pd.date_range("2000", periods=24, freq="M")
+        )
+        expected = pd.Series(
+            range(12), index=pd.date_range("2000", periods=12, freq="M"), name="A"
+        )
+        result = df.loc["2000", "A"]
+        tm.assert_series_equal(result, expected)


Code review comments:
diff_hunk: @@ -1704,6 +1704,11 @@ def _is_scalar_access(self, key: Tuple):
             if isinstance(ax, MultiIndex):
                 return False
 
+            if isinstance(k, str) and ax.is_all_dates:
path: pandas/core/indexing.py
text: @author:
Is `is_all_dates` the appropriate way to check whether partial string indexing is supported?

@author:
My concern is that it's potentially expensive. Why not just `isinstance(ax, DatetimeIndex)`?

@user1:
could alias `is_all_dates` to `_partial_str_indexing_supported` (or something less verbose) to make this more obvious?

For non-object Index is_all_dates is pretty cheap

@user2:
this is not the right place for this, rather you just call: 

self._get_partial_string_timestamp_match_key(key, labels)

@author:
Where would you do that? The problem with the current structure is that `_LocIndexer.__getitem__` if `self._is_scalar_access(key)` returns True, we go straight to `return self._getitem_scalar(key)`, which is incorrect for partial string indexing.

It doesn't look like `_get_partial_string_timestamp_match_key` will work, at least not directly. That doesn't convert the partial string key to a slice or array indexer.

```python
(Pdb) pp key
('2000', 'A')
(Pdb) pp self._get_partial_string_timestamp_match_key(key[0], self.obj.axes[0])
'2000'
```

Stepping back, I think that `_is_scalar_access` *should* return False for partial string indexing, since it's not actually a scalar access.

@author:
> could alias is_all_dates to _partial_str_indexing_supported (or something less verbose) to make this more obvious?

> For non-object Index is_all_dates is pretty cheap
 
It's the object case that I'm worried about, as I think this always hit.

Rather than inferring, why don't we define whether an index type supports partial string indexing as a class attribute? That skips the (potentially) expensive inference step, which IIUC would only be useful if (object-type) `Index` objects supported partial string indexing, which I don't believe they do.

@author:
0fb9bbe6e6b95d7bd1e47233e7fa659a48141a4b has that, as a POC. All the indexing tests pass locally.