Repository: pandas-dev/pandas
Pull request: PERF: Faster Series.__getattribute__
Closes #19764

Problem statement:
PERF: Very slow printing of Series with DatetimeIndex
Consider this series with a datetime index:

```
s = pd.Series(np.random.randn(int(1e6)), 
              index=pd.date_range("2012-01-01", freq='s', periods=int(1e6)))
```

Showing this in the console or notebook (doing just `s`) is very slow (noticable delay of 1 to 2 seconds in console, around 8 seconds in the notebook or JupyterLab ). But, on the other hand, in a plain Python console the display is instantly.

Further, on the other hand, if the series has no DatetimeIndex, the display appears instantly. Also, if it is a frame (doing `s.to_frame()`) the display is instantly. And when doing `print(s)` explicitly, the display is actually identical and also instantly. When it is another datetime-like index, eg `s.to_period()`, the display is instant.

So it seems we are doing some work under the hood that is not needed. And it is somehow triggered by doing this in an IPython context, and specifically for DatetimeIndex.

This already seems to be present on some older versions of pandas as well.

Hints from issue discussion:
cc @takluyver what is the mechanism how in general an object is displayed? As the `__repr__` or `__str__` itself is fast, but displaying in an IPython console is not.
Since seems to be present in IPython >= 6.1 (I don't see it in an environment with IPython 6.0). And in an ipython console, the overhead comes from `MimeBundleFormatter.__call__` calling `get_real_method`, which in the end does a `getattr` on the series to check if a `_repr_mimebundle_` exists.

So in the end it is due to `getattr` taking a huge amount of time on a Series with a DatetimeIndex. With the example from above:

```
In [29]: %timeit getattr(s, '_bla', None)
1.39 s ± 31.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [31]: s2 = s.reset_index(drop=True)

In [32]: %timeit getattr(s2, '_bla', None)
3.54 µs ± 15.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

In [33]: df = s.to_frame()

In [34]: %timeit getattr(df, '_bla', None)
3.64 µs ± 67 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

```

And from the same profiling, it seems it is doing expensive `get_loc` calls on the index a few times, which stem from a 'contains' call in generic.py `__getattr__` (`name in s.index`), and timings for this:

```
In [41]: %timeit '_bla' in s.index
517 ms ± 69.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [42]: %timeit '_bla' in s2.index
1.4 µs ± 79.1 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

In [43]: %timeit '_bla' in s3.index
2.91 µs ± 149 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
```

which in turn steps from a very slow `get_loc` of the DatetimeIndex engine. I suppose it is doing (for this case) way too much work because it tries to interpret the string as a datetime string.
I didn't know we did dot-attribute indexing for a Series - seems odd - something we want to support / actually gets used?

```python
In [10]: s = pd.Series([1, 2, 3], index=['a', 'b', 'c'])

In [11]: s.c
Out[11]: 3
```

I guess it is fairly prominent in the indexing docs, just missed it.

https://pandas.pydata.org/pandas-docs/stable/indexing.html#attribute-access
Yeah, the dataframe equivalent is more known and used, I personally don't think many people do that for series as well.

For the original issues, I think we should use in `__getattr__` somehow a 'smarter' contains check, because I don't think we want to support any string-like to be used as attribute for DatetimeIndex (I am thinking if there is actually one that would be a valid attribute name, as it cannot start with a number).


There also seems to be fundamental performance problem with `DatetimeIndex.get_loc` - maybe same as #17754

```python
In [26]: dti = s.index

In [27]: dti.get_loc('2012-01-01')
Out[27]: slice(0, 86400, None)

In [28]: %timeit dti.get_loc('2012-01-01')
1.24 s ± 25.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [29]: %timeit dti.get_loc(pd.Timestamp('2012-01-01'))
11 µs ± 126 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
```
Explored a bit what `DatetimeIndex.get_loc` and `DatetimeEngine.get_loc` are doing, and a lot is going on:

- Something like `dti.get_loc('blabla')` ultimately calls `DatetimeEngine.get_loc` 2 times, and `dti.get_loc('2012-01-01 00:00:00')` as well. While we can know in advance that `DatetimeEngine.get_loc` will fail as this method does not work for strings (it needs a datetime-like)
- A single call to `DatetimeEngine.get_loc('string')`, although it will never work, takes a lot of time. The ultimate reason for this long time is that it is trying a `values.searchsorted(key)` where `values` are the int64 data and `key` is the string. The numpy searchsorted will then cast under the hood the int values to string or object (whather the common dtype between values and key), which is slow. The value that then comes out of it, in case of a string, is outside the range (just added to the right of the values) and we check this and raise a KeyError. While in case of a string we just never should have tried.
- When I would have used `int(1e6) - 1` instead of `int(1e6)` values, we are below the size threshold, and it takes a different code path. In that case the situation is better (we still do unneeded tries of `get_loc`, but they are not that slow in that case)
Correctness question, is this the right output?

```python
In [5]: pd.date_range('2017', periods=12)
Out[5]:
DatetimeIndex(['2017-01-01', '2017-01-02', '2017-01-03', '2017-01-04',
               '2017-01-05', '2017-01-06', '2017-01-07', '2017-01-08',
               '2017-01-09', '2017-01-10', '2017-01-11', '2017-01-12'],
              dtype='datetime64[ns]', freq='D')

In [6]: pd.date_range('2017', periods=12).get_loc("2017")
Out[6]: slice(0, 12, None)
```

The docstring says in int is returned. The closes analog I can think of is `IntervalIndex` w/ overlapping intervals.

```python
In [12]: pd.IntervalIndex.from_tuples([(0, 1), (0, 2), (0, 3)]).get_loc(0.5)
Out[12]: array([0, 1, 2])
```
Second question, should `DatetimeIndex.get_loc(int)` ever work? We test that it does.

```python
In [11]: s.index.get_loc(s.index[0].value)
Out[11]: 0
```

That's taking the underlying integer representation. cc @jreback
Not really sure, should dive into the code again.

But, to fix the actual regression, I think the easier path will be to avoid any `get_loc` call that is coming from `__getattr__` altogether, as I mentioned in https://github.com/pandas-dev/pandas/issues/19764#issuecomment-366760838 (not that `get_loc` shouldn't be fixed)

Commit message: PERF: Faster Series.__getattribute__

Closes #19764

Patch to review:
diff --git a/doc/source/whatsnew/v0.23.0.txt b/doc/source/whatsnew/v0.23.0.txt
index ffa4f1068f84d..7082850488326 100644
--- a/doc/source/whatsnew/v0.23.0.txt
+++ b/doc/source/whatsnew/v0.23.0.txt
@@ -958,6 +958,7 @@ Performance Improvements
 - Improved performance of :func:`pandas.core.groupby.GroupBy.any` and :func:`pandas.core.groupby.GroupBy.all` (:issue:`15435`)
 - Improved performance of :func:`pandas.core.groupby.GroupBy.pct_change` (:issue:`19165`)
 - Improved performance of :func:`Series.isin` in the case of categorical dtypes (:issue:`20003`)
+- Improved performance of ``Series.__getattribute__`` when the Series has certain index types. This manifiested in slow printing of large Series with a ``DatetimeIndex`` (:issue:`19764`)
 - Fixed a performance regression for :func:`GroupBy.nth` and :func:`GroupBy.last` with some object columns (:issue:`19283`)
 
 .. _whatsnew_0230.docs:
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
index 86342b6996abf..f78b2cb996f42 100644
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -4375,7 +4375,7 @@ def __getattr__(self, name):
                 name in self._accessors):
             return object.__getattribute__(self, name)
         else:
-            if name in self._info_axis:
+            if self._info_axis._is_dotable and name in self._info_axis:
                 return self[name]
             return object.__getattribute__(self, name)
 
diff --git a/pandas/core/indexes/base.py b/pandas/core/indexes/base.py
index 2ceec1592d49b..c87ca9c3c8d04 100644
--- a/pandas/core/indexes/base.py
+++ b/pandas/core/indexes/base.py
@@ -244,6 +244,12 @@ class Index(IndexOpsMixin, PandasObject):
     _engine_type = libindex.ObjectEngine
 
     _accessors = set(['str'])
+    # Whether items can be selected from NDFrame.<item>
+    # Some indexes (DatetimeIndex, Int64Index) cannot contain
+    # valid Python identifiers. Setting _is_dotable = False is an
+    # optimization.
+    # https://github.com/pandas-dev/pandas/issues/19764
+    _is_dotable = True
 
     str = CachedAccessor("str", StringMethods)
 
diff --git a/pandas/core/indexes/interval.py b/pandas/core/indexes/interval.py
index 54800d0d76d2e..82123062cd57a 100644
--- a/pandas/core/indexes/interval.py
+++ b/pandas/core/indexes/interval.py
@@ -207,6 +207,7 @@ class IntervalIndex(IntervalMixin, Index):
     _typ = 'intervalindex'
     _comparables = ['name']
     _attributes = ['name', 'closed']
+    _is_dotable = False  # can't contain Python identifiers
 
     # we would like our indexing holder to defer to us
     _defer_to_indexing = True
diff --git a/pandas/core/indexes/numeric.py b/pandas/core/indexes/numeric.py
index 1fe0c8fa289e6..4267bde342932 100644
--- a/pandas/core/indexes/numeric.py
+++ b/pandas/core/indexes/numeric.py
@@ -31,6 +31,7 @@ class NumericIndex(Index):
 
     """
     _is_numeric_dtype = True
+    _is_dotable = False  # Can't contain Python identifiers
 
     def __new__(cls, data=None, dtype=None, copy=False, name=None,
                 fastpath=False):
diff --git a/pandas/core/indexes/period.py b/pandas/core/indexes/period.py
index 4a224d4e6ee7f..a8ab01c2cb2b0 100644
--- a/pandas/core/indexes/period.py
+++ b/pandas/core/indexes/period.py
@@ -204,6 +204,7 @@ class PeriodIndex(DatelikeOps, DatetimeIndexOpsMixin, Int64Index):
     """
     _typ = 'periodindex'
     _attributes = ['name', 'freq']
+    _is_dotable = False  # Can't contain Python identifiers
 
     # define my properties & methods for delegation
     _other_ops = []
diff --git a/pandas/tests/indexes/datetimelike.py b/pandas/tests/indexes/datetimelike.py
index 7d01a2a70145d..20a03ccaff696 100644
--- a/pandas/tests/indexes/datetimelike.py
+++ b/pandas/tests/indexes/datetimelike.py
@@ -8,6 +8,10 @@
 
 class DatetimeLike(Base):
 
+    def test_is_dotable(self):
+        idx = self.create_index()
+        assert idx._is_dotable is False
+
     def test_shift_identity(self):
 
         idx = self.create_index()
diff --git a/pandas/tests/indexes/test_base.py b/pandas/tests/indexes/test_base.py
index 7cd880b51661d..bda58e9935cc4 100644
--- a/pandas/tests/indexes/test_base.py
+++ b/pandas/tests/indexes/test_base.py
@@ -66,6 +66,10 @@ def generate_index_types(self, skip_index_keys=[]):
             if key not in skip_index_keys:
                 yield key, idx
 
+    def test_is_dotable(self):
+        idx = self.create_index()
+        assert idx._is_dotable is True
+
     def test_new_axis(self):
         new_index = self.dateIndex[None, :]
         assert new_index.ndim == 2
diff --git a/pandas/tests/indexes/test_category.py b/pandas/tests/indexes/test_category.py
index e9fddfde90348..7d72a121d34e0 100644
--- a/pandas/tests/indexes/test_category.py
+++ b/pandas/tests/indexes/test_category.py
@@ -33,6 +33,10 @@ def create_index(self, categories=None, ordered=False):
         return CategoricalIndex(
             list('aabbca'), categories=categories, ordered=ordered)
 
+    def test_is_dotable(self):
+        ci = self.create_index(categories=list('abcd'))
+        assert ci._is_dotable is True
+
     def test_construction(self):
 
         ci = self.create_index(categories=list('abcd'))
diff --git a/pandas/tests/indexes/test_multi.py b/pandas/tests/indexes/test_multi.py
index da11ac645ae07..184aa1f576fbd 100644
--- a/pandas/tests/indexes/test_multi.py
+++ b/pandas/tests/indexes/test_multi.py
@@ -48,6 +48,10 @@ def setup_method(self, method):
     def create_index(self):
         return self.index
 
+    def test_is_dotable(self):
+        idx = self.create_index()
+        assert idx._is_dotable is True
+
     def test_boolean_context_compat2(self):
 
         # boolean context compat
diff --git a/pandas/tests/indexes/test_numeric.py b/pandas/tests/indexes/test_numeric.py
index bafb6ae2e45f4..f09bc1550dca8 100644
--- a/pandas/tests/indexes/test_numeric.py
+++ b/pandas/tests/indexes/test_numeric.py
@@ -64,6 +64,10 @@ def test_index_rdiv_timedelta(self, scalar_td, index):
 
 class Numeric(Base):
 
+    def test_is_dotable(self):
+        idx = self.create_index()
+        assert idx._is_dotable is False
+
     def test_numeric_compat(self):
         pass  # override Base method
 
diff --git a/pandas/tests/indexes/test_range.py b/pandas/tests/indexes/test_range.py
index 8990834ebe91a..a7ec91a6afe13 100644
--- a/pandas/tests/indexes/test_range.py
+++ b/pandas/tests/indexes/test_range.py
@@ -44,6 +44,10 @@ def check_binop(self, ops, scalars, idxs):
                     expected = op(Int64Index(idx), scalar)
                     tm.assert_index_equal(result, expected)
 
+    def test_is_dotable(self):
+        idx = self.create_index()
+        assert idx._is_dotable is False
+
     def test_binops(self):
         ops = [operator.add, operator.sub, operator.mul, operator.floordiv,
                operator.truediv]


Code review comments:
diff_hunk: @@ -958,6 +958,7 @@ Performance Improvements
 - Improved performance of :func:`pandas.core.groupby.GroupBy.any` and :func:`pandas.core.groupby.GroupBy.all` (:issue:`15435`)
 - Improved performance of :func:`pandas.core.groupby.GroupBy.pct_change` (:issue:`19165`)
 - Improved performance of :func:`Series.isin` in the case of categorical dtypes (:issue:`20003`)
+- Improved performance of ``Series.__getattribute__`` when the Series has certain index types. This manifiested in slow printing of large Series with a ``DatetimeIndex`` (:issue:`19764`)
path: doc/source/whatsnew/v0.23.0.txt
text: @user1:
It's getattr instead of getattribute I think