Repository: pandas-dev/pandas

Issue:
PERF: Very slow printing of Series with DatetimeIndex
Consider this series with a datetime index:

```
s = pd.Series(np.random.randn(int(1e6)), 
              index=pd.date_range("2012-01-01", freq='s', periods=int(1e6)))
```

Showing this in the console or notebook (doing just `s`) is very slow (noticable delay of 1 to 2 seconds in console, around 8 seconds in the notebook or JupyterLab ). But, on the other hand, in a plain Python console the display is instantly.

Further, on the other hand, if the series has no DatetimeIndex, the display appears instantly. Also, if it is a frame (doing `s.to_frame()`) the display is instantly. And when doing `print(s)` explicitly, the display is actually identical and also instantly. When it is another datetime-like index, eg `s.to_period()`, the display is instant.

So it seems we are doing some work under the hood that is not needed. And it is somehow triggered by doing this in an IPython context, and specifically for DatetimeIndex.

This already seems to be present on some older versions of pandas as well.





Discussion:
cc @takluyver what is the mechanism how in general an object is displayed? As the `__repr__` or `__str__` itself is fast, but displaying in an IPython console is not.
Since seems to be present in IPython >= 6.1 (I don't see it in an environment with IPython 6.0). And in an ipython console, the overhead comes from `MimeBundleFormatter.__call__` calling `get_real_method`, which in the end does a `getattr` on the series to check if a `_repr_mimebundle_` exists.

So in the end it is due to `getattr` taking a huge amount of time on a Series with a DatetimeIndex. With the example from above:

```
In [29]: %timeit getattr(s, '_bla', None)
1.39 s ± 31.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [31]: s2 = s.reset_index(drop=True)

In [32]: %timeit getattr(s2, '_bla', None)
3.54 µs ± 15.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

In [33]: df = s.to_frame()

In [34]: %timeit getattr(df, '_bla', None)
3.64 µs ± 67 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

```

And from the same profiling, it seems it is doing expensive `get_loc` calls on the index a few times, which stem from a 'contains' call in generic.py `__getattr__` (`name in s.index`), and timings for this:

```
In [41]: %timeit '_bla' in s.index
517 ms ± 69.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [42]: %timeit '_bla' in s2.index
1.4 µs ± 79.1 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

In [43]: %timeit '_bla' in s3.index
2.91 µs ± 149 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
```

which in turn steps from a very slow `get_loc` of the DatetimeIndex engine. I suppose it is doing (for this case) way too much work because it tries to interpret the string as a datetime string.
I didn't know we did dot-attribute indexing for a Series - seems odd - something we want to support / actually gets used?

```python
In [10]: s = pd.Series([1, 2, 3], index=['a', 'b', 'c'])

In [11]: s.c
Out[11]: 3
```

I guess it is fairly prominent in the indexing docs, just missed it.

https://pandas.pydata.org/pandas-docs/stable/indexing.html#attribute-access
Yeah, the dataframe equivalent is more known and used, I personally don't think many people do that for series as well.

For the original issues, I think we should use in `__getattr__` somehow a 'smarter' contains check, because I don't think we want to support any string-like to be used as attribute for DatetimeIndex (I am thinking if there is actually one that would be a valid attribute name, as it cannot start with a number).


There also seems to be fundamental performance problem with `DatetimeIndex.get_loc` - maybe same as #17754

```python
In [26]: dti = s.index

In [27]: dti.get_loc('2012-01-01')
Out[27]: slice(0, 86400, None)

In [28]: %timeit dti.get_loc('2012-01-01')
1.24 s ± 25.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [29]: %timeit dti.get_loc(pd.Timestamp('2012-01-01'))
11 µs ± 126 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
```
Explored a bit what `DatetimeIndex.get_loc` and `DatetimeEngine.get_loc` are doing, and a lot is going on:

- Something like `dti.get_loc('blabla')` ultimately calls `DatetimeEngine.get_loc` 2 times, and `dti.get_loc('2012-01-01 00:00:00')` as well. While we can know in advance that `DatetimeEngine.get_loc` will fail as this method does not work for strings (it needs a datetime-like)
- A single call to `DatetimeEngine.get_loc('string')`, although it will never work, takes a lot of time. The ultimate reason for this long time is that it is trying a `values.searchsorted(key)` where `values` are the int64 data and `key` is the string. The numpy searchsorted will then cast under the hood the int values to string or object (whather the common dtype between values and key), which is slow. The value that then comes out of it, in case of a string, is outside the range (just added to the right of the values) and we check this and raise a KeyError. While in case of a string we just never should have tried.
- When I would have used `int(1e6) - 1` instead of `int(1e6)` values, we are below the size threshold, and it takes a different code path. In that case the situation is better (we still do unneeded tries of `get_loc`, but they are not that slow in that case)
Correctness question, is this the right output?

```python
In [5]: pd.date_range('2017', periods=12)
Out[5]:
DatetimeIndex(['2017-01-01', '2017-01-02', '2017-01-03', '2017-01-04',
               '2017-01-05', '2017-01-06', '2017-01-07', '2017-01-08',
               '2017-01-09', '2017-01-10', '2017-01-11', '2017-01-12'],
              dtype='datetime64[ns]', freq='D')

In [6]: pd.date_range('2017', periods=12).get_loc("2017")
Out[6]: slice(0, 12, None)
```

The docstring says in int is returned. The closes analog I can think of is `IntervalIndex` w/ overlapping intervals.

```python
In [12]: pd.IntervalIndex.from_tuples([(0, 1), (0, 2), (0, 3)]).get_loc(0.5)
Out[12]: array([0, 1, 2])
```
Second question, should `DatetimeIndex.get_loc(int)` ever work? We test that it does.

```python
In [11]: s.index.get_loc(s.index[0].value)
Out[11]: 0
```

That's taking the underlying integer representation. cc @jreback
Not really sure, should dive into the code again.

But, to fix the actual regression, I think the easier path will be to avoid any `get_loc` call that is coming from `__getattr__` altogether, as I mentioned in https://github.com/pandas-dev/pandas/issues/19764#issuecomment-366760838 (not that `get_loc` shouldn't be fixed)
Indeed, I got curious though :) Perhaps I should keep performance questions in #17754.

I'll investigate the getattr stuff now (for 0.23).
So currently, `Series/DataFrame.__getattr__` does a `if name in self._info_axis`, which then calls `Index.__contains__`, which does the engine lookup.

So a quick dirty "hack" would be to add a `_contains_getattr` method to the base `Index` class which is equal to `__contains__`, and then `DatetimeLikeIndex` can override this to just return False (as a valid attribute name can never be contained in the index). 

Is that too dirty of a hack?

Resolving patch:
diff --git a/asv_bench/benchmarks/series_methods.py b/asv_bench/benchmarks/series_methods.py
--- a/asv_bench/benchmarks/series_methods.py
+++ b/asv_bench/benchmarks/series_methods.py
@@ -121,3 +121,16 @@ def setup(self):
 
     def time_dir_strings(self):
         dir(self.s)
+
+
+class SeriesGetattr(object):
+    # https://github.com/pandas-dev/pandas/issues/19764
+    goal_time = 0.2
+
+    def setup(self):
+        self.s = Series(1,
+                        index=date_range("2012-01-01", freq='s',
+                                         periods=int(1e6)))
+
+    def time_series_datetimeindex_repr(self):
+        getattr(self.s, 'a', None)
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -4375,7 +4375,7 @@ def __getattr__(self, name):
                 name in self._accessors):
             return object.__getattribute__(self, name)
         else:
-            if name in self._info_axis:
+            if self._info_axis._can_hold_identifiers_and_holds_name(name):
                 return self[name]
             return object.__getattribute__(self, name)
 
diff --git a/pandas/core/indexes/base.py b/pandas/core/indexes/base.py
--- a/pandas/core/indexes/base.py
+++ b/pandas/core/indexes/base.py
@@ -2083,6 +2083,19 @@ def __getitem__(self, key):
         else:
             return result
 
+    def _can_hold_identifiers_and_holds_name(self, name):
+        """
+        Faster check for ``name in self`` when we know `name` is a Python
+        identifier (e.g. in NDFrame.__getattr__, which hits this to support
+        . key lookup). For indexes that can't hold identifiers (everything
+        but object & categorical) we just return False.
+
+        https://github.com/pandas-dev/pandas/issues/19764
+        """
+        if self.is_object() or self.is_categorical():
+            return name in self
+        return False
+
     def append(self, other):
         """
         Append a collection of Index options together


Test patch:
diff --git a/pandas/tests/indexes/datetimelike.py b/pandas/tests/indexes/datetimelike.py
--- a/pandas/tests/indexes/datetimelike.py
+++ b/pandas/tests/indexes/datetimelike.py
@@ -8,6 +8,11 @@
 
 class DatetimeLike(Base):
 
+    def test_can_hold_identifiers(self):
+        idx = self.create_index()
+        key = idx[0]
+        assert idx._can_hold_identifiers_and_holds_name(key) is False
+
     def test_shift_identity(self):
 
         idx = self.create_index()
diff --git a/pandas/tests/indexes/test_base.py b/pandas/tests/indexes/test_base.py
--- a/pandas/tests/indexes/test_base.py
+++ b/pandas/tests/indexes/test_base.py
@@ -66,6 +66,11 @@ def generate_index_types(self, skip_index_keys=[]):
             if key not in skip_index_keys:
                 yield key, idx
 
+    def test_can_hold_identifiers(self):
+        idx = self.create_index()
+        key = idx[0]
+        assert idx._can_hold_identifiers_and_holds_name(key) is True
+
     def test_new_axis(self):
         new_index = self.dateIndex[None, :]
         assert new_index.ndim == 2
diff --git a/pandas/tests/indexes/test_category.py b/pandas/tests/indexes/test_category.py
--- a/pandas/tests/indexes/test_category.py
+++ b/pandas/tests/indexes/test_category.py
@@ -33,6 +33,11 @@ def create_index(self, categories=None, ordered=False):
         return CategoricalIndex(
             list('aabbca'), categories=categories, ordered=ordered)
 
+    def test_can_hold_identifiers(self):
+        idx = self.create_index(categories=list('abcd'))
+        key = idx[0]
+        assert idx._can_hold_identifiers_and_holds_name(key) is True
+
     def test_construction(self):
 
         ci = self.create_index(categories=list('abcd'))
diff --git a/pandas/tests/indexes/test_multi.py b/pandas/tests/indexes/test_multi.py
--- a/pandas/tests/indexes/test_multi.py
+++ b/pandas/tests/indexes/test_multi.py
@@ -48,6 +48,11 @@ def setup_method(self, method):
     def create_index(self):
         return self.index
 
+    def test_can_hold_identifiers(self):
+        idx = self.create_index()
+        key = idx[0]
+        assert idx._can_hold_identifiers_and_holds_name(key) is True
+
     def test_boolean_context_compat2(self):
 
         # boolean context compat
diff --git a/pandas/tests/indexes/test_numeric.py b/pandas/tests/indexes/test_numeric.py
--- a/pandas/tests/indexes/test_numeric.py
+++ b/pandas/tests/indexes/test_numeric.py
@@ -64,6 +64,11 @@ def test_index_rdiv_timedelta(self, scalar_td, index):
 
 class Numeric(Base):
 
+    def test_can_hold_identifiers(self):
+        idx = self.create_index()
+        key = idx[0]
+        assert idx._can_hold_identifiers_and_holds_name(key) is False
+
     def test_numeric_compat(self):
         pass  # override Base method
 
diff --git a/pandas/tests/indexes/test_range.py b/pandas/tests/indexes/test_range.py
--- a/pandas/tests/indexes/test_range.py
+++ b/pandas/tests/indexes/test_range.py
@@ -44,6 +44,11 @@ def check_binop(self, ops, scalars, idxs):
                     expected = op(Int64Index(idx), scalar)
                     tm.assert_index_equal(result, expected)
 
+    def test_can_hold_identifiers(self):
+        idx = self.create_index()
+        key = idx[0]
+        assert idx._can_hold_identifiers_and_holds_name(key) is False
+
     def test_binops(self):
         ops = [operator.add, operator.sub, operator.mul, operator.floordiv,
                operator.truediv]