PERF: restore DatetimeIndex.__iter__ performance by using non-EA implementation
State: closed

The `timeseries.TimeIteration.time_iter` benchmark shows a `6.4x` regression here: https://qwhelan.github.io/pandas/#timeseries.Iteration.time_iter?p-time_index=%3Cfunction%20date_range%3E&commits=08395af4-fc24c2cd . An iteration of `asv find` blames https://github.com/pandas-dev/pandas/commit/1fc76b80#diff-26a6d2ca7adfca586aabbb1c9dd8bf36R74 . I believe what is happening is:
- `pandas.core.indexes.datetimelike.ea_passthrough` was modified from calling `getattr()` to being passed a classmethod directly
- The corresponding change in `DatetimeIndexOpsMixin.__init__` then implicitly changed `__iter__` from `DatetimeArray.__iter__` to `DatetimeLikeArrayMixin.__iter__`. The latter must assume an `ExtensionArray`, so is intrinsically slower.
- This regression was then hidden in `asv` due to inadvertent inclusion of memory addresses when using certain parameters. This is resolved in https://github.com/airspeed-velocity/asv/pull/771 and explains why my `asv` results shows this regression while http://pandas.pydata.org/speed/pandas/#timeseries.Iteration.time_iter? does not

Reverting back to old behavior yields a `7x` speedup:
```
$ asv compare HEAD^ HEAD --only-changed
       before           after         ratio
     [649ad5c9]       [4d0c13cc]
     <any_all_fix>       <datetime_iter>
-      3.30±0.02s          469±6ms     0.14  timeseries.Iteration.time_iter(<function date_range>)
-        34.7±1ms       11.2±0.2ms     0.32  timeseries.Iteration.time_iter_preexit(<function date_range>)
```
- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Datetime
description: Datetime data dtype
name: Performance
description: Memory or execution speed performance

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26702?src=pr&el=h1) Report
> Merging [#26702](https://codecov.io/gh/pandas-dev/pandas/pull/26702?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/649ad5c91aabb95d7f200eec84b82cae1459fc65?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26702/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26702?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26702      +/-   ##
==========================================
- Coverage   91.88%   91.87%   -0.01%     
==========================================
  Files         174      174              
  Lines       50701    50702       +1     
==========================================
- Hits        46588    46584       -4     
- Misses       4113     4118       +5
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.41% <100%> (ø)` | :arrow_up: |
| #single | `41.91% <100%> (-0.11%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26702?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexes/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/26702/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZXMucHk=) | `96.37% <100%> (ø)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26702/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `78.94% <0%> (-10.53%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/26702/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97% <0%> (-0.12%)` | :arrow_down: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/26702/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `90.6% <0%> (-0.11%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26702?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26702?src=pr&el=footer). Last update [649ad5c...4d0c13c](https://codecov.io/gh/pandas-dev/pandas/pull/26702?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26702?src=pr&el=h1) Report
> Merging [#26702](https://codecov.io/gh/pandas-dev/pandas/pull/26702?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/3ff4f38f2b4572af92946af9ce9d50481413358d?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26702/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26702?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26702      +/-   ##
==========================================
- Coverage   91.88%   91.88%   -0.01%     
==========================================
  Files         174      174              
  Lines       50702    50703       +1     
==========================================
- Hits        46590    46587       -3     
- Misses       4112     4116       +4
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.42% <100%> (ø)` | :arrow_up: |
| #single | `41.9% <100%> (-0.11%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26702?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexes/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/26702/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZXMucHk=) | `96.37% <100%> (ø)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26702/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `78.94% <0%> (-10.53%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/26702/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97% <0%> (-0.12%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26702?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26702?src=pr&el=footer). Last update [3ff4f38...a1e703f](https://codecov.io/gh/pandas-dev/pandas/pull/26702?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

LGTM.  Only question is if in principle we should be wrapping `self._data.__iter__`, but I don't think its worth the hassle.
@TomAugspurger Done
Thanks. In theory, a similar chunking trick could be done for PeriodArray & TimedeltaArray, right?  The only thing to change would be the function doing the conversion in

```python
            converted = tslib.ints_to_pydatetime(data[start_i:end_i],
                                                 tz=self.tz, freq=self.freq,
                                                 box="timestamp")
 ```

Happy to leave that as a followup.
thanks @qwhelan 
@jreback Thanks for merging!

@TomAugspurger Yes, definitely a possibility. There's probably mid-single-digit factor speedups available there. Unfortunately, I'm headed to Europe for vacation next week and starting a new job shortly thereafter so can't promise to make any headway on those.
No worries, I opened https://github.com/pandas-dev/pandas/issues/26713 to
track this.

On Fri, Jun 7, 2019 at 2:21 PM Christopher Whelan <notifications@github.com>
wrote:

> @jreback <https://github.com/jreback> Thanks for merging!
>
> @TomAugspurger <https://github.com/TomAugspurger> Yes, definitely a
> possibility. There's probably mid-single-digit factor speedups available
> there. Unfortunately, I'm headed to Europe for vacation next week and
> starting a new job shortly thereafter so can't promise to make any headway
> on those.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/26702?email_source=notifications&email_token=AAKAOITHGCNLCEIOHQRJW5TPZKYMNA5CNFSM4HVT2QQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXGYLQI#issuecomment-500008385>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIRY3W43FBBZU7WWP3LPZKYMNANCNFSM4HVT2QQQ>
> .
>