[WIP] perf improvements for strftime
State: closed

Changelog

 - Added the default format "%Y-%m-%d %H:%M:%S" as basic format (same as format=None), as suggested by @auderson.

- [ ] closes #44764 , closes #46252, closes #46319
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.
- [ ] Post `timeit` results showing the perf improvements

Labels:
name: Datetime
description: Datetime data dtype
name: Performance
description: Memory or execution speed performance
name: Stale

Comments:
Can you post %timeit results showing the perf improvement?  (edit OP)
Sure. I edited the todo list accordingly.
Note: there seem to be a related (but distinct) issue in #29461 , solved by #34668 . 

Still I will stick to the current road where I modify the `format_array_from_datetime` function only.
Hello @smarie! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









































There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-03-27 14:05:05 UTC
First milestone reached: all utility functions required to create the fast datetime formatting strings are there.

A `fast_strftime` util function can be used to demonstrate (and test) their performance on plain old `datetime` objects. Added relevant tests for this.

Next step: modify `format_array_from_datetime` so that it relies on the same underlying function `convert_dtformat` in order to format `npy_datetimestruct` instances.

> Are these issues all connected? In general we prefer if one pr tackels one topic, not multiple at once

Agreed, with the current diff, this PR is likely too large for someone to review in earnest.
Noted - I'll at least try to move the two small fixes for `Period` into separate PRs, before this final big one about `fast_strftime` is reviewed.
Quick status on this PR: today I reached a point where all CI checks pass (finally !!!), and therefore I was able to try reproducing the benchmark in #44764 . Results are pretty good !

![image](https://user-images.githubusercontent.com/3236794/158021597-5363b303-54ed-45c9-a6a6-f84241e7f411.png)

 - (blue) the "legacy" curve corresponds to using `strftime`. It seems even worse on my machine (windows 10) than the result reported in #44764 !
 - (orange) the "manual" curve corresponds to the same manual method as in #44764. It is as good as reported in #44764
 - (green) the "new and legacy default" represents the case where date_format=None (default format). It is as good as reported in #44764
 - (red) the "new custom" is the main use case why this PR was created: a custom date format. It was improved a lot as compared to the blue curve !
 - (violet) the "new basic formats" represents the case reported by @auderson where the custom date format is actually identical to the default format (green). We see that both now have the same perf

<details>
```python
import time
import numpy as np
import pandas as pd
from joblib import Parallel, delayed


def timer(f):
    def inner(*args, **kwargs):
        s = time.time()
        result = f(*args, **kwargs)
        e = time.time()
        return e - s
    return inner


@timer
def method_legacy(index):
    return index.strftime("%Y-%m-%d_%H:%M:%S", fast_strftime=False)


@timer
def method_manual(index):
    attrs = ('year', 'month', 'day', 'hour', 'minute', 'second')
    parts = [getattr(index, at) for at in attrs]
    b = []
    for year, month, day, hour, minute, second in zip(*parts):
        b.append(f'{year}-{month:02}-{day:02}_{hour:02}:{minute:02}:{second:02}')
    b = pd.Index(b)
    return b


@timer
def method_new_and_legacy_default(index):
    return index.strftime(None)


@timer
def method_new_custom(index):
    return index.strftime("%Y-%m-%d_%H:%M:%S")


@timer
def method_new_basic_formats(index):
    return index.strftime("%Y-%m-%d %H:%M:%S")


index = pd.date_range('2000', '2020', freq='1min')


@delayed
def profile(p):
    n = int(10 ** p)
    time_legacy = method_legacy(index[:n])
    time_manual = method_manual(index[:n])
    time_new_and_legacy_default = method_new_and_legacy_default(index[:n])
    time_new_custom = method_new_custom(index[:n])
    time_new_basic_formats = method_new_basic_formats(index[:n])
    return n, time_legacy, time_manual, time_new_and_legacy_default, \
           time_new_custom, time_new_basic_formats


records = Parallel(10, verbose=10, max_nbytes='500M')(
    profile(p) for p in np.arange(1, 7.1, 0.1)
)


pd.DataFrame(records, columns=['n', 'time_legacy', 'time_manual',
                               'time_new_and_legacy_default', 'time_new_custom',
                               'time_new_basic_formats'
                               ]).set_index(
    'n').plot(figsize=(10, 8))

import matplotlib.pyplot as plt
plt.show()
```
</details>

@smarie thins way too large a PR

pls split this up into logical pieces
@jreback yes I was going to split it as explained above, but I wanted first to reach a "clean" state where CI is happy and the benchmark shows good results. Now I will try (next week) to split this into pieces, but as already mentioned, there is not much I can do here: `strftime` was used in many places (`Period`, `Timestamp`, csv formatter, excel formatter, sqlite adapter, some `repr` methods, etc.) and all of them needed replacement. I will do my best do separate between

 - bug fixes
 - quick fixes for all repr and the like (not requiring core changes)
 - the "core" replacement for datetime arrays
 - the same for periods
 - ... maybe other similar
This is impressive😀
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
bot: please leave this open. #46759 and #46405 are the next bits of this PR to consider
Hello @smarie, it seems there is a flake8 issue (`pandas/tests/io/formats/test_format.py:3253:5: F811 redefinition of unused 'test_period_custom' from line 3201`) to fix and a conflict to solve before this can be merged. Could you do that please ? Feature is really interesting !

Thanks @Paul-AUB ! Actually this PR is just here as a reference, it is split in several pieces. https://github.com/pandas-dev/pandas/pull/46759 and https://github.com/pandas-dev/pandas/pull/46405 are the next bits of this PR to consider, the first is actually quite close to be merged. The second has a side effect in the CI concerning some locale-dependent setting, that needs to be investigated. Once #46405 is merged, I'll finally update this PR or create a new one with the same contents, so that it contains the remaining bit (the engine able to improve any strftime format)
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
Hi @mroeschke, yes I plan to finish this now that most of its "other aspects" have been merged (through other PRs #46361, #46759, #46405, #47570). I'll try to find some time later this month