[READY] perf improvements for strftime
State: open

This PR is a new clean version of #46116

- [x] closes #44764
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.




Labels:
name: Datetime
description: Datetime data dtype
name: Performance
description: Memory or execution speed performance
name: Stale

Comments:
This PR is still pretty big. Any reason why you are introducing a new `fast_strftime` keyword instead of just trying to improve performance inplace? I think that would help to reduce the size, though still probably need to break up in smaller subsets. The bigger a PR is, the harder it is to review so ends up in a long review cycle
Thanks @WillAyd for the comment ! I filled the what's new doc to help with the review process.
I will also try to move part of the mods in a separate PR, I'll tell you when this is done
@WillAyd  I split this PR in two: first part is #51459 , and once merged this PR will cover the rest
@WillAyd would you be able to review #51459 ? That would help remove some code from this PR
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
> This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.

Please leave this PR open, we are waiting for https://github.com/pandas-dev/pandas/pull/51459 to be merged to reduce the volume of changes, but otherwise this is ready 
This is now ready for review @MarcoGorelli or @mroeschke or @WillAyd .

The PR is still pretty big, but it is a single feature. I moved everything I could to other PRs (#51459, #53003, #46405, #47570, #46759, #46361) since original PR #46116, ... 15 months ago :)

The only items I can think of removing to make this (marginally) smaller would be :

- these two test files : pandas/tests/arrays/categorical/test_repr.py and pandas/tests/io/formats/test_to_csv.py . Indeed the tests do not contain the fast_strftime parametrization.
- the ASVs

But it would not really reduce much the volume of the PR, and these items are also clearly related to the PR ; so not sure it is worth spending the time doing the splitting.

I propose to add the `fast_strftime` parameter everywhere mostly as a safety belt, 

- even if the tests are now covering all the aspects we can think of 
- even if there is an automatic fallback to the c-strftime when the formatting pattern can not be converted (`UnsupportedStrFmtDirective` caught in calls to `convert_strftime_format`).

Indeed, in spite of all the care taken in writing this PR, if there is a problem, it will be much easier to indicate a workaround (`fast_strftime=False`) than a version downgrade (use `pandas<2.1`) in my opinion.

EDIT: below the ASV results. All improvements are in line with expectations, huge perf gains ! The only item I need to check again is `ToCSVDatetimeIndex.time_frame_date_formatting_index_custom`, ~~I suspect that `date_format` is not taken into account hence the lack of any improvement.~~

EDIT2: I found the cause of bad perf for `ToCSVDatetimeIndex.time_frame_date_formatting_index_custom` and fixed it. I updated the ASVs below. ~~I will propose this fix in an independent PR as it is quite straightforward.~~ Since this fix does not provide any improvement without the rest of this PR (I tested it on HEAD), I leave it in this PR

> asv compare 935244a9 f9e3b5f9

<details>

| Change   | Before [935244a9] | After [537b9f73] | Ratio   | Benchmark (Parameter)                                                                             |
|----------|-------------------|------------------|---------|---------------------------------------------------------------------------------------------------|
| -        | 13.9±0.2ms     | 3.79±0.1ms    | 0.27    | io.csv.ToCSVDatetime.time_frame_date_formatting_custom                                            |
|          | 2.93±0.1ms     | 2.96±0.07ms   | 1.01    | io.csv.ToCSVDatetime.time_frame_date_formatting_default                                           |
| -        | 9.23±0.3ms     | 3.65±1ms      | 0.40    | io.csv.ToCSVDatetime.time_frame_date_formatting_default_explicit                                  |
| -        | 18.3±0.3ms     | 8.00±0.3ms    | 0.44    | io.csv.ToCSVDatetimeBig.time_frame_date_formatting_custom(1000)                                   |
| -        | 167±7ms        | 62.2±1ms      | 0.37    | io.csv.ToCSVDatetimeBig.time_frame_date_formatting_custom(10000)                                  |
| -        | 1.64±0.02s     | 611±8ms       | 0.37    | io.csv.ToCSVDatetimeBig.time_frame_date_formatting_custom(100000)                                 |
|          | 6.08±0.3ms     | 6.19±0.2ms    | 1.02    | io.csv.ToCSVDatetimeBig.time_frame_formatting_default(1000)                                       |
|          | 46.8±1ms       | 47.3±0.4ms    | 1.01    | io.csv.ToCSVDatetimeBig.time_frame_formatting_default(10000)                                      |
|          | 467±6ms        | 469±9ms       | 1.00    | io.csv.ToCSVDatetimeBig.time_frame_formatting_default(100000)                                     |
| -        | 867±20ms       | 251±20ms      | 0.29    | io.csv.ToCSVDatetimeIndex.time_frame_date_formatting_index_custom                                 |
|          | 189±0.7ms      | 198±7ms       | 1.05    | io.csv.ToCSVDatetimeIndex.time_frame_date_formatting_index_default                                |
| -        | 841±20ms       | 192±1ms       | 0.23    | io.csv.ToCSVDatetimeIndex.time_frame_date_formatting_index_default_explicit                       |
|          | 3.92±0.07ms    | 3.94±0.2ms    | 1.01    | io.csv.ToCSVPeriod.time_frame_period_formatting_custom(1000, 'D')                                 |
|          | 4.20±0.06ms    | 4.44±0.07ms   | 1.06    | io.csv.ToCSVPeriod.time_frame_period_formatting_custom(1000, 'H')                                 |
|          | 29.3±0.3ms     | 30.9±0.6ms    | 1.05    | io.csv.ToCSVPeriod.time_frame_period_formatting_custom(10000, 'D')                                |
|          | 35.0±0.9ms     | 33.4±0.2ms    | 0.95    | io.csv.ToCSVPeriod.time_frame_period_formatting_custom(10000, 'H')                                |
|          | 3.80±0.06ms    | 3.90±0.09ms   | 1.03    | io.csv.ToCSVPeriod.time_frame_period_formatting_default(1000, 'D')                                |
|          | 4.16±0.1ms     | 4.27±0.1ms    | 1.03    | io.csv.ToCSVPeriod.time_frame_period_formatting_default(1000, 'H')                                |
|          | 30.2±1ms       | 30.2±0.4ms    | 1.00    | io.csv.ToCSVPeriod.time_frame_period_formatting_default(10000, 'D')                               |
|          | 32.1±0.4ms     | 32.7±0.2ms    | 1.02    | io.csv.ToCSVPeriod.time_frame_period_formatting_default(10000, 'H')                               |
|          | 3.80±0.04ms    | 3.70±0.05ms   | 0.97    | io.csv.ToCSVPeriod.time_frame_period_formatting_default_explicit(1000, 'D')                       |
|          | 4.19±0.2ms     | 4.17±0.1ms    | 1.00    | io.csv.ToCSVPeriod.time_frame_period_formatting_default_explicit(1000, 'H')                       |
|          | 29.1±0.5ms     | 29.6±0.3ms    | 1.02    | io.csv.ToCSVPeriod.time_frame_period_formatting_default_explicit(10000, 'D')                      |
|          | 34.1±1ms       | 33.0±1ms      | 0.97    | io.csv.ToCSVPeriod.time_frame_period_formatting_default_explicit(10000, 'H')                      |
| -        | 10.2±0.2ms     | 3.81±0.08ms   | 0.37    | io.csv.ToCSVPeriodIndex.time_frame_period_formatting_index_custom(1000, 'D')                      |
| -        | 9.89±0.2ms     | 4.12±0.2ms    | 0.42    | io.csv.ToCSVPeriodIndex.time_frame_period_formatting_index_custom(1000, 'H')                      |
| -        | 86.8±4ms       | 27.9±0.2ms    | 0.32    | io.csv.ToCSVPeriodIndex.time_frame_period_formatting_index_custom(10000, 'D')                     |
| -        | 88.8±1ms       | 28.5±0.5ms    | 0.32    | io.csv.ToCSVPeriodIndex.time_frame_period_formatting_index_custom(10000, 'H')                     |
|          | 2.12±0.03ms    | 2.07±0.04ms   | 0.97    | io.csv.ToCSVPeriodIndex.time_frame_period_formatting_index_default(1000, 'D')                     |
|          | 2.46±0.06ms    | 2.40±0.03ms   | 0.98    | io.csv.ToCSVPeriodIndex.time_frame_period_formatting_index_default(1000, 'H')                     |
|          | 11.9±0.3ms     | 12.0±0.1ms    | 1.01    | io.csv.ToCSVPeriodIndex.time_frame_period_formatting_index_default(10000, 'D')                    |
|          | 15.8±0.7ms     | 15.6±0.07ms   | 0.99    | io.csv.ToCSVPeriodIndex.time_frame_period_formatting_index_default(10000, 'H')                    |
| -        | 4.35±0.06ms    | 2.24±0.04ms   | 0.51    | io.csv.ToCSVPeriodIndex.time_frame_period_formatting_index_default_explicit(1000, 'D')            |
| -        | 5.22±0.2ms     | 2.60±0.02ms   | 0.50    | io.csv.ToCSVPeriodIndex.time_frame_period_formatting_index_default_explicit(1000, 'H')            |
| -        | 34.9±3ms       | 12.6±0.1ms    | 0.36    | io.csv.ToCSVPeriodIndex.time_frame_period_formatting_index_default_explicit(10000, 'D')           |
| -        | 37.8±0.6ms     | 16.5±0.8ms    | 0.44    | io.csv.ToCSVPeriodIndex.time_frame_period_formatting_index_default_explicit(10000, 'H')           |
|          | 312±10ms          | 313±3ms          | 1.00    | io.json.ToJSONISO.time_iso_format('columns')                                                      |
|          | 341±20ms          | 339±6ms          | 0.99    | io.json.ToJSONISO.time_iso_format('index')                                                        |
|          | 294±10ms          | 314±30ms         | 1.07    | io.json.ToJSONISO.time_iso_format('records')                                                      |
|          | 341±8ms           | 326±4ms          | 0.96    | io.json.ToJSONISO.time_iso_format('split')                                                        |
|          | 294±10ms          | 287±9ms          | 0.98    | io.json.ToJSONISO.time_iso_format('values')                                                       |
|          | 2.43±0.07ms       | 2.59±0.06ms      | 1.07    | strftime.BusinessHourStrftime.time_frame_offset_repr(1000)                                        |
|          | 24.4±2ms          | 24.0±0.3ms       | 0.98    | strftime.BusinessHourStrftime.time_frame_offset_repr(10000)                                       |
|          | 2.79±0.2ms        | 2.66±0.1ms       | 0.95    | strftime.BusinessHourStrftime.time_frame_offset_str(1000)                                         |
|          | 25.5±2ms          | 23.9±0.4ms       | 0.94    | strftime.BusinessHourStrftime.time_frame_offset_str(10000)                                        |
| -        | 7.06±2ms          | 1.20±0.03ms      | 0.17    | strftime.DatetimeStrftime.time_frame_date_formatting_custom(1000, False)                          |
| -        | 7.68±0.4ms        | 2.54±0.03ms      | 0.33    | strftime.DatetimeStrftime.time_frame_date_formatting_custom(1000, True)                           |
| -        | 77.1±5ms          | 10.1±0.2ms       | 0.13    | strftime.DatetimeStrftime.time_frame_date_formatting_custom(10000, False)                         |
| -        | 81.8±5ms          | 23.2±1ms         | 0.28    | strftime.DatetimeStrftime.time_frame_date_formatting_custom(10000, True)                          |
|          | 755±4μs           | 759±20μs         | 1.01    | strftime.DatetimeStrftime.time_frame_date_formatting_default(1000, False)                         |
|          | 4.53±0.5ms        | 4.65±0.09ms      | 1.03    | strftime.DatetimeStrftime.time_frame_date_formatting_default(1000, True)                          |
|          | 6.55±0.7ms        | 5.97±0.3ms       | 0.91    | strftime.DatetimeStrftime.time_frame_date_formatting_default(10000, False)                        |
|          | 42.2±0.8ms        | 46.6±1ms         | ~1.11   | strftime.DatetimeStrftime.time_frame_date_formatting_default(10000, True)                         |
| -        | 6.72±0.3ms        | 1.06±0.06ms      | 0.16    | strftime.DatetimeStrftime.time_frame_date_formatting_index_custom(1000, False)                    |
| -        | 8.39±2ms          | 2.51±0.3ms       | 0.30    | strftime.DatetimeStrftime.time_frame_date_formatting_index_custom(1000, True)                     |
| -        | 67.5±1ms          | 9.85±0.09ms      | 0.15    | strftime.DatetimeStrftime.time_frame_date_formatting_index_custom(10000, False)                   |
| -        | 75.3±1ms          | 24.6±2ms         | 0.33    | strftime.DatetimeStrftime.time_frame_date_formatting_index_custom(10000, True)                    |
|          | 1.31±0.05ms       | 1.33±0.04ms      | 1.02    | strftime.DatetimeStrftime.time_frame_date_formatting_index_default(1000, False)                   |
|          | 5.15±0.2ms        | 4.24±0.1ms       | ~0.82   | strftime.DatetimeStrftime.time_frame_date_formatting_index_default(1000, True)                    |
|          | 21.6±10ms         | 12.4±0.3ms       | ~0.58   | strftime.DatetimeStrftime.time_frame_date_formatting_index_default(10000, False)                  |
|          | 41.9±0.5ms        | 44.2±1ms         | 1.06    | strftime.DatetimeStrftime.time_frame_date_formatting_index_default(10000, True)                   |
|          | 644±10μs          | 682±40μs         | 1.06    | strftime.DatetimeStrftime.time_frame_date_to_str(1000, False)                                     |
|          | 4.97±0.3ms        | 4.61±0.2ms       | 0.93    | strftime.DatetimeStrftime.time_frame_date_to_str(1000, True)                                      |
|          | 8.12±1ms          | 6.06±0.2ms       | ~0.75   | strftime.DatetimeStrftime.time_frame_date_to_str(10000, False)                                    |
|          | 48.5±4ms          | 46.4±5ms         | 0.96    | strftime.DatetimeStrftime.time_frame_date_to_str(10000, True)                                     |
| -        | 8.25±0.05ms       | 1.90±0.08ms      | 0.23    | strftime.DatetimeStrftime.time_frame_datetime_formatting_custom(1000, False)                      |
| -        | 11.7±1ms          | 3.31±0.08ms      | 0.28    | strftime.DatetimeStrftime.time_frame_datetime_formatting_custom(1000, True)                       |
| -        | 80.8±7ms          | 17.3±0.4ms       | 0.21    | strftime.DatetimeStrftime.time_frame_datetime_formatting_custom(10000, False)                     |
| -        | 104±3ms           | 30.5±0.8ms       | 0.29    | strftime.DatetimeStrftime.time_frame_datetime_formatting_custom(10000, True)                      |
|          | 1.72±0.2ms        | 1.44±0.03ms      | ~0.84   | strftime.DatetimeStrftime.time_frame_datetime_formatting_default(1000, False)                     |
|          | 4.38±0.1ms        | 4.89±0.3ms       | ~1.12   | strftime.DatetimeStrftime.time_frame_datetime_formatting_default(1000, True)                      |
|          | 12.4±0.1ms        | 12.3±0.09ms      | 0.99    | strftime.DatetimeStrftime.time_frame_datetime_formatting_default(10000, False)                    |
|          | 56.0±20ms         | 45.0±2ms         | ~0.80   | strftime.DatetimeStrftime.time_frame_datetime_formatting_default(10000, True)                     |
|          | 1.37±0.03ms       | 1.29±0.01ms      | 0.94    | strftime.DatetimeStrftime.time_frame_datetime_formatting_default_explicit(1000, False)            |
| -        | 8.16±0.06ms       | 3.32±0.1ms       | 0.41    | strftime.DatetimeStrftime.time_frame_datetime_formatting_default_explicit(1000, True)             |
|          | 11.7±0.4ms        | 11.9±0.3ms       | 1.02    | strftime.DatetimeStrftime.time_frame_datetime_formatting_default_explicit(10000, False)           |
| -        | 96.4±7ms          | 33.4±3ms         | 0.35    | strftime.DatetimeStrftime.time_frame_datetime_formatting_default_explicit(10000, True)            |
|          | 871±100μs         | 783±50μs         | ~0.90   | strftime.DatetimeStrftime.time_frame_datetime_formatting_default_explicit_date_only(1000, False)  |
| -        | 7.01±0.06ms       | 2.52±0.09ms      | 0.36    | strftime.DatetimeStrftime.time_frame_datetime_formatting_default_explicit_date_only(1000, True)   |
|          | 6.28±0.2ms        | 6.27±0.2ms       | 1.00    | strftime.DatetimeStrftime.time_frame_datetime_formatting_default_explicit_date_only(10000, False) |
| -        | 78.1±10ms         | 24.1±0.4ms       | 0.31    | strftime.DatetimeStrftime.time_frame_datetime_formatting_default_explicit_date_only(10000, True)  |
|          | 1.60±0.01ms       | 1.59±0.01ms      | 0.99    | strftime.DatetimeStrftime.time_frame_datetime_formatting_default_with_float(1000, False)          |
| -        | 10.0±0.3ms        | 3.25±0.04ms      | 0.32    | strftime.DatetimeStrftime.time_frame_datetime_formatting_default_with_float(1000, True)           |
|          | 14.7±0.1ms        | 14.9±0.7ms       | 1.02    | strftime.DatetimeStrftime.time_frame_datetime_formatting_default_with_float(10000, False)         |
| -        | 102±5ms           | 32.4±6ms         | 0.32    | strftime.DatetimeStrftime.time_frame_datetime_formatting_default_with_float(10000, True)          |
| -        | 9.22±0.8ms        | 1.98±0.05ms      | 0.21    | strftime.DatetimeStrftime.time_frame_datetime_formatting_index_custom(1000, False)                |
| -        | 11.1±3ms          | 3.21±0.3ms       | 0.29    | strftime.DatetimeStrftime.time_frame_datetime_formatting_index_custom(1000, True)                 |
| -        | 88.8±30ms         | 16.4±0.1ms       | 0.18    | strftime.DatetimeStrftime.time_frame_datetime_formatting_index_custom(10000, False)               |
| -        | 88.9±4ms          | 31.6±3ms         | 0.36    | strftime.DatetimeStrftime.time_frame_datetime_formatting_index_custom(10000, True)                |
|          | 1.53±0.01ms       | 1.50±0.02ms      | 0.98    | strftime.DatetimeStrftime.time_frame_datetime_formatting_index_default(1000, False)               |
|          | 4.52±0.2ms        | 4.47±0.1ms       | 0.99    | strftime.DatetimeStrftime.time_frame_datetime_formatting_index_default(1000, True)                |
|          | 12.4±0.1ms        | 13.1±0.6ms       | 1.06    | strftime.DatetimeStrftime.time_frame_datetime_formatting_index_default(10000, False)              |
|          | 53.3±6ms          | 43.8±2ms         | ~0.82   | strftime.DatetimeStrftime.time_frame_datetime_formatting_index_default(10000, True)               |
|          | 3.26±0.1ms        | 2.60±0.07ms      | ~0.80   | strftime.DatetimeStrftime.time_frame_datetime_formatting_iso8601_map(1000, False)                 |
|          | 3.63±0.07ms       | 3.59±0.1ms       | 0.99    | strftime.DatetimeStrftime.time_frame_datetime_formatting_iso8601_map(1000, True)                  |
|          | 25.0±0.1ms        | 25.4±3ms         | 1.01    | strftime.DatetimeStrftime.time_frame_datetime_formatting_iso8601_map(10000, False)                |
|          | 32.7±0.4ms        | 33.5±1ms         | 1.02    | strftime.DatetimeStrftime.time_frame_datetime_formatting_iso8601_map(10000, True)                 |
| -        | 7.94±0.3ms        | 2.02±0.2ms       | 0.25    | strftime.DatetimeStrftime.time_frame_datetime_formatting_iso8601_strftime_Z(1000, False)          |
| -        | 9.80±1ms          | 3.35±0.2ms       | 0.34    | strftime.DatetimeStrftime.time_frame_datetime_formatting_iso8601_strftime_Z(1000, True)           |
| -        | 83.9±8ms          | 16.0±0.3ms       | 0.19    | strftime.DatetimeStrftime.time_frame_datetime_formatting_iso8601_strftime_Z(10000, False)         |
| -        | 83.6±2ms          | 28.4±2ms         | 0.34    | strftime.DatetimeStrftime.time_frame_datetime_formatting_iso8601_strftime_Z(10000, True)          |
|          | 7.74±0.5ms        | 7.58±0.09ms      | 0.98    | strftime.DatetimeStrftime.time_frame_datetime_formatting_iso8601_strftime_offset(1000, False)     |
|          | 10.00±0.2ms       | 9.96±0.2ms       | 1.00    | strftime.DatetimeStrftime.time_frame_datetime_formatting_iso8601_strftime_offset(1000, True)      |
|          | 74.1±1ms          | 71.8±0.4ms       | 0.97    | strftime.DatetimeStrftime.time_frame_datetime_formatting_iso8601_strftime_offset(10000, False)    |
|          | 100±9ms           | 96.0±1ms         | 0.96    | strftime.DatetimeStrftime.time_frame_datetime_formatting_iso8601_strftime_offset(10000, True)     |
|          | 1.80±0.3ms        | 1.31±0.01ms      | ~0.73   | strftime.DatetimeStrftime.time_frame_datetime_to_str(1000, False)                                 |
|          | 8.70±4ms          | 4.89±0.5ms       | ~0.56   | strftime.DatetimeStrftime.time_frame_datetime_to_str(1000, True)                                  |
|          | 12.2±0.2ms        | 12.3±0.3ms       | 1.01    | strftime.DatetimeStrftime.time_frame_datetime_to_str(10000, False)                                |
|          | 41.7±2ms          | 46.4±5ms         | ~1.11   | strftime.DatetimeStrftime.time_frame_datetime_to_str(10000, True)                                 |
| -        | 7.09±1ms          | 2.03±0.04ms      | 0.29    | strftime.PeriodStrftime.time_frame_period_formatting_custom(1000, 'D')                            |
| -        | 6.62±0.8ms        | 2.39±0.3ms       | 0.36    | strftime.PeriodStrftime.time_frame_period_formatting_custom(1000, 'H')                            |
| -        | 54.8±0.6ms        | 20.7±3ms         | 0.38    | strftime.PeriodStrftime.time_frame_period_formatting_custom(10000, 'D')                           |
| -        | 56.4±0.9ms        | 20.5±1ms         | 0.36    | strftime.PeriodStrftime.time_frame_period_formatting_custom(10000, 'H')                           |
|          | 715±20μs          | 711±20μs         | 0.99    | strftime.PeriodStrftime.time_frame_period_formatting_default(1000, 'D')                           |
|          | 1.56±0.2ms        | 1.22±0.7ms       | ~0.79   | strftime.PeriodStrftime.time_frame_period_formatting_default(1000, 'H')                           |
|          | 6.42±0.2ms        | 6.38±0.2ms       | 0.99    | strftime.PeriodStrftime.time_frame_period_formatting_default(10000, 'D')                          |
|          | 9.20±0.2ms        | 8.66±0.2ms       | 0.94    | strftime.PeriodStrftime.time_frame_period_formatting_default(10000, 'H')                          |
|          | 1.25±0.4ms        | 723±20μs         | ~0.58   | strftime.PeriodStrftime.time_frame_period_formatting_default_explicit(1000, 'D')                  |
|          | 986±40μs          | 1.07±0.05ms      | 1.09    | strftime.PeriodStrftime.time_frame_period_formatting_default_explicit(1000, 'H')                  |
|          | 8.51±0.8ms        | 6.57±0.2ms       | ~0.77   | strftime.PeriodStrftime.time_frame_period_formatting_default_explicit(10000, 'D')                 |
| -        | 10.6±1ms          | 8.56±0.2ms       | 0.81    | strftime.PeriodStrftime.time_frame_period_formatting_default_explicit(10000, 'H')                 |
|          | 709±100μs         | 632±8μs          | ~0.89   | strftime.PeriodStrftime.time_frame_period_formatting_index_default(1000, 'D')                     |
|          | 916±20μs          | 863±30μs         | 0.94    | strftime.PeriodStrftime.time_frame_period_formatting_index_default(1000, 'H')                     |
|          | 7.02±2ms          | 6.30±0.06ms      | ~0.90   | strftime.PeriodStrftime.time_frame_period_formatting_index_default(10000, 'D')                    |
|          | 9.35±0.2ms        | 8.84±0.2ms       | 0.95    | strftime.PeriodStrftime.time_frame_period_formatting_index_default(10000, 'H')                    |
|          | 636±20μs          | 671±30μs         | 1.06    | strftime.PeriodStrftime.time_frame_period_formatting_index_default_explicit(1000, 'D')            |
|          | 875±10μs          | 892±50μs         | 1.02    | strftime.PeriodStrftime.time_frame_period_formatting_index_default_explicit(1000, 'H')            |
|          | 9.19±4ms          | 7.66±0.5ms       | ~0.83   | strftime.PeriodStrftime.time_frame_period_formatting_index_default_explicit(10000, 'D')           |
|          | 9.69±2ms          | 8.76±0.2ms       | ~0.90   | strftime.PeriodStrftime.time_frame_period_formatting_index_default_explicit(10000, 'H')           |
| -        | 5.23±0.1ms        | 2.25±0.1ms       | 0.43    | strftime.PeriodStrftime.time_frame_period_formatting_iso8601_strftime_Z(1000, 'D')                |
| -        | 6.20±0.8ms        | 2.16±0.06ms      | 0.35    | strftime.PeriodStrftime.time_frame_period_formatting_iso8601_strftime_Z(1000, 'H')                |
| -        | 52.2±3ms          | 19.1±0.7ms       | 0.37    | strftime.PeriodStrftime.time_frame_period_formatting_iso8601_strftime_Z(10000, 'D')               |
| -        | 53.7±1ms          | 19.8±0.2ms       | 0.37    | strftime.PeriodStrftime.time_frame_period_formatting_iso8601_strftime_Z(10000, 'H')               |
| -        | 5.41±0.3ms        | 2.95±0.02ms      | 0.54    | strftime.PeriodStrftime.time_frame_period_formatting_iso8601_strftime_offset(1000, 'D')           |
| -        | 7.03±1ms          | 3.40±0.6ms       | 0.48    | strftime.PeriodStrftime.time_frame_period_formatting_iso8601_strftime_offset(1000, 'H')           |
|          | 58.0±8ms          | 32.3±20ms        | ~0.56   | strftime.PeriodStrftime.time_frame_period_formatting_iso8601_strftime_offset(10000, 'D')          |
|          | 56.2±1ms          | 33.5±10ms        | ~0.60   | strftime.PeriodStrftime.time_frame_period_formatting_iso8601_strftime_offset(10000, 'H')          |
|          | 2.16±0.1ms        | 2.38±2ms         | ~1.10   | strftime.PeriodStrftime.time_frame_period_repr(1000, 'D')                                         |
|          | 2.35±0.04ms       | 3.26±1ms         | ~1.39   | strftime.PeriodStrftime.time_frame_period_repr(1000, 'H')                                         |
|          | 25.3±6ms          | 21.9±4ms         | ~0.86   | strftime.PeriodStrftime.time_frame_period_repr(10000, 'D')                                        |
|          | 23.5±2ms          | 24.9±0.3ms       | 1.06    | strftime.PeriodStrftime.time_frame_period_repr(10000, 'H')                                        |
|          | 2.59±1ms          | 1.94±0.05ms      | ~0.75   | strftime.PeriodStrftime.time_frame_period_str(1000, 'D')                                          |
|          | 2.88±0.9ms        | 2.06±0.05ms      | ~0.72   | strftime.PeriodStrftime.time_frame_period_str(1000, 'H')                                          |
|          | 18.1±1ms          | 20.1±1ms         | ~1.11   | strftime.PeriodStrftime.time_frame_period_str(10000, 'D')                                         |
|          | 21.1±1ms          | 19.1±2ms         | ~0.91   | strftime.PeriodStrftime.time_frame_period_str(10000, 'H')                                         |
|          | 779±100μs         | 674±10μs         | ~0.87   | strftime.PeriodStrftime.time_frame_period_to_str(1000, 'D')                                       |
|          | 923±20μs          | 903±5μs          | 0.98    | strftime.PeriodStrftime.time_frame_period_to_str(1000, 'H')                                       |
|          | 8.13±0.3ms        | 7.39±1ms         | ~0.91   | strftime.PeriodStrftime.time_frame_period_to_str(10000, 'D')                                      |
|          | 9.54±0.5ms        | 9.68±0.9ms       | 1.01    | strftime.PeriodStrftime.time_frame_period_to_str(10000, 'H')                                      |

</details>
thanks for updating

I'll take a look when I get a chance - perf improvements generally aren't highest-priority (compared with regressions / bugs / API changes), but this would be nice to have, the improvement is quite impressive. Just saying this to manage expectations, it may take some time to get this in
Generally speaking, do you know if the `strftime` performance is ever a bottleneck for anyone? It would be good improve it, but if I'm reading the results correctly, it doesn't look critical?

Not totally sure about having a separate option though, would rather not introduce complexity. Is there any case when you can envisage this failing?
> Generally speaking, do you know if the `strftime` performance is ever a bottleneck for anyone? It would be good improve it, but if I'm reading the results correctly, it doesn't look critical?

Well, it is actually quite critical for industrial applications. Actually, the very reason why I opened this thread of PRs more than one year ago was that our cloud applications in production rely on pydantic + pandas to expose dataframes as payloads in web services, and we identified that our main bottleneck in perfs was indeed pandas strftime, used to write the output payloads of the web services.

A significant speed improvement (see actual graphics below) will definitely help

![image](https://user-images.githubusercontent.com/3236794/158021597-5363b303-54ed-45c9-a6a6-f84241e7f411.png)

_Originally posted by @smarie in https://github.com/pandas-dev/pandas/issues/46116#issuecomment-1065894261_
makes sense, thanks for explaining!
> Is there any case when you can envisage this failing?

No, I think that I found most edge cases already. The mechanism is based on two items:

 - the `convert_strftime_format` template converter, that moslty contains string "is in" and "replace" statements.  https://github.com/pandas-dev/pandas/pull/51298/files#diff-fd90258af6c8946c90f0af064c63f1a5941102ba948e798c2c5b248308197219R133
 - the python %-formatting mechanisms with keywords https://github.com/pandas-dev/pandas/pull/51298/files#diff-81022a29ff44a1036a4aca0409893e34a78526bf7b5a016171fa9f4bd743bc83R1484 https://github.com/pandas-dev/pandas/pull/51298/files#diff-6e86989165ac69c298669127b6fefcc0807ac6d601c61677a9348b2ec46ef433R231
 
Note that unsupported directives (%z, %Z, %j, %U, %W, %c, %x and %X) lead to an error in `convert_strftime_format`, that is caught by callers and let them fall back into the current behaviour of using `strftime` for each array element (slower).

Both above functions seem simple enough to not hide unexpected behaviours. However maybe on some platforms (Pypy, embedded...) some users will want to use the c-strftime ? For some reason that I don't know ?

Thanks again for your support and careful reviews
Hello @MarcoGorelli , did you have a chance to start reviewing this ? 

Basically there is not much here : 

- new file pandas/_libs/tslibs/strftime.py providing `convert_strftime_format` and associated exception.
- Timestamp arrays (`DatetimeArray`, `DatetimeLikeArrayMixin`): `format_array_from_datetime` has a new `fast_strftime: bool = True` argument, it is propagated to `strftime`.
- Period arrays: `period_array_strftime` has a new `fast_strftime: bool = True` argument. It relies on `period_format` which was improved to support it.
- Scalars: `Timestamp` and `_Period` have a new method `fast_strftime`. The caller needs to call `convert_strftime_format` beforehand to use these so this is "advanced".
- `to_csv`: new `fast_strftime` propagated to `format_with_header`. `CSVFormatter` also integrates it
- ASVs
- Tests

If this is too much to review, maybe I can split this PR in three: 

- first only for timestamps (not periods).
- then CSV formatting
- then period and periodarrays

Would that help you ?

~~
Note that I have a minor error in CI, that I have no clue how to solve:

- `mypy` : `pandas/io/formats/csvs.py:190: error: Item "PeriodIndex" of "Union[DatetimeIndex, PeriodIndex]" has no attribute "strftime"  [union-attr]`: related to [this change](https://github.com/pandas-dev/pandas/pull/51298/files#diff-502016ad632639a35cc160a8194afc60ccaab9bdacde65f104bdd35d3c69b40cR190). For this one I am a bit surprised because `PeriodIndex` is decorated with `@inherit_names` as follows:

```python
@inherit_names(
    ["strftime", "start_time", "end_time"] + PeriodArray._field_ops,
    PeriodArray,
    wrap=True,
)
@inherit_names(["is_leap_year", "_format_native_types"], PeriodArray)
class PeriodIndex(DatetimeIndexOpsMixin):
```

could it be that `@inherit_names` is not compliant with mypy ?
~~

EDIT : the answer is "yes" :) and it was actually written in the code comments of `PeriodIndex`. I therefore fixed the issue the way it was suggested in the comment.

@MarcoGorelli I finally managed to have all green lights from CI :)
> If this is too much to review, maybe I can split this PR in three:
> 
>  * first only for timestamps (not periods).
> 
>  * then CSV formatting
> 
>  * then period and periodarrays
> 
> Would that help you ?

Gentle ping - @MarcoGorelli , just to get you opinion

hey - sorry for the delay, I was away the past two weeks

I have some priorities at the moment I'm afraid, and very limited maintenance time available - but yes, any effort to split it up would certainly help, thanks
Thanks @MarcoGorelli for taking the time to answer. Ok, I'll try to split this in 3 in the upcoming weeks.
Also if other reviewers already familiar with the previous similar PRs can help, it can maybe be a solution to accelerate review time?
 (@mroeschke , @jreback or @jbrockmendel  )
Thanks again, we'll see how this goes
sure - will try to get this into the 2.x series anyway, if not 2.1 it'll be 2.2
Just to disclose priorities - a couple of high-priority items I think really need addressing for the 2.x releases:
- reviewing #52064
- addressing #34916

If you felt like helping to address either, then the sooner we can get those in, the sooner we can move forwards with this one

Thanks for your patience and understanding generally, interactions with you have been very positive
Thanks @MarcoGorelli for the positive feedback !
If I can find some time one of these weeks I'll have a look, otherwise I'll wait until this is completed
> ```python
> idx.strftime('%-d')
> ```

Thanks @MarcoGorelli ! Indeed this is not recognized as a valid pattern. I do not know this pattern, it is not indicated in [python docs](https://docs.python.org/3/library/datetime.html#strftime-strptime-behavior) but still appears to be recognized by the C strftime. Still, I cannot find any explanation about it in the doc: https://man7.org/linux/man-pages/man3/strftime.3.html

Is this a nominal behaviour or a side-effect ?

If this is desirable (and I guess so for retrocompatibility), but we do not manage to find an exhaustive documentation of all these desirable behaviours, then I suppose that the safest fix is to replace [these lines](https://github.com/pandas-dev/pandas/pull/51298/files#diff-fd90258af6c8946c90f0af064c63f1a5941102ba948e798c2c5b248308197219R270-R271) by "if there is a remaining `%`, raise an `UnsupportedStrFmtDirective` error". This will trigger a fallback to the legacy `strftime` usage.

Still if we are able to list some of these undocumented but valid patterns, it would be great to include them as valid patterns so that they also benefit from the perf improvements

ah maybe it's OK, thanks - I was looking at https://strftime.org/, but it says "platform-specific". still, would be good to preserve behaviour there if possible
> then I suppose that the safest fix is to replace [these lines](https://github.com/pandas-dev/pandas/pull/51298/files#diff-fd90258af6c8946c90f0af064c63f1a5941102ba948e798c2c5b248308197219R270-R271) by "if there is a remaining %, raise an UnsupportedStrFmtDirective error". This will trigger a fallback to the legacy strftime usage.

sounds fine

can we remove the `fast_strftime` option if we want to get this in for pandas 2.x please? if we're confident in it, let's just use it

can we also add a hypothesis test for this please? something like
```python
import hypothesis.strategies as st
from hypothesis import given
import datetime as dt
@given(
    datetimes=st.lists(st.datetimes(min_value=dt.datetime(1700, 1, 1), max_value=dt.datetime(2200, 1, 1))),
    fmt=st.sets(
    st.sampled_from(
        [
        '%a',
        '%A',
        '%w',
        '%d',
        '%-d',
        '%b',
        '%B',
        '%m',
        '%-m',
        '%y',
        '%Y',
        '%H',
        '%-H',
        '%I',
        '%-I',
        '%p',
        '%M',
        '%-M',
        '%S',
        '%-S',
        '%f',
        '%z',
        '%Z',
        '%j',
        '%-j',
        '%U',
        '%-U',
        '%W',
        '%-W',
        '%c',
        '%x',
        '%X',
        '%%',
        ]
    ), min_size=1)
)
def test_me(datetimes, fmt):
    fmt = ''.join(fmt)
    idx = pd.DatetimeIndex(datetimes)
    result = idx.strftime(fmt)
    expected = pd.Index([i.strftime(fmt) for i in datetimes])
    pd.testing.assert_index_equal(result, expected)
```
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
Sorry @mroeschke and @MarcoGorelli I have had much trouble finding time to work on this lately, but I still aim at completing it in the upcoming weeks. 
awesome thanks - feel free to ping us when we should reopen (or just open a new PR, up to you)
@mroeschke @MarcoGorelli : please reopen, I have updated with the latest main branch. (https://github.com/pandas-dev/pandas/pull/55603 was a bit painful , note for myself. But I think I managed to find the "new" right destination for all tests)

Once reopened, I will apply the two changes we discussed above (removing faststrftime arg everywhere and defaulting to legacy method in presence of an unresolved %)

Thanks!
@mroeschke @MarcoGorelli : now ready for review.

All tests pass, the remaining CI failures are not relevant to this PR it seems, but to a warning in numpy related to encoding when scipy `sem` is used


@mroeschke @MarcoGorelli : would you have bandwidth to review this PR before end of year ? We could make it in 2023 :)
didn't manage to finish this, but I've put it in my tasks for next week

quick comment - I wouldn't introduce `pd.convert_strftime_format`, I don't think it should be a top-level function. maybe in `pandas.tseries.api` (I think that's it)?
Indeed. I made the change, it is now only in `pandas.tseries.api.convert_strftime_format`