BUG refactor datetime parsing and fix 8 bugs
State: closed

- [ ] closes #50051
- [ ] closes #50071
- [ ] closes #14487
- [ ] closes #50254
- [ ] closes #21422
- [ ] closes #26493
- [ ] closes #17410 
- [ ] closes #50359
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

this'd solve a number of issues

work in progress

---

Performance: this maintains the fastness for ISO formats:
```python
format = '%Y-%d-%m %H:%M:%S%z'
dates = pd.date_range('1900', '2000').tz_localize('+01:00').strftime(format).tolist()
```

upstream/main:
```python
In [2]: %%timeit
   ...: pd.to_datetime(dates, format=format)
   ...: 
   ...: 
241 ms ± 3.43 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```
here
```python
In [2]: %%timeit
   ...: pd.to_datetime(dates, format=format)
   ...: 
   ...: 
221 ms ± 5.58 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

---

Demo of how this addresses https://github.com/pandas-dev/pandas/issues/17410

```python
In [8]: s = pd.Series(['20120101']*1000000)

In [9]: %timeit pd.to_datetime(s, cache=False)  # no format
72.7 ms ± 929 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [10]: %timeit pd.to_datetime(s, cache=False, format='%Y%m%d')  # slightly faster, as it doesn't need to guess the format
72.2 ms ± 665 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [11]: %timeit pd.to_datetime(s, cache=False, format='%Y%d%m')  # by comparison, non-ISO is much slower
1.12 s ± 52.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

---

Now real difference for non-ISO formats:

1.5.2:
```python
In [16]: format = "%m-%d-%Y"

In [17]: dates = pd.date_range('1900', '2000').tz_localize('+01:00').strftime(format).tolist()

In [18]: %%timeit
    ...: pd.to_datetime(dates, format=format)
    ...:
    ...:
43.5 ms ± 280 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

here:
```python
In [2]: format = "%m-%d-%Y"

In [3]: dates = pd.date_range('1900', '2000').tz_localize('+01:00').strftime(format).tolist()

In [4]: %%timeit
   ...: pd.to_datetime(dates, format=format)
   ...: 
   ...: 
42.4 ms ± 405 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```


note
----
gonna try to get https://github.com/pandas-dev/pandas/issues/50361 in first, so marking as draft for now

Labels:
name: Bug
name: Datetime
description: Datetime data dtype

Comments:
this might be superseded by #50258 

EDIT: maybe not, this one causes less breakage
Only fixes 8 bugs?
> Nice!

Thanks!

Can I ask that we get https://github.com/pandas-dev/pandas/pull/50366 in first though? That'll reduce the diff in this one
> Can I ask that we get #50366 in first though? That'll reduce the diff in this one

Cool, that's in, and I've rebased.

Thanks for your reviews and approvals - @jbrockmendel any further thoughts?
> Nice, thanks for being persistent on this

Thanks!

@WillAyd @mroeschke any further comments, or good-to-merge?
Thanks @MarcoGorelli