BUG: Do not fail when parsing pydatetime objects in pd.to_datetime
State: closed

- [x] closes #49298 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Granted, there's already a PR open to address this (https://github.com/pandas-dev/pandas/pull/49338), it may be stale. As this is a blocker for PDEP0004, I've taken it forwards, but have taken care to include @aaossa as commit author.
I've fixed it up a bit to handle the objections I had and have re-written the tests

Labels:
name: Datetime
description: Datetime data dtype

Comments:
havent looked at the tests yet, but the non-test changes look good
Thanks for keeping this updated @MarcoGorelli , should I close #49338 in favor of this PR?
thanks @aaossa ! sure, yes please
> havent looked at the tests yet, but the non-test changes look good

merging for now then to unblock the PDEP4 PR, if you have comments on the tests I'll address them separately

thanks all for your reviews!
I seem to haven gotten a lot of failed tests from this PR, especially in `pandas/tests/tools/test_to_datetime.py`where I get 155 failed tests.

The error message are like `TypeError: array_strptime() got an unexpected keyword argument 'utc'`

Example error report:

```bash
[gw2] darwin -- Python 3.9.7 /opt/homebrew/Caskroom/miniconda/base/envs/pandas/bin/python

self = <pandas.tests.tools.test_to_datetime.TestToDatetimeInferFormat object at 0x1404f9070>, cache = False

    def test_to_datetime_infer_datetime_format_series_start_with_nans(self, cache):
        ser = Series(
            np.array(
                [
                    np.nan,
                    np.nan,
                    "01/01/2011 00:00:00",
                    "01/02/2011 00:00:00",
                    "01/03/2011 00:00:00",
                ],
                dtype=object,
            )
        )

        tm.assert_series_equal(
            to_datetime(ser, infer_datetime_format=False, cache=cache),
>           to_datetime(ser, infer_datetime_format=True, cache=cache),
        )

pandas/tests/tools/test_to_datetime.py:2405:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pandas/core/tools/datetimes.py:1083: in to_datetime
    values = convert_listlike(arg._values, format)
pandas/core/tools/datetimes.py:438: in _convert_listlike_datetimes
    res = _to_datetime_with_format(
pandas/core/tools/datetimes.py:544: in _to_datetime_with_format
    res = _array_strptime_with_fallback(
pandas/core/tools/datetimes.py:478: in _array_strptime_with_fallback
    result, timezones = array_strptime(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   'p': 18,  # an additional key, only with I
E   TypeError: array_strptime() got an unexpected keyword argument 'utc'

pandas/_libs/tslibs/strptime.pyx:56: TypeError
___________________________ TestToDatetimeInferFormat.test_infer_datetime_format_tz_name[UTC-0] ____________________________
[gw2] darwin -- Python 3.9.7 /opt/homebrew/Caskroom/miniconda/base/envs/pandas/bin/python

self = <pandas.tests.tools.test_to_datetime.TestToDatetimeInferFormat object at 0x1404f9220>, tz_name = 'UTC', offset = 0

    @pytest.mark.parametrize(
        "tz_name, offset", [("UTC", 0), ("UTC-3", 180), ("UTC+3", -180)]
    )
    def test_infer_datetime_format_tz_name(self, tz_name, offset):
        # GH 33133
        ser = Series([f"2019-02-02 08:07:13 {tz_name}"])
>       result = to_datetime(ser, infer_datetime_format=True)

pandas/tests/tools/test_to_datetime.py:2414:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pandas/core/tools/datetimes.py:1083: in to_datetime
    values = convert_listlike(arg._values, format)
pandas/core/tools/datetimes.py:438: in _convert_listlike_datetimes
    res = _to_datetime_with_format(
pandas/core/tools/datetimes.py:544: in _to_datetime_with_format
    res = _array_strptime_with_fallback(
pandas/core/tools/datetimes.py:478: in _array_strptime_with_fallback
    result, timezones = array_strptime(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   'p': 18,  # an additional key, only with I
E   TypeError: array_strptime() got an unexpected keyword argument 'utc'

pandas/_libs/tslibs/strptime.pyx:56: TypeError
```


My OS is MacOs 12.5.1 and AFAIKS my packages are up-to-date:

<details>

INSTALLED VERSIONS
------------------
commit           : 2804681c45b5f748d0977e3998bbfc91d4653b10
python           : 3.9.7.final.0
python-bits      : 64
OS               : Darwin
OS-release       : 21.6.0
Version          : Darwin Kernel Version 21.6.0: Wed Aug 10 14:28:35 PDT 2022; root:xnu-8020.141.5~2/RELEASE_ARM64_T8101
machine          : arm64
processor        : arm
byteorder        : little
LC_ALL           : None
LANG             : None
LOCALE           : None.UTF-8

pandas           : 2.0.0.dev0+805.g2804681c45
numpy            : 1.23.4
pytz             : 2022.1
dateutil         : 2.8.2
setuptools       : 65.5.0
pip              : 22.2.2
Cython           : 0.29.32
pytest           : 7.1.2
hypothesis       : 6.37.0
sphinx           : 5.0.2
blosc            : 1.10.6
feather          : None
xlsxwriter       : 3.0.3
lxml.etree       : 4.9.1
html5lib         : None
pymysql          : 1.0.2
psycopg2         : 2.9.3
jinja2           : 3.1.2
IPython          : 7.29.0
pandas_datareader: None
bs4              : 4.11.1
bottleneck       : 1.3.5
brotli           :
fastparquet      : 0.8.3
fsspec           : 2022.10.0
gcsfs            : None
matplotlib       : 3.5.3
numba            : 0.56.4
numexpr          : 2.8.4
odfpy            : None
openpyxl         : 3.0.10
pandas_gbq       : None
pyarrow          : 9.0.0
pyreadstat       : None
pyxlsb           : 1.0.10
s3fs             : 2022.10.0
scipy            : 1.9.1
snappy           : None
sqlalchemy       : 1.4.43
tables           : 3.7.0
tabulate         : 0.9.0
xarray           : None
xlrd             : 2.0.1
zstandard        : 0.18.0
tzdata           : None
qtpy             : None
pyqt5            : None
</details>



this is fine in CI - does this function in `pandas/_libs/tslibs/strptime.pyx` have a `utc` argument? did you re-compile your C extensions? not sure what else it could be
ah wait, you're right @topper-123 , the PY39 macos job has started timing out

I'll take a look

---

EDIT: looks like it's passing in recent runs. So I think it might be unrelated to this PR
Hmm I'll try out various stuff, maybe reinstall the python environment anew.

Thanks for responding so quickly.
A short follow-up: I got this fixed by compiling the c code using `python setup.py build_ext --inplace --force -j 4` instead of the usual `python setup.py build_ext -j 4`.

So this error was caused by something in my build process and is unrelated to this PR.