ENH/WIP: resolution inference in pd.to_datetime, DatetimeIndex
State: closed

- [x] closes #54048
- [x] closes #55014
- [x] closes #55096
- [x] closes #55487
- [x] closes #55564
- [x] closes #56733
- [x] closes #57080
- [x] closes #57349
- [x] closes #58300
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This is an API change so will have to wait for 3.0, posting it to get feedback on desired API in mostly-IO places where this causes API changes.

This does not yet handle Timedelta scalar, TimedeltaIndex, or pd.to_timedelta.  I'd like to get those done for 3.0 but its a much lower priority than the datetime side.

I've been splitting things off to reduce the diff, will try to keep doing that.  Open to suggestions for splittable components.  Currently #55741 has some of the cython implementation.

19 tests are failing locally.  11 of those are stata tests that would be addressed by #55642.  4 are SAS and have a similar option available.  One is in JSON and may be hard to solve, xref #55827.  The rest are caused by #55737.


Labels:
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
@willayd can i get your input on how to handle the failing json test here (xref #55827)
I haven't figured it out but my suspicion is that the json bug stems from the `def _try_convert_dates()` function in Python; not from anything in the read/write extensions. A quick debug of this versus main I didn't see the integral column going through the _try_convert_dates path on this branch, whereas on main it does
OK I think where this breaks down is here:

https://github.com/pandas-dev/pandas/blob/4f080b8f2e5c93d1b0dd1ceb5487fba67feb02ab/pandas/io/json/_json.py#L1326

The json python parser has some concept of what it thinks a "min_stamp" is for different precisions, and when reading back integral values like 1356 and 1357 that fails.

Generally I am not sure that `convert_dates` in `read_json` should be doing this. In a world where you can have different units of precision, we would either need to set that argument to always assume `ns` (?), change it to allow the user to provide a precision, and/or deprecate
Would you be OK with just xfailing the relevant test?  a long-term solution is out of scope for this PR, but medium-term (keep in mind the goal is for this to be merged shortly after 2.2 is released so this goes live in 3.0) we could do something like warn when doing to_json with non-nano
I have to think more but I am wary of doing that in pieces - it will make it really confusing if you try to load JSON files across different pandas versions and get very different results.

The issue isn't as much on the writing part as the reading part. But what we could do to at least keep things consistent for now is make sure that the writer continues to write nanosecond integral values for dates. That in turn helps the reader by being able to continue to assume nanoseconds.

I think being lossy on the unit is better than tossing the whole value out the window
> The issue isn't as much on the writing part as the reading part. But what we could do to at least keep things consistent for now is make sure that the writer continues to write nanosecond integral values for dates. That in turn helps the reader by being able to continue to assume nanoseconds.

The "continue to write nanosecond integral values" isn't quite right.  In main, if you have a non-nano dt64, you're going to trigger the bad behavior (#55827).  This PR just makes it so that users are much more likely to get non-nano in the wild.

How about a separate PR that warns when writing non-nanos as integers?
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
@WillAyd iirc you worked on ADBC stuff, maybe you can help troubleshoot the min-version test_api_custom_dateparsing_error failure?
postgres stores DATEs internally with microsecond precision so the expectation of those tests looks correct. Do you know where the coercion to nanoseconds is happening here?
Could be wrong but I don't believe the issue has anything to do with the ADBC version - the result showing in CI currently I think is expected behavior
> Could be wrong but I don't believe the issue has anything to do with the ADBC version - the result showing in CI currently I think is expected behavior

I think when you made this comment the CI was passing bc I had edited the min-versions dependencies to use adbc-driver-postgresql==0.10.0 instead of 0.8.0.  It is now restored to 0.8.0 and is failing again.
Ah OK. Well if upgrading to 0.10.0 fixes it then I say we should just bump that minimum. The ADBC drivers are new and released fast enough that I don't think we need to spend a lot of time on backwards compatibility right now
added whatsnew section, could use eyeballs.  both on content and bc it is breaking the docbuild for reasons that are unclear to me
@mroeschke thoughts on getting this across the finish line?
/preview
I had 2 comments otherwise looks OK. Do you think we can also do the same for timedelta before 3.0?
> Do you think we can also do the same for timedelta before 3.0?

Not likely.  IIRC even the scalar case doesnt do inference on strings yet.
Nice to finally get this in @jbrockmendel