BUG: catch out-of-bounds datetime64 in Series/DataFrame constructor
State: closed

- [x] closes https://github.com/pandas-dev/pandas/issues/26206
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

TODO: Need to add a whatsnew and test for DataFrame.

Question:

I am now letting this raise an error. But, an alternative could be to return an object array (by simply removing the `except OutOfBoundsDatetime: raise`, as the code below that fall backs to object array if no `dtype` is specified. 

This would be more consistent with if it is not an array but list of scalars:

```
In [5]: a = np.array(['2262-04-12'], dtype='datetime64[s]')

In [6]: pd.Series(a)   # now raising on this PR
...
OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 2262-04-12 00:00:00

In [7]: pd.Series([a[0]])    # already falling back to object array on master
Out[7]: 
0    2262-04-12T00:00:00
dtype: object
```

That is somewhat the general logic of the `_try_cast`. Of course, if you specify the required dtype explicitly, you should still get an error (which is actually also a bug on master, I see now).

However, that would be inconsistent with the current `pd.Index` behaviour, which raises when passing out of bounds datetime64 (but also returns object dtype with list of datetime64 scalars)

cc @jbrockmendel @mroeschke 


Labels:
name: Datetime
description: Datetime data dtype
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26848?src=pr&el=h1) Report
> Merging [#26848](https://codecov.io/gh/pandas-dev/pandas/pull/26848?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/b9b081dc6b510c8290ded12fe751b1216843527e?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26848/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26848?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26848      +/-   ##
==========================================
- Coverage   91.97%   91.97%   -0.01%     
==========================================
  Files         180      180              
  Lines       50756    50760       +4     
==========================================
+ Hits        46685    46686       +1     
- Misses       4071     4074       +3
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.56% <100%> (ø)` | :arrow_up: |
| #single | `41.84% <33.33%> (-0.07%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26848?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/dtypes/cast.py](https://codecov.io/gh/pandas-dev/pandas/pull/26848/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2Nhc3QucHk=) | `90.56% <100%> (+0.03%)` | :arrow_up: |
| [pandas/core/internals/construction.py](https://codecov.io/gh/pandas-dev/pandas/pull/26848/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL2NvbnN0cnVjdGlvbi5weQ==) | `95.95% <100%> (+0.02%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26848/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `88.88% <0%> (-11.12%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/26848/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.89% <0%> (-0.12%)` | :arrow_down: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/26848/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `90.94% <0%> (+0.1%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26848?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26848?src=pr&el=footer). Last update [b9b081d...369eead](https://codecov.io/gh/pandas-dev/pandas/pull/26848?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26848?src=pr&el=h1) Report
> Merging [#26848](https://codecov.io/gh/pandas-dev/pandas/pull/26848?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/8374a1d2a58245067669cd6424793b134a695971?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26848/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26848?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26848      +/-   ##
==========================================
- Coverage   91.86%   91.85%   -0.01%     
==========================================
  Files         179      179              
  Lines       50700    50703       +3     
==========================================
- Hits        46576    46575       -1     
- Misses       4124     4128       +4
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.45% <100%> (ø)` | :arrow_up: |
| #single | `41.12% <40%> (-0.06%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26848?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/dtypes/cast.py](https://codecov.io/gh/pandas-dev/pandas/pull/26848/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2Nhc3QucHk=) | `90.53% <100%> (ø)` | :arrow_up: |
| [pandas/core/internals/construction.py](https://codecov.io/gh/pandas-dev/pandas/pull/26848/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL2NvbnN0cnVjdGlvbi5weQ==) | `95.96% <100%> (+0.03%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26848/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `88.88% <0%> (-11.12%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/26848/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.88% <0%> (-0.12%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26848?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26848?src=pr&el=footer). Last update [8374a1d...c5fc3c6](https://codecov.io/gh/pandas-dev/pandas/pull/26848?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

> not sure how this is inconsistent

Because we have been doing the opposite for some releases?

Before 0.23.0, all of Index, Series and DataFrame constructors have been consistently raising an error when out-of-bounds datetime64 array was passed, even if no explicit dtype was specified. 

So yes, I consider converting to object an option (this is why I raised the question in the top post), but it's not necessarily the "only correct" way. For example, for Index, this would be a change in behaviour.



> Before 0.23.0, all of Index, Series and DataFrame constructors have been consistently raising an error when out-of-bounds datetime64 array was passed, even if no explicit dtype was specified.

this is likely a bug / regression

Series constructor was designed to not raise and rather coerce 


> Series constructor was designed to not raise and rather coerce

I suppose you don't mean "coerce" here in meaning of `to_datetime`, as that is returning NaT ?
@jreback looked at some different cases and made a general issue here: https://github.com/pandas-dev/pandas/issues/26853
Hello @jorisvandenbossche! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-06-21 06:16:22 UTC
One question about the tests: I want to add the exact same for DataFrame and Index constructors. I can copy an edited version to the respective testing files for DataFrame and Index, but on the other hand, I think there is also benefit in keeping the test together (less duplication of the boiler plate around it, could do a parametrize). Do we have a place to put general constructor tests that check for consistency? (because doing it here in the series tests is also a bit out of place)
> One question about the tests: I want to add the exact same for DataFrame and Index constructors. I can copy an edited version to the respective testing files for DataFrame and Index, but on the other hand, I think there is also benefit in keeping the test together (less duplication of the boiler plate around it). Do we have a place to put general constructor tests that check for consistency? (because doing it here in the series tests is also a bit out of place)

we have 2 mechanisms for this, neither complete solutions. test_generic you can do Series/DataFrame and test_base you can do Series/Index.

generally for this as its really the Series you are fixing is to put it in test_base and then copy a similar example to the appropriate palace in frame tests.
My last commit does an example parametrize of Index/DataFrame/Series

> generally for this as its really the Series you are fixing

yes, but that also fixes DataFrame (with passing a dict) as it is using the same code. For Index it is indeed only adding tests to ensure consistency.
Will move to test_base, that seems most appropriate (it indeed already has some tests that checks consistency across Series, Index)
What both the dataframe and series fix have in common is that the *code* (where the fixes are applied) is now centralized in `construction.py`. So it might make sense to also have a `test_construction.py` at some point.
This should be ready. I opened a new issue for the xfail DataFrame case that I added, as this is a more general issue that we are doing unsafe astype: https://github.com/pandas-dev/pandas/issues/26919
pls merge master. lgtm. merge on green.