Add date dtype
State: closed

- [ ] closes #32473

This is the beginning of the date data type, and it definitely works properly on a high level. For some of the places where strings might need to be converted and such I used the cython code that was implemented in tslib. The time complexity is still linear, but some of those methods may need to be rewritten for dates in cython,  which I'm happy to do.


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Needs Discussion
description: Requires discussion from core team before further action
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Stale

Comments:
Hello @zbrookle! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:















* In the file [`pandas/core/internals/blocks.py`](https://github.com/pandas-dev/pandas/blob/068e9bc9cc6698e0da6e728013483ec84b4e4914/pandas/core/internals/blocks.py):

> [Line 2040:1](https://github.com/pandas-dev/pandas/blob/068e9bc9cc6698e0da6e728013483ec84b4e4914/pandas/core/internals/blocks.py#L2040): [E302](https://duckduckgo.com/?q=pep8%20E302) expected 2 blank lines, found 1
> [Line 2049:1](https://github.com/pandas-dev/pandas/blob/068e9bc9cc6698e0da6e728013483ec84b4e4914/pandas/core/internals/blocks.py#L2049): [E302](https://duckduckgo.com/?q=pep8%20E302) expected 2 blank lines, found 1



* In the file [`pandas/tests/arrays/test_dates.py`](https://github.com/pandas-dev/pandas/blob/068e9bc9cc6698e0da6e728013483ec84b4e4914/pandas/tests/arrays/test_dates.py):

> [Line 5:89](https://github.com/pandas-dev/pandas/blob/068e9bc9cc6698e0da6e728013483ec84b4e4914/pandas/tests/arrays/test_dates.py#L5): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (94 > 88 characters)
> [Line 17:9](https://github.com/pandas-dev/pandas/blob/068e9bc9cc6698e0da6e728013483ec84b4e4914/pandas/tests/arrays/test_dates.py#L17): [E126](https://duckduckgo.com/?q=pep8%20E126) continuation line over-indented for hanging indent
> [Line 20:5](https://github.com/pandas-dev/pandas/blob/068e9bc9cc6698e0da6e728013483ec84b4e4914/pandas/tests/arrays/test_dates.py#L20): [E121](https://duckduckgo.com/?q=pep8%20E121) continuation line under-indented for hanging indent
> [Line 79:1](https://github.com/pandas-dev/pandas/blob/068e9bc9cc6698e0da6e728013483ec84b4e4914/pandas/tests/arrays/test_dates.py#L79): [E302](https://duckduckgo.com/?q=pep8%20E302) expected 2 blank lines, found 1
> [Line 93:1](https://github.com/pandas-dev/pandas/blob/068e9bc9cc6698e0da6e728013483ec84b4e4914/pandas/tests/arrays/test_dates.py#L93): [E302](https://duckduckgo.com/?q=pep8%20E302) expected 2 blank lines, found 1
> [Line 102:1](https://github.com/pandas-dev/pandas/blob/068e9bc9cc6698e0da6e728013483ec84b4e4914/pandas/tests/arrays/test_dates.py#L102): [E302](https://duckduckgo.com/?q=pep8%20E302) expected 2 blank lines, found 1
> [Line 114:1](https://github.com/pandas-dev/pandas/blob/068e9bc9cc6698e0da6e728013483ec84b4e4914/pandas/tests/arrays/test_dates.py#L114): [E302](https://duckduckgo.com/?q=pep8%20E302) expected 2 blank lines, found 1
> [Line 127:1](https://github.com/pandas-dev/pandas/blob/068e9bc9cc6698e0da6e728013483ec84b4e4914/pandas/tests/arrays/test_dates.py#L127): [E302](https://duckduckgo.com/?q=pep8%20E302) expected 2 blank lines, found 1
> [Line 134:1](https://github.com/pandas-dev/pandas/blob/068e9bc9cc6698e0da6e728013483ec84b4e4914/pandas/tests/arrays/test_dates.py#L134): [E302](https://duckduckgo.com/?q=pep8%20E302) expected 2 blank lines, found 1
> [Line 179:1](https://github.com/pandas-dev/pandas/blob/068e9bc9cc6698e0da6e728013483ec84b4e4914/pandas/tests/arrays/test_dates.py#L179): [E305](https://duckduckgo.com/?q=pep8%20E305) expected 2 blank lines after class or function definition, found 1
> [Line 184:40](https://github.com/pandas-dev/pandas/blob/068e9bc9cc6698e0da6e728013483ec84b4e4914/pandas/tests/arrays/test_dates.py#L184): [W292](https://duckduckgo.com/?q=pep8%20W292) no newline at end of file



##### Comment last updated at 2020-05-29 16:47:47 UTC
could DateArray just be an alias for PeriodArray[D]?
> What's the motivation for solving just date frequency, rather than making a generic datetime array with arbitrary frequency? What's the additional complexity in allowing user-configurable frequency?

@TomAugspurger The motivation is that most data frameworks have a dedicated date datatype. It's pretty common in databases and spread sheets that an end user might want just the date type when exporting. I think for all other uses the datetime64 type is fine because that lines up well with the "timestamp" type that's pretty commonly used in databases and other places. However that type doesn't line up when a user wants a date type 
> could DateArray just be an alias for PeriodArray[D]?

@jbrockmendel I know you had suggested this on the original issue #32473 but I think that there a couple reasons to have a different dedicated type. 

1. When a user prints out a dataframe, thinking they're using a date, I think it would be really misleading to have Period[D] displayed as the dtype rather than date.

2. When you export to other formats, like parquet for example, the cleanest way to map to what other frameworks have as a date type, is for there to be a dedicated pandas date type as well.
I think a `date` dtype would be cool to have in pandas, but I think there needs to be some more high-level discussion first:

- Since this is a new dtype, IMO it should subclass BaseMaskedArray, and not the DatetimeLikeArray
- Related to the above, IMO we should directly use `pd.NA` as missing value indicator
- There are actually multiple ways this could be stored. Now it is "days since 1970-01-01", but it could also be microseconds since 1970-01-01, which still gives a good date range but would make a conversion between date dtype <-> future timestamp dtype with µs resolution a no-op (this exists in Arrow, but will ask there whether someones knows is that is a common type)

So maybe we first return to the issue to get a bit more agreement on those discussion items before further working on the PR.
(and @zbrookle, thanks a lot for looking into this!)
> What's the motivation for solving just date frequency, rather than making a generic datetime array with arbitrary frequency? What's the additional complexity in allowing user-configurable frequency?

> I think a date dtype would be cool to have in pandas, but I think there needs to be some more high-level discussion first:

I've been looking at what it would take to support non-nano dt64/td64, am putting together an email to pandas-dev about the options/tradeoffs.  From an implementation standpoint, the hard parts are a) timezones and b) combing the code for all the places where we assume always-nanos.

Date does have the advantage that we probably wouldn't want a tzaware date, which simplifies the implementation.
> I think a `date` dtype would be cool to have in pandas, but I think there needs to be some more high-level discussion first:
> 
> * Since this is a new dtype, IMO it should subclass BaseMaskedArray, and not the DatetimeLikeArray
> * Related to the above, IMO we should directly use `pd.NA` as missing value indicator
> * There are actually multiple ways this could be stored. Now it is "days since 1970-01-01", but it could also be microseconds since 1970-01-01, which still gives a good date range but would make a conversion between date dtype <-> future timestamp dtype with µs resolution a no-op (this exists in Arrow, but will ask there whether someones knows is that is a common type)
> 
> So maybe we first return to the issue to get a bit more agreement on those discussion items before further working on the PR.

- I agree that those things should definitely be discussed further, which is why I held off on writing any cython code yet. Most of the changes that I've made can be easily adapted to use days or microseconds, since I considered both options.

- The reason I used days was because numpy uses days since 1970-01-01 in it's datetime64[D] implementation so I figured it should be consistent, and if a user explicitly specifies date, using days will in theory allow a user to expand further out from epoch than with microseconds. 

- The pd.NA is something that I'd prefer as I think it ends up being easier to work with from my experience especially when using arrow.

- I think that if we want to use pd.NA then maybe it would be worth having a nullable datetimelike that this would then inherit from
great idea. this PR is stale however and would need quite a bit to rebase. Please open a new one if interested.