API: 'D' and offset.Day always represents calendar day
State: closed

- [x] closes #22864
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Frequency
description: DateOffsets

Comments:
Hello @mroeschke! Thanks for updating the PR.

 - There are no PEP8 issues in the file [`pandas/core/arrays/datetimes.py`](https://github.com/mroeschke/pandas/blob/78369f521615db3488983948adc778aeffe16e20/pandas/core/arrays/datetimes.py) !

 - There are no PEP8 issues in the file [`pandas/core/indexes/datetimes.py`](https://github.com/mroeschke/pandas/blob/78369f521615db3488983948adc778aeffe16e20/pandas/core/indexes/datetimes.py) !

 - There are no PEP8 issues in the file [`pandas/core/indexes/interval.py`](https://github.com/mroeschke/pandas/blob/78369f521615db3488983948adc778aeffe16e20/pandas/core/indexes/interval.py) !

 - There are no PEP8 issues in the file [`pandas/core/indexes/period.py`](https://github.com/mroeschke/pandas/blob/78369f521615db3488983948adc778aeffe16e20/pandas/core/indexes/period.py) !

 - There are no PEP8 issues in the file [`pandas/core/resample.py`](https://github.com/mroeschke/pandas/blob/78369f521615db3488983948adc778aeffe16e20/pandas/core/resample.py) !

 - There are no PEP8 issues in the file [`pandas/tests/indexes/datetimes/test_date_range.py`](https://github.com/mroeschke/pandas/blob/78369f521615db3488983948adc778aeffe16e20/pandas/tests/indexes/datetimes/test_date_range.py) !

 - There are no PEP8 issues in the file [`pandas/tests/indexes/datetimes/test_indexing.py`](https://github.com/mroeschke/pandas/blob/78369f521615db3488983948adc778aeffe16e20/pandas/tests/indexes/datetimes/test_indexing.py) !

 - There are no PEP8 issues in the file [`pandas/tests/indexes/datetimes/test_timezones.py`](https://github.com/mroeschke/pandas/blob/78369f521615db3488983948adc778aeffe16e20/pandas/tests/indexes/datetimes/test_timezones.py) !

 - There are no PEP8 issues in the file [`pandas/tests/indexes/timedeltas/test_timedelta_range.py`](https://github.com/mroeschke/pandas/blob/78369f521615db3488983948adc778aeffe16e20/pandas/tests/indexes/timedeltas/test_timedelta_range.py) !

 - There are no PEP8 issues in the file [`pandas/tests/resample/test_datetime_index.py`](https://github.com/mroeschke/pandas/blob/78369f521615db3488983948adc778aeffe16e20/pandas/tests/resample/test_datetime_index.py) !

 - There are no PEP8 issues in the file [`pandas/tests/resample/test_period_index.py`](https://github.com/mroeschke/pandas/blob/78369f521615db3488983948adc778aeffe16e20/pandas/tests/resample/test_period_index.py) !

 - There are no PEP8 issues in the file [`pandas/tests/scalar/timedelta/test_arithmetic.py`](https://github.com/mroeschke/pandas/blob/78369f521615db3488983948adc778aeffe16e20/pandas/tests/scalar/timedelta/test_arithmetic.py) !

 - There are no PEP8 issues in the file [`pandas/tests/series/test_timezones.py`](https://github.com/mroeschke/pandas/blob/78369f521615db3488983948adc778aeffe16e20/pandas/tests/series/test_timezones.py) !

 - In the file [`pandas/tests/tseries/offsets/test_offsets.py`](https://github.com/mroeschke/pandas/blob/78369f521615db3488983948adc778aeffe16e20/pandas/tests/tseries/offsets/test_offsets.py), following                are the PEP8 issues :
> [Line 3229:66](https://github.com/mroeschke/pandas/blob/78369f521615db3488983948adc778aeffe16e20/pandas/tests/tseries/offsets/test_offsets.py#L3229): [W292](https://duckduckgo.com/?q=pep8%20W292) no newline at end of file


 - There are no PEP8 issues in the file [`pandas/tests/tseries/offsets/test_ticks.py`](https://github.com/mroeschke/pandas/blob/78369f521615db3488983948adc778aeffe16e20/pandas/tests/tseries/offsets/test_ticks.py) !

 - There are no PEP8 issues in the file [`pandas/tseries/offsets.py`](https://github.com/mroeschke/pandas/blob/78369f521615db3488983948adc778aeffe16e20/pandas/tseries/offsets.py) !



##### Comment last updated on December 13, 2018 at 23:43 Hours UTC
One test is failing because of https://github.com/pandas-dev/pandas/issues/23270

https://github.com/pandas-dev/pandas/pull/23278 should fix that test.
Update: 

In general, I have removed the usage of `CalendarDay` and `'CD'` and replaced `CalendarDay` with `_Day` (to ultimately turn into the new `Day`) and the core tests are passing.

I am slowly introducing `FutureWarnings`, but I anticipating this taking a while. I am chugging through Timedelta warnings currently, but I will also need to introduce `FutureWarnings` to `Day` arithmetic, `.shift`, etc. The ending diff will probably be really large.
@jreback What's our policy about catching warnings in the tests? There are already ~1000 `FutureWarning`s for Timedelta usage with 'D', and I will need to add more `FutureWarnings` for Timestamp arithmetic, shift, etc. Do I need to catch/test them all?
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22867?src=pr&el=h1) Report
> Merging [#22867](https://codecov.io/gh/pandas-dev/pandas/pull/22867?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/d43ac979fdf3038c713de93c7b27b4140be6c9c5?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/22867/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/22867?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #22867      +/-   ##
==========================================
+ Coverage   92.22%   92.22%   +<.01%     
==========================================
  Files         162      162              
  Lines       51789    51788       -1     
==========================================
  Hits        47762    47762              
+ Misses       4027     4026       -1
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.62% <100%> (-0.01%)` | :arrow_down: |
| #single | `43.01% <66.66%> (+0.01%)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/22867?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexes/period.py](https://codecov.io/gh/pandas-dev/pandas/pull/22867/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9wZXJpb2QucHk=) | `93.06% <ø> (ø)` | :arrow_up: |
| [pandas/core/indexes/interval.py](https://codecov.io/gh/pandas-dev/pandas/pull/22867/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9pbnRlcnZhbC5weQ==) | `95.26% <ø> (ø)` | :arrow_up: |
| [pandas/core/indexes/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/22867/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZXMucHk=) | `96.34% <ø> (ø)` | :arrow_up: |
| [pandas/core/resample.py](https://codecov.io/gh/pandas-dev/pandas/pull/22867/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzYW1wbGUucHk=) | `96.83% <100%> (+0.02%)` | :arrow_up: |
| [pandas/core/arrays/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/22867/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2RhdGV0aW1lcy5weQ==) | `98.57% <100%> (+0.01%)` | :arrow_up: |
| [pandas/tseries/offsets.py](https://codecov.io/gh/pandas-dev/pandas/pull/22867/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvb2Zmc2V0cy5weQ==) | `96.85% <100%> (ø)` | :arrow_up: |
| [pandas/util/\_decorators.py](https://codecov.io/gh/pandas-dev/pandas/pull/22867/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvX2RlY29yYXRvcnMucHk=) | `91.34% <0%> (-0.17%)` | :arrow_down: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/22867/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `87.41% <0%> (-0.1%)` | :arrow_down: |
| [pandas/io/json/normalize.py](https://codecov.io/gh/pandas-dev/pandas/pull/22867/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2pzb24vbm9ybWFsaXplLnB5) | `96.87% <0%> (-0.07%)` | :arrow_down: |
| [pandas/io/formats/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/22867/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvaHRtbC5weQ==) | `98.61% <0%> (-0.04%)` | :arrow_down: |
| ... and [3 more](https://codecov.io/gh/pandas-dev/pandas/pull/22867/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22867?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22867?src=pr&el=footer). Last update [d43ac97...78369f5](https://codecov.io/gh/pandas-dev/pandas/pull/22867?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@mroeschke I will try to give this a look (finally, sorry about that). Can you give a short status update? Are there still problems you are encountering, .. ?
One of the example I used in the discussion is now errorring:

```
In [1]: idx = pd.date_range("2016-10-30", freq='H', periods=4*24, tz='Europe/Helsinki')                                                                                         

In [2]: pd.Series(index=idx).resample('D').count()                                                                                                                              
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-aaf4c03a8f51> in <module>
----> 1 pd.Series(index=idx).resample('D').count()

~/scipy/pandas/pandas/core/resample.py in f(self, _method)
    864 for method in ['count']:
    865     def f(self, _method=method):
--> 866         return self._downsample(_method)
    867     f.__doc__ = getattr(GroupBy, method).__doc__
    868     setattr(Resampler, method, f)

~/scipy/pandas/pandas/core/resample.py in _downsample(self, how, **kwargs)
    990         **kwargs : kw args passed to how function
    991         """
--> 992         self._set_binner()
    993         how = self._is_cython_func(how) or how
    994         ax = self.ax

~/scipy/pandas/pandas/core/resample.py in _set_binner(self)
    170         """
    171         if self.binner is None:
--> 172             self.binner, self.grouper = self._get_binner()
    173 
    174     def _get_binner(self):

~/scipy/pandas/pandas/core/resample.py in _get_binner(self)
    178         """
    179 
--> 180         binner, bins, binlabels = self._get_binner_for_time()
    181         bin_grouper = BinGrouper(bins, binlabels, indexer=self.groupby.indexer)
    182         return binner, bin_grouper

~/scipy/pandas/pandas/core/resample.py in _get_binner_for_time(self)
    979         if self.kind == 'period':
    980             return self.groupby._get_time_period_bins(self.ax)
--> 981         return self.groupby._get_time_bins(self.ax)
    982 
    983     def _downsample(self, how, **kwargs):

~/scipy/pandas/pandas/core/resample.py in _get_time_bins(self, ax)
   1428         # general version, knowing nothing about relative frequencies
   1429         bins = lib.generate_bins_dt64(
-> 1430             ax_values, bin_edges, self.closed, hasnans=ax.hasnans)
   1431 
   1432         if self.closed == 'right':

~/scipy/pandas/pandas/_libs/lib.pyx in pandas._libs.lib.generate_bins_dt64()

ValueError: Values falls after last bin
```
Sorry, I have been primarily working on reverting things and getting the test to pass again by rolling back `CalendarDay`.

**TODO**
Add all the depreciation warnings (I anticipate there being quite a few of these). This needs to be included in the following:

- `Day` tick arithmetic with other `Tick`s, `Timedeltas`, and `DatetimeTZ`
- `DatetimeIndex.shift` (tz-aware only)

**Migration**
The plan is for `_Day` (formerly `CalendarDay`) to just replace `Day` once the prior `Day` behavior is replaced.

**Concern**
During the last dev chat, there was interest for `'D'` to be compatible freq argument/offset with both `Timedelta` and `Datetime`. I don't see a clear way to make this possible without adding a lot of monkeypatching.

Example: `timedelta_range(..., freq='D')`;  `to_offset('D')` will return `_Day` in the future and this offset will need to increment a `Timedelta`, but `_Day` + `Timedelta` is an invalid operation.
Closing in favor of https://github.com/pandas-dev/pandas/pull/24330 for now (reverts behavior to pre-`CalendarDay`). We can revisit this after the 0.24.0 release.