DEPR offsets: rename 'M' to 'ME'
State: closed

This pr is related to issue #9586.

Added warning to to_offset(freq) in case freq='M'. This is the first step to deprecate 'M' for MonthEnd. 
TODO: let 'M' keep working for Period

example to reproduce:
```
for orig_freq, target_freq in [('10D', '1M'), ('3M', 'Y')]:
    print(f"{orig_freq}->{target_freq}")
    ind = pd.date_range('2022-01-01', freq=orig_freq, periods=5)
    s = pd.Series(np.arange(5), ind)
    print(s) 
    print(s.resample(target_freq, closed='right').asfreq())
```

both `date_range` and `resample` produce warnings.
```
pandas/core/resample.py:1604: UserWarning: 'M' will be deprecated, please use 'ME' for 'month end'
  freq = to_offset(freq)
```

Labels:
name: Frequency
description: DateOffsets
name: Deprecate
description: Functionality to remove in pandas

Comments:
The `_prefix` in the class `MonthEnd` is redefined from “M” to "ME”.
In `to_offset` we check, if  “M” is passed, then raise a warning and change “M” to “ME”.

Now we can apply both, “M” and “ME”. The example below raises one warning for freq=‘3M’
```
for orig_freq, target_freq in [('10D', '1ME'), ('3M', 'Y')]:
    print(f"{orig_freq}->{target_freq}")
    ind = pd.date_range('2022-01-01', freq=orig_freq, periods=5)
    s = pd.Series(np.arange(5), ind)
    print(s) 
    print(s.resample(target_freq, closed='right').asfreq())
```
In order to reduce the number of test errors I also changed “M” to “ME” in pandas/tests/arrays/test_datetimelike.py:35 
```
@pytest.fixture(params=["D", "B", "W", "ME", "Q", "Y"])
```
This reduced number of test errors from 1400 to 68.
Maybe `to_offset` could have a `is_period` boolean argument, which would be passed as `True` when called when instantiating a Period, i.e.

https://github.com/pandas-dev/pandas/blob/52c653b437db67815203872de6179b1f7adeb148/pandas/core/arrays/period.py#L965

And then, instead of warning if `name == 'M'`, if you warn if `name == 'M' and not is_period`
> However, it also throws a warning for `PeriodIndex(..., freq='M')`, which as per [#9586 (comment)](https://github.com/pandas-dev/pandas/issues/9586#issuecomment-1454859723) we need to keep working
> 
> So, we need to figure out how to pass `'M'` to PeriodIndex without it warning, but such that passing `'M'` to `date_range` would warn

Thanks for helping me with the issue.

I noticed the same inconsistency for Period. It also throws a warning for `Period(..., freq='M')`:
```
>>> pd.Period('2017Q3', freq='M')
Period('2017-07', 'ME')
```

I will work on fixing it.
we might need to update

https://github.com/pandas-dev/pandas/blob/072402b58cbd87c2106413a37213c0b6020e34b1/pandas/_libs/tslibs/period.pyx#L2304-L2307

so that, when it's printed, it still shows `'M'`
Looks like this one might need updating

https://github.com/pandas-dev/pandas/blob/8e2b974d9a5a9a5fa8c523db6eaa5c7831e76dc9/pandas/_libs/tslibs/dtypes.pyx#L143-L155
> Looks like this one might need updating
> 
> https://github.com/pandas-dev/pandas/blob/8e2b974d9a5a9a5fa8c523db6eaa5c7831e76dc9/pandas/_libs/tslibs/dtypes.pyx#L143-L155

Thank you, I updated `_attrname_to_abbrevs `as you suggested and
https://github.com/pandas-dev/pandas/blob/8e2b974d9a5a9a5fa8c523db6eaa5c7831e76dc9/pandas/_libs/tslibs/dtypes.pyx#L104
and
https://github.com/pandas-dev/pandas/blob/8e2b974d9a5a9a5fa8c523db6eaa5c7831e76dc9/pandas/_libs/tslibs/dtypes.pyx#L306-L325
and
https://github.com/pandas-dev/pandas/blob/8e2b974d9a5a9a5fa8c523db6eaa5c7831e76dc9/pandas/_libs/tslibs/dtypes.pyx#L342-L345
as well
and it fixed my tests.


but I still have failures in some other tests and I keep working on fixing them.
thanks @natmokval - looks like CI didn't run for some reason, could you fetch and merge upstream/main please so I can take a look at any test failures? or just let me know which tests are failing for you and I'll try running them
> thanks @natmokval - looks like CI didn't run for some reason, could you fetch and merge upstream/main please so I can take a look at any test failures? or just let me know which tests are failing for you and I'll try running them

Because I didn't want to run CI, I added '[skip ci]' to the commit message.  Maybe it's better to run CI for every commit.

I did fetch, merge main, and made a new commit. Could you please take a look at the tests:
pandas/tests/indexes/period/test_indexing.py
and
pandas/tests/groupby/test_timegrouper.py 

You'll need to update

https://github.com/pandas-dev/pandas/blob/8e2b974d9a5a9a5fa8c523db6eaa5c7831e76dc9/pandas/core/resample.py#L1602

too

For how I found this: the failing test has `df.groupby(Grouper(freq="6M"))`. So I looked at the definition of `Grouper`, which led to `TimeGrouper`, which led to that line
> You'll need to update
> 
> https://github.com/pandas-dev/pandas/blob/8e2b974d9a5a9a5fa8c523db6eaa5c7831e76dc9/pandas/core/resample.py#L1602
> 
> too
> 
> For how I found this: the failing test has `df.groupby(Grouper(freq="6M"))`. So I looked at the definition of `Grouper`, which led to `TimeGrouper`, which led to that line

Thank you, I updated `end_types`
Hi @MarcoGorelli , I corrected tests, and looks like there are no failures in the tests now. I think all failures are related to docs. Should I update docs and change M to ME for MonthEnd?
Wow, solid effort here! Well done fixing the tests!

Going through this, I'm wondering whether `Period` should be special-cased at all - in case you missed it, I left a comment about it here https://github.com/pandas-dev/pandas/issues/9586#issuecomment-1503351225
> Wow, solid effort here! Well done fixing the tests!
> 
Thanks :)
> Going through this, I'm wondering whether `Period` should be special-cased at all - in case you missed it, I left a comment about it here [#9586 (comment)](https://github.com/pandas-dev/pandas/issues/9586#issuecomment-1503351225)

I am also unsure if we should keep "M" for `Period`, while we have "ME" for the `offsets`.

All checks except Doc Build and Upload pass. Now I am trying to figure out how to deal with failures in Doc Build.
When I build the documentation locally I get a few warnings like
`UserWarning: \'M\' will be deprecated, please use \'ME\'`, but not errors. 

Can these warnings be the reason for ci failure? @MarcoGorelli, could you please help me with this?
yup, the docs build fails if there's warnings

sure, I'll take a look tomorrow

EDIT: didn't manage to get to this, will take a look soon when I get a chance
looks like there's a few places in the docs which need changing, such as

https://github.com/pandas-dev/pandas/blob/8b2ebdba47106ca2f787ba2239c30f3465be3dc3/doc/source/getting_started/comparison/includes/time_date.rst#L8

but it looks like the tests are all working now. Well done! 🙌 

From https://github.com/pandas-dev/pandas/issues/9586#issuecomment-1505801217, we need to figure out to preserve `'M'` for Period. I'll take a closer look in the week, but I'd like to think it wouldn't be too hard at this point, and that we can simply change it when constructing `Period` and when printing it (`__repr__`)
Hi @MarcoGorelli, could you please take a look at my last changes? I renamed 'M' to 'ME' in docstrings in *.rst files. Now, all checks pass without failures.

I don't understand so far what to do to preserve 'M' to `Period`.
wow, well done! ok I'll try something out and will get back to you
Right, I think it's time to bring `is_period` back to `to_offset`. By default, it will be `False`. Then:
- if `False`, then `'M'` will raise a FutureWarning advising to use `'ME'`, and `name` will be changed to `'ME'`
- if `True`, then it will not raise a warning, but `name` will still be changed to `'ME'`. But if the user passed `'ME'` to `Period`, then that should raise

Then, within `_maybe_convert_freq` in `pandas/_libs/tslibs/period.pyx`, you can pass `is_period=True` in `to_offset`, and likewise in `validate_dtype_freq` in `pandas/core/arrays/period.py`

Finally, you'll also need to convert back to "'M'" from `"'ME'` in `PeriodDtype.name` and in `_Period.__repr__`

Please do ask if this is unclear or if anything here trips you up
I took a look, and the issue with `test_feather` is that `ArrowPeriodType` is getting initialised with `'ME'` here:

https://github.com/pandas-dev/pandas/blob/4f1320ff6a1f273878e4f1be52c76f3e3ae22768/pandas/core/arrays/period.py#L411

So I think there you'll want to add another if/then (which we will later use a dict for), like:
```diff
-        period_type = ArrowPeriodType(self.freqstr)
+        if self.freqstr == 'ME':
+            period_type = ArrowPeriodType('M')
+        else:
+            period_type = ArrowPeriodType(self.freqstr)
```
You'll also need to put `is_period=True` in `_parse_dtype_strict`
> I took a look, and the issue with `test_feather` is that `ArrowPeriodType` is getting initialised with `'ME'` here:
> 
> https://github.com/pandas-dev/pandas/blob/4f1320ff6a1f273878e4f1be52c76f3e3ae22768/pandas/core/arrays/period.py#L411
> 
> So I think there you'll want to add another if/then (which we will later use a dict for), like:
> 
> ```diff
> -        period_type = ArrowPeriodType(self.freqstr)
> +        if self.freqstr == 'ME':
> +            period_type = ArrowPeriodType('M')
> +        else:
> +            period_type = ArrowPeriodType(self.freqstr)
> ```
> 
> You'll also need to put `is_period=True` in `_parse_dtype_strict`

Thank you, @MarcoGorelli, I did as you suggested. After the correction, there are no failures in `test_feather.py`.
Generally looks really solid!

@jbrockmendel @mroeschke do you want to take a look?

@rhshadrach - we've had to modify part of the `resample` code, in order to tell whether the index is `PeriodIndex` or `DatetimeIndex` in order to know whether to emit a deprecation warning or not - tagging in case you had any objections to those changes

---

To summarise the changes:
- 'M' is deprecated in favour of 'ME' for offsets
- for Period, where there is no distinction between "month begin" and "month end", we keep 'M'
- if this is accepted, then PRs to make a similar change for "Y" -> "YE" (and some others) will follow. But the basic architecture is in place now, that should just be a matter of updating the remapping dictionary and adjusting tests / docs accordingly
> might be good to start working on a separate branch in which we start fixing up the tests for other similar deprecations ('Y' -> 'YE', 'Q' -> 'QE', and some others)

I would like to work on follow-on this PR. I will start with replacing `'Y'` with `'YE'` for `offsets`.

@MarcoGorelli is this ready to merge (once the merge conflicts are addressed)?
yes, looks good to me
Great work here @natmokval 
Thanks @MarcoGorelli, @mroeschke, @jbrockmendel for helping me with this PR and for  reviewing it.
Renaming "M" to "ME" overall seems like a nice change -- just curious, though, was this PR expected to change results for examples like the following?

#### Prior to this PR

```
>>> import numpy as np; import pandas as pd
>>> index = pd.date_range("2000-01-01T12:07:01", periods=5, freq="3MS")
>>> series = pd.Series(np.arange(100.0, 100.0 + index.size), index=index)
>>> series.resample("MS", closed="right", label="right").mean()
2000-01-01      NaN
2000-02-01    100.0
2000-03-01      NaN
2000-04-01      NaN
2000-05-01    101.0
2000-06-01      NaN
2000-07-01      NaN
2000-08-01    102.0
2000-09-01      NaN
2000-10-01      NaN
2000-11-01    103.0
2000-12-01      NaN
2001-01-01      NaN
2001-02-01    104.0
Freq: MS, dtype: float64
```

#### After this PR

```
>>> import numpy as np; import pandas as pd
>>> index = pd.date_range("2000-01-01T12:07:01", periods=5, freq="3MS")
>>> series = pd.Series(np.arange(100.0, 100.0 + index.size), index=index)
>>> series.resample("MS", closed="right", label="right").mean()
2000-01-01    100.0
2000-02-01      NaN
2000-03-01      NaN
2000-04-01    101.0
2000-05-01      NaN
2000-06-01      NaN
2000-07-01    102.0
2000-08-01      NaN
2000-09-01      NaN
2000-10-01    103.0
2000-11-01      NaN
2000-12-01      NaN
2001-01-01    104.0
```

For context, we implement a generalized version of `resample` in xarray that also works for non-standard calendars, and we test that the results of our implementation match the results of pandas for the standard calendar (for a set of occasionally unusual examples).  This is one example of a test failure that appeared recently in our upstream build (our implementation expects the former result to hold): https://github.com/pydata/xarray/issues/8091.
thanks @spencerkclark - was this fixed by https://github.com/pandas-dev/pandas/pull/55283 ? 

edit: yeah trying this on `main` it seems the previous results are preserved
Oh nice, thanks, I hadn't noticed that!  Yes, that fixes most of the failures.  The remaining ones seem to be a little more subtle; I will try and isolate how they relate to pandas when I get a chance.