CLN/ENH: Stop instantiating all offsets on load.
State: closed

Implements @wesm's suggestion and clears the way for new offsets.

Very simple to extend - map a string to your class in mapping, define a `from_name`
classmethod, then class gets passed everything after the `-` (if
anything) to its method `from_name()`. Caches to dict afterwards (right
now doesn't use cacheable, but could take advantage of that).

cc @cancan101

For reference, here are all the existing offset: output aliases (using
types because otherwise prints exactly the same).
[(None, NoneType),
('A-APR', pandas.tseries.offsets.YearEnd),
('A-AUG', pandas.tseries.offsets.YearEnd),
('A-DEC', pandas.tseries.offsets.YearEnd),
('A-FEB', pandas.tseries.offsets.YearEnd),
('A-JAN', pandas.tseries.offsets.YearEnd),
('A-JUL', pandas.tseries.offsets.YearEnd),
('A-JUN', pandas.tseries.offsets.YearEnd),
('A-MAR', pandas.tseries.offsets.YearEnd),
('A-MAY', pandas.tseries.offsets.YearEnd),
('A-NOV', pandas.tseries.offsets.YearEnd),
('A-OCT', pandas.tseries.offsets.YearEnd),
('A-SEP', pandas.tseries.offsets.YearEnd),
('AS-APR', pandas.tseries.offsets.YearBegin),
('AS-AUG', pandas.tseries.offsets.YearBegin),
('AS-DEC', pandas.tseries.offsets.YearBegin),
('AS-FEB', pandas.tseries.offsets.YearBegin),
('AS-JAN', pandas.tseries.offsets.YearBegin),
('AS-JUL', pandas.tseries.offsets.YearBegin),
('AS-JUN', pandas.tseries.offsets.YearBegin),
('AS-MAR', pandas.tseries.offsets.YearBegin),
('AS-MAY', pandas.tseries.offsets.YearBegin),
('AS-NOV', pandas.tseries.offsets.YearBegin),
('AS-OCT', pandas.tseries.offsets.YearBegin),
('AS-SEP', pandas.tseries.offsets.YearBegin),
('B', pandas.tseries.offsets.BusinessDay),
('BA-APR', pandas.tseries.offsets.BYearEnd),
('BA-AUG', pandas.tseries.offsets.BYearEnd),
('BA-DEC', pandas.tseries.offsets.BYearEnd),
('BA-FEB', pandas.tseries.offsets.BYearEnd),
('BA-JAN', pandas.tseries.offsets.BYearEnd),
('BA-JUL', pandas.tseries.offsets.BYearEnd),
('BA-JUN', pandas.tseries.offsets.BYearEnd),
('BA-MAR', pandas.tseries.offsets.BYearEnd),
('BA-MAY', pandas.tseries.offsets.BYearEnd),
('BA-NOV', pandas.tseries.offsets.BYearEnd),
('BA-OCT', pandas.tseries.offsets.BYearEnd),
('BA-SEP', pandas.tseries.offsets.BYearEnd),
('BAS-APR', pandas.tseries.offsets.BYearBegin),
('BAS-AUG', pandas.tseries.offsets.BYearBegin),
('BAS-DEC', pandas.tseries.offsets.BYearBegin),
('BAS-FEB', pandas.tseries.offsets.BYearBegin),
('BAS-JAN', pandas.tseries.offsets.BYearBegin),
('BAS-JUL', pandas.tseries.offsets.BYearBegin),
('BAS-JUN', pandas.tseries.offsets.BYearBegin),
('BAS-MAR', pandas.tseries.offsets.BYearBegin),
('BAS-MAY', pandas.tseries.offsets.BYearBegin),
('BAS-NOV', pandas.tseries.offsets.BYearBegin),
('BAS-OCT', pandas.tseries.offsets.BYearBegin),
('BAS-SEP', pandas.tseries.offsets.BYearBegin),
('BM', pandas.tseries.offsets.BusinessMonthEnd),
('BMS', pandas.tseries.offsets.BusinessMonthBegin),
('BQ', pandas.tseries.offsets.BQuarterEnd),
('BQ-APR', pandas.tseries.offsets.BQuarterEnd),
('BQ-AUG', pandas.tseries.offsets.BQuarterEnd),
('BQ-DEC', pandas.tseries.offsets.BQuarterEnd),
('BQ-FEB', pandas.tseries.offsets.BQuarterEnd),
('BQ-JAN', pandas.tseries.offsets.BQuarterEnd),
('BQ-JUL', pandas.tseries.offsets.BQuarterEnd),
('BQ-JUN', pandas.tseries.offsets.BQuarterEnd),
('BQ-MAR', pandas.tseries.offsets.BQuarterEnd),
('BQ-MAY', pandas.tseries.offsets.BQuarterEnd),
('BQ-NOV', pandas.tseries.offsets.BQuarterEnd),
('BQ-OCT', pandas.tseries.offsets.BQuarterEnd),
('BQ-SEP', pandas.tseries.offsets.BQuarterEnd),
('BQS', pandas.tseries.offsets.BQuarterBegin),
('BQS-APR', pandas.tseries.offsets.BQuarterBegin),
('BQS-AUG', pandas.tseries.offsets.BQuarterBegin),
('BQS-DEC', pandas.tseries.offsets.BQuarterBegin),
('BQS-FEB', pandas.tseries.offsets.BQuarterBegin),
('BQS-JAN', pandas.tseries.offsets.BQuarterBegin),
('BQS-JUL', pandas.tseries.offsets.BQuarterBegin),
('BQS-JUN', pandas.tseries.offsets.BQuarterBegin),
('BQS-MAR', pandas.tseries.offsets.BQuarterBegin),
('BQS-MAY', pandas.tseries.offsets.BQuarterBegin),
('BQS-NOV', pandas.tseries.offsets.BQuarterBegin),
('BQS-OCT', pandas.tseries.offsets.BQuarterBegin),
('BQS-SEP', pandas.tseries.offsets.BQuarterBegin),
('Q-APR', pandas.tseries.offsets.QuarterEnd),
('Q-AUG', pandas.tseries.offsets.QuarterEnd),
('Q-DEC', pandas.tseries.offsets.QuarterEnd),
('Q-FEB', pandas.tseries.offsets.QuarterEnd),
('Q-JAN', pandas.tseries.offsets.QuarterEnd),
('Q-JUL', pandas.tseries.offsets.QuarterEnd),
('Q-JUN', pandas.tseries.offsets.QuarterEnd),
('Q-MAR', pandas.tseries.offsets.QuarterEnd),
('Q-MAY', pandas.tseries.offsets.QuarterEnd),
('Q-NOV', pandas.tseries.offsets.QuarterEnd),
('Q-OCT', pandas.tseries.offsets.QuarterEnd),
('Q-SEP', pandas.tseries.offsets.QuarterEnd),
('QS', pandas.tseries.offsets.QuarterBegin),
('QS-APR', pandas.tseries.offsets.QuarterBegin),
('QS-AUG', pandas.tseries.offsets.QuarterBegin),
('QS-DEC', pandas.tseries.offsets.QuarterBegin),
('QS-FEB', pandas.tseries.offsets.QuarterBegin),
('QS-JAN', pandas.tseries.offsets.QuarterBegin),
('QS-JUL', pandas.tseries.offsets.QuarterBegin),
('QS-JUN', pandas.tseries.offsets.QuarterBegin),
('QS-MAR', pandas.tseries.offsets.QuarterBegin),
('QS-MAY', pandas.tseries.offsets.QuarterBegin),
('QS-NOV', pandas.tseries.offsets.QuarterBegin),
('QS-OCT', pandas.tseries.offsets.QuarterBegin),
('QS-SEP', pandas.tseries.offsets.QuarterBegin),
('S', pandas.tseries.offsets.Second),
('T', pandas.tseries.offsets.Minute),
('U', pandas.tseries.offsets.Micro),
('W-FRI', pandas.tseries.offsets.Week),
('W-MON', pandas.tseries.offsets.Week),
('W-SAT', pandas.tseries.offsets.Week),
('W-SUN', pandas.tseries.offsets.Week),
('W-THU', pandas.tseries.offsets.Week),
('W-TUE', pandas.tseries.offsets.Week),
('W-WED', pandas.tseries.offsets.Week),
('WOM-1FRI', pandas.tseries.offsets.WeekOfMonth),
('WOM-1MON', pandas.tseries.offsets.WeekOfMonth),
('WOM-1THU', pandas.tseries.offsets.WeekOfMonth),
('WOM-1TUE', pandas.tseries.offsets.WeekOfMonth),
('WOM-1WED', pandas.tseries.offsets.WeekOfMonth),
('WOM-2FRI', pandas.tseries.offsets.WeekOfMonth),
('WOM-2MON', pandas.tseries.offsets.WeekOfMonth),
('WOM-2THU', pandas.tseries.offsets.WeekOfMonth),
('WOM-2TUE', pandas.tseries.offsets.WeekOfMonth),
('WOM-2WED', pandas.tseries.offsets.WeekOfMonth),
('WOM-3FRI', pandas.tseries.offsets.WeekOfMonth),
('WOM-3MON', pandas.tseries.offsets.WeekOfMonth),
('WOM-3THU', pandas.tseries.offsets.WeekOfMonth),
('WOM-3TUE', pandas.tseries.offsets.WeekOfMonth),
('WOM-3WED', pandas.tseries.offsets.WeekOfMonth),
('WOM-4FRI', pandas.tseries.offsets.WeekOfMonth),
('WOM-4MON', pandas.tseries.offsets.WeekOfMonth),
('WOM-4THU', pandas.tseries.offsets.WeekOfMonth),
('WOM-4TUE', pandas.tseries.offsets.WeekOfMonth),
('WOM-4WED', pandas.tseries.offsets.WeekOfMonth)]


Labels:

Comments:
@wesm @jtratner This might be a good time to answer: #5015

@jtratner I wonder if this is anything worth cherry picking from #5148

I'll make it `_named`, that seems reasonable.

Looking through, the rule code is pretty clear, it's `prefix-OffsetSpecificEnding`, where prefix is something like `B`, `Q`, etc. and the OffsetSpecificEnding is something like month, day, etc.  There are a bunch of aliases for backwards compatibility and that's about it.

Looking at the regex you can see there are a few more rules: for example numbers are allowed only right after the dash.

Yeah, but that's really just for validation: it covers the set of rules that already exist (as well as the legacy rules from scikits.timeseries, etc). (since you only end up with #s in the string for WOM) If we wanted to add some crazy additional rule_code, you'd change the regex to make it pass the validation.

I agree that we can always change the regex since the regex is just used for validation and not for parsing. We just want to make sure that we can express the parameterization for a given offset in a reasonable manner.

Places where rule_code actually matters:
- is_subperiod/is_superperiod (I guess different WOM can't downsample to each other)
- tseries/tools - where it checks that rule_code == 'M' (so would be fine with static rule_code)
- Timegrouper - which actually has to hack around rule_code to remove the suffix and just check for the prefix
- generic - used for an error message
- `_get_freq` in tseries/plotting - used to get period alias [probably could just have offsets know their period aliases)

Okay, I have to go, but I think it's simple: you split on '-', first element needs to define class for lookup, you can have whatever else you want can go afterwards (and it's up to the dateoffset to handle it)

Agreed.

I would like to cleanup the is_subperiod/is_superperiod eventually. One of the nice features of the 52-53 week fiscal years is that by design they upsample to weeks.

actually not sure whether we should deprecate rule_code, just want to work around it.

@cancan101 good suggestion about adding a prefix attribute, ended up meaning that I could remove most of the special rule_codes and simplify things! (I ended up leaving comments with the actual prefixes just for clarity :smile:)

@jtratner An additional location where rule_code matters is `_FrequencyInferer`. See for example `get_freq`, `_get_wom_rule`, `_get_monthly_rule`.

okay well this doesn't change rule_code.

That may be the case but I would be in favor of at the very least moving these over to using constants (along lines of https://github.com/pydata/pandas/pull/5189#discussion_r6932636) if not writing this code to check against a DateOffset object rather than the freq_str / rule_code.

I want to push that to a separate refactor. This keeps the behavior relatively the same and will enable you to merge your 52-53 week PR.

That's totally fine. I was just going through looking at the upsample/downsample code and was reminded of those string literals...

frankly, you probably have a better sense of this than I, just figured I could simplify a bit for you first, make sure we have a working framework, then it's easier to make more advanced tweaks later.

Okay, I think this is in a pretty good place at this point. Thanks for the feedback @cancan101! Any other comments? @wesm - does this match what you were thinking?

@jtratner NP. 

@jreback and others - okay?

ok by me

okay, going to merge this soon? any other comments?

Is there a vbench for pandas load time? 

No, I don't think so (hard to time), but this is mostly to make your PR
simpler and easier to support.

@jtratner I'm totally fine with commit. I was just curious as to start up gain. 

It's a good point, I've definitely thought about it too. Not sure how you'd
test within Python, don't know how you could unload it to load again.
Memory footprint might be a better metric, but even that isn't so high.

Maybe something from:
http://stackoverflow.com/questions/437589/how-do-i-unload-reload-a-python-module
On Oct 14, 2013 9:34 PM, "Jeff Tratner" notifications@github.com wrote:

> It's a good point, I've definitely thought about it too. Not sure how you'd
> test within Python, don't know how you could unload it to load again.
> Memory footprint might be a better metric, but even that isn't so high.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/5189#issuecomment-26303269
> .

Go for it - I'd certainly be interested to see the result.