ENH: PeriodIndex can accept freq with mult
State: closed

Closes #7811.
- [x] Change `Period.freq` and `PeriodIndex.freq` to store offsets.
  - [x] Add `freqstr` to `Period`, `PeriodIndex` can use `DatetimeIndexOpsMixin`'s logic 
  - [x] Logic and tests for pickles taken in prev versions
- [x] Perform shift/arithmetic considering freq's mult. 
  - [x] Test all offsets has accessible `n` proeprties  (#10350)
- [x] Explicit tests for `asfreq` and `to_timestamp` using freq with mult.
- [x] Ops with different base freq must be prohibited. Change freq comparison to base comparison.
- [x] Fix partial string slicing bug (some tests are commented out because of this)
- [x] Fix order bug (#10295, removing temp logic for `PeriodIndex`)
- ~~Decide a policy for legacy freq aliases, like "WK"~~ (handled in #10878)
- [x] Update doc.


Labels:
name: Enhancement
name: Frequency
description: DateOffsets
name: Period
description: Period data type

Comments:
I think this might need an example in the docs as well http://pandas.pydata.org/pandas-docs/stable/timeseries.html#periodindex-and-period-range (you can use same example in release notes)

this is a bit odd though when you operate on these:

you are not propogating the given freq to the actual period?

```
In [10]: PeriodIndex(start='2014-01', freq='3M', periods=4).tolist()
Out[10]: 
[Period('2014-01', 'M'),
 Period('2014-04', 'M'),
 Period('2014-07', 'M'),
 Period('2014-10', 'M')]

In [11]: (PeriodIndex(start='2014-01', freq='3M', periods=4) + 1).tolist()
Out[11]: 
[Period('2014-02', 'M'),
 Period('2014-05', 'M'),
 Period('2014-08', 'M'),
 Period('2014-11', 'M')]
```

@sinhrks can you rebase

@sinhrks can you respond to my comments above?

@sinhrks ?

@sinhrks can you rebase / revisit

@jreback Indeed, the frequency given to `PeriodIndex` is not propagated to each `Period` within. Do you think it should be propagated? Your earlier response indicates that you do, but I'm checking in to verify.
This unwanted behavior is also shown using `period_range`:

``` python
In [11]: pd.period_range('2-1-2014 00:00', '2-1-2014 3:00', freq='30Min').tolist()

Out[11]:

[Period('2014-02-01 00:00', 'T'),
 Period('2014-02-01 00:30', 'T'),
 Period('2014-02-01 01:00', 'T'),
 Period('2014-02-01 01:30', 'T'),
 Period('2014-02-01 02:00', 'T'),
 Period('2014-02-01 02:30', 'T'),
 Period('2014-02-01 03:00', 'T')]
```

@blbradley yes that sounds right. When the periods are created during iteration (e.g. the `tolist()`) then the freq _should_ (and currently is passed).

It should also be true when this PR is implementd.

@jreback Thanks. I started trying to implement that. We'll see how it goes.

@jreback @blbradley OK. In #7811, I initially thought that created `PeriodIndex` doesn't need custom freq, but it is more natural. I'll consider the implementation.

`Period` would greatly benefit from being rewritten in Cython (probably as part of `tslib`) as to take advantage of `convert_to_tsobject`.

@blbradley this is true. But that's a whole separate project (though not THAT difficult). feel free!

@sinhrks I've been revisiting this. One problem is that `Period.asfreq` does not handle multiples. I think this could be handled by creating a `PeriodIndex` the same way as `DatetimeIndex` in `pandas.tseries.resample.asfreq`.

@sinhrks I would love to collaborate on this. Please respond soon cause I'm actively working on it.

@sinhrks ?
@blbradley interested in working on this one?

@jreback This year sometime! I did some code spikes towards it before, but they were not fruitful.

Maybe first change `Period` to take multiple freqs, then revisit this. I'll look into `Period` code.

@sinhrks can you rebase this. This would be a welcome enhancement.

Rebased. 

I've considered a little to make `Period` and `PeriodIndex` to have multiple freq, but not sure what the good behavior is.

```
import pandas as pd

pidx1 = pd.period_range('2011', freq='3A', periods=3)
pidx1
# PeriodIndex(['2011', '2014', '2017'], dtype='int64', freq='A-DEC')

# -> currently, above expr has the same as below
pd.PeriodIndex(['2011', '2014', '2017'], freq='A')

# internal representation is the same as standard frequency.
pidx1.asi8
# array([41, 44, 47])

# -> In case of multiple, it should also be continuous like array([k, k+1, k+2])?

# shift slides one freq. 
pidx1.shift(1)
# PeriodIndex(['2012', '2015', '2018'], dtype='int64', freq='A-DEC')

# -> It should slide "one multiple unit" (3 years in above case) like below?
# pidx1.shift(1)
# PeriodIndex(['2014', '2017', '2020'], dtype='int64', freq='A-DEC')
```

show the same example with timestamps/DatetimeIndex and see what it does, I _think_ that the freqs are  multiples themselves, e.g. `3A-DEC`

Correct. Let me organize `DatetimeIndex` (`TimedeltaIndex`) and current `PeriodIndex` behavior and `freq` definition differences, and would like to clarify some points.

#### Definitions
- `DatetimeIndex` and `TimedeltaIndex`: Use `freq` to indicate whether all the values are exactly on the frequency as descent order.
- `PeriodIndex`: Use `freq` to specify base frequency regardless of value order.

```
pd.DatetimeIndex(['2011-12-31', '2012-12-31', '2013-12-31'], freq='A')
# DatetimeIndex(['2011-12-31', '2012-12-31', '2013-12-31'], dtype='datetime64[ns]', freq='A-DEC', tz=None)

# Unable to instanciate DatetimeIndex in non-monotonic order
pd.DatetimeIndex(['2012-12-31', '2011-12-31', '2013-12-31'], freq='A')
# ValueError: Inferred frequency None from passed dates does notconform to passed frequency A-DEC

# But PeriodIndex can does
pd.PeriodIndex(['2012-12-31', '2011-12-31', '2013-12-31'], freq='A')
# PeriodIndex(['2011', '2013', '2012'], dtype='int64', freq='A-DEC')
```

#### Internal repr
- `DatetimeIndex` and `TimedeltaIndex`: `offsets` instance. 
- `PeriodIndex`: `str`. 

```
didx = pd.DatetimeIndex(['2011-12-31', '2013-12-31', '2015-12-31'], freq='2A')
type(didx.freq)
# pandas.tseries.offsets.YearEnd

pidx = pd.PeriodIndex(['2012-12-31', '2011-12-31', '2013-12-31'], freq='A')
type(pidx.freq)
# str
```

#### Shift / Scalar arithmetic ops
- `DatetimeIndex` and `TimedeltaIndex`: Shift by number of `mult`
- `PeriodIndex`: Shift by 1 (because `mult` is always 1).

```
# results in 2 years shift
didx + 1
DatetimeIndex(['2013-12-31', '2015-12-31', '2017-12-31'], dtype='datetime64[ns]', freq='2A-DEC', tz=None)
```

#### Other ops
- `DatetimeIndex` and `TimedeltaIndex`: Ops with different `base` freq results in no `freq`. Ops with different `mult` may preserve `freq` if possible.
- `PeriodIndex`: Ops with different `freq` is prohibited. 

```
didx2 = pd.DatetimeIndex(['2012-12-31', '2013-12-31', '2014-12-31'], freq='A')
didx.union(didx2)
# DatetimeIndex(['2011-12-31', '2012-12-31', '2013-12-31', '2014-12-31', '2015-12-31'],
#               dtype='datetime64[ns]', freq='A-DEC', tz=None)

pidx2 = pd.PeriodIndex(['2012-12-31', '2011-12-31', '2013-12-31'], freq='D')
pidx.union(pidx2)
# ValueError: Only like-indexed PeriodIndexes compatible for join (for now)
```

### Required changes for `PeriodIndex`

I think following should be basic rules?
- Internal repr: Change `PeriodIndex.freq` to store `offsets`. Pickle needs to be cared.
- Shift / Arithmetic Ops: Perform shift/arithmetic considering with `mult`
- Other ops: Ops with different `base` is prohibited. Always preserve `base` freq, and preserve `mult` if possible. 

OK, this should almost work now. I'd like to clarify the freq defition of `PeriodIndex`.

As described above, `DatetimeIndex.freq` guarantees each timestamps are exactly on the freq.

```
import pandas as pd
pd.DatetimeIndex(['2011-12-31', '2013-12-31', '2015-12-31'], freq='2A')
# DatetimeIndex(['2011-12-31', '2013-12-31', '2015-12-31'], dtype='datetime64[ns]', freq='2A-DEC', tz=None)

pd.DatetimeIndex(['2011-12-31', '2012-12-31', '2013-12-31'], freq='2A')
# ValueError: Inferred frequency A-DEC from passed dates does not conform to passed frequency 2A-
```

Otherwise, `PeriodIndex.freq` only defines the base frequency of the `Period`. Thus, following is valid.

```
pd.PeriodIndex(['2011', '2011', '2011'], freq='A')
# PeriodIndex(['2011', '2011', '2011'], dtype='int64', freq='A-DEC')
```

The point is whether `PeriodIndex` using freq with mult should guarantee the values are on the freq (like `DatetimeIndex`) or not.

### 1. `PeriodIndex` construction using freq with mult

Is this valid, or raise `ValueError` because values are not exactly on the freq with mult?

```
# current impl allows following construction.
pd.PeriodIndex(['2011', '2011', '2011'], freq='2A')
# PeriodIndex(['2011', '2011', '2011'], dtype='int64', freq='2A-DEC')
```

### 2. `PeriodIndex` slicing using freq with mult.

`DatetimeIndex` multiplies its freq via slicing with step.

```
pd.DatetimeIndex(['2011-12-31', '2012-12-31', '2013-12-31'], freq='A')[::2]
# DatetimeIndex(['2011-12-31', '2013-12-31'], dtype='datetime64[ns]', freq='2A-DEC', tz=None)
```

The same rule should be applied to `PeriodIndex`. The below case must be valid because index is monotonic.

```
# multiply freq if monotonic
pd.PeriodIndex(['2011', '2012', '2013'], freq='A')[::2]
# PeriodIndex(['2011', '2013'], dtype='int64', freq='2A-DEC')
```

In non-monotonic case, not sure whether the freq should be multiplied, or keep base freq ("A-DEC" in below example).

```
# current impl always multiplies freq.
pd.PeriodIndex(['2011', '2011', '2011'], freq='A')[::2]
# PeriodIndex(['2011', '2011'], dtype='int64', freq='2A-DEC')
```

Others will have more contextual view, but here's my $0.02, as a heavy user of `PeriodIndex`:

A `Period` is a span of time; its `freq` represents the length of that span. It exists independently of what else is in the Index (unlike `DatetimeIndex`, where the `freq` is a 'commentary' on how far the points are apart).

So
- This should be explicit in the `Period` (like it is now for quarters, as below `2011Q1`). How does the 2A example work? Is that 2011-2012? Should it clearly display that?
- I don't think you want to do any changing of the `freq` by just selecting things from an index (i.e. neither of the last two examples), because you'd be transforming the objects inside the index. Particularly the last example, or, even more extreme, if you had `pd.PeriodIndex(['2000', '2005', '2006'], freq='A')[::2]`, how would that work? 

Overall this is awesome and I'm excited for this to be in.

---

```
In [61]:
pd.Period('2011', freq='Q')
Out[61]:
Period('2011Q1', 'Q-DEC') 
```

> its freq represents the length of that span

Thanks. Based on the definition, it must behave like below. I'll update PR if there is no other suggestions within few days.

```
# it is valid, each elements represents 2 years span starting from 2011 
pd.PeriodIndex(['2011', '2011', '2011'], freq='2A')
# PeriodIndex(['2011', '2011', '2011'], dtype='int64', freq='2A-DEC')

# slicing must not change freq
pd.PeriodIndex(['2011', '2012', '2013'], freq='A')[::2]
# PeriodIndex(['2011', '2013'], dtype='int64', freq='A-DEC')

# ops with different freq is meaningless. must be prohibited 
pd.PeriodIndex(['2011', '2011', '2011'], freq='2A').union(pd.PeriodIndex(['2011', '2011', '2011'], freq='A'))
# ValueError
```

And current repl is like below:

```
pd.Period('2011', freq='2Q')
# Period('2011Q1', '2Q-DEC')
```

That's superb!

My point on the `2011Q1` example was that the definitions of the quarterly Periods are currently clear, and I wonder if the `mult` periods could be made clearer, either in the repr or the docs. 

For example, what does `pd.Period('2011', freq='2A')` represent? Is it the period from the start of 2011 until the end of 2012? 
And then, and I don't have a strong view here, should these be represented explicitly?

```
pd.Period('2011', freq='2Q') <- the period from Jan-Jun 2011 inclusive
a) # Period('2011Q1', '2Q-DEC')
b) # Period('2011Q1-Q2', '2Q-DEC')
c) # Period('2011Q1-2011Q2', '2Q-DEC')
```

```
pd.Period('2011', freq='2A') <- the period beginning in 2011, and ending in December two years later?
a) # Period('2011', '2A-DEC') 
b) # Period('2011-2012', '2A-DEC')
```

More generally, it seems like the general rule for these is 'the earliest possible valid period that starts within the date supplied'. So the first example is the first quarter of 2011, the second is the two years from the start of 2011. And `Period('2011', '2Q-NOV')` would be the Mar-May 2011 inclusive. 
Is that right? 

@maximilianr comments are reasonable. Yes the freq is a fundamental part of the `Period`.

When I get around to making a `period` dtype, it WILL have the `freq` as a parameter (like I am doing with the `tz` for a datetimes, e.g. `datetime64[ns, US/Eastern]` (the `ns` is a parameter as well, though not actually used ATM).

Notice that `freq` is NOT part of the dtype, rather its a property of the index.

So I would be onboard with having `2M` as the actual freq (stored internally prob as `base_freq` and `span` maybe)

However a `PeriodIndex` is just a collection of `Periods` that happen to have the same frequency. Yes a `DatetimeIndex` with a freq has this same property.

Note that `PeriodIndex` is really just a time-based version of `IntervalIndex` (and maybe we can eventually change the impl to actually do that). cc @shoyer ....

So a period is a span, constructed from the beginning of the period. When converted it can actually convert to either the start (default) or end point.

```
In [22]: p = pd.Period('2011', freq='A')

In [23]: p
Out[23]: Period('2011', 'A-DEC')

In [24]: p.to_timestamp(how='end')
Out[24]: Timestamp('2011-12-31 00:00:00')

In [25]: p.to_timestamp(how='start')
Out[25]: Timestamp('2011-01-01 00:00:00')
```

@sinhrks hows this coming

Updated based on above definition, and now ready for review. Tests are locally passed.

It may better to be a separate PR, can legacy freq be deprecated?

@sinhrks what's the legacy freq issue?