ENH: Add SemiMonthEnd and SemiMonthBegin offsets #1543
State: closed

- [x] closes #1543
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Enhancement
name: Frequency
description: DateOffsets

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13315?src=pr) is **84.28%**

> Merging [#13315](https://codecov.io/gh/pydata/pandas/pull/13315?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **0.03%**

``` diff
@@             master     #13315   diff @@
==========================================
  Files           138        138          
  Lines         50810      50934   +124   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          42809      42931   +122   
- Misses         8001       8003     +2   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [07761c5...fe221b2](https://codecov.io/gh/pydata/pandas/compare/07761c519154a6ed8a9345512476152226499ba6...fe221b21a54ed2d4d533e2ce1cb6d4a11ceec477)

cc @sinhrks 

@jreback I have made the requested changes - I'll squash the new commit(s) once it's approved.

@jreback Any additional changes you see? Thanks.

there are some benchmarks for `.apply` in `asv_bench/benchmarks/timeseries`, pls add this new freq there as well.

Will do.

Added benchmarks and optimized `.apply` to not use inefficient looping.

benchmark results:

``` bash

[  0.00%] · For pandas commit hash e7d8257c:
[  0.00%] ·· Building for conda-py2.7-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt.......................................
[  0.00%] ·· Benchmarking conda-py2.7-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
...
[  0.01%] ··· Running timeseries.timeseries_semi_month_offset.time_semi_month_begin_apply   69.11μs
[  0.01%] ··· Running timeseries.timeseries_semi_month_offset.time_semi_month_begin_decr    77.51μs
[  0.01%] ··· Running timeseries.timeseries_semi_month_offset.time_semi_month_begin_decr_n  90.41μs
[  0.01%] ··· Running timeseries.timeseries_semi_month_offset.time_semi_month_begin_incr    79.20μs
[  0.01%] ··· Running timeseries.timeseries_semi_month_offset.time_semi_month_begin_incr_n  99.60μs
[  0.01%] ··· Running timeseries.timeseries_semi_month_offset.time_semi_month_end_apply     76.50μs
[  0.01%] ··· Running timeseries.timeseries_semi_month_offset.time_semi_month_end_decr      125.04μs
[  0.01%] ··· Running timeseries.timeseries_semi_month_offset.time_semi_month_end_decr_n    113.05μs
[  0.01%] ··· Running timeseries.timeseries_semi_month_offset.time_semi_month_end_incr      81.48μs
[  0.01%] ··· Running timeseries.timeseries_semi_month_offset.time_semi_month_end_incr_n    95.23μs
```

this looks good. can you see if you can add an `apply_index` (high-perf apply).

I am probably missing something obvious, but should `SemiMonthBegin` be the 1st & 16th of every month? That way, a 30 day month has an equal number of days in each semi-month?

(I'm ignoring February - but maybe that should be a day earlier...)

I think this is pretty standard to do semi-montly on 1/15th. @adrienemery thoughts?

@MaximilianR @jreback I've seen both personally but how about we make it flexible such that the 15th becomes a parameter. That way people can use it for their application as needed. I like keeping 15th as the default since it provides a nice default symmetry.

See latest commit for proposed change - I haven't added tests yet, but if we agree to go this route I will, otherwise will rollback that change.

I did test it in a notebook and it works nicely:

<img width="777" alt="screenshot 2016-06-01 13 45 42" src="https://cloud.githubusercontent.com/assets/4293818/15725121/3042a8c6-27ff-11e6-8fa1-38a49078a333.png">

I could also update `_from_name` such that it allows setting pivot day such as `SMS-16` 

``` python
    @classmethod
    def _from_name(cls, suffix=None):
        if not suffix:
            pivot = 15
        else:
            pivot = int(suffix)
        return cls(pivot=pivot)
```

<img width="734" alt="screenshot 2016-06-01 13 54 34" src="https://cloud.githubusercontent.com/assets/4293818/15725367/678cafc4-2800-11e6-9bdf-a5c288c94700.png">

yeah that's prob reasonable though need to add routines to parse SMS-16 then

look at the last week freq to see what the parameters are named (eg not pivot)

> look at the last week freq to see what the parameters are named (eg not pivot)

 I don't understand what you mean by this.

you are thinking of allowing `SMS-16`, so needs to parse this

```
In [2]: pd.tseries.frequencies.to_offset('1MS')
Out[2]: <MonthBegin>

In [3]: pd.tseries.frequencies.to_offset('2M')
Out[3]: <2 * MonthEnds>
```

- added `.apply_index` methods
- added flexible `day_of_month` parameter that defaults to the 15th
- added support to parse `day_of_month` from frequency string

I'd like to add a few more tests for `.apply_index` but wanted to get feedback on current change in case there are any issues/changes to be made.

Updated asv benchmark output with `.apply_index` included:

``` bash
[  0.01%] ··· Running timeseries.timeseries_semi_month_offset.time_semi_month_begin_apply           58.91μs
[  0.01%] ··· Running timeseries.timeseries_semi_month_offset.time_semi_month_begin_apply_index     49.61ms
[  0.01%] ··· Running timeseries.timeseries_semi_month_offset.time_semi_month_begin_decr            68.74μs
[  0.01%] ··· Running timeseries.timeseries_semi_month_offset.time_semi_month_begin_decr_n          80.07μs
[  0.01%] ··· Running timeseries.timeseries_semi_month_offset.time_semi_month_begin_decr_rng        70.12ms
[  0.01%] ··· Running timeseries.timeseries_semi_month_offset.time_semi_month_begin_incr            75.89μs
[  0.01%] ··· Running timeseries.timeseries_semi_month_offset.time_semi_month_begin_incr_n          73.19μs
[  0.01%] ··· Running timeseries.timeseries_semi_month_offset.time_semi_month_begin_incr_rng        56.28ms
[  0.01%] ··· Running timeseries.timeseries_semi_month_offset.time_semi_month_end_apply             73.79μs
[  0.01%] ··· Running timeseries.timeseries_semi_month_offset.time_semi_month_end_apply_index       57.93ms
[  0.01%] ··· Running timeseries.timeseries_semi_month_offset.time_semi_month_end_decr              77.41μs
[  0.01%] ··· Running timeseries.timeseries_semi_month_offset.time_semi_month_end_decr_n            83.09μs
[  0.01%] ··· Running timeseries.timeseries_semi_month_offset.time_semi_month_end_decr_rng          56.51ms
[  0.01%] ··· Running timeseries.timeseries_semi_month_offset.time_semi_month_end_incr              66.83μs
[  0.01%] ··· Running timeseries.timeseries_semi_month_offset.time_semi_month_end_incr_n            70.97μs
[  0.01%] ··· Running timeseries.timeseries_semi_month_offset.time_semi_month_end_incr_rng          55.86ms
```

@chris-b1 can you have a look.

few comments, looks good overall

I have addressed comments cc/ @jreback @chris-b1 

Looks good to me.  If you ever hit the wall on vectorized performance, it probably wouldn't be hard to modify or clone this [cython helper](https://github.com/pydata/pandas/blob/master/pandas/tslib.pyx#L4889) to handle your case, but definitely not necessary, what you're doing is pretty quick already.

- updated whatsnew
- added more tests (including failing cases)

@adrienemery looks real good. just some minor comments. ping when pushed and green.

@jreback should be good to go - I have addressed comments and build is green

@adrienemery thanks! nice PR. these have a lot of pieces for a full patch. Nice job!

give a check on the built docs (prob take a few hours), and if needs corrections, pls issue a followup PR.

@jreback thanks for your help on this PR - especially as this is my first time contributing to pandas. Also thanks for all your work on pandas 👍