PERF: Improve performance CustmBusinessDay - 2nd
State: closed

New attempt to close #8236

CustomBusinessDay is now fast with holiday calendar and incrementing in different ways. `busdaycalendar` is cached in self.kwds. 

```
In [15]:
date = pd.Timestamp('20120101')
cbday = pd.offsets.CustomBusinessDay()
date + cbday
%timeit date + cbday
100000 loops, best of 3: 14.9 µs per loop
In [16]:
hdays = [dt.datetime(2013,1,1) for ele in range(1000)]
cbdayh = pd.offsets.CustomBusinessDay(holidays=hdays)
cbme = pd.offsets.CustomBusinessMonthEnd(holidays=hdays)
cbmb = pd.offsets.CustomBusinessMonthBegin(holidays=hdays)
%timeit date + cbdayh
%timeit date + 2 * cbdayh
%timeit date - 1 * cbdayh
%timeit date - 10 * cbday
%timeit date + cbmb
%timeit date + 10 * cbmb
%timeit date - 10 * cbmb
%timeit date + cbme
%timeit date + 10 * cbme
%timeit date - 10 * cbme
```

100000 loops, best of 3: 14.8 µs per loop
10000 loops, best of 3: 20.1 µs per loop
10000 loops, best of 3: 24 µs per loop
10000 loops, best of 3: 23.7 µs per loop
10000 loops, best of 3: 108 µs per loop
10000 loops, best of 3: 104 µs per loop
10000 loops, best of 3: 117 µs per loop
10000 loops, best of 3: 118 µs per loop
10000 loops, best of 3: 188 µs per loop
1000 loops, best of 3: 147 µs per loop


Labels:
name: Performance
description: Memory or execution speed performance
name: Frequency
description: DateOffsets

Comments:
can u show the vbench results ?

I'm trying to run the vbench as follows: 

```
python test_perf.py -b 50bd7ca -t e6a4980 -r timeseries_custom
```

The repo gets cloned ok but I get empty results. I tried `timeseries_custom.*`. When I run `-r timeserries` I get results but the vbenches I added are missing. 

Any idea what I'm doing wrong? 

`./test_perf.sh -b master -t HEAD -r timeseries_custom`

`test_perf.sh` is a shell script

I marked for 0.15.1, but if you are finished in say next week or so can put in 0.15.0

I cannot test against master because the benchmarks are added in this PR. The first commit (50bd7ca) adds them. 

When I run (that's the same command the batch file calls)

```
python vb_suite/test_perf.py -b 50bd7ca -t e6a4980 -r timeseries
```

I get

```
*** Processing results...
***
Invoked with :
--ncalls: 3
--repeats: 3


-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
timeseries_large_lookup_value                |   0.0190 |   0.0440 |   0.4314 |
timeseries_slice_minutely                    |   0.0513 |   0.0723 |   0.7099 |
timeseries_1min_5min_ohlc                    |   0.6993 |   0.8721 |   0.8019 |
timeseries_to_datetime_YYYYMMDD              |   7.6191 |   8.9304 |   0.8532 |
timeseries_add_irregular                     |  17.5420 |  20.1940 |   0.8687 |
timeseries_asof_nan                          |   9.0024 |   9.7530 |   0.9230 |
timeseries_asof                              |   9.1070 |   9.7996 |   0.9293 |
timeseries_timestamp_downsample_mean         |   4.4064 |   4.7113 |   0.9353 |
timeseries_iter_periodindex                  | 1045.1464 | 1111.9016 |   0.9400 |
timeseries_sort_index                        |   8.4510 |   8.7680 |   0.9638 |
timeseries_iter_periodindex_preexit          |  10.6723 |  11.0544 |   0.9654 |
timeseries_infer_freq                        |   5.9141 |   6.0170 |   0.9829 |
timeseries_to_datetime_iso8601               |   4.2920 |   4.3263 |   0.9921 |
timeseries_1min_5min_mean                    |   0.6184 |   0.6216 |   0.9948 |
timeseries_period_downsample_mean            |  10.5550 |  10.5963 |   0.9961 |
timeseries_asof_single                       |   0.0269 |   0.0270 |   0.9971 |
timeseries_is_month_start                    |   2.4337 |   2.4350 |   0.9994 |
frame_assign_timeseries_index                |   0.6769 |   0.6613 |   1.0237 |
timeseries_iter_datetimeindex                | 595.7903 | 574.1064 |   1.0378 |
timeseries_iter_datetimeindex_preexit        |  10.9403 |  10.4793 |   1.0440 |
timeseries_timestamp_tzinfo_cons             |   0.0226 |   0.0123 |   1.8387 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [e6a4980] : Take out prints
Base   [50bd7ca] : add vbench
```

Somehow the new benchmarks are not included even though the base commit is the one where the benchmarks are included. 

IIRC. push u up your branch (1 commit or more), then the command I gave WILL pick up all of the benches (new or not)

Found the problem. There was a typo in the `setup` string which caused the whole set of benchmarks to be skipped. 

vbench against master

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
timeseries_custom_bday_cal_incr_neg_n        |   0.0250 |  11.5357 |   0.0022 |
timeseries_custom_bday_cal_decr              |   0.0247 |  10.8493 |   0.0023 |
timeseries_custom_bday_cal_incr              |   0.0214 |   5.4567 |   0.0039 |
timeseries_custom_bmonthbegin_incr_n         |   0.1256 |  20.6480 |   0.0061 |
timeseries_custom_bmonthbegin_decr_n         |   0.1147 |  17.3107 |   0.0066 |
timeseries_custom_bday_cal_incr_n            |   0.0374 |   5.4900 |   0.0068 |
timeseries_custom_bmonthend_decr_n           |   0.1403 |  12.0017 |   0.0117 |
timeseries_custom_bmonthend_incr_n           |   0.1294 |   5.6400 |   0.0229 |
timeseries_custom_bmonthend_incr             |   0.1146 |   0.1373 |   0.8345 |
timeseries_custom_bday_decr                  |   0.0210 |   0.0227 |   0.9231 |
timeseries_custom_bday_apply                 |   0.0147 |   0.0153 |   0.9585 |
timeseries_custom_bday_incr                  |   0.0157 |   0.0163 |   0.9610 |
timeseries_custom_bday_apply_dt64            |   0.0180 |   0.0184 |   0.9827 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
```

@jreback Let me know if you would like me to make any other changes for this. 

Btw, you referred to another [implementation](https://github.com/pydata/pandas/pull/8253#discussion_r17474873) of `_to_dt64` which I couldn't find. Could you point me to it? 

these need a pickle test

self.round_trip_pickle() (look in utils/testing/Testcase)

checking backward compat is a bit tougher

can you rebase and squash to a single commit. hard to see what is changing here.

Done

I implemented the changes you suggested: 

You proposed to take out `holidays`, `calendars` etc. out of `kwds`. The operators defined in `DateOffset` depend on this however: 

```
def __mul__(self, someInt):
    return self.__class__(n=someInt * self.n, normalize=self.normalize, **self.kwds)
```

Right now, I add them back into kwds to make sure the operators function. To fully take them out of kwds we could overwrite the operators in the `CustomBusinessDay`. 

Regarding the `cache_readonly` decorator. We could do something like this

```
@cache_readonly
def calendar(self,weekmask=None, holidays=None, calendar=None):
    # Compute calendar
```

and then in **init**

```
calendar = self.calendar(weekmask=weekmask, holidays=holidays, calendar=calendar)
```

I don't see any added value to the way it is implemented right now though. 

Did some code cleanup and squashed. Speed is still enhanced as described above. Hope this can make it in 0.15. 

Any comments? 

Added pickle tests. Not sure how to check compat to 0.14.1...

create a pickle in 0.14.1 Nd add like here: https://github.com/pydata/pandas/commit/fccd7feabfe0bf15c30d712f3a4a4ff71e76ad4a
read it in, create and assert equal to one created at run time

Added the compat check. Also ran the vbench again - unfortunately the performance has suffered a bit during the refactoring. 

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
timeseries_custom_bday_cal_incr_neg_n        |   0.1706 |  10.8820 |   0.0157 |
timeseries_custom_bday_cal_decr              |   0.1750 |  10.9700 |   0.0160 |
timeseries_custom_bday_cal_incr_n            |   0.0943 |   5.4903 |   0.0172 |
timeseries_custom_bday_cal_incr              |   0.0950 |   5.4743 |   0.0173 |
timeseries_custom_bmonthbegin_decr_n         |   0.3400 |  16.5097 |   0.0206 |
timeseries_custom_bmonthbegin_incr_n         |   0.3503 |  16.6264 |   0.0211 |
timeseries_custom_bmonthend_decr_n           |   0.2886 |  11.1147 |   0.0260 |
timeseries_custom_bmonthend_incr_n           |   0.2010 |   5.6076 |   0.0358 |
timeseries_custom_bday_decr                  |   0.0220 |   0.0230 |   0.9585 |
timeseries_custom_bday_incr                  |   0.0157 |   0.0163 |   0.9610 |
timeseries_custom_bday_apply_dt64            |   0.0177 |   0.0184 |   0.9654 |
timeseries_custom_bmonthend_incr             |   0.1307 |   0.1153 |   1.1330 |
timeseries_custom_bday_apply                 |   0.0183 |   0.0153 |   1.1917 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
```

Fixed performance

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
timeseries_custom_bday_cal_decr              |   0.0257 |  10.7663 |   0.0024 |
timeseries_custom_bday_cal_incr_neg_n        |   0.0264 |  10.7176 |   0.0025 |
timeseries_custom_bday_cal_incr              |   0.0260 |   5.3866 |   0.0048 |
timeseries_custom_bday_cal_incr_n            |   0.0266 |   5.3720 |   0.0050 |
timeseries_custom_bmonthbegin_incr_n         |   0.2057 |  16.2563 |   0.0127 |
timeseries_custom_bmonthbegin_decr_n         |   0.2673 |  16.2613 |   0.0164 |
timeseries_custom_bmonthend_decr_n           |   0.2924 |  10.9464 |   0.0267 |
timeseries_custom_bmonthend_incr_n           |   0.2050 |   5.5127 |   0.0372 |
timeseries_custom_bday_apply                 |   0.0157 |   0.0160 |   0.9801 |
timeseries_custom_bmonthend_incr             |   0.1113 |   0.1109 |   1.0036 |
timeseries_custom_bday_apply_dt64            |   0.0190 |   0.0180 |   1.0529 |
timeseries_custom_bday_decr                  |   0.0234 |   0.0220 |   1.0614 |
timeseries_custom_bday_incr                  |   0.0203 |   0.0157 |   1.2995 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
```

Do you ever run the tests for different Python versions locally (e.g. via tox)? Find it a bit cumbersome to debug via Travis output. 

rename the .pkl to .pickle and I think they will work (not being picked up correctly by the setup routines)

I think I have to fix the other . pkl as well

I agree the code needs refactoring. I'm not sure it is so easy though. I won't have time to do it this weekend so I opened https://github.com/pydata/pandas/issues/8458. 

To test for py26 and py27 you have two separate directories and virtualenvs? 

yep create a completely separate ve

But also two separate pandas clones (directories) because of the cython extensions, right?

yep they r completely separate

@bjonen ok, looks fine, is green, but lmk when done / ready to merge

Checked speed. Ready from my side

@bjonen thanks!