Holiday calendars with Pandas. It’s very common to hear that a data... | by Francisco Gárate | Towards Data Science

It’s very common to hear that a data scientist is a statistician who works with Python on Mac. In part, that’s correct. However, anyone who has worked within the academic world knows that, for a statistician, cows are spherical and months have 365.25/12 days. Instead, in the real world, neither two cows are the same nor is one month equal to any other.
I hope this post adds value to all that extra work and time spent by those who work with data avoiding statistical simplifications, whether they are data scientists or whatever they are called.
Pandas is the de facto analysis tool for data management in Python. It allows you to work easily with numerical tables and time series. A useful feature in Pandas is to do date ranges easily with the pd.date_range() function, which includes the following parameters (exactly three must be specified):
start: Start of range. Left limit for generating dates
end: End of range. Right limit
periods: Number of periods to generate
freq: Frequency strings (e.g. D: calendar day frequency, M: month end frequency, Y: year end frequency) see list of Offset Aliases.
A few lines are enough:
         Date0  2019-01-311  2019-02-282  2019-03-313  2019-04-304  2019-05-315  2019-06-306  2019-07-317  2019-08-318  2019-09-309  2019-10-3110 2019-11-3011 2019-12-31
In certain areas, mainly financial and actuarial, it’s especially useful to manage date ranges where the days in the series correspond to the first or last business day of the month (for example, when projecting coupon payments or annuities).
To do this, the pd.date_range() function has different values for the frequency parameter. In the following example, ‘BM’ corresponds to BusinessMonthEnd (last working day of the month):
         Date0  2019-01-311  2019-02-282  2019-03-293  2019-04-304  2019-05-315  2019-06-286  2019-07-317  2019-08-308  2019-09-309  2019-10-3110 2019-11-2911 2019-12-31
However, if we want more accuracy, we must consider bank holidays (for example, if we calculate costs that depend on the exact days between coupon and coupon, 1 day off in 20 is a 5% error). Knowing the holidays within a period is especially useful when estimating human habits and behaviors (medical care, travels, etc.). In the insurance sector, these patterns can directly affect accounting reserves; for example, when calculating costs incurred but not reported (IBNR). In fact, in some insurance companies it’s common to slightly increase the claims ratio in leap years due to having one calendar day more than the rest.
In the following example we introduce a list of two bank holidays (February 28 and May 31) just to see how it works. We then calculate the days between payments if it occurs on the last working day of the month:
         Date  n_days0  2019-01-31     0.01  2019-02-27    27.02  2019-03-29    30.03  2019-04-30    32.04  2019-05-30    30.05  2019-06-28    29.06  2019-07-31    33.07  2019-08-30    30.08  2019-09-30    31.09  2019-10-31    31.010 2019-11-29    29.011 2019-12-31    32.0
As shown, 2019–02–28 and 2019–05–31 don’t appear in the payment calendar.
Finally, there is the option of writing rules for calculating public holidays (as these are usually carried over to the next working day).
In certain countries, such as the United States, there are laws (Uniform Monday Holiday Act of 1968), whose rules are included in Pandas, print(USFederalHolidayCalendar.rules) as an example for developing other calendars.
However, in many other countries there are no rules to determine the holidays of the year (as occurs in Europe), where the holidays are set by regulations each year. For holidays that occur on fixed dates (e.g. US Memorial Day or July 4th) an observance rule determines when that holiday is observed if it falls on a weekend or some other non-observed day. Defined observance rules are:
In that case, it’s possible to write the rules of the labor calendars using some common premises for calculating the bank holidays as days on Sunday are moved to the following Monday (observance=sunday_to_monday instruction). E.g.: 2018 Canada day (July 1st) fell on Sunday and got moved to July 2nd. These should be overwritten on the AbstractHolidayCalendar class to have the range apply to all calendar subclasses.
Later, I show the labor days in July 2018 to check how it works:
         Date0  2018-07-031  2018-07-052  2018-07-063  2018-07-094  2018-07-105  2018-07-116  2018-07-127  2018-07-138  2018-07-169  2018-07-1710 2018-07-1811 2018-07-1912 2018-07-2013 2018-07-2314 2018-07-2415 2018-07-2516 2018-07-2617 2018-07-2718 2018-07-3019 2018-07-31
You can see how Sunday, July 1st is replaced by Monday, July 2nd
Like this, once we have created our own national holidays calendar, we can use it to see the banks holidays in the coming years, for example 2020:
DatetimeIndex(['2019-01-01', '2019-01-07', '2019-03-18',                '2019-04-01', '2019-04-19', '2019-05-01',                '2019-07-01', '2019-07-04', '2019-11-01',                '2019-12-25'],              dtype='datetime64[ns]', freq=None)
Regional and local holidays can be included in the same way, although noted that in certain countries it’s usually the moveable fests (a religious holiday that is not on the same day every year). In that case, it becomes necessary to create a new function based on Easter() or sunday_to_monday() pandas function). For instance, Whit Monday is the 7th Monday after Easter in France.
Our custom calendar can be also used in other libraries such as Prophet. Prophet is an advanced machine learning library created by Facebook, focused on non-linear over time data regression models, which requires a list of holidays provided by the user.
For the creation of rules in the Jewish calendar, see these rules done by Ulrich and David Greve.
(En español, aquí)