Offsetting the Model — Logic to Implementation | by Ajay Tiwari | Towards Data Science

In our property and casualty insurance world very often we use a term called ‘offset’ which is widely used for modeling rate (count/exposure) such as the number of claims per exposure unit. This helps the model to transform the response variable from rate to count keeping coefficient as 1 by using simple algebra.
This term may confuse the beginners, I had this question when I first introduced to this approach. Why can’t we manually derive a new response variable by simply dividing the count by its respective exposure instead of using an offset term?
I will walk you through the log-likelihood functions of both versions, with and without offset to show the difference and highlight any discrepancies followed in the approach.
One important point to be noted, offset should be included on the same “scale” as the linear predictor. In the case of a log link model, this requires the offset variable to be logged before inclusion in the model.
Here is the regression model in its original form with an offset variable (Si), response variable Y, xi as independent variables and a set of parameters θ that can be estimated by maximum likelihood. This is achieved by just multiplying the exposure variable to the original form.
Further, we can simplify this equation by taking natural logarithm both sides
Now, this equation can be rewritten as a likelihood function in terms of θ as below
Finally, we arrive at the log-likelihood equation of the first version (with offset), only terms dependent on θ will remain in the equation, other terms gets dropped out during differentiation:
Let’s take a look at the other case when we are modeling rate (Yi/si) as the response variable, here is our new model equation.
Again, we arrive at the same equation as we had in the first case.
So, what is the difference? Both equations look the same.
In the first case, we are assuming yi (count response) following a Poisson distribution, whereas in the second case we are assuming (yi/si)has a Poisson distribution, this second assumption is not appropriate which makes our model incorrect.
Let’s take a look at the new likelihood function:
The new equation after keeping only θ dependent terms during differentiation
Now, we can compare log-likelihood functions (1) and (2), and observe the difference, both the approach will produce different outcomes. Also, we have seen, the second equation is based on a wrong assumption about the Poisson distribution, data is no more integers. Therefore, we can conclude that using a transformed rate variable as the response and without an offset is an incorrect approach.
However, still, we have an option to correct the latter approach by applying a weight equivalent to exposure (si) to each observation. We can see the following equation is equivalent to equation (1).
Let’s understand the difference in the implementation through the following R code examples. Similarly, these can be implemented in other modeling packages as well, both approaches will give the same output.
#Implementation of equation (1) in Rpoi_r <- glm(numclaims ~ x1+x2+x3,data=train, family = "poisson", offset=log(exposure))#Implementation of equation (3) in Rpoi_r <- glm(numclaims/exposure ~ x1+x2+x3,data=train, family = "poisson", weight = exposure)
There are cases when we already know the effect of an explanatory variable and rather than estimating parameters β for this variable, it is better to use available information.
Another scenario, boosting a new model using an existing model output, e.g., you trained a model for 200 iterations, and later wanted to restart from there to finish another 300 iterations, instead of re-starting the process from scratch you can provide previous model output as base margin or offset.
This implementation is quite easy and straight forward in R, SAS, Emblem (insurance industry-specific GLM software from Willis Tower Watson) and in statsmodels (a statistical modeling counterpart of sci-kit learn in python), whereas it’s less popular in machine learning models. Instead, they use some function to set the base score before training a new model, this base score can be used from an existing model or any known effect of a variable.
As an alternative, this function of ML algorithms can help us in implementing offset while modeling rates. This implementation is possible in all the cases where Poisson, Gamma and Tweedie loss functions have been used with log-link.
Offset in the case of a GLM in R can be achieved using the offset() function, similar implementation is possible for pscl, GBM, and glmnet packages.
poi_r <- glm(numclaims ~ x1+x2+x3,data=train, family = "poisson", offset=log(exposure))
Offset in the case of a GLM in Python (statsmodels) can be achieved using the exposure() function, one important point to note here, this doesn’t require logged variable, the function itself will take care and log the variable.
poi_py = sm.GLM(y_train, X_train, exposure = df_train.exposure, family=sm.families.Poisson()).fit()
Offset in the case of XGBOOST in Python can be achieved using the set_base_margin() function, which requires a logged variable. In lightgbm, we can use set_init_score() before training.
#Convert training data into DMatrixdtrain_offset = xgb.DMatrix(data=X_train.iloc[:,2:29],label=y_train)#Before training the model set a base margin into DMatrix  dtrain_offset.set_base_margin(np.log(X_train['exposure']))
We discussed how Poisson regression can be used to model rate using an offset variable and how this functionality can be implemented in machine learning models. The examples of algorithms given here are not an exhaustive list, readers can explore other algorithms such as model-based decision trees and neural networks.
Thanks for reading! I hope you find this article informative.
xgboost.readthedocs.io
lightgbm.readthedocs.io
[3]Duncan Anderson and Team, 2007, A Practitioner’s Guide to Generalized Linear Models