When and How to use Weighted Least Squares (WLS) Models | by Jonathan Balaban | Towards Data Science

At Metis, one of the first machine learning models I teach is the Plain Jane Ordinary Least Squares (OLS) model that most everyone learns in high school. Excel has a way of removing the charm from OLS modeling; students often assume there’s a scatterplot, some magic math that draws a best fit line, then an r2 in the corner that we’d like to get close to 1. Truth is, there’s so much more than meets the eye with OLS, and after about a week, students are crying for mercy (disclaimer: no students are actually harmed!) as we plunge into the depths of a domain that initially seemed so simple.
With this widely-applicable model understood, a natural response is to use OLS for anything and everything. Which isn’t a terrible idea: although OLS requires four — some say five or six — assumptions to be met with raw or “massaged” data, the modeling paradigm is quite robust and can often perform well, as long as we regularize and use proper complexity, log transforms, intercepts, etc.
However, OLS is only one of a distinguished family tree:
Weighted Least Squares (WLS) is the quiet Squares cousin, but she has a unique bag of tricks that aligns perfectly with certain datasets!
Another of my students’ favorite terms — and commonly featured during “Data Science Hangman” or other happy hour festivities — is heteroskedasticity. Coming from the ancient Greek hetero, meaning “different”, and skedasis, meaning “dispersion”, it can also be found in the anglicized “Heteroscedasticity” (notice the additional ‘c’) form. In a nutshell, data that is heteroskedastic has variability that changes as a function of the inputs.
The truth of the matter is, lots of data exhibits this “Heteroskedasticity”. Draw up some example feature-response relationships and we can often intuitively explain why:
As age increases, net worths tend to diverge
As company size increases, revenues tend to diverge
Or, as infant height increases, weight tends to diverge
One of OLS’ major assumptions is that the data — and therefore, the residuals — are homeskedastic. Uh-oh! Well, the good news is that OLS can handle a certain level of heteroskedasticity. Search online and you might find different rules-of-thumb, like “the highest variability shouldn’t be greater than four times that of the smallest”. There are also a number of tests to statistically determine the scale of your problem.
Fortunately, OLS’ assumptions are not black and white, binary enforcements. There’s a gray area where the model still works rather well.
“But what if I have terrible — over 4x heteroskedasticity — regression, master?”
“Then we shall turn to WLS, young Padawan!”
Let’s take a look at how WLS is implemented in one of my favorite machine learning environments, scikit-learn.
Let’s generate some fake data:
import numpy as npimport pandas as pdimport seaborn as snsimport statsmodels.api as sm%matplotlib inline# generate random datanp.random.seed(24)x = np.random.uniform(-5,5,25)ε = 2*np.random.randn(25)y = 2*x+ε
# alternate error as a function of xε2 = ε*(x+5)y2 = 2*x+ε2sns.regplot(x,y);sns.regplot(x,y2);
Notice that the sets come from the same ground truth function, but the increasing variance as a function of x causes the orange model to fit a line different than the blue. In another random draw, the slope may be lower than blue, but will be more volatile in general.
# add a strong outlier for high xx_high = np.append(x,5)y_high = np.append(y2,160)# add a strong outlier for low xx_low = np.append(x,-4)y_low = np.append(y2,160)
The first append above mimics a common scenario where an area of high variance (expectedly) sees an extreme observation. This will affect OLS more than WLS, as WLS will de-weight the variance and its“penalty”.
To calculate sample weights, remember that the errors we added varied as a function of (x+5); we can use this to inversely weight the values. As long as the relative weights are consistent, an absolute benchmark isn’t needed.
# calculate weights for sets with low and high outliersample_weights_low = [1/(x+5) for x in x_low]sample_weights_high = [1/(x+5) for x in x_high]
# reshape for compatibilityX_low = x_low.reshape(-1, 1)X_high = x_high.reshape(-1, 1)---------# import and fit an OLS model, check coefficientsfrom sklearn.linear_model import LinearRegressionmodel = LinearRegression()model.fit(X_low, ymod)# fit WLS using sample_weightsWLS = LinearRegression()WLS.fit(X_low, ymod, sample_weight=sample_weights_low)print(model.intercept_, model.coef_)print('WLS')print(WLS.intercept_, WLS.coef_)# run this yourself, don't trust every result you see online =)
Notice how the slope in WLS is MORE affected by the low outlier, as it should. The low region should have low variability, so the outlier is magnified above what OLS does, pushing the slope more negative. Let’s see below how the high outlier is suppressed in WLS.
model = LinearRegression()model.fit(X_high, ymod)WLS.fit(X_high, ymod, sample_weight=sample_weights_high)print(model.intercept_, model.coef_)print('WLS')print(WLS.intercept_, WLS.coef_)
You’ll notice how outliers in areas where variance is expected are reduced in impact on the parameter estimation. Remember, use WLS when outliers are not all considered equal!
There you have it! Implementing WLS can be somewhat tricky;sklearn doesn’t have a distinct WLS model because the argument functionality (that’s also used in Decision Trees and other models) secretly supports our needs.
This was a basic intro to WLS, and there’s plenty more in this space to explore, including the promising Huber-White ‘sandwich’ estimator approach.
And finally, here’s a weighting approach recommended in the book Introduction to Linear Regression Analysis by Douglas C. Montgomery, Elizabeth A. Peck, and G. Geoffrey Vining. For example:
Always seek to use experience or prior information when modeling.Using residuals of the model — for example if var(εi)=σ2x_i*var(εi)=σ2x_i — then we may decide to use w_i=1/x_i.If the responses are the average of n observation, something like var(y_i)=var(ε_i)=σ2/n_i*var(y_i)=var(ε_i)=σ2/n_i, then we may decide to use w_i=n_i.Sometime we know that different observations have been measured by different instruments that have some (known or estimated) accuracy. In this case we may decide to use weights as inversely proportional to the variance of measurement errors.
Always seek to use experience or prior information when modeling.
Using residuals of the model — for example if var(εi)=σ2x_i*var(εi)=σ2x_i — then we may decide to use w_i=1/x_i.
If the responses are the average of n observation, something like var(y_i)=var(ε_i)=σ2/n_i*var(y_i)=var(ε_i)=σ2/n_i, then we may decide to use w_i=n_i.
Sometime we know that different observations have been measured by different instruments that have some (known or estimated) accuracy. In this case we may decide to use weights as inversely proportional to the variance of measurement errors.
As with most data science endeavors, your approach must be flexible to the type of data you have.
Happy modeling! And as always, thanks for reading, connecting, and sharing!