Error-in-Variables Models: Deming Regression | by Dr. Robert Kübler | Towards Data Science

Machine learning is often all about the following question:
Given a dataset (X, y), where X is a feature matrix and y is a target vector, find an f with f(X) ≈ y.
We usually do not enforce a strict equality à la f(X) = y because there are errors in the target values y. These errors arise from the fact that we usually cannot observe everything in the universe and put it into our feature matrix X. And even if we could, quantum mechanics tells us that there might still be randomness left in the system, making it possible for us to receive different outcomes given the same inputs.
Side note: This indeterminism makes it possible to design algorithms for quantum computers that allow us to factor giant numbers into a product of prime numbers — for example, 900 = 2 * 32 * 5, just much larger — to break important encryption schemes such as RSA which are used everywhere on the internet nowadays. It also allows us to search an unsorted array of length n in time √n, which is a rather surprising result achieved with Grover’s algorithm.
Long story short, labels are noisy, and we deal with this by introducing an error term ɛ, such as in y = a*x + b + ɛ in the case of linear regression. Nothing new so far. But answer me the following question:
Have you ever thought of errors in the feature variables X?
I certainly didn’t. So let me show you when this might be important.
I’m a fan of classic, unremarkable examples. They don’t distract you and let you focus on new things, step by step. So, let’s use the following one: Predict the height of a person, given their weight! Never done this one before, right? 😉
Your goal is to find the relationship between weight and height. If a person is 1 kg heavier than another person, what can you say about their heights? You want to use a linear model, so basically the task is to compute the slope of the line.
The story: You invited 500 people to your innovative study and took their weights and heights.
The twist: After the experiment is over, you noticed that your scale is quite inaccurate — after repeatedly weighing yourself, you got a lot of numbers that are all over the place, although their mean is at least around your real weight. And this is the same for the other 500 weights, too. Oops. How to deal with this situation? Re-running the whole experiment is definitely not an option.
We start by generating some data that you could have obtained running the experiment: First, we generate the real weights x_true and derive heights y from them. Then, we add some noise to the weights x_true and call the result x.
Note: I use x instead of X, because we only have a single feature here.
import numpy as npnp.random.seed(0)n = 500x_true = 10*np.random.randn(n) + 70 # the true, unobserved weightsy = x_true + 100 + 5*np.random.randn(n) # the observed heightsx = x_true + 10*np.random.randn(n) # the noisy, observed weights
In real life, we are only given x and y to work with, so this is what we see:
If we would not know about the broken scale — or simply don’t care — we could apply a linear regression model to the noisy x and y, for example like this:
from sklearn.linear_model import LinearRegressionlr = LinearRegression()lr.fit(x.reshape(-1, 1), y)print(f'height = {lr.coef_[0]:.2f}*weight + {lr.intercept_:.2f}')# OUTPUT: height = 0.50*weight + 134.09
Well, it looks like a reasonable result: the higher the weight, the higher the height. Sanity check passed. However, look back at how we generated the height values:
y = x_true + 100 + 5*np.random.randn(n)
The real coefficient is actually 1, the intercept is 100, which is a far cry from the coefficient 0.5 and intercept 134 we got from running linear regression. If this is a problem depends on you, though.
If you continue using your broken scale and you just want to predict, the current model height = 0.50*weight + 134.09 is the way to go since it's perfectly adapted to this setting. You just trained it that way.
On the other hand, if you want to learn the true relationship between height and true weight, you have to be smarter than that. The problem is that your slope is too small, which is also known as regression dilution or regression attenuation. We will now see how to solve this problem.
There are several ways to retrieve the correct coefficients even with dirty data. One of the simplest ones is the so-called Deming regression, a variant of ordinary least squares to account for the error.
Deming regression is named after Dr. William Edwards Deming who, by the way, didn’t even invent this method, but made it popular. Credits also go to the inventors R. J. Adcock and C. H. Kummell.
Before we start, let us go quickly over some notations. You have probably seen these things already, but let’s get everybody on the same page here. We assume that x=(x1, x2, x3, ..., xn) and y=(y1, y2, y3, ..., yn) is our dataset. Then, we define the following:
Before we come to the Deming regression formula, let me stress that with these notations we can also recover our ordinary least squares estimator coefficient again via
which in Python translates to
s_xy = ((x - x.mean()) * (y - y.mean())).sum() s_xx = ((x - x.mean())**2).sum()print(f'slope = {s_xy / s_xx:.2f}')# Output: slope = 0.50
You can also calculate the intercept from this, which I omit here. We know already from sklearn that it is 134.09.
Now, let’s do Deming, Adcock, and Kummell some justice and present the adjusted slope and intercept:
Alright, looks slightly more complicated, but still manageable. But what is this δ in the formula? Sadly, this is the main disadvantage of this approach. It’s nothing that we can compute, we just have to know or guess it.
δ is the ratio between the variance of the errors in y divided by the variance of the errors in x. I mean the following: the regression formula y = a*x + b + ɛ, contains a normally distributed random variable ɛ, so we assume. It has some variance σγ. This is what I mean with “variance of the errors in y”.
We further assume that our observed weights x are also obtained by adding Gaussian noise to the real weights x_true, which also has some variance, the “variance of the errors in x”.
In our example, we are given everything. Let me paste the formulas that we used to create the dataset:
x_true = 10*np.random.randn(n) + 70y = x_true + 100 + 5*np.random.randn(n)x = x_true + 10*np.random.randn(n)
Here, we can see that the variance of the errors in y is exactly 52 = 25. The variance of the errors in x is 102 = 100, thus δ = 25/100 = 1/4. Plugging everything in gives us
x_mean = x.mean()y_mean = y.mean()s_xy = ((x - x.mean()) * (y - y.mean())).sum() s_xx = ((x - x.mean())**2).sum()s_yy = ((y - y.mean())**2).sum()delta = 1/4deming_slope = (s_yy - delta*s_xx + np.sqrt((s_yy - delta*s_xx)**2 + 4*delta*s_xy**2)) / (2*s_xy)deming_intercept = y_mean - x_mean*deming_slopeprint(f'Deming slope = {deming_slope:.2f}')print(f'Deming intercept = {deming_intercept:.2f}')# Output: Deming slope = 1.03, Deming intercept = 97.01
This is close enough to the true values 1 and 100! Sure, it’s not 100% accurate, but that is the usual deal with maximum likelihood estimates. We can visualize the results:
We can again see that the regression line trained on the noisy data is too flat — the so-called regression dilution.
I said earlier that Deming regression is a variant of simple linear regression with one variable. To be more precise, it is even a generalization of linear regression. What I mean is the following:
If there are no errors in the features x, Deming regression and simple linear regression yield the same result.
For me, this is rather intuitive, but looking at the formulas, it’s difficult to see. I will not prove it here, but I try to make you believe.
If x has no errors, the variance of the errors in x is zero, implying that there is actually no δ that we could compute, as we would divide by zero. But let us naively say that δ is infinity in this case because for s>0 we have
or for the non-mathematicians:
A positive number divided by an extremely tiny positive number is a giant number.
Since we can’t plug in infinity directly, we use a poor man’s trick and simulate it by setting δ to a huge number.
x_mean = x.mean()y_mean = y.mean()s_xy = ((x - x.mean()) * (y - y.mean())).sum() s_xx = ((x - x.mean())**2).sum()s_yy = ((y - y.mean())**2).sum()delta = 99999deming_slope = (s_yy - delta*s_xx + np.sqrt((s_yy - delta*s_xx)**2 + 4*delta*s_xy**2)) / (2*s_xy)deming_intercept = y_mean - x_mean*deming_slopeprint(f'Deming slope = {deming_slope:.2f}')print(f'Deming intercept = {deming_intercept:.2f}')# Output: Deming slope = 0.50, Deming intercept = 134.09
This is exactly our simple linear regression result, nice! Careful: This is not a proof, yet it gives a nice feeling for it.
And think about the other extreme: What happens if the errors in the features are extremely large, let’s say infinity? Well, then δ becomes zero and the whole Deming slope term becomes just zero as well, which is easy to calculate (do it!). The intercept is then the mean of the labels y. Basically, the model is only a flat line then.
This also makes sense intuitively since the features are essentially useless in this case, so it’s good to ignore them and act like there are no features at all. And the best model without features that minimizes the sum of squares error is just the constant mean prediction.
In this article, we introduced the problem of errors in the features X and how they can influence the true relationships between variables. Simple linear regression yields a slope that is usually substantially lower than the truth, called regression dilution.
However, smart people invented another kind of regression to adjust for these errors: Deming regression. It makes it possible to still come to the correct conclusion without re-running the experiment another time, which can save time, money, and nerves.
Deming regression is great, but it is tricky to guess the correct values for δ, which is a major drawback because the result of the regression relies heavily on δ. Still, it’s nice that you can even make a statement in the case of features with errors.
A natural question to ask now is the following: What happens if we have more than one feature? Mathematicians got us covered, there is the total least squares method which is a direct generalization of Deming regression to more than one variable. And there are many more models, as you can see here.
Now you know that it makes sense to also pay attention to errors in all of your measurements and act accordingly to weaken their effects as well as possible.
I hope that you learned something new, interesting, and useful today. Thanks for reading!
As the last point, if you
want to support me in writing more about machine learning andplan to get a Medium subscription anyway,
want to support me in writing more about machine learning and
plan to get a Medium subscription anyway,
why not do it via this link? This would help me a lot! 😊
To be transparent, the price for you does not change, but about half of the subscription fees go directly to me.
Thanks a lot, if you consider supporting me!
If you have any questions, write me on LinkedIn!