Quantile Regression. When performing regression analysis, It... | by Steven Dye | Towards Data Science

When performing regression analysis, It isn’t enough to come up with a numerical prediction to a problem, you also need to express how confident you are in that prediction. For example, if you’re looking at the price for a home in a particular market and your model predicts a house will sell for $262,458.45, how confident can you be that your model’s prediction is exactly right? Hopefully your gut says that there is an infinitesimally small chance that this is true, but maybe there’s a chance that your model is close to the actual answer. We need a way to predict a range of values while having a certain amount of confidence in that range.
Enter quantile regression. Unlike regular linear regression which uses the method of least squares to calculate the conditional mean of the target across different values of the features, quantile regression estimates the conditional median of the target. Quantile regression is an extension of linear regression that is used when the conditions of linear regression are not met (i.e., linearity, homoscedasticity, independence, or normality).
Traditionally, the linear regression model for calculating the mean takes the form
where p is equal to the number of features in the equation and n is the number of data points. The best linear regression line is found by minimizing the mean square error, which is found with the equation
Now for quantile regression, you’re not limited to just finding the median, but you can calculate any quantile (percentage) for a particular value in the features variables. For example, if we were to find the 25th quantile for the price of a particular home, that would mean that there is a 25% chance the actual price of the house is below the prediction, while there is a 75% chance that the price is above.
Taking a similar structure to the linear regression model, the quantile regression model equation for the τth quantile is
This means that instead of being constants, the beta coefficients are now functions with a dependency on the quantile. Finding the values for these betas at a particular quantile value involves almost the same process as it does for regular linear quantization, except now we have to reduce the median absolute deviation.
Here the function ρ is the check function which gives asymmetric weights to the error depending on the quantile and the overall sign of the error. Mathematically, ρ takes the form
In this case, u is the error of a single data point and the max function returns the largest value in the parentheses. This means that if the error is positive, then the check function multiples the error by τ, and if the error is negative, then the check function multiplies the error by (1- τ).
For example, if you want the median of the 10th percentile, that means you want 90% of the errors to be positive and 10% to be negative. In order to find the smallest MAD while having this statement be true, weights have to added to the errors. In the case of the the 10th quantile, a weight of 0.9 is added the negative weights while a weight of 0.1 is added to the positive ones.
Let’s look at quantile regression in action. Let’s examine the python statsmodels example for QuantReg, which takes a look at the relationship between income and expenditures on food for a sample of working class Belgian households in 1857, and see what kind of statistical analysis we can do.
import statsmodels.api as smimport statsmodels.formula.api as smfdata = sm.datasets.engel.load_pandas().datadata.head()
mod = smf.quantreg('foodexp ~ income', data)res = mod.fit(q=.5)print(res.summary())
As you can see, you can create a regression line for a particular quantile and perform a statistical analysis on it the same way that you can with a regular linear regression model.
Here is a link to a paper written by SAS about Quantile Regression if you want to learn more. Other sources are provided below.