3 Weeks Beginners Guide to Ace Data Science Interview: #Day 10
By: Vinay Vikram
Tags: Interview Questions, Machine Learning, Interview, Python, Data Science

3 Weeks Beginners Guide to Ace Data Science Interview: #Day 10

Interview Questions on Model Evaluation Techniques for Regression Algorithms

About the Series Data Science field is an exciting career choice and seeing a lot of hiring across fresh, lateral and experienced job positions. It’s one thing to know the concepts and totally another to crack the rigorous interviews for data science positions. If a candidate is aware of the different questions and the interview process, he is on the right path to an excellent career in the evolving Data Science field. This 3-week beginners guide to Ace Data Science Interview will be a useful asset for individuals who are preparing for the Data Science interviews. Every day for the next 21 days, we will talk about the different areas of the Data Science field and cover them elaborately. So sit back and start reading the article to get a finer understanding of the Data Science field and go prepared for the interviews.

While training a model is a key step, how the model generalizes or performs on unseen data is an equally important aspect that should be considered in every machine learning pipeline. We need to know whether it actually works similarly like it if for the train data and, consequently, if we can trust its predictions.

Could the model be merely memorizing the data it is fed with, and therefore unable to make good predictions on future samples, or samples that it hasn’t seen before?

In this article, we explain the techniques used in evaluating how well a Regression machine learning model generalizes to new, previously unseen data.

Question1: State out the evaluations matrix used for generalization of regression algorithms.

Answer:-

Mean Absolute Error

Mean Square Error

Root Mean Squared Error (RMSE)

Root Mean Squared Logarithmic Error

R-squared/Adjusted

Question2: Out of three given three evaluation metrics (MAE| MSE|RMSE) which evaluation metrics give rise to a better generalization model.

Answer:-

Claiming about any of the metrics is certainly baseless all depends on the kind of data we have.

(1) Mean Squared Error (MSE):

Useful if we have unexpected values that we should care about. We should pay attention to a very high or low value.

If we make a single very bad prediction, the squaring will make the error even worse and it may skew the metric towards overestimating the model’s badness.

(2) Mean Absolute Error (MAE):

The MAE is a linear score which means that all the individual differences are weighted equally in the average. For example, the difference between 10 and 0 will be twice the difference between 5 and 0. However, the same is not true for RMSE.

in the average. For example, the difference between 10 and 0 will be twice the difference between 5 and 0. However, the same is not true for RMSE. It penalizes huge errors not as badly as MSE does. Thus, it’s not that sensitive to outliers as a mean square error.

Thus, it’s not that sensitive to outliers as a mean square error. Eg: MAE is widely used in finance, where $10 error is usually exactly two times worse than $5 error. On the other hand, MSE metric thinks that $10 error is four times worse than $5 error. MAE is easier to justify than RMSE.

(3) Root Mean Squared Error (RMSE):

RMSE is just the square root of MSE. The square root is introduced to make the scale of the errors to be the same as the scale of targets.

Question3: Suppose you are given a student's score data. After all, preprocessing of data, when you plot a scatter plot between your features and the target variable you found there are many data points that deviate from the rest data points(Outliers). If you drop these outliers you are certainly gonna lose a large amount of data. So you decide to train your model as it is now, which of the following evaluation metrics(Mean Square Error or Mean Absolute Error)will tell about the betterness of your model?

Answer:-

MAE is more robust (less sensitive to outliers) than MSE.

The following questions help you to decide:

MSE vs MAE

Question4: What Is the Coefficient of Determination?

Answer:-

The coefficient of determination is a measure used in statistical analysis that assesses how well a model explains and predicts future outcomes. It is indicative of the level of explained variability in the data set.

Generally, it is known by the term of R-squared

Question5: From the below image explain the term SSregression & SStotal.

Answer:-

SSregression: The sum of squares due to regression, or SSR . It is the sum of the difference between the predicted value and the mean of the dependent variable .

The due to regression, or . It is the sum of the difference between the and the . SStotal: The sum of squares total, denoted SST, is the squared differences between the observed dependent variable and its mean.

Question6: In a linear regression problem, we are using “R-squared” to measure goodness-of-fit. We add a feature in linear regression model and retrain the same model. Which of the following options is true?

(A) If R Squared increases, this variable is significant.

(B) If R Squared decreases, this variable is not significant.

(C) Individually R squared cannot tell about variable importance. We can’t say anything about it right now.

(D) None of these.

Answer:- C

“R-squared” individually can’t tell whether a variable is significant or not because each time when we add a feature, “R squared” can either increase or stay constant. But, it is not true in case of “Adjusted R squared” (increases when features found to be significant).

Question7: Why do we need Adjusted R-squared when we have R-squared already?

Answer:-

The adjusted R-squared is a modified version of R-squared that has been adjusted for the number of predictors in the model. The adjusted R-squared increases only if the new term improves the model more than would be expected by chance. It decreases when a predictor improves the model by less than expected by chance. The adjusted R-squared can be negative, but it’s usually not. It is always lower than the R-squared.

Note: The adjusted R-squared is always smaller than the R-squared, as it penalizes the excessive use of variables.

Question8: Can we have -ve R-squared value? This one is a bit dubious, Some books suggest it is and some suggest it is not.

Answer:-

R2 compares the fit of the chosen model with that of a horizontal straight line (the null hypothesis). If the chosen model fits worse than a horizontal line, then R2 is negative. Note that R2 is not always the square of anything, so it can have a negative value without violating any rules of math. R2 is negative only when the chosen model does not follow the trend of the data, so fits worse than a horizontal line.

Common Misconception: A lot of articles in the web state that the range of R² lies between 0 and 1 which is not actually true. The maximum value of R² is 1 but the minimum can be minus infinity. For example, consider a really crappy model predicting highly negative value for all the observations even though y_actual is positive. In this case, R² will be less than 0. This is a highly unlikely scenario but the possibility still exists.

Example: fit data to a linear regression model constrained so that the Y-intercept must equal 1500.

Guys this is a just case to understand when will be the R-squared is negative. But when you train your model you never found such a worse model.

Question9: MAE and RMSE — Which Metric is Better?

Answer:-

Similarities: Both MAE and RMSE express average model prediction error in units of the variable of interest. Both metrics can range from 0 to ∞ and are indifferent to the direction of errors. They are negatively-oriented scores, which means lower values are better.

Differences: Taking the square root of the average squared errors has some interesting implications for RMSE. Since the errors are squared before they are averaged, the RMSE gives a relatively high weight to large errors. This means the RMSE should be more useful when large errors are particularly undesirable. The three tables below show examples where MAE is steady and RMSE increases as the variance associated with the frequency distribution of error magnitudes also increases.

Conclusion

RMSE has the benefit of penalizing large errors more so it can be more appropriate in some cases, for example, if being off by 10 is more than twice as bad as being off by 5. But if being off by 10 is just twice as bad as being off by 5, then MAE is more appropriate.

From an interpretation standpoint, MAE is clearly the winner. RMSE does not describe average error alone and has other implications that are more difficult to tease out and understand.

On the other hand, one distinct advantage of RMSE over MAE is that RMSE avoids the use of taking the absolute value, which is undesirable in many mathematical calculations.

Eg: The squared difference has nicer mathematical properties; it’s continuously differentiable (nice when you want to minimize it), it’s a sufficient statistic for the Gaussian distribution. The mean absolute deviation (the absolute value notation you suggest) is also used as a measure of dispersion, but it’s not as “well-behaved” as the squared error.