3 Weeks Beginners Guide to Ace Data Science Interview: #Day 19
By: Vinay Vikram
Tags: Interview Questions, Machine Learning, Interview, Python, Data Science

3 Weeks Beginners Guide to Ace Data Science Interview: #Day 19

Interview Question on Ensemble Learning & Stacking

About the Series Data Science field is an exciting career choice and seeing a lot of hiring across fresh, lateral and experienced job positions. It’s one thing to know the concepts and totally another to crack the rigorous interviews for data science positions. If a candidate is aware of the different questions and the interview process, he is on the right path to an excellent career in the evolving Data Science field. This 3-week beginners guide to Ace Data Science Interview will be a useful asset for individuals who are preparing for the Data Science interviews. Every day for the next 21 days, we will talk about the different areas of the Data Science field and cover them elaborately. So sit back and start reading the article to get a finer understanding of the Data Science field and go prepared for the interviews.

Ensemble learning is the process by which multiple models, such as classifiers or experts, are logically combined to solve a particular computational intelligence problem. In simple terms, you can call Ensemble Learning the

“wisdom of the crowd”

The famous Netflix Prize competition took almost 3 years before the goal of a 10% improvement was reached. The winner's used gradient boosted decision trees to combine over 500 models.

Question Time

Question 1: What is the idea behind ensemble learning?

Different models used in ensemble improves overall variance, There are main 3 widely used ensembles learning techniques are:

Bagging Boosting Stacking

So if you have different models built for the same data and the same target variable, you can use one of the above three (bagging, boosting, stacking) methods to build an ensemble model. As every model used in the ensemble has its own performance measures, some of the models may perform better than the ultimate ensemble model and some of them may perform poorer than or equal to the ensemble model. But overall the ensemble methods will improve overall model performance and stability of the model.

Question 2: Explain bagging(stands for Bootstrap Aggregating).

Answer:

Bagging, or Bootstrap Aggregating, is an ensemble method in which the dataset is first divided into multiple subsets through resampling ( method of drawing repeated samples from the original data set to create multiple data subsets).

Then, each subset is used to train a separate model, and the final predictions are made through

Voting(Classification problem)or Averaging(Regression problem)the component models.

Bagging is performed in parallel.

Question 3: What is the difference in bias and variance in respective to the following algorithms. - Random forest - Gradient Boosting

Why this difference?

Answer:

Random Forest: It’s a tree-based algorithm that has Low bias and High Variance .

and . Gradient Boosting: It’s a weak learner(decision tree) based ensemble model that has High bias and Low variance.

Because, Boosting is based on weak learners (high bias, low variance). In terms of decision trees, weak learners are shallow trees, sometimes even as small as decision stumps (trees with two leaves).On the other hand, Random Forest uses fully grown decision trees (low bias, high variance).

Question 4: Why are ensemble methods superior to individual models?

Answer:

They average out biases, reduce variance, and are less likely to overfit.

There’s a common line in machine learning which is:

“Do Ensemble and get 2%” Which means perform ensemble learning and enhance model performance by 2 percent.

Question 5: Explain Stacking and Boosting.

Answer:

Boosting is a two-step approach, where one first uses subsets of the original data to produce a series of averagely performing models and then “boosts” their performance by combining them together using a particular cost function (=majority vote). Unlike bagging, in the classical boosting the subset creation is not random and depends upon the performance of the previous models: every new subset contains the elements that were (likely to be) misclassified by previous models. Stacking is similar to boosting: you also apply several models to your original data. The difference here is, however, that you don’t have just an empirical formula for your weight function, rather you introduce a meta-level and use another model/approach to estimate the input together with outputs of every model to estimate the weights or, in other words, to determine what models perform well and what badly given these input data.

Question 6: Which of the following is/are true about weak learners used in the ensemble model?

This question I recently got in one of my assignment-task.

1. They have low variance and they don’t usually overfit

2. They have a high bias, so they can not solve hard learning problems

3. They have high variance and they don’t usually overfit

Answer:

Solution: (1)

Weak learners basically shallow decision trees and they don’t try to fit every data point. So they usually don’t overfit which means that weak learners have low variance and high bias.

Question:7 What is the bias-variance decomposition of classification error in the ensemble method?

Answer:

The expected error of a learning algorithm can be decomposed into bias and variance. A bias term measures how closely the average classifier produced by the learning algorithm matches the target function. The variance term measures how much the learning algorithm’s prediction fluctuates for different training sets.

Question 8: Which ensemble technique converts a set of weak learners to a strong learner?

Answer:

The term Boosting refers to a family of algorithms which converts weak learner to strong learners. Boosting is an ensemble method for improving the model predictions of any given learning algorithm. The idea of boosting is to train weak learners sequentially, each trying to correct its predecessor.

Question 9: Bagging or Boosting, which of the ensemble model has independent weak learners?

Answer:

In Bagging, each individual trees are independent of each other because they consider the different subset of features and samples operates in parallel to each other.

In boosting tree individual weak learners are not independent of each other because each tree corrects the results of the previous tree. Bagging and boosting both can be considered as improving the base learner's results.

Question 10: Why is the decision tree a weak learner?

Answer:

The classic weak learner is a decision tree. By changing the maximum depth of the tree you can control the model fitting to the data. Decision trees used as weak learners are shallow in depth so they don’t overfit the data. This makes them incredibly popular for boosting. … One simple example is a 1-level decision tree called decision stump applied in bagging or boosting

Question 11: What is the difference between AdaBoost and Gradient Boosting?

Answer:

Both methods use a set of weak learners. They try to boost these weak learners into a strong learner(which is the basic function of boosting algorithm).

Gradient Boosting

Gradient boosting generates learners during the learning process. It build the first learner to predict the values/labels of samples and calculate the loss (the difference between the outcome of the first learner and the real value). It will build a second learner to predict the loss after the first step. The step continues to learn the third, fourth… until a certain threshold.

Read more in detail...

AdaBoost(Adaptive Boosting)

Adaboost requires users specify a set of weak learners (alternatively, it will randomly generate a set of weak learner before the real learning process). It will learn the weights of how to add these learners to be a strong learner. The weight of each learner is learned by whether it predicts a sample correctly or not. If a learner is mispredict a sample, the weight of the learner is reduced a bit. It will repeat such process until converge.

Read more in detail…