Ensemble learning: A case study from the 1994 US Census database | by Vinícius Campos | Towards Data Science

Before we start to understand what is ensemble learning and how is it works, we need to know the dataset used in this case study.
The dataset used is a subset of the 1994 US Census database, which was provided by Barry Becker and is available at UCI Machine Learning Repository. The prediction task of this repository is to determine whether a person makes over 50K a year, for this the following attributes and values are provided:
high_income: target class.
age: continuous.
workclass: Private, Self-emp-not-inc, Self-emp-inc, Federal-gov, Local-gov, State-gov, Without-pay, Never-worked.
fnlwgt: continuous.
education: Bachelors, Some-college, 11th, HS-grad, Prof-school, Assoc-acdm, Assoc-voc, 9th, 7th-8th, 12th, Masters, 1st-4th, 10th, Doctorate, 5th-6th, Preschool.
education-num: continuous.
marital-status: Married-civ-spouse, Divorced, Never-married, Separated, Widowed, Married-spouse-absent, Married-AF-spouse.
occupation: Tech-support, Craft-repair, Other-service, Sales, Exec-managerial, Prof-specialty, Handlers-cleaners, Machine-op-inspct, Adm-clerical, Farming-fishing, Transport-moving, Priv-house-serv, Protective-serv, Armed-Forces.
relationship: Wife, Own-child, Husband, Not-in-family, Other-relative, Unmarried.
race: White, Asian-Pac-Islander, Amer-Indian-Eskimo, Other, Black.
sex: Female, Male.
capital-gain: continuous.
capital-loss: continuous.
hours-per-week: continuous.
native-country: United-States, Cambodia, England, Puerto-Rico, Canada, Germany, Outlying-US(Guam-USVI-etc), India, Japan, Greece, South, China, Cuba, Iran, Honduras, Philippines, Italy, Poland, Jamaica, Vietnam, Mexico, Portugal, Ireland, France, Dominican-Republic, Laos, Ecuador, Taiwan, Haiti, Columbia, Hungary, Guatemala, Nicaragua, Scotland, Thailand, Yugoslavia, El-Salvador, Trinadad&Tobago, Peru, Hong, Holand-Netherlands.
Ensemble learning is a technique that combine other machine learning models to optimize and create a better one.
The are some different types of ensemble methods, such as: boosting, stacking, pasting, bagging and random forest. In this case study the focus is the last two methods.
Bootstrap Aggregating or Bagging is a technique which combines bootstrapping and aggregating methods. The first method consists in divide a dataset in n subsets with replacement, the idea of second method is create n models, one for each subset and aggregate them to yield in a final prediction.
Pasting method works similar to bagging, but the difference is that bootstrapping step, where the split is made without replacement.
The random forest is a special type of ensemble algorithm that uses multiple decision trees to built its model. Each tree is trained with different portion of dataset. Usually, this split is made similarly with bagging technique and the final model is built of several decision tree models which combined produce the prediction of model.
For this project we will use python as programming language, and the libraries: pandas and scikit-learn. Initially, let’s know the “face” of our data:
# load the datasetincome = pd.read_csv("income.csv")income.head()
Before we continue, we need to preprocess the data to change the categorical variables to numerical variables. After this process, this is a sample of the data:
Now, we’ll use two great scikit-learn features, called Pipeline and GridSearchCV, which allow us to test automatically several hyperparameters of different models.
# split-out train/validation and test datasetX_train, X_test, y_train, y_test = train_test_split(income.drop(labels="high_income",axis=1),                                                    income["high_income"],test_size=0.20,random_state=seed,shuffle=True,stratify=income["high_income"])
# The full pipeline as a step in another pipeline with an estimator as the final steppipe = Pipeline(steps = [("clf",RandomForestClassifier())])# create a dictionary with the hyperparameterssearch_space = [{"clf":[DecisionTreeClassifier()],                 "clf__criterion": ["gini","entropy"],                 "clf__splitter": ["best","random"],                 "clf__random_state": [seed],                 "fs__score_func":[chi2],                 "fs__k":[4,6,8]},                {"clf":[RandomForestClassifier()],                 "clf__n_estimators": [200,300],                 "clf__criterion": ["gini","entropy"],                 "clf__max_leaf_nodes": [32,64,128],                 "clf__random_state": [seed],                 "fs__score_func":[chi2],                 "fs__k":[4,6,8]},                {'clf' [BaggingClassifier(DecisionTreeClassifier(random_state=42))],                 "clf__base_estimator__criterion": ['gini','entropy'],                 "clf__base_estimator__splitter": ['best','random'],                 "clf__oob_score": [True],                 "clf__n_estimators": [200,300],                 "clf__bootstrap":[True],                 "fs__score_func":[chi2],                 "fs__k":[4,6,8]},                {'clf': [BaggingClassifier(DecisionTreeClassifier(random_state=42))],                 "clf__base_estimator__criterion": ['gini','entropy'],                 "clf__base_estimator__splitter": ['best','random'],                 "clf__oob_score": [False],                 "clf__n_estimators": [200,300],                 "clf__bootstrap":[False],                 "fs__score_func":[chi2],                 "fs__k":[4,6,8]}]# create grid searchkfold = KFold(n_splits=num_folds,random_state=seed)grid = GridSearchCV(estimator=pipe,                     param_grid=search_space,                    cv=kfold,                    scoring=scoring,                    return_train_score=True,                    n_jobs=-1,                    refit="AUC", verbose=10)
# fit grid searchbest_model = grid.fit(X_train,y_train)
This configuration allow the algorithm uses all the available cores, which will test 960 different configuration of decision trees, random forests and bagging classifiers(with decision trees as internal model) in a parallel way.
After this process, the best model produced by the GridSearchCV was a random forest model with the following configuration:
Before we define if this is the best model, let’s check the accuracy of the model to train and test datasets. The purpose of this comparison is verify if the model is underfitted or overfitted.
As we can see above, the accuracy for both sets are good, and more than that, the values of train and test accuracy are very close. Thus, this results show us that the best model produced by GridSearchCV was well generalized.
In a random forest classifier we can set a hyperparameter called bootstrap, which defines weather samples will be trained with replacement or not. Although the best model was selected with this parameter as false, which tries to help the model to minimize the chance of overfitting, several other models present similar results when the parameter was set as true, as we can see in the image below.
So, for this dataset, we achieved good results regardless of the value of the bootstrap variable. However, the worst results due to possible overfitting came with bootstrap equal to true.
Now, lets check the importance of each feature of the dataset for our model. For this task, we used two tools: The feature importance, from random forest classifier and the library SHAP (SHapley Additive exPlanations), which is a unified approach to explain the output of any machine learning model.
The left image was produced by feature_importances_ from scikit-learn and the right image was produced by SHAP. It is important to see that the both results presented similar results and the 4 most important features were the same, changing only the positions. With these results it is easy to know the information that most influences the model, which greatly helps understanding and solving the problem.