Data Preprocessing and Model Comparison Techniques you must know | by Fan Yuan | Towards Data Science

As we all know, when doing data science projects, it’s always more than fitting the model on data and getting the model performance. Actually, in terms of good projects, it’s more about exploring the data, cleaning, preprocessing data and finally comparing several models’ perform to get the best one.
In this post, I’ll use U.S. census data taken from the UCI Machine Learning Repository and go through the full process. At the end of the process, we’ll solve the following several problems:
Construct a model that accurately predicts whether an individual makes more than $50,000.What are the key factors contributing to high vs. low income?Are there any significant gaps in these Census attributes by gender or race?Any underneath clusters (group) based on census data?
Construct a model that accurately predicts whether an individual makes more than $50,000.
What are the key factors contributing to high vs. low income?
Are there any significant gaps in these Census attributes by gender or race?
Any underneath clusters (group) based on census data?
Have a quick check on whether there’s any huge missing value in columns or rows which may largely affect the later analysis.
The result above shows there’s no `null` value in the dataset. But according to data notes provided, unknown data was converted into ‘?’. Therefore, next, we’ll convert ‘?’ to NaNs.
Since the missing data ‘?’ is in a small volume, here I choose to just remove the unknown data which flagged as ‘?’ here. But if the missing data is in a large volume, need to consider imputing NaNs with more advanced methods.
The largest missing percentage by column level is 5% in the dataset, and most columns are complete enough. Therefore, here I’ll remove the NaN values instead of manually imputing.
The goal is to identify if an individual has an income over 50k or not, so first, have an overlook at how the income distributes in the data set. A cursory investigation of the dataset will determine how many individuals fit into each group and will tell us about the percentage of these individuals making more than $50,000.
Here I’ll generate some variables to help analysis as below:- The total number of records- The number of individuals making more than $50,000 annually- The number of individuals making at most $50,000 annually- The percentage of individuals making more than $50,000 annually
Note:Since in EDA process, we don’t need the test data, I’ll combine the train and test data in this section just for getting a better and more general distribution of data
From the graph above, we can see that most of ‘>50K’ group are people in middle career and the male is a bit more than female in ‘>50K’ especially around 50 years old.
We can see the top three biggest groups for people who have income larger 50K are professional in Exec-managerial, Prof-specialty, and Sales.
Since the race is a little bit unbalanced in the data set, White accounts most of ‘>50K’ group. And within each race, ‘<50K’ population is much larger than ‘>50K’.
Then we take a quick view of how education correlates to the income level. We can see the biggest group is Bachelors following by HS-grad and Some-college. And after moving the missing value:
Total number of records: 45222Individuals making more than $50,000: 11208Individuals making at most $50,000: 34014Percentage of individuals making more than $50,000: 24.78%
Before data can be used as input for machine learning algorithms, it often must be cleaned, formatted, and restructured. After processing the missing entries, there are some qualities about certain features that must be adjusted. This preprocessing can help tremendously with the outcome and predictive power of nearly all learning algorithms.
Skewness may violate model assumptions or may impair the interpretation of feature importance. Therefore, here I will apply the logarithmic transformation on the skewed data.
As shown in the graph, there seems skewness in ‘capital-gain’ and ‘capital-loss’ features. Use the quantitative result to confirm if I need to transform skewness in these two variables.
In addition to performing transformations on features that are highly skewed, here will perform some type of scaling on numerical features. Applying a scaling to the data does not change the shape of each feature’s distribution (such as ‘capital-gain’ or ‘capital-loss’ above); however, it is useful to scale the input attributes for a model that relies on the magnitude of values, such as distance measures used in k-nearest neighbors and in the preparation of coefficients in regression.
There are several features for each record that are non-numeric. Typically, learning algorithms expect the input to be numeric, which requires that non-numeric features (called *categorical variables*) be converted. Here convert categorical variables by using the **one-hot encoding** scheme.
Additionally, as with the non-numeric features, I need to convert the non-numeric target label, `’income’` to numerical values for the learning algorithm to work. Since there are only two possible categories for this label (“<=50K” and “>50K”), we can simply encode these two categories as `0` and `1`, respectively. Therefore, we will
Use `sklearn.OneHotEncoder` to perform one-hot encoding on the `’features_log_minmax_transform’` data.  — Note: Since the test data is separate, in case there are unseen categories in test data which will fail the model, here use sklearn.OneHotEncoder rather the pd.get_dummies()Convert the target label `’income_raw’` to numerical entries. Set records with “<=50K” to `0` and records with “>50K” to `1`.
Use `sklearn.OneHotEncoder` to perform one-hot encoding on the `’features_log_minmax_transform’` data.  — Note: Since the test data is separate, in case there are unseen categories in test data which will fail the model, here use sklearn.OneHotEncoder rather the pd.get_dummies()
Convert the target label `’income_raw’` to numerical entries. Set records with “<=50K” to `0` and records with “>50K” to `1`.
In this section, I will investigate five different algorithms, and determine which is best at modeling the data. Four of these algorithms will be supervised learners, and the fifth algorithm is known as a naive predictor.
Generate a naive predictor to show what a base model without any intelligence would look like. That is if we chose a model that always predicted an individual made more than $50,000, what would that model’s accuracy and F-score be on this dataset? Here assume that we consider more about to correctly predict individual who has income over 50K.
Therefore, a model’s ability to precisely predict those that make more than $50,000 is more important than the model’s ability to recall those individuals. We can use F-beta score as a metric that considers both precision and recall:
Apart from the bench mark model, I’ll choose four other models Logistic Regression, Random Forest, Ensemble Methods (AdaBoost) and Support Vector Machines (SVM) as a candidate to build the predictive model.
Logistic RegressionRandom ForestEnsemble Methods — AdaBoostSupport Vector Machines (SVM)
Logistic Regression
Random Forest
Ensemble Methods — AdaBoost
Support Vector Machines (SVM)
To properly evaluate the performance of each model chosen above more efficiently, it’s helpful to create a training and predicting pipeline that can quickly and effectively train models using various sizes of training data and perform predictions on the testing data.
The pipeline will: — Fit the learner to the sampled training data and record the training time. — Perform predictions on the test data `X_test`, and also on the first 300 training points `X_train[:300]`. — Record the total prediction time. — Calculate the accuracy score for both the training subset and the testing set. — Calculate the F-score for both the training subset and the testing set.
Next, I will choose from the four supervised learning models the *best* model to use on the test data. I will then perform a grid search optimization for the model over the entire training set (`X_train` and `y_train`) to improve upon the untuned model’s F-score.
According to the model performance graph above, although the Random Forest performs best on the training set the AdaBoost Classifier finally predicts best on testing data. Although the accuracy of AdaBoost Classifier is quite similar to the performance of other models, F-score of AdaBoost is better on both training and testing data when the model is applied to the whole data set. Also, in contrast to Support Vector Classifier which takes dramatically more time to train and predict, the AdaBoost is faster. In terms of binary classification, AdaBoost will also perform well in this case.
Fine-tune the chosen model. Use grid search (`GridSearchCV`). Use the entire training set for this.
The optimized model’s accuracy on testing data is 0.8701 and F-score is 0.7518. Both of those scores are better than the unoptimized model. Also, the optimized model performs much better than the benchmarks
Generally, it’s useful to know which features provide the most predictive power when performing supervised learning on a dataset like the census data here. In this case, it means we wish to identify a small number of features that most strongly predict whether an individual makes at most or more than $50,000.
Here will choose a scikit-learn classifier (e.g., adaboost, random forests) that has a `feature_importance_` attribute. Fit this classifier to training set and use this attribute to determine the top 5 most important features for the census dataset.
From the result, the feature importance put ‘capital-loss’ the most important feature. It’s probably because the bigger capital loss means that the person has to have that volume of money to invest. The ‘age’ ranks the second one which may because the elder the people the more salary they will have to the donor. The ‘hours-per-week’ and ‘sex_Female’ ranks the forth and fifth which probably because it’s not that sure cases. It’s true since maybe the person works longer but have a lower unit salary.
From the visualization above, we see that the top five most important features contribute more than half of the importance of all features present in the data. This hints that we can attempt to reduce the feature space and simplify the information required for the model to learn. The code cell below will use the same optimized model found earlier, and train it on the same training set with only the top five important features.
As a result above show, the model performs a bit worse if I only used important features. The accuracy is 5% lower and the f-score is 7% lower, so in this case, I’ll still choose to use all feature to build the model unless when the time for fitting model matters a lot.
For a more detailed view of my analysis, feel free to check out my corresponding GitHub repository: