Machine Learning Classification Project: Finding Donors | by Victor Roman | Towards Data Science

In this project, we will use a number of different supervised algorithms to precisely predict individuals’ income using data collected from the 1994 U.S. Census.
We will then choose the best candidate algorithm from preliminary results and further optimize this algorithm to best model the data. Our goal with this implementation is to build a model that accurately predicts whether an individual makes more than $50,000.
This sort of task can arise in a non-profit setting, where organizations survive on donations. Understanding an individual’s income can help a non-profit better understand how large of a donation to request, or whether or not they should reach out to begin with. As from our previous research we have found out that the individuals who are most likely to donate money to a charity are the ones that make more than $50,000.
The dataset for this project originates from the UCI Machine Learning Repository. The datset was donated by Ron Kohavi and Barry Becker, after being published in the article “Scaling Up the Accuracy of Naive-Bayes Classifiers: A Decision-Tree Hybrid”. You can find the article by Ron Kohavi online. The data we investigate here consists of small changes to the original dataset, such as removing the 'fnlwgt' feature and records with missing or ill-formatted entries.
The modified census dataset consists of approximately 32,000 data points, with each datapoint having 13 features. This dataset is a modified version of the dataset published in the paper “Scaling Up the Accuracy of Naive-Bayes Classifiers: a Decision-Tree Hybrid”, by Ron Kohavi. You may find this paper online, with the original dataset hosted on UCI.
Features
age: Age
workclass: Working Class (Private, Self-emp-not-inc, Self-emp-inc, Federal-gov, Local-gov, State-gov, Without-pay, Never-worked)
education_level: Level of Education (Bachelors, Some-college, 11th, HS-grad, Prof-school, Assoc-acdm, Assoc-voc, 9th, 7th-8th, 12th, Masters, 1st-4th, 10th, Doctorate, 5th-6th, Preschool)
education-num: Number of educational years completed
marital-status: Marital status (Married-civ-spouse, Divorced, Never-married, Separated, Widowed, Married-spouse-absent, Married-AF-spouse)
occupation: Work Occupation (Tech-support, Craft-repair, Other-service, Sales, Exec-managerial, Prof-specialty, Handlers-cleaners, Machine-op-inspct, Adm-clerical, Farming-fishing, Transport-moving, Priv-house-serv, Protective-serv, Armed-Forces)
relationship: Relationship Status (Wife, Own-child, Husband, Not-in-family, Other-relative, Unmarried)
race: Race (White, Asian-Pac-Islander, Amer-Indian-Eskimo, Other, Black)
sex: Sex (Female, Male)
capital-gain: Monetary Capital Gains
capital-loss: Monetary Capital Losses
hours-per-week: Average Hours Per Week Worked
native-country: Native Country (United-States, Cambodia, England, Puerto-Rico, Canada, Germany, Outlying-US(Guam-USVI-etc), India, Japan, Greece, South, China, Cuba, Iran, Honduras, Philippines, Italy, Poland, Jamaica, Vietnam, Mexico, Portugal, Ireland, France, Dominican-Republic, Laos, Ecuador, Taiwan, Haiti, Columbia, Hungary, Guatemala, Nicaragua, Scotland, Thailand, Yugoslavia, El-Salvador, Trinadad&Tobago, Peru, Hong, Holand-Netherlands)
Target Variable
income: Income Class (<=50K, >50K)
We will first load the Python libraries that we are going to use, as well as the census data. The last column will be our target variable, ‘income’, and the rest will be the features.
# Import libraries necessary for this projectimport numpy as npimport pandas as pdfrom time import timefrom IPython.display import display # Import supplementary visualization code visuals.pyimport visuals as vs# Pretty display for notebooks%matplotlib inline# Load the Census datasetdata = pd.read_csv("census.csv")# Success - Display the first recorddisplay(data.head(n=1))
An initial exploration of the dataset will show us how many individuals fit in each group and the proportion of them that earn more than $50,000.
# Total number of recordsn_records = data.shape[0]# Number of records where individual's income is more than $50,000n_greater_50k = data[data['income'] == '>50K'].shape[0]# Number of records where individual's income is at most $50,000n_at_most_50k = data[data['income'] == '<=50K'].shape[0]# Percentage of individuals whose income is more than $50,000greater_percent = (n_greater_50k / n_records) * 100# Print the resultsprint("Total number of records: {}".format(n_records))print("Individuals making more than $50,000: {}".format(n_greater_50k))print("Individuals making at most $50,000: {}".format(n_at_most_50k))print("Percentage of individuals making more than $50,000: {}%".format(greater_percent))
Data must be preprocessed in order to be used in Machine Learning algorithms. This preprocessing phase includes the cleaning, formatting and reestructuring of the data.
For this dataset there aren’t empty entries nor invalid ones, but, there are some features that must be adjusted. This task will dramatically improve the results and the predictive power of our model.
Transforming Skewed Continuous Fetures
Skewed distributions on the feature’s values can make an algorithm to underperfom if the range is not normalized.
In our dataset, there are two features with this distribution:
‘capital-gain’
‘capital-loss’
# Split the data into features and target labelincome_raw = data['income']features_raw = data.drop('income', axis = 1)# Visualize skewed continuous features of original datavs.distribution(data)
For this type of disttributions, is very typical to apply logarithmic transformationon the data, so the outliers do not affect negatively the machine learning model’s performance.
However, we should carefully treat the 0 values, as the log(0) is undefined, so we will translate these values to a small amount above 0 to apply the logarithm succesfully.
# Log-transform the skewed featuresskewed = ['capital-gain', 'capital-loss']features_log_transformed = pd.DataFrame(data = features_raw)features_log_transformed[skewed] = features_raw[skewed].apply(lambda x: np.log(x + 1))# Visualize the new log distributionsvs.distribution(features_log_transformed, transformed = True)
Normalizing Numerical Features
It is recommended to perform some type of scaling on numerical features. This scaling will not change the shape of the feature’s distribution but it ensures the equal treatment of each feature when supervised models are applied.
# Import sklearn.preprocessing.StandardScalerfrom sklearn.preprocessing import MinMaxScaler# Initialize a scaler, then apply it to the featuresscaler = MinMaxScaler() # default=(0, 1)numerical = ['age', 'education-num', 'capital-gain', 'capital-loss', 'hours-per-week']features_log_minmax_transform = pd.DataFrame(data = features_log_transformed)features_log_minmax_transform[numerical] = scaler.fit_transform(features_log_transformed[numerical])# Show an example of a record with scaling applieddisplay(features_log_minmax_transform.head(n = 5))
Preprocessing Categorial Features
If we take a look at the previous table, we can see that there are some features like ‘occupation’ or ‘race’ that are not numerical, they are categorical. Machine learning algorithms expect to work with numerical values, so these categorical features should be transformed.
One of the most popular categorical transformations is called ‘one-hot encoding’. In one-hot encoding, there is created a ‘dummy’ variable for each possible category of the categorical feature.
For a better understanding, take a look at the following table:
In addition, we should transform our target variable ‘income’. It can only take to possible values, “<=50K” and “>50K”, so we will avoid one-hot encoding and codify the categories as 0 and 1, respectively.
features_log_minmax_transform.head(1)
# One-hot encode the 'features_log_minmax_transform' data features_final = pd.get_dummies(features_log_minmax_transform)# Encode the 'income_raw' data to numerical valuesincome = income_raw.map({'<=50K':0,'>50K':1})# Print the number of features after one-hot encodingencoded = list(features_final.columns)print("{} total features after one-hot encoding.".format(len(encoded)))# See the encoded feature namesprint (encoded)
Shuffle and Split Data
When all categorical variables are transformed, and all numerical features normalized, we need to split our data into training and test sets. We’ll use 80% of the data to training and 20% for testing,
# Import train_test_split from sklearn.model_selection import train_test_split  # Split the 'features' and 'income' data into training and testing sets X_train, X_test, y_train, y_test = train_test_split(features_final,                                                      income,                                                      test_size = 0.2,                                                      random_state = 0)  # Show the results of the split print("Training set has {} samples.".format(X_train.shape[0])) print("Testing set has {} samples.".format(X_test.shape[0]))
In this section, we will study four different algorithms, and determine the best at modeling and predicting our data. One of these algorithms will be a naive predictor which will serve as a baseline of performance and, the other three will be supervised learners.
Metrics and The Naive Predictor
The objective of the project is to correctly identify what individuals make more than 50k$ per year, as they are the group most likely to donate money to charity. Therefore we should choose wisely our evaluation metric.
Note: Evaluation Metrics reminder
When making predictions on events we can get 4 type of results: True Positives, True Negatives, False Positives and False Negatives. All of these are represented in the following classification matrix:
Accuracy measures how often the classifier makes the correct prediction. It’s the ratio of the number of correct predictions to the total number of predictions (the number of test data points).
Precision tells us what proportion of events we classified as a certain class, actually were that class. It is a ratio of true positives to all positives.
Recall (sensitivity) tells us what proportion of events that actually were the of a certain class were classified by us as that class. It is a ratio of true positives to all the positives.
For classification problems that are skewed in their classification distributions like in our case, accuracy by itself is not an appropiate metric. Instead, precision and recall are much more representative.
These two metrics can be combined to get the F1 score, which is weighted average(harmonic mean) of the precision and recall scores. This score can range from 0 to 1, with 1 being the best possible F1 score(we take the harmonic mean as we are dealing with ratios).
In addition, since we are searching individuals willing to donate, the model’s ability to precisely predict those that make more than $50,000 is more important than the model’s ability to recall those individuals. We can use F-beta score as a metric that considers both precision and recall.
Concretely, for β = 0.5, it is given more importance on precision.
If we take a look at the distribution of classes of the ‘income’ variable, it is evident that there are a lot more people that make at most $50K per year than those who make more. So, we could make the naive predicition of taking a random individual, and predicting that he/she will not make more than $50K per year. It is called naive because we haven’t considered any relevant information to substantiate the claim.
This naive prediction will serve us as a benchmark to determine if our model is performing well or not. It is important to notice that it is pointless to use this type of prediction by itself as all individuals would be classified as not donors.
Naive Predictor Performance
We’ll run the following code to determine our naive predictor performance:
# Counting the ones as this is the naive case. Note that 'income' is the 'income_raw' data encoded to numerical values done in the data preprocessing step.TP = np.sum(income) # Specific to the naive caseFP = income.count() - TP# No predicted negatives in the naive caseTN = 0 FN = 0 # Calculate accuracy, precision and recallaccuracy = TP / (TP + FP + TN + FN)recall = TP / (TP + FN)precision = TP / (TP + FP)# Calculate F-score using the formula above for beta = 0.5 and correct values for precision and recall.beta = 0.5fscore = (1 + beta**2) * ((precision * recall) / ((beta**2) * precision + recall))# Print the results print("Naive Predictor: [Accuracy score: {:.4f}, F-score: {:.4f}]".format(accuracy, fscore))
Considering the shape of our data:
45222 datapoints
103 Features
The number of datapoints is not very large but there is a significat amount of features and not all supervised algorithms are suitable to handle quantity appropiately.
Some of the available Classification algorithms in scikit-learn:
Gaussian Naive Bayes (GaussianNB)
Decision Trees
Ensemble Methods (Bagging, AdaBoost, Random Forest, Gradient Boosting)
K-Nearest Neighbors (KNeighbors)
Stochastic Gradient Descent Classifier (SGDC)
Support Vector Machines (SVM)
Logistic Regression
Considering their characteristics and our dataset, we’ll choose the following three:
a) Gaussian Naive Bayes
The strenghts of the model are: It is simple and fast classifier that provides good results with little tunning of the model’s hyperparameters. In addition, it does not require a large amount of data to be propperly trained.
The weaknesses of the model are: It has a strong feature independece assumptions. If we do not have ocurrences of a class label and a certain attribute value together (e.g. class = ‘nice’, shape = ‘sphere’) then the frequency-based probability estimate will be zero, so given the conditional independence assumption, when all the probabilities are multiplied we will get zero, which will affect the posterior probabilty estimate.
One possible real world application where this model can be applied is for text learning.
It is a good candidate because it is an efficient model and can deal with many features (the data set contains 98 features).
b) Random Forests
The strenghts of the model are: It works well with binary features, as it is an ensembling of decision trees. It does not expect linear features. It works well with high dimensional spaces and large number of training examples.
The main weakness is that it may overfit when dealing wih noisy data.
One possible real world application where this model can be applied is for predicting stock market prices.
It is a good candidate because it is often a quite accurate classificator and works well with binary features and high dimensional datasets.
c) Support Vector Machines Classifier
The strenghts of the model are: It works well with no linearly separable data and high dimensional spaces.
The main weakness is that it may be quite inefficient to train, so it is no suitable for “industrial scale” applications.
One possible real world application where this model can be applied is for classifying people with and without common diseases.
It is a good candidate because it is often a quite accurate classificator and works well with binary features and high dimensional datasets.
Creating a Training and Predicting Pipeline
To properly evaluate the performance of each model, we will create a training and predicting pipeline that will allow us to quickly and effectively train models using various sizes of training data and perform predictions on the testing data.
# Import two metrics from sklearn - fbeta_score and accuracy_scoredef train_predict(learner, sample_size, X_train, y_train, X_test, y_test):     '''    inputs:       - learner: the learning algorithm to be trained and predicted on       - sample_size: the size of samples (number) to be drawn from training set       - X_train: features training set       - y_train: income training set       - X_test: features testing set       - y_test: income testing set    '''        results = {}        # Fit the learner to the training data     start = time() # Get start time    learner = learner.fit(X_train[:sample_size], y_train[:sample_size])    end = time() # Get end time        # Calculate the training time    results['train_time'] = end - start             # Get the predictions on the test set    start = time() # Get start time    predictions_test = learner.predict(X_test)    predictions_train = learner.predict(X_train[:300])    end = time() # Get end time        # Calculate the total prediction time    results['pred_time'] = end -start                # Compute accuracy on the first 300 training samples     results['acc_train'] = accuracy_score(y_train[:300], predictions_train)            # Compute accuracy on test set using accuracy_score()    results['acc_test'] = accuracy_score(y_test, predictions_test)        # Compute F-score on the the first 300 training samples    results['f_train'] = fbeta_score(y_train[:300], predictions_train, beta=0.5)            # Compute F-score on the test set which is y_test    results['f_test'] = fbeta_score(y_test, predictions_test, beta=0.5)           # Success    print("{} trained on {} samples.".format(learner.__class__.__name__, sample_size))            # Return the results    return results
# Import the three supervised learning models from sklearnfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.naive_bayes import GaussianNBfrom sklearn.svm import SVC# Initialize the three modelsrandom_state = 42clf_A = RandomForestClassifier(random_state=random_state)clf_B = GaussianNB()clf_C = SVC(random_state=random_state)# Calculate the number of samples for 1%, 10%, and 100% of the training datasamples_100 = len(y_train)samples_10 = int(len(y_train)/10)samples_1 = int(len(y_train)/100)# Collect results on the learnersresults = {}for clf in [clf_A, clf_B, clf_C]:    clf_name = clf.__class__.__name__    results[clf_name] = {}    for i, samples in enumerate([samples_1, samples_10,   samples_100]):        results[clf_name][i] = \        train_predict(clf, samples, X_train, y_train, X_test, y_test)# Run metrics visualization for the three supervised learning models chosenvs.evaluate(results, accuracy, fscore)
Finally, in this section we will choose the best model to use on our data, and then, perform a grid search optimization for ther model over the entire training set (X_train and y_train) by tunning parameters to improve the model’s F-score.
Choosing the Best Model
Based on the results of the evaluation, the most appropiate model to identify potential donors is the Random Forest Classifier as it yields the same F-score of the Support Vector Classifier but in much less time.
This is coherent with our knowledge of the algorithm as it is a very good choice when dealing with high-dimensional datasets, in other words, datasets with a large number of features.
So among the evaluated models, this is the more efficient one and best the suited to work with our dataset.
Describing Random Forest in Layman’s Terms
To understand Random Forests Classificators we need first to introduce the concept of Decision Trees. A Decision Tree is a flowchart-like structure, in which each internal node represents a test on an attribute of the dataset, each brand represents the outcome of the test and each leaf represents a class label. So the algorithm will make the tests on the data, finding out which are the most relevant features of the dataset to predict a certain outcome, and separating accordingly the dataset.
The following is an example of a decision tree to decide wheter or not you lend your car to someone:
A random forest is a meta estimator that fits a number of decision tree classifiers on various sub-samples of the dataset and uses averaging to improve the predictive accuracy of the model, and control overfitting by preventing it to become too complex and unable to generalize on unseen data. It randomly selects a number of features and trains each decision tree classifier in every sub-set of the features. Then, it each decision tree to make predictions by making them vote for the correct label.
Random Forests classifiers are quite popular in classification problems due to its simplicity to use, efficiency and its good predicting accuracy.
Model Tunning
We will now fine tune the chosen model, using GridSearchCV.
# Import the necessary librariesfrom sklearn.model_selection import GridSearchCVfrom sklearn.metrics import make_scorer# Initialize the classifierclf = RandomForestClassifier(random_state = 42)# Create the parameters list parameters =  {    'max_depth': [10,20,30,40],    'max_features': [2, 3],    'min_samples_leaf': [3, 4, 5],    'min_samples_split': [8, 10, 12],    'n_estimators': [50,100,150]}# Make an fbeta_score scoring object using make_scorer()scorer = make_scorer(fbeta_score, beta=0.5)# Perform grid search on the classifier grid_obj = GridSearchCV(estimator=clf, param_grid=parameters, scoring=scorer)# Fit the grid search object to the training data grid_fit = grid_obj.fit(X_train, y_train)# Get the estimatorbest_clf = grid_fit.best_estimator_# Make predictions using the unoptimized and modelpredictions = (clf.fit(X_train, y_train)).predict(X_test)best_predictions = best_clf.predict(X_test)# Report the before-and-afterscoresprint("Unoptimized model\n------")print("Accuracy score on testing data {:.4f}".format(accuracy_score(y_test, predictions)))print("F-score on testing data: {:.4f}".format(fbeta_score(y_test, predictions, beta = 0.5)))print("\nOptimized Model\n------")print("Final accuracy score on the testing data: {:.4f}".format(accuracy_score(y_test, best_predictions)))print("Final F-score on the testing data: {:.4f}".format(fbeta_score(y_test, best_predictions, beta = 0.5)))
# Show the best classifier hyperparametersbest_clf
Observations
The optimized model’s accuracy and F-score on testing data are: 84.8% and 71.38% respectively.
These scores are slightly better than the ones of the unoptimized model but the computing time is far larger.
The naive predictor benchmarks for the accuracy and F-score are 24.78% and 29.27% respectively, which are much worse than the ones obtained with the trained model.
Extra: Feature Importance
An important task when performing supervised learning on a dataset like his one is determining which features provide the most predictive power. By focusing on the relationship between only a few crucial features and the target label we simplify our understanding of the phenomenon, which is most always a useful thing to do.
In the case of this project, that means we wish to identify a small number of features that most strongly predict whether an individual makes at most or more than $50,000.
Intuitively, of the 13 features available on the original datasetwe can infer that the most important ones for predicting the income are:
1) age
2) education
3) native-country
4) occupation
5) hours per week
The rank’s order logic is the following:
It is likely that a person’s income would increase over the years and older people tend to earn more money than younger ones.
In additon, people who have higher education tend to get higher paid jobs and this is a factor that is also heavily related with the native country as, usually, people that are native of economically stronger countries tend to have access to higher education.
Occupation is an important feature to take into consideration as yearly income vary a lot depending on the industry and sector.
Finally, hours per week as normally, people who work more hours tend to earn more.
We will now chek the accuracy of our logic by passing our data through a model that hast the feature_importance_ method:
# Import Ada Boost Classifierfrom sklearn.ensemble import AdaBoostClassifier# Train the supervised model on the training model = AdaBoostClassifier().fit(X_train, y_train)# Extract the feature importances using .feature_importances_ importances = model.feature_importances_# Plotvs.feature_plot(importances, X_train, y_train)
The intution followed in the previous section was partially right as the AdaBoost tests shows that feautures like age, hours per week and education are quite relevant to predict the income. However, we didn’t indentified capital-loss and capital-gain.
Feature Selection
Now it is reasonable to ask the following:
How does a model perform if we only use a subset of all the available features in the data?
With less features required to train, the expectation is that training and prediction time is much lower — at the cost of performance metrics. From the visualization above, we see that the top five most important features contribute more than half of the importance of all features present in the data. This hints that we can attempt to reduce the feature space and simplify the information required for the model to learn.
# Import functionality for cloning a modelfrom sklearn.base import clone# Reduce the feature spaceX_train_reduced = X_train[X_train.columns.values[(np.argsort(importances)[::-1])[:5]]]X_test_reduced = X_test[X_test.columns.values[(np.argsort(importances)[::-1])[:5]]]# Train on the "best" model found from grid search earlierclf = (clone(best_clf)).fit(X_train_reduced, y_train)# Make new predictionsreduced_predictions = clf.predict(X_test_reduced)# Report scores from the final model using both versions of dataprint("Final Model trained on full data\n------")print("Accuracy on testing data: {:.4f}".format(accuracy_score(y_test, best_predictions)))print("F-score on testing data: {:.4f}".format(fbeta_score(y_test, best_predictions, beta = 0.5)))print("\nFinal Model trained on reduced data\n------")print("Accuracy on testing data: {:.4f}".format(accuracy_score(y_test, reduced_predictions)))print("F-score on testing data: {:.4f}".format(fbeta_score(y_test, reduced_predictions, beta = 0.5)))
Observations of the Effects of Feature Selection
Both accuracy and f-score are lower on the reduced data than on the original dataset. Specially f-score 71.38% vs 67.57%
Considering the metrics obtained in the evaluations made on the default model, the optimized model and the reduced dataset, the best option would be to use the default model version with the complete dataset as it yields a good combination of accuracy and f-score in a good training time.
Throughout this article we made a machine learning classification project from end-to-end and we learned and obtained several insights about classification models and the keys to develop one with a good performance.
This was the third of the machine learning projects that have been explained inthis series. If you liked it, check out the previous one:
Regression Project
Naive Bayes Project
Stay tuned for the next article! Which will be an introduction to the theory and concepts regarding to Unsupervised Learning.
If you liked this post then you can take a look at my other posts on Data Science and Machine Learning here.
If you want to learn more about Machine Learning, Data Science and Artificial Intelligence follow me on Medium, and stay tuned for my next posts!