Predictive Modeling and Multiclass Classification: Tackling the Taarifa Challenge | by Antonio Stark | Towards Data Science

Classification predictive problems are one of the most encountered problems in data science. In this article, we’re going to solve a multiclass classification problem using three main classification families: Nearest Neighbors, Decision Trees, and Support Vector Machines (SVMs).
The dataset and original code can be accessed through this GitHub link.
This article tackles the same challenge introduced in this article. While this article is a standalone for predictive modeling and multiclass classification, if you are wondering how I cleaned the dataset for use in modeling, you can check out that article as well! (also, if you came straight from that article, feel free to skip this section!)
The challenge I’ll use for this article is taken from Drivendata.org. You can think of it as a Kaggle for social impact challenges. You still get the same perks for winning and pretty well-formatted datasets, with the additional benefit that you’ll be making a positive impact on the world!
The particular challenge that we’re using for this article is called “Pump it Up: Data Mining the Water Table.” The challenge is to create a model that will predict the condition of a particular water pump (“waterpoint”) given its many attributes.
The data is provided by Taarifa, an open-source API that gathers this data and presents it to the world. If you’re curious about their work and what their datapoints represent, make sure to check out their website (and GitHub!)
Our original dataset (as provided by the challenge) had 74,000 data points of 42 features. In the previous article about data preprocessing and exploratory data analysis, we converted that into a dataset of 74,000 data points of 114 features. The increased number of features is mainly from one-hot encoding where we expanded categorical features into multiple features per category. We’ve actually eliminated more than half of the features before one-hot encoding, from 42 features to just 20. One-hot encoding on the remaining 20 features led us to the 114 features we have here.
Currently, our test dataset has no labels associated with them. In order to see the accuracy of our models, we need labels for our test dataset as well. So as painful as it is, we’re going to discard the test dataset for now. We can re-use this after we’ve trained all of our models and decided which model we want to use for the final submission.
train = df[df.train==True].drop(columns=['train'])X = train.drop(columns=['status_group'])y = train.status_group
We’ll create an artificial test dataset from our training data as the train data all have labels. Let’s take a one-third random sample from our training dataset and designate that as our testing set for our models. We can use the train_test_split package from scikit-learn (or, “sklearn”).
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.33, random_state=1)
Let’s quickly re-check our label balances here.
It seems like our random splitting did pretty well! Our output balance is pretty identical to both our training and testing dataset. The output classes are a bit imbalanced, we’ll get to that later.
While there are many types of classifiers we can use, they are generally put into these three families: nearest neighbors, decision trees, and support vector machines. We’re going to look at one example model from each family of models.
Do note that our task is a multi-class classification problem. While there are ways to do multi-class logistic regression, we’re not doing it here.
There are also a lot of relationships between our features. This is either because they correspond to similar aspects (e.g. latitude and longitude), or are results of one-hot encoding. That’s why we won’t be doing a Naive Bayes model here as well.
The metric employed by Taarifa is the “classification rate” — the percentage of correct classification by the model. Let’s look at the classification rate and run time of each model.
# packages required for metricsimport time, mathfrom sklearn.metrics import accuracy_scorecols_results=['family','model','classification_rate','runtime']results = pd.DataFrame(columns=cols_results)results
For our Nearest Neighbors classifier, we’ll employ a K-Nearest Neighbor (KNN) model. A KNN is a “lazy classifier” — it does not build any internal models, but simply “stores” all the instances in the training dataset.
Therefore, a KNN has no “training time” — instead, it takes a lot of time in prediction. It is especially awful when we have a large dataset and the KNN has to evaluate the distance between the new data point and existing data points.
Let’s see how a KNN does in accuracy and time for k = 1 to 9. (Remember a KNN of k=1 is just the nearest neighbor classifier)
from sklearn.neighbors import KNeighborsClassifierkVals = range(1,10)knn_names = ['KNN-'+str(k) for k in kVals]for k in kVals:    knn = KNeighborsClassifier(n_neighbors=k)    knn.fit(X_train, y_train)        time_start = time.time()    y_pred = knn.predict(X_test)    time_run = time.time()-time_start        results = results.append(pd.DataFrame([['KNN',knn_names[k-1],accuracy_score(y_test,y_pred),time_run]],columns=cols_results),ignore_index=True)results[results.family=='KNN']
Okay, so we have our KNNs here. Let’s visualize how well they’ve done and how much time they’ve took
fig, ax = plt.subplots()ax.plot(kVals,results[results.family=='KNN'].classification_rate,color='blue',marker='o')ax.set_xlabel('k-value for KNN models')ax.set_ylabel('classification rate (blue)')ax2=ax.twinx()ax2.plot(kVals,results[results.family=='KNN'].runtime,color='red',marker='o')ax2.set_ylabel('runtime (seconds; red)')plt.show()
The accuracy does fluctuate a bit at first but gradually stabilizes around 67% as we take more neighbors into account. The runtime generally increases linearly with k-value.
For our Decision-Tree based model, we’ll create a random forest. Random forest is a method of creating multiple decision trees over a subset of the training dataset and taking the consensus result. Because of this random subsetting method, random forests are resilient to overfitting but takes longer time to train than a single decision tree.
Let’s see how random forests of 1 (this is just a single decision tree), 10, 100, and 1,000 trees fare.
from sklearn.ensemble import RandomForestClassifierrVals = range(1,4)rf_names = ['RF-'+str(int(math.pow(10,r))) for r in rVals]for r in rVals:    clf = RandomForestClassifier(n_estimators=int(math.pow(10,r)),random_state=0)    time_start = time.time()    clf.fit(X_train,y_train)    time_run = time.time()-time_start    y_pred=clf.predict(X_test)        results = results.append(pd.DataFrame([['RF',rf_names[r-1],accuracy_score(y_test,y_pred),time_run]],columns=cols_results),ignore_index=True)    results[results.family=='RF']
Let’s see how they visualize.
The runtime increases almost exactly linearly with the number of decision trees in the random forest, as to be expected. The accuracy, however, does not increase much and is in the 80% ballpark.
SVMs utilize what’s known as a “kernel trick” to create hyperplane separators for your data (i.e. linear separators for non-linear problems). While SVMs “could” overfit in theory, the generalizability of kernels usually makes it resistant from small overfitting.
SVMs do tend to take a lot of time, and its success is highly dependent on its kernel. A pure SVM on this dataset (40k data points of 100 features) will take forever to run, so we’ll create a “bagged classifier” using the BaggingClassifier library offered by sklearn. The BaggingClassifier will take a base model (for us, the SVM), and train multiple of it on multiple random subsets of the dataset. For us, let’s train 10 SVM models per kernel on 1% of the data (about 400 data points) each time.
Let’s see how well our model works for three different kernels: linear, RBF, and sigmoid. The Sklearn SVC library also gives us the poly kernel, but it was taking forever to train even in the reduced dataset, so we’re not doing it here.
Let’s also visualize the accuracy and run time of these SVM models.
The linear kernel does show the highest accuracy, but it has a horrible training time. Considering that we took a bagging approach that will take at maximum 10% of the data (=10 SVMs of 1% of the dataset each), the accuracy is actually pretty impressive.
Let’s compare the accuracy and runtime of all of our models!
It seems like random forests give the best results — nearly 80% accuracy! A random forest with just 100 trees already achieves one of the best results with only nominal training time. Linear SVMs and KNN models give the next best level of results. But apart from comparing models against each other, how can we “objectively” know how well our models have done?
For any classification task, the base case is a random classification scheme. One way to do this is to create a random classifier that will classify the input randomly and compare the results. The random assignment of labels will follow the “base” proportion of the labels given to it at training.
from sklearn.dummy import DummyClassifierclf = DummyClassifier(strategy='stratified',random_state=0)clf.fit(X_train,y_train)y_pred = clf.predict(X_test)print('Accuracy of a random classifier is: %.2f%%'%(accuracy_score(y_test,y_pred)*100))
It seems like for a base accuracy of 45%, all of our models have done pretty well in terms of accuracy.
Lastly, we come back to the class imbalance problem that we’ve mentioned at the beginning. The imbalance in labels leads classifiers to bias towards the majority label. For our case, it’s towards the ‘functional’ label.
We’ve already seen that a classifier that predicts the ‘functional’ label for half the time (‘functional’ label takes up 54.3% of the dataset) will already achieve 45% accuracy. This is already far better than a uniform random guess of 33% (1/3).
Class imbalance may not affect classifiers if the classes are clearly separate from each other, but in most cases, they aren’t. To see whether or not class imbalance affected our models, we can undersample the data.
Balanced undersampling means that we take a random sample of our data where the classes are ‘balanced.’ This can be done using the imblearn library’s RandomUnderSampler class. Think of imblearn as a sklearn library for imbalanced datasets.
from imblearn.under_sampling import RandomUnderSamplerrus = RandomUnderSampler(random_state=0)X_rus_train,y_rus_train = rus.fit_resample(X_train,y_train)X_rus_test,y_rus_test = rus.fit_resample(X_test,y_test)
While our original X_train had almost 40,000 data points, our undersampled dataset only has about 8,700 data points. Let’s run it through our most successful model — random forests — and see if undersampling affects our model accuracy.
Let’s retrain our most successful models — our random forests — on this undersampled dataset.
rf_rus_names = ['RF_rus-'+str(int(math.pow(10,r))) for r in rVals]for r in rVals:    clf = RandomForestClassifier(n_estimators=int(math.pow(10,r)),random_state=0)    time_start = time.time()    clf.fit(X_rus_train,y_rus_train)    time_run = time.time()-time_start    y_rus_pred=clf.predict(X_rus_test)        results = results.append(pd.DataFrame([['RF',rf_rus_names[r-1],accuracy_score(y_rus_test,y_rus_pred),time_run]],columns=cols_results),ignore_index=True)    results[results.family=='RF']
So our model accuracy has decreased from close to 80% to under 70%. A part of this is from the fact that the model has had a reduced dataset to work with. But another factor is that our original Random Forest models were getting a falsely “inflated” accuracy due to the majority class bias, which is now gone after classes have been imbalanced. At the same time, balancing of classes does lead to an objectively more accurate model, albeit not a more effective one. The ultimate decision is yours to make — would you care about “inflated” accuracy, or would these “false positives” deter you from using the original models?
As our “false positives” may lead us to declare non-functional or in-need-of-repair waterpoints to go unaddressed, we might want to err the other way, but the choice is up to you.