Machine Learning Mini-Project: Early Stage Diabetes Prediction
By: Peijin Chen
Tags: Machine Learning, Data Science Courses, Diabetes, Data Science, Random Forest Classifiers

Start to finish on some open source data.

As a teacher, I’ve had to rely on video conferencing this last half year, but until now I have never bothered sharing any of my personal projects or explorations via video, until now. I have a YouTube playlist where I take you through various aspects of a simple machine learning project. In this case, it’s Early Stage Diabetes Prediction, a basic binary outcome supervised classification project that involves a modest number of mostly categorical features. In this post, I’ll give you a TLDR type overview as well as some details, and you can check out my videos or my public notebook to really delve into the code and the ideas.

DATA

The dataset is publicly available here, as part of the UCI Machine Learning Repository:

The paper is entitled Likelihood Prediction of Diabetes at Early Stage Using Data Mining Techniques (Islam et al., 2019), and you can find it here, though unfortunately it is behind a paywall. I have it, and will share some of it, hopefully in a way that doesn’t get me in hot water.

My Google Colab Notebook with all the code is here, and should be publicly available to read:

My YouTube series on this project, so far, 8 videos, taking you through some of the code that you’ll find in the Colab notebook above. This is a living playlist, meaning that I might add some extra videos on this topic, especially if there are requests or audience feedback. The videos are labeled 1–8 and include the subtopic covered in the video as well.

ORIGINAL PAPER RESULTS: Random Forest Regressor is the best and can achieve 97.4% accuracy. They compare this against decision trees and other learners. They use both 10-fold cross-validation and 80/20 split.

OUR RESULTS: We confirm that RF is probably indeed the best, but we compare to a wider range of learners, including GBDTs like CatBoost, LightGBM, XGBoost, as well as support vector machines and multi-layer perceptrons. We find that CatBoost and LightGBM are the second and third best choices. Unlike the authors, we use repeated stratified k-fold cross-validation to get a more “confident” estimate of the learner’s performance. Also, we don’t use balanced accuracy or F1 scores to judge the classifier. We use the Matthews Correlation Coefficient (MCC, more about this later). Furthermore, we offer some feature importance analysis that the original paper does not, using SHAP values and decision plots.

EXPLORATORY DATA ANALYSIS

The authors of the paper state that they had 520 questionnaire responses but that 20 of them had missing values, so that they ended up using 500 rows. You can see in their results, shown below, that they have a 400/100 split when they do 80/20 and 500 when they do cross-validation. However, when you download the data set from UCI, there are no missing values at all. All 520 values seem to be there. I’m not sure if they explained this anywhere, and frankly, for our purposes, which are mostly educational, I don’t care that much. But know that the results I am sharing here with you are for 520 data records. Here’s a glimpse of what the data looks like

Subset of the Data Set

DATA PREPROCESSING — Binning

D.isna().sum() #You'll find no NAs even though the authors said there were!!! #All the features, except age, are categorical. Heck, just make age categorical while we’re at it. How about 5 categories? D["Age"] = pd.qcut(D["Age"], q = 5).astype("category").cat.codes #qcut bins the data, 5 equal bins. Convert into category so its clear that this categorical data.

The target is the last column. So once you separate features and target into X and Y, do this:

X1 = X.copy()

Y1 = Y.copy() for col in catcols:

X1[col] = X1[col].astype("category")

X1[col] = X1[col].cat.codes

Y1 = Y1.astype("category")

Y1 = Y1.cat.codes #The cat.codes part will convert the string data {"Yes","No"} or {"Male","Female"} into {0,1}. Everything is 0 and 1 and now, except for the Age category which is {1,2,3,4,5}

EXPERIMENTS

MODEL SELECTION

The first thing I chose in designing these experiments was the error measure: the Matthews Correlation Coefficient. Why ? Well, it is a balanced measure, unlike accuracy or F1, precision or recall, which can be influenced by the threshold (e.g., the value of the logit being above 0.5 or not) as well as the (im)balance of the classes, where large classes have a disproportionate effect on the error metrics, thus influencing them to seem better than they are. Of course, variants of some measures can be made more balanced, but for simplicity, I’m sticking to one that we know offers a balanced view of performance. For the interested reader, there is a paper that argues that the MCC should be the error measure of choice when it comes to binary classification.

From Wikipedia

We want to use cross-validation to get a better sense of how generalizable our models are, that is, how well they do unseen data. We use the sklearn library’s RepeatedStratifiedKFold cross-validation scheme to help us get a more realistic sense of how our models perform on unseen or highly variegated data. We’ll use 10 splits of the data (in part because the authors of the original paper did as well) and 5 repeats of the experiment (because I don’t have the patience for 100 much less 1000 trials).

rs = np.random.randint(1000) ; print("Seed", rs) rskf = RepeatedStratifiedKFold(n_splits = 10, n_repeats = 5, random_state = rs)

The authors of the paper used Logistic Regression, the J48 Decision Tree, Random Forests, and the Naive Bayes Classifier. On the other hand, for this project we use Stochastic Gradient Descent Classifier, Multilayer Perceptron Classifier, Decision Trees, Random Forest, XGBoost, LightGBM, CatBoost, Logistic Regression, Ridge Classifier, K Nearest Neighbors Classifier, and the Support Vector Machine.

I would like to start off with “tryouts”, wherein each learner is trained with their default values. Pictured below is an excerpt from that code. To evaluate these models, I created a series of empty lists, one for each model/learner, because the cross_val_score function in sklearn, which does the CV and evaluation, returns a list of scores from each fold — but I would rather use the mean of these scores to get just one overall score. This code excerpt below includes just 3 learners (for the sake of space) — but you can see the entirety of the code in my Colab notebook. Let’s see who makes the cut.

rfscores = []

xgbscores = []

lrscores = [] reg = RandomForestClassifier(verbose = 0,random_state = rs)

reg.fit(X1,Y1)

rfscores.append(cross_val_score(reg, X1, Y1, scoring = mcc, cv = rskf)) reg = XGBClassifier(random_state = rs)

reg.fit(X1,Y1)

xgbscores.append(cross_val_score(reg, X1, Y1, scoring = mcc, cv = rskf)) reg = LogisticRegressionCV(cv = 5, refit = True, random_state = rs, max_iter = 500)

reg.fit(X1,Y1)

lrscores.append(cross_val_score(reg, X1, Y1, scoring = mcc, cv = rskf))

Now, we find the mean scores of each list of CV scores and then compare:

The First Round of Tryouts

We can see, even without any tuning, that the tree-based algorithms do the best. Random Forest seems to be head and shoulders above the rest, which is consistent with results from the original paper. However, it might be worth noting that the MLP isn’t that bad either and might even surprise us if we spend time tuning the hyperparameters. I will not claim here that my testing has been exhaustive, but I can say with some measure of confidence that the Support Vector Machine (SVM), which might normally get eliminated given the performance above, can in fact, find itself in the running for best model — it’s all in the tuning, my friend.

TUNING HYPERPARAMETERS

The code I used for each learner is almost identical; just change the learner and the hyperparameters. If you are not sure which hyperparameters to tune, first of all, Google something like “most important xgboost hyperparameters”. You will get a sense from everything you read about which ones are the most important, and you will what other people have used as value ranges. The approach that I am using here is not grid search (exhaustive testing of various combinations of HP values, which is time-consuming), nor is it random search (randomly selecting “grid points” in the HP space). This is called Bayesian optimization, and here is the explanation from Wikipedia:

Applied to hyperparameter optimization, Bayesian optimization builds a probabilistic model of the function mapping from hyperparameter values to the objective evaluated on a validation set. By iteratively evaluating a promising hyperparameter configuration based on the current model, and then updating it, Bayesian optimization, aims to gather observations revealing as much information as possible about this function and, in particular, the location of the optimum…In practice, Bayesian optimization has been shown to obtain better results in fewer evaluations compared to grid search and random search, due to the ability to reason about the quality of experiments before they are run.

That said, there are many python libraries for handling Bayesian type optimization. The one that I use, which I think is fairly simple but reliable, is bayesian-optimization, and I recommend it not only for that but because it has been better than some other libraries in terms of, you know, actually finding optimal (or at least really good) values.

I made choices as to which hyperparameters were chosen for optimization, and the ranges as well, some of which is based on what others do, and some of it just learned through trial and error — on this data set. Don’t assume that the values will be the same for other data sets. Sometimes you should start with a small range around the default values, just to see if you can get an incremental improvement. If you’re not satisfied with that, I would expand the HP ranges — but don’t necessarily expand all of them at once. If you suddenly expand the HP space too much, the algorithm can get lost, with one “bad apple” HP ruining the performance of the learner as a whole.

Random Forest Regressor

The optimization algorithm for the 2 HP’s I tested here resulted in max_depth = 14 and n_estimators = 173. Then I used these values and refit the model to the data, finding both CV-based MCC scores.

Random Forest Results, after tuning. The CV-MCC score is over 0.95, where 1 is the maximum value.

You can see that it has a good CV-MCC score of .952, and makes a total of 3 errors when trained on the whole set. I believe the CV-MCC score is a better estimate of this model’s power. Remember, the values of the MCC range between -1 and 1, where 1 means perfect correlation and therefore a perfect classifier.

Support Vector Machine

Support Vector Classifier results after tuning.

With the SVM hyperparameter C (cost) and gamma (bandwidth) tuned properly, the CV-MCC score comes out to 0.939, which is almost as good as the Random Forest. Trained on the whole data set, it performs exactly the same as RF model, making 3 errors. On other runs of this experiment, I have seen the SVM scores reach as high as 0.95, putting it very close to the Random Forest Model. However, there is a bit more variance in the SVC scores than the RF scores, leading me to believe that RF might be better in terms of consistency.

LightGBM

LightGBM after tuning

LightGBM is fast and efficient GBDT implementation, and the results after tuning also look decent, compared with the other methods. The overall CV-MCC score of 0.937 isn’t bad at all.

CatBoost

CatBoost is yet another fast and popular GBDT algorithm, and handles categorical data quite easily (hence the name CATboost). This comes in handy, since all of our features are categorical. Here are the results:

Catboost after tuning.

You can see that the CV-MCC score of 0.943 beats out the LightGBM and the SVM, but is still second place to the Random Forest . When trained on the entire data set, it makes 3 errors, like the other top learners.

Extra note: I did try stacking, but it didn’t seem to help that much.

Conclusion: The best learners for this problem are probably Random Forest and CatBoost, though the SVM with radial basis function kernel should be considered as well.

FEATURE IMPORTANCE ANALYSIS

SHAP VALUES: SHAP values are one way of measuring feature importance. The core of the idea came from the game theorist Edward Shapley, who studied how coalitions affect the outcomes of games. In our machine learning context, we think of each feature as a “player” of the game, and certain groupings or match-ups of players tends to have certain results. Think of tug of war — if you are evenly matched, the rope the stays in the middle (the mean value of the model’s predictions/output); however, if you have a strong “player” on one side — like the feature Polyuria or the feature Gender — the model will predict something either higher or lower than the mean. Of course, that depends on what Gender shows up today at the tug of war match. If, in this game (i.e., singular data point), it is a 1 (as opposed to 0), and teams up with Polyuria as a 1, then you might see the model pulled more decisively to 1 or 0. There are some nice properties to SHAP values — for example, additivity — which means that the effects of each player adds up to the outcome of every particular game. The point of all of this, in the modern machine learning context, is that we want more interpretable machine learning models. We want to know what features really matter, and why. With that in mind, let’s take a look at some bar charts, where we look at the average SHAP value of each feature over the entire data set. Remember, a “player” like Polyuria might sometimes have a great effect on the model (maybe when it has a value of 1), and sometimes have a smaller effect. But we can measure the overall impact size, and that’s what the following graphs will do.

Feature Importance/Random Forest. Here’s a feature importance bar graph for the Random Forest model:

SHAP values for the RF model

We can see that Polyuria (abnormally large volumes of dilute urine), Polydipsia (abnormally great thirst), and Gender are the 3 most important features.

Feature Importance/CatBoost

SHAP values for CatBoost

From the graph, we can see that CatBoost agrees on the fact that Polyuria and Polydipsia are important, as well as Gender. However, itching, irritability, and sudden weight loss — considered somewhat important by the Random Forest model — take on a greater importance with CatBoost. Both learners agree that muscle stiffness, delayed healing, weakness, and obesity tend to be of lesser importance in terms of influencing the model’s output.

Feature Importance/SVM

SHAP values on a random sample of 300 of the 520 total data points. We chose a sample because this graph takes a lot longer than the ones for tree-based learners.

This is interesting — delayed healing and visual blurring are considered important features by the SVM, whereas the GBDT methods tend to rank those quite a bit lower. However, the presence of Polyuria, Polydipsia, and Gender as the most important features seems consistent with the results from the other models. Those most likely really are the most important features, no matter what model you use.

DECISION FUNCTIONS

But how do these learners actually make decisions?Let’s take a look at what the SHAP value decision plot for the Random Forest model, which, to the best of our knowledge, is the best overall model. Trace the path for the data points, starting at the bottom and ending at the top.

SHAP value decision plot. You can see strong “branches” that roughly show how certain features push the model output in one direction or another.

One can see that the real “separation” happens with 8–10 most important, most discriminating features. The amount of separability of the data is actually quite amazing, in that there isn’t a lot of “crossover” — instances that might start at the bottom seeming to be in one class, only to make a huge “jump” once it reaches a more important feature. The two classes are really “pushed” apart by the first 3 features, which confirms our view that they are truly essential to any good model, regardless of underlying algorithm.

FEATURE INTERACTIONS

The nice thing about GBDTs is that they make for nonlinear models. Now you could model all kinds of nonlinear interactions and effects if you were to expand your features to include, say, polynomial powers and interaction terms of the original features. By this I mean that maybe your target Y is not a simple linear function of X, but instead is a function of X². Maybe you have 2 features, such as X1 and X2, and the interaction term of X1*X2 is important. You might imagine something like Race*Gender or Age*Education as interactions terms that matter in some econometric or sociological study. Sometimes, in order to achieve greater Gaussianity of your features, which is helpful in traditional regression methods, you might want to use some log-transformed version of X, such as log(1 + X). In this project, I have not used any Box-Cox/Yeo-Johnson nor any polynomial features in addition to the original feature set. This is not because I know this approach doesn’t work (I haven’t tested it), but for the sake of simplicity in presenting this project. That said, regression tree methods do allow for first-order interactions of the X1*X2 type, and we use the shap library to graph which interactions are actually important to the model:

Interaction Values between Features, as interpreted through SHAP values. Notice that the Gender*Polyuria interaction seems fairly high up in terms of importance.

We can see that Gender*Polyuria, Polyuria*Polydipsia, Gender*Polydipsia are all important feature interactions, and are in fact more important than many of the original features. This offers a more nuanced view than that of the original bar chart of feature importances. Alopecia, which is a kind of hair loss, shows up among the interaction terms as well.

ERRORS

What is going on with the 3 data points that even the best models can’t seem to get right? Let’s take a look at the features for those 3 records :

A subset of the features for the 3 records that even the Random Forest model can’t get right.

So we are talking about an age group that has 0 for every possible feature — meaning they do not exhibit any of the symptoms on the questionnaire. You don’t have to be a doctor to classify someone like that as not being at risk for diabetes, and yet these records were classified as positive. This begs questions such as “is that data accurate? Or is something wrong with those 3 records?” or “what other factors might we want to add in as features to better help discriminate between the two classes?”

CONCLUSION

Here’s something from the original paper:

The authors used Naive Bayes (NB), Logistic Regression (LR), J48 Decision Tree, and Random Forest (RF). These are the results from their 10-fold cross-validation based experiments.

I hope you’ve enjoyed this little tour of a project that I worked on in my spare time. I enjoy finding (manageable) data sets on the internet, whether from UCI or Kaggle or others, and then running various models to see what kind of performance we can achieve, and what really matters to the model. With this mini-report, we found that, as did the authors, that the random forest model is well-suited to this problem. Furthermore, we included various GBDT learners such as CatBoost and LightGBM, and found they were comparable, as as SVM with radial basis function kernel. Results such as these are not mentioned in the original paper. Lastly, when we look at SHAP values, we see that each model has a different ranking of feature importance, but that Polyuria, Polydipsia, Gender, Age, and other such factors tend to consistently rank higher than others. We also saw the influence of interaction effects, especially with Polyuria.

Please check out my YouTube channel and my Google Colab notebook for more details. I invite any criticism or suggestions, and by the way, I am looking for a new job, so if you think that any of my machine learning or data science skills might be of use to me, feel free to contact me on LinkedIn or right here in the comments. There will be more projects and tutorials coming soon!