Machine Learning Mini-Project 2: Hepatitis C Prediction from Blood Samples.
By: Peijin Chen
Tags: Logistic Regression, Artificial Intelligence, Classification Algorithms, Data Science, Machine Learning

Photo by Karolina Grabowska from Pexels

This is the second in my Machine Learning Mini Project series, where I take some public data sets, such as those found in the UCI Machine Learning Repository (which are often used in peer reviewed papers, so have some level of vetting), and see what I can learn using basic and advanced machine learning models. I don’t pretend to have any expert knowledge, nor am I sharing any state-of-the-art results or methods. These are meant as explorations, with my notebooks and code being publicly accessible. As with anything else, machine learning is a practice, and practitioners should play around with a variety of data and learn what methods and workflows are germane to their subject, or to the aims of their research.

The link above cites two papers:

The Enhanced Liver Fibrosis (ELF) score: normal values, influence factors and proposed cut-off values. Lichtinghagen, R. et al. J Hepatol 2013; 59: 236–42

The second article focuses more on machine learning contributions:

Hoffmann, Georg, et al. “Using machine learning techniques to generate laboratory diagnostic pathways — a case study.” J Lab Precis Med 3 (2018): 58.

This is a SUPERVISED LEARNING/CLASSIFICATION problem with 5 classes. Here are the 5 target categories that we are trying to predict: (values: ‘0=Blood Donor’, ‘0s=suspect Blood Donor’, ‘1=Hepatitis’, ‘2=Fibrosis’, ‘3=Cirrhosis’)

As you can see from the graph below, the classes are highly imbalanced, with most of the instances being the healthy blood donors.

An imbalanced data set.

Preliminaries: Python Libraries

The libraries that I use are numpy, pandas, smote_variants, lightgbm, sklearn, missingno, collections, seaborn. I’m running this whole thing in Google Colab notebook. Most of these can be installed with a quick !pip install smote_variants type command in your notebook or IDE of choice.

Data Explanations

FEATURES

Here’s an explanation of what some of the features stand for. Some of them are diagnostic blood tests used to detect forms of liver damage or disease. I read the papers, and somehow there doesn’t seem to be a complete explanation of every feature in this data set, leading me to believe they either didn’t use all of it or just used part of it and combined it with some data of their own. Anyway, here’s what I could figure out.

ALB = albumin

BIL = bilirubin

CHE = choline esterase

GGT = γ-glutamyl-transferase

AST = aspartate amino-transferase

ALT = alanine amino-transferase

CREA = creatinine?

PROT = ??

ALP = ?

CHOL = cholesterol?

Here’s the definition of the AST from From WebMD:

The aspartate aminotransferase (AST) test is a blood test that checks for liver damage. Your doctor might order this test to find out if you have liver disease and to monitor your treatment.

Data Preprocessing

MISSING VALUES

Missing Values, as visualized by the missingno Python library

One of our most obvious problems are missing values, but thankfully there aren’t that many, so we should just use some form of imputation to fill them out. I tried “median” and “mean”, that is, just putting in the mean and median values of the column where the missing value is located. Here’s the code for that:

from sklearn.impute import * imp_mean = SimpleImputer(missing_values=np.nan, strategy = 'median')

Data = pd.DataFrame(imp_mean.fit_transform(Data), columns = Data.columns)

POWER TRANSFORMS

Generally, you want your features, if possible, to have more Gaussian (normal, non-skewed) distributions. This is because traditional statistical methods work better under this assumption. That said, I have found that the probability distribution function (PDF) of these features doesn’t always matter much for tree-based classifiers such as the Random Forest or XGBoost, but I did the transform anyway, with the exception being gender, the one categorical variable, which will keep its binary encoding of 0 or 1. Here’s an example of how the PowerTransformer() from sklearn changes the PDF of the AST feature.

The Power Transform results in a more Gaussian(normal) distribution.

Here’s the code we can use to accomplish that:

numcols = list(X.columns)

numcols.remove("Sex") for c in numcols:

pt = PowerTransformer()

X[c] = pt.fit_transform(X[c].values.reshape(-1,1))

Model Selection

Let’s first try the DummyClassifier() from sklearn, and we can see that there are various “strategies” for this naive classifier to take, i.e. stratified, most_frequent, and prior. You might think, “why would I want a dumb classifier” — but you do, because you should never lose sight of how much better (or worse) a machine learning algorithm is compared to a “dummy” that just guesses outcomes based on simple rules. If a dummy could guess 96% of the outcomes by always guessing the majority class, then the fact that your learner can achieve 96.1% accuracy isn’t something you should be too excited about. Let’s try a DummyClassifier out, using 3 strategies.

from sklearn.dummy import * rkf = RepeatedStratifiedKFold(n_splits = 4, n_repeats = 5, random_state = 0)

strats = ["stratified", "most_frequent", "prior"] for strat in strats:

scores = cross_val_score(DummyClassifier(strategy=strat, X, Y,

cv = rkf,

scoring = "accuracy") print(np.mean(scores)) >>0.7509061200237671 0.8666709107885578 0.8666709107885578

See? Guessing the most frequent class the whole time is going to get an accuracy of 0.867. That’s not skill, that’s just class imbalance. OK, let’s actually try a real algorithm now, like Logistic Regression (LogisticRegression() in sklearn):

mcc = make_scorer(matthews_corrcoef) rkf = RepeatedStratifiedKFold(n_splits = 4, n_repeats = 5, random_state = 0) scores = cross_val_score(LogisticRegression(), X, Y,

cv = rkf, scoring = "accuracy") print(np.mean(scores)) >>>0.942451829216535

So we can see that using Logistic Regression, we can achieve an accuracy of 0.942. I’ll test a few others as well. Here’s example code for testing LightGBM:

rkf = RepeatedStratifiedKFold(n_splits = 4, n_repeats = 5, random_state = 3) scores = cross_val_score(LGBMClassifier(), X, Y, cv = rkf,

scoring = "accuracy") print(np.mean(scores)) >>>0.9297512944571767

Using default hyper-parameter values, we see that the Logistic Regression model (0.942) outperforms the LightGBM model (0.930), at least in terms of accuracy.

OTHER METRICS: THE MCC

Let’s consider other error metrics as well. One that I tend to use a lot is the Matthews Correlation Coefficient (MCC), which takes into consideration all “quadrants” of the confusion matrix — meaning true positives, true negatives, false positives, and false negatives, which makes it a more balanced metric than accuracy or any other measure that doesn’t take into account class weights. The MCC ranges between +1 and -1, like the Pearson correlation coefficient, with 1 being perfect correlation, while 0 is something like random guessing. In the first line of the code snippet below, I turn the MCC implemention in sklearn.metrics (matthews_corrcoef) into a sklearn scoring metric using the make_scorer() function. From there on, the code is similar to what we used above:

mcc = make_scorer(matthews_corrcoef) rkf = RepeatedStratifiedKFold(n_splits = 4, n_repeats = 5, random_state = 0) scores = cross_val_score(MLPClassifier(), X, y = Y,

cv = rkf,

scoring = mcc)

print(np.mean(scores)) >>>0.7192459136506957

I tested out the MLPClassifier, among other learners, with the following cross-validated results:

NaiveClassifier: 0.0

LogisticRegression: 0.766

LightGBM: 0.688

MLPClassifier: 0.719

RandomForestClassifier: 0.651

XGBClassifier: 0.697

SGDClassifier: 0.731

It looks like Logistic Regression is clearly the best classifier for this dataset, followed by SGDClassifier and the MLPClassifier in second and third place, respectively. There is a chance that tuning hyper-parameters might alter the rankings, but through my own somewhat cursory experimentation, the gains seem marginal and probably not worth all the time it would take to find (near)-optimal values.

AVERAGE CONFUSION MATRICES

What about the confusion matrix? Since we are using repeated stratified k-fold cross-validation, we could create a list and append for every trained model on its respective validation fold. Since I used 4 folds and 5 repeats, that means that I essentially fitted a model 20 times, and tested the model on 20 validation folds. There’s a slight problem that you might run into with this and similar datasets — that is, having a minority class with so few instances that you can’t stratify it with the normal 10 or 20 folds. If you try this, what happens is that the folds that don’t have that class, then confusion_matrix from sklearn is going to omit that class, and that means that your matrices will not all be of the same shape. For example, in some validation folds, I only had 4 of the 5 classes present, so I would get some 4x4 and some 5x5 matrices, which I couldn’t then easily average using np.mean. Therefore, just try to keep the folds to something like 3–5. Here’s the average confusion matrix for Logistic Regression

Average Confusion Matrix for Logistic Regression Model.

Here is the average confusion matrix for the SGDClassifier:

Average Confusion Matrix for the Stochastic Gradient Descent Classifier (SGDClassifier)

Obviously, there isn’t a huge difference. It looks like the Logistic Regression model might be slightly better in discriminating the 4th class, but not by much. Of course, SGD is just a linear classifier in which you can change the loss function. If you use loss = “log”, you’ve got a Logistic Regression, and if you use loss = “hinge” (the default), you’re training a linear support vector machine (SVM).

Feature Importance: SHAP values

Here’s the code you need to find the SHAP values. Run this and you’ll get a pretty bar graph showing overall feature importance for the Logistic Regression model.

clf = LogisticRegression(fit_intercept=True)

clf.fit(X, Y) explainer = shap.LinearExplainer(clf, data = X)

shap_values = explainer.shap_values(X)

shap.summary_plot(shap_values, X, plot_type="bar")

Feature importance as measured by SHAP values. AST and BIL are the most impactful features, overall. That does not mean they are the most important for each case.

Let’s take a look at how the decision came about for one of the data points (the 5th one, in this case):

preds = clf.predict_proba(X) def class_labels(row_index):

global preds

return [f'Class {i} {preds[row_index][i]}' for i in range(5)] row_index = 5 shap.multioutput_decision_plot(base_values = list(expected_value),

shap_values = shap_values,

row_index = row_index,

feature_names = list(X.columns),

highlight[np.argmax(preds[row_index])])

The dotted line indicates the class that won out and became the model’s prediction. In this case, we can see that Class 2 was in the running for awhile, but that the AST feature value cemented this as a Class 0.

For contrast, let’s take a look at the feature importance for the SGDClassifier model:

SHAP value based feature importance for the SGDClassifier model. AST and ALB are the most important, the margin of difference isn’t as great as in the Logistic Regression Model.

FEATURE INTERACTIONS

The nice thing about gradient boosting decision tree type learners (e.g. LightGBM, XGBoost, CatBoost, NGBoost, ThunderGBM) is that they capture nonlinear interaction terms between the features, which our regular linear models do not. Let’s train a LightGBM model and then use the shap library to see what interaction terms are important.

shap_interaction_values = explainer.shap_interaction_values(X, Y) expected_value = explainer.expected_value shap.decision_plot(expected_value, shap_interaction_values,

features = X,

link='identity', ignore_warnings = True,

feature_order = "hclust")

This results in the following summary plot:

Decision Paths for the Data Set. ALT*AST, ALP*AST are among the important interaction terms. Gender never appears in any interaction term, though we see Age appear in Age*CHOL. We assume that CHOL means cholesterol.

Just eyeballing , we can see that AST, ALB, and BIL are among the top features, while CHOL, Sex, and Age are usually among the least important features overall.

Synthetic Minority Oversampling Technique (SMOTE)

SMOTE methods are used to augment your data by generating synthetic data, which you hope is realistic data and better defines your decision functions, that is, somehow demarcates more clearly the separation between the classes. In particular, the idea is to help rectify class imbalance by creating some synthetic data points for the smallest classes. The smote-variants library in Python collects a wide range of SMOTE variants that you can use. There is no way to know a priori which one of the variants is going to help your model improve its performance. There is one issue that I want to bring up. First, take a look at the code below:

oversampler = MulticlassOversampling(sv.Supervised_SMOTE(proportion = 0.1)) X_samp, y_samp = oversampler.sample(X, Y) print(X.shape, X_samp.shape) Counter(Y), Counter(y_samp) >>>(615, 12) (2663, 12) >>>(Counter({0.0: 533, 1.0: 7, 2.0: 24, 3.0: 21, 4.0: 30}),

Counter({0.0: 533, 1.0: 532, 2.0: 533, 3.0: 532, 4.0: 533}))

What we can see from this is that firstly, we went from 615 data records to having 2663. The reason is because we now have 532–533 instances of each data set, meaning that each non-majority class is now on par, in terms of instances, with the majority class. Here’s the code for running repeated experiments to see if SMOTE helps. I am using TRIM_SMOTE here just as an example. I haven’t tested all the variants exhaustively, but of the ones that I tested (a handful) none of them seemed to help the classifier. Note: this is NOT CROSS-VALIDATION! I say this because if you use either the smote_variant’s built-in cross-validation function or just use sklearn, you are running the risk of data leakage. If you are synthesizing data and then spreading it throughout the folds, the classifier is going to be “more familiar” with the data in the validation fold than it would normally be, since some of that data is derived from the training data. Better than to just split it 75/25 each trial, and then augment the training fold alone. Train 2 classifiers — one on the augmented training fold, one on the original training fold. Then compare them on the same validation fold.

In any case, I don’t think there is a strong case for using SMOTE. Here’s a look at the mean MCC score of the two classifiers, over 50 runs, as well as the standard deviation of the scores.

print(np.mean(Scores1), np.std(Scores1), np.mean(Scores2), np.std(Scores2)) >>0.7521941508693497 0.06086768680379448 0.6866459807113643 0.0717005989090920

Here, the first score — 0.752 — is the average MCC of the classifier on the original train data. The other score — 0.687 — is the average MCC of the classifier trained on the augmented data. That alone tells me that SMOTE isn’t an obvious choice in terms of things we can do to easily boost our classifier’s skill.

DISCUSSION

We explored various classification models, and found that the Logistic Regression and Stochastic Gradient Descent models perform the best. In terms of the MCC, we can see that the Logistic Regression model outperforms many of the other learners and algorithms that we chose for comparison. The models are skillful in predicting the healthy (majority) class, but falter a bit on the other classes, with the possible exception of the last class. We found that SMOTE doesn’t seem to help the classifier by creating more minority class data points.

Well, that’s it, for now. Keep your eyes out for the companion video on YouTube, which I will link to in the comments once it’s fully complete. As always, I welcome your comments and criticisms, and I am looking for any job opportunities or internships that involve data science and machine learning, so if you hear of anything, feel free to drop me a line.