# Machine Learning-Based Performance Comparison to Diagnose Anterior Cruciate Ligament Tears

Authors: Mazhar Javed Awan; Mohd Shafry Mohd Rahim; Naomie Salim; Amjad Rehman; Haitham Nobanee
Journal: Journal of Healthcare Engineering
 (2022)

---

## Abstract

In recent times, knee joint pains have become severe enough to make daily tasks difficult. Knee osteoarthritis is a type of arthritis and a leading cause of disability worldwide. The middle of the knee contains a vital portion, the anterior cruciate ligament (ACL). It is necessary to diagnose the ACL ruptured tears early to avoid surgery. The study aimed to perform a comparative analysis of machine learning models to identify the condition of three ACL tears. In contrast to previous studies, this study also considers imbalanced data distributions as machine learning techniques struggle to deal with this problem. The paper applied and analyzed four machine learning classification models, namely, random forest (RF), categorical boosting (Cat Boost), light gradient boosting machines (LGBM), and highly randomized classifier (ETC) on the balanced, structured dataset of ACL. After oversampling a hyperparameter adjustment, the above four models have achieved an average accuracy of 95.72%, 94.98%, 94.98%, and 98.26%. There are 2070 observations and eight features in the collection of three diagnosis ACL classes after oversampling. The area under curve value was approximately 0.998, respectively. Experiments were performed using twelve machine learning algorithms with imbalanced and balanced datasets. However, the accuracy of the imbalanced dataset has remained under 76% for all twelve models. After oversampling, the proposed model may contribute to the investigation of ACL tears on magnetic resonance imaging and other knee ligaments efficiently and automatically without involving radiologists.


---

## Body

## 1. Introduction

Knee bone and joint diseases are ubiquitous in almost all groups of age and sex. These are anterior cruciate ligament (ACL) injuries, osteoarthritis (OA), and osteoporosis (OP) [1–3]. The knee joint comprises the femur, tibia, patella, and the synovial membrane, which contains synovial fluid. The end of the femur is covered by articular cartilage. It moves against the articular cartilage of the tibia. The thin layers of rigid, slippery tissue called cartilage act as a protective cushion to allow the bones to move more freely [4, 5]. The knee ligaments are strong bands of tissue that connect one bone to another. Ligament bones limit movements and stabilize joints and durable bands of fibrous tissue, which can connect the bones and strength. The four main ligaments in Figure 1 are included the anterior cruciate ligament (ACL), the posterior cruciate ligament (PCL), the medial cruciate ligament (MCL), and the lateral cruciate ligament (LCL) [6–8].Figure 1
The four structures of the knee ligament. (a) ACL and PCL stabilized the knee. (b) MCL inner side of the knee. (c) LCL outer side of the knee.
(a)(b)(c)The ACL tear is a strong band of tissue in the center and an essential part of the knee [9]. The ACL ligament cannot regenerate; unlike muscle, around 100,000 to 200,000 individuals tear it each year, and 500 million dollars are spent on ACL treatment annually [10]. The ACL tear often causes osteoarthritis or wearing down of the bone and cartilage in the knee [11]. The mechanism of injury to the ACL is usually a noncontact, pivoting injury. The muscles are attached to tendons and then bones. Osteoarthritis is figured out when the cartilage begins to thin or roughen; this happens naturally as part of aging. New bits of bone known as osteophytes may start to grow within the joints, and fluid can build up inside [12]. It reduces the space within the joints, which means that the joint does not move as smoothly as it used to and might feel stiff and painful (see Figure 2) [13, 14].Figure 2
The knee bone anatomy and injury mechanism. (a) Structure of knee ACL injury. (b) Osteoarthritis due to joint space reduction mechanism.
(a)(b)ML-based classification models are strongly affected by imbalanced data, especially in the medical field. The class imbalance is one of the common problems which affects the prediction accuracy and could lead to biases in the result. It is required to balance the data by increasing the minority class or decreasing the majority class (undersampling). The distribution can vary from a slight bias to a severe imbalance [15–18].The paper aims to apply extensive machine learning models to efficiently predict ACL tears in the early stage to avoid ACL injury efficiently. In this paper, we compare and analyze the results of the class imbalance problem in the context of structured data contained multiclasses through oversampling technique.As per our knowledge, there is no study to identify the three classes of ACL tears on structured data. Therefore, this paper presented class imbalanced ACL data and evaluated the performance of twelve machine learning classifiers with and without oversampling.The significant contributions of the paper are the following:(i)
Enhanced the distributions of partial and ruptured ACL classes through oversampling to balance all three categories.(ii)
Applied extensive data visualization for the case of imbalanced and balanced datasets as well.(iii)
As per our knowledge, there is no such study we applied and compared twelve machine learning classifier models on an imbalanced and balanced dataset.(iv)
After adjusting hyperparameters and oversampling class balancing, the four machine learning models achieved above 95% accuracy, precision, recall, and F1-score.(v)
The extra tree classifier model accuracy is 98.28%, the highest among all machine learning models.The paper is organized in the following: Section2 is about the work related to machine learning prediction of the knee and other diseases. Section 3 is connected to material and methodology, data exploration, and methods of various machine learning models with random forest and extra tree cat boost used in our study. Section 4 compares the classification results with accuracy, confusion matrix, and other metrics. Conclusions are given in Section 5.


## 2. Related Work

The medical data are usually extensive and very hard to analyze and interpret by humans quickly. For this purpose, the machine learning-based models showed promising results in all medical fields to diagnose and predict various diseases efficiently [19–25]. The early detection of knee OA and OP disease progression is complex and challenging in the case of classification problems [26, 27]. The machine learning models can quantify anterior cruciate injury risk better for sports player injuries, synovial fluid of human OA knees, and joint angles prediction [28–32].Machine learning is used widely in sports injuries prediction because many models performed better results. Jauhiainen, Kauppi [33] was used motion analysis and physical datasets of severe knee injuries of 318 cases. The random forest and logistic regression machine learning model achieved with receiver operative curves (ROC) only under 0.63 and 0.65. These were highly prevalent among athletes, and injury follow-up lasted for 12 months. Kotti, Duffell’s [34] study used a locomotion dataset of 47 osteoarthritides and 47 healthy knees and applied a random forest model with nine features. Three per axis was achieved for the discriminative features with an accuracy of 74.4% only. The study was not good for temporal information, and the parameters were strictly quantitative. Tiulpin, Klein’s [35] analysis was used a machine learning-based approach for predicting structural knee OA development using data collected during a single clinical visit has been developed. The most important conclusion of this study is that patients with KL-0 and KL-1 at baseline were predicted to advance. Du et al. [36] discussed the Cartilage Damage Index (CDI) as a tool for determining how far osteoarthritis has progressed in the knee. Stajduhar et al. [37]’s study was related to our dataset knee ACL.Recently comparative analysis approaches in classifying imbalanced and balanced datasets are widespread in the literature. The study by Vijayvargiya et al. [38] was used various machine learning models on the original normal and abnormal subjects about knee from electromyography (EMG) data. The extra tree classifier found the best accuracy after oversampling at 93.3%. There was no improvement in the performance metrics through various class balancing techniques.The literature suggested that machine learning, the ensemble of classifiers, and boosting are known to increase the accuracy of solving the class imbalance problem. Our study uses a machine learning classification model on structured data for three classes and differs from most other studies examined in the related work. Some of the studies applied machine learning to structured data. Still, our approach differs from these studies because we compared the performance of machine learning models before and after class balancing.Above all literature, traditional machine learning models are applied chiefly to unstructured data such as MRI and X-rays to predict the anterior cruciate ligament injury and osteoarthritis in most existing state-of-the-art. Moreover, several researchers have developed diagnosis methods to identify other diseases through machine learning. However, there is no such study to detect the three ACL classes through machine learning comparative analysis. These issues are addressed in this research article to diagnose early ACL rupture tears.


## 3. Materials and Methods

This section presents the methods and materials used in this study. Section3.1 is the dataset description. Section 3.2 is the proposed framework of the study. Section 3.3 is the oversampling technique handling. Section 3.4 is the data exploration analysis of balanced datasets. The proposed machine learning models are explained in Section 3.5.


### 3.1. Data Description

We used the anterior cruciate ligament metadata file for our experiments. The 917 samples containing three ACL classes that are healthy, partial, and full ruptured were acquired from Clinical Hospital Centre Rijeka. These are 75.2% for healthy and 18.8%, 6% for partial and injured tears, respectively. The three classes’ volumes are 690, 172, and 55, respectively, are shown in Figure3.Figure 3
The three ACL tear class numbers in the bar graph.The feature names with unique and mean values of each feature are described in Table1.Table 1
The feature description with unique and mean values.
FeaturesUnique valuesMean valueExam id909739320.042530SerialNo115.367503aclDiagnosis30.307525KneeLR20.511450roiX63114.497274roiY89109.318430roiZ1113.992366roiHeight5891.758997roiWidth5990.736096roiDepth53.359869


### 3.2. Proposed Framework

This section of the article discusses the proposed anterior cruciate ligament injury prediction system consists of many steps which are ideally linked to each other to get the desired results.Step I.
The dataset is considered only in a structured form, imbalanced in nature, and its details have already been discussed in the section data description.Step II.
The dataset was prepared, which included checking for unique values, NULL values, string values, and converting imbalanced data into balanced data by the oversampling technique described in Section3.3.Step III.
For better understanding, the data exploration analysis (EDA) was visualized through various libraries like Matplotlib and Seaborn, which have been used to plot correlation heatmap, typical distribution plots, and count plots.Step IV.
After this, the data were split into training and testing set in 75% and 25%.Step V.
The training data have been applied to twelve supervised machine learning models, and the four machine learning models trained well after adjustment in the hyperparameters.Step VI.
With the help of test data, all models were evaluated through a confusion matrix, mean accuracy, precision, recall, F1-score. The receiver operative characteristics (ROC) were only considered the best four models.Step VII.
At the last stage, the prediction of three classes was compared without class balancing and with the oversampling balancing of all twelve machine learning models.
Figure4 shows the overall proposed framework for the process and its septs.Figure 4
The proposed framework of our approach.


### 3.3. Handling Class Imbalanced Data

The class imbalance is a big problem in machine learning and image-related datasets [39]. It can handle undersampling [40], oversampling [41], and hybrid sampling techniques efficiently [42]. Our current dataset is an imbalance in nature, as shown in Figure 3. We applied the Scikit library and import resample [43]. Here, we are using oversampling in partial and ruptured tears classes. After applied oversampling, the ratios of the three categories are now equal, as shown in Figure 5.Figure 5
The percentage ratio in a pie chart.After oversampling, the data are shown with equal proportions that are 690 samples and 33.3% ratio of each sample percentage as shown in Figure6.Figure 6
After oversampling data ratio and percentage.


### 3.4. Data Exploration and Visualization

Data exploration and visualization are critical to evaluate machine learning models through the python libraries of Matplotlib [44] and Seaborn [45]. There are the following various plots after oversampling balanced datasets.


#### 3.4.1. Heatmap Correlation Matrix

The correlation matrix indicates the highest correlation, namely roiWidth and roiHeight features for predicting a diagnosis of ACL tears. Figure7 shows the relationship covariance of each feature with the after-oversampling class balanced.(1)ρY1,Y2=CovarY1,Y2σY1σY2,where Covar means covariance measure and features Y1 and Y2 are computed for every pair in equations (1) and (2).(2)CovarY1,Y2=EY1−EY1Y2−EY2.Figure 7
Heatmap correlation matrix of a balanced dataset.


#### 3.4.2. Normal Distribution of Data

Figure8 is related to various distribution plots of all components, and ROI height and ROI width are generally distributed for both cases.Figure 8
Distribution plot of balanced dataset plot.


#### 3.4.3. Histogram Plots

Figure9 shows the histogram counts of each feature after oversampling.Figure 9
Histogram plot of balanced dataset plot.


#### 3.4.4. Distribution of Class

Figure10 shows the distribution of three classes for every feature. Series 5 feature has contained healthy and partial tears much greater.Figure 10
Distribution plot of each feature in each class after a balanced dataset.


### 3.5. Machine Learning Approaches

We applied twelve various machine learning models out of eight classifier models, logistic regression [46], support vector machine [47], decision tree [48], k-nearest neighbour [49], Gaussian Naïve Bayes [50], AdaBoosting [51], gradient boosting [52], extreme gradient boosting [53] used for experiment results only. The following four proposed models are discussed in Section 3.5.1. Random Forest [54], Section 3.5.2. Extra Tree Classifier [55], Section 3.5.3. Categorical Boosting [56], and Section 3.5.4. LGBM Classifier. We have explained this because it performs better results for our datasets.


#### 3.5.1. Random Forest

There areM Features and N Rows. In a random forest, it grows multiple trees such that each tree comprises the square root of the total number of features that are present. In our case, we have M features, so each tree would have a square root of M features to train on; additionally, it uses bootstrap samples or samples with replacement. Figure 11 shows the structure of a random forest tree [57].Figure 11
Random Forest structure ofN tree and three classes.The algorithm of random forest is shown in Table2.Table 2
The random forest classifier algorithm.
Input: Randomly selectm features from all number of parts where m << DTFor noded, calculate the best split point among them feature until n number of nodesSplit the node into two daughter nodes using the best splitEnd: Build your forest by repeating step in the loop for several trees constructed based on the highest votingThe final prediction (final Pred) is by taking the majority of the decision tree DT1 (m), DT2 (m) from m features(3)Final Pred=modeDT1m,DT2m,…,DTnm.Generally, it is written as(4)Final Pred=mode∑n=1nDTnm.


#### 3.5.2. Extra Tree Classifier

An extremely randomized or extra tree classifier (ETC) is an ensemble algorithm that uses many unpruned decision trees from the training datasets [55]. The algorithm of ETC is described in Table 3.Table 3
Extra tree classifier algorithm.
Input: The local learning subsetk parameter corresponding to the number of splits to tryFor each of those splits is done on a randomly chosen feature, with a randomly chosen cut-pointFor an ordinal variable, pick uniformly in the range [min (xi, max (xj)] for a nominal variable, select one of the categories at randomEnd: Only optimize over theK random splitsThe extra tree is also a bootstrapping and bagging algorithm. Still, the big difference between ETC and RF is that a random forest is like a greedy algorithm that uses the best available parameter at each node for the split based on Gini or entropy. The process of ETC is random but not greedy. The extra used all the records of the samples [58].LetO be training samples with n possible classes (O = O1, O2,…, On).The entropy (En) is obtained by the following mathematical formula:(5)EnO=−∑j=1nPj.logPj.The entropy afterO samples were portioned in Oj with some features is obtained; M is given as follows:(6)EnO,M=∑j=1nPj.EnOj.The information gain (IG) in the equation is defined as follows:(7)I.G=EnO−EnO,M,(8)Gini=1−∑kPk2,where p is the probability number of samples of class k and a total number of samples.Extra tree classifier is much faster than random forest. There are three differences.(i)
The extra tree classifier is selected samples for every decision tree without replacement. All models are unique.(ii)
The total number of features selected remains the same, that is, the square root of the total number of features, in the case of the classification task.(iii)
The main difference between a random forest and an extra tree classifier is that instead of computing the locally optimal split for a feature combination, a random value is selected for the split for the extra tree. These are not the best split for features.The whole idea is rather than not spending time finding the best splitting point. The best criteria are randomly picking up a point and spit based on that; this leads to more diversified trees and fewer splitters to evaluate when training and extremely random forest. In the case of readily available datasets, if observed during testing with noisy features, the extra tree classifiers seemed to outperform the random forest.


#### 3.5.3. Categorical Boost Classifier

A categorical boosting (CatBoost) method focuses on processing categorical features and boosting trees with some ordering principle without showing conversion error. A target leakage problem occurred in gradient boosting and the standard way of categorical features to numbers. The ordering principle can apply to target encoding, categorical features, and boosting trees [59].(1) Mean Target Encoding. It is an efficient way to deal with categorical variables to substitute them with numerical values. The mean target encoding can apply to categorical variables with the mean target value. Figure 12 explains the mean target encoding with a simple example. There are color features (red, blue, and green) in unique categories, and the target is either zero or one. Then, each type, red, blue, and green, is calculated by the target mean. The new feature column is named as encoded-color replaced with target mean value against each category. The advantage of target encoding was the explosion of the feature space compared with one-hot encoding, just adding one extra column at the end.Figure 12
Target means calculated using each color value and encoded to the color target.Target encoding could also smooth the calculation with a prior term as shown in the following formula.(9)mean_target=class_inclass+Priortotal_count+1,where, in the equation, count_inclass were the number of counts the label value equal to 1 for the objects against the categorical feature value, prior value can be assumed was determined the starting parameters, and total count means the total number of things with the categorical feature value.(2) Ordering Boosting. The ordered target encoding technique helps prevent overfitting due to target leakage.The encoded value estimates the expected target value against each feature category.(10)Estb|ai=aki.Boost implements an efficient modification of the ordered boosting on the basic decision tree. It was good for small datasets, support training with pairs, good quality with default parameters, extensive support of models formats, stable and model analysis tool. The classical boosting uses multiple trees and whole datasets with the residuals, which causes overfitting. The ordered boosting does not use the whole datasets to calculate residuals.Assuming modelMi was trained on the first data points, then calculating the residuals at each point i using model Mi − 1. The idea is that the tree did not see the data points as before, so it cannot overfit. Figure 13 shows the N separate trees with data point M4 [56].Figure 13
Ordering boosting to avoid overfitting problem on four data points.The model was trained on four data points,M4. The residuals are shown in equation (1).(11)rx5,y5=y5−M4x5,where N trees are not feasible, and it works with trees at location 2, where j = 1, 2, … log2 (n).


#### 3.5.4. Light Gradient Boosting (LGBM)

LightGBM is a gradient boosting framework that uses a decision-tree-based learning algorithm fast, distributed, and reduces the memory usage designed by Microsoft Research Asia [60].(1) Gradient-based one-side Sampling (GOSS). This method focuses more on the under-trained part of the dataset, which tried to learn more aggressively. The slight gradient means that it contains minor errors, which means the data points are learned well. The large gradient implies significant errors, which means the data points are not known well. The algorithm is supported for large gradients, and it is much essential. The algorithm of GOSS in Table 4 first sorts the data points according to their absolute gradient value.Table 4
The gradient-based one-side sampling algorithm.
Input: [Tr = training data, iter = iterations, LGA = sample of large gradient data, SGD = sample of small gradient data, L = loss function, Lr = weak learner]Models ← [ ], fact ← 1 − LGA/SGDtopNumber ← LGA × len (Tr)randNumber ← SGD × len (Tr)Fori = 1 to iter dopreds ← models.predict (Tr) h ← L (Tr, preds),k ← [1, 1, ...] sorted ← GetSortedIndices (abs (h)) topSet ← sorted [1: topNumber] randSet ← RandomPick (sorted [topNumber:len (Tr)], randNumber) usedSet ← topSet + randSet k [randSet] × = fact (Assigning weight for small gradient) newModel ← Lr (I [usedSet], −h [usedSet], k [usedSet])models.append (newModel)Then, the top sampling ratio of the large gradient of data (LGD) points × 100% instances was considered. Then, it randomly samples the proportion of small gradient data (SGD) × 100% instances from the rest of the data points. In the end, GOSS amplified the sampled data with a small gradient by multiplying 1 − LGA/SGD when calculating the information gain. We focused more on the under-trained instances without changing the original data distribution by much.Figure14 explains the light GBM split tree leaf-wise.Figure 14
Light GBM classifier growth leaf-wise.(2) Exclusive Feature Bundling (FEB). It efficiently represents sparse features such as 100 encoded features, reducing the total number of features.It is designed to be a distributed, high-performance gradient boosting framework based on a decision tree algorithm with lower memory usage and capable of large-scale handling data [61].


## 3.1. Data Description

We used the anterior cruciate ligament metadata file for our experiments. The 917 samples containing three ACL classes that are healthy, partial, and full ruptured were acquired from Clinical Hospital Centre Rijeka. These are 75.2% for healthy and 18.8%, 6% for partial and injured tears, respectively. The three classes’ volumes are 690, 172, and 55, respectively, are shown in Figure3.Figure 3
The three ACL tear class numbers in the bar graph.The feature names with unique and mean values of each feature are described in Table1.Table 1
The feature description with unique and mean values.
FeaturesUnique valuesMean valueExam id909739320.042530SerialNo115.367503aclDiagnosis30.307525KneeLR20.511450roiX63114.497274roiY89109.318430roiZ1113.992366roiHeight5891.758997roiWidth5990.736096roiDepth53.359869


## 3.2. Proposed Framework

This section of the article discusses the proposed anterior cruciate ligament injury prediction system consists of many steps which are ideally linked to each other to get the desired results.Step I.
The dataset is considered only in a structured form, imbalanced in nature, and its details have already been discussed in the section data description.Step II.
The dataset was prepared, which included checking for unique values, NULL values, string values, and converting imbalanced data into balanced data by the oversampling technique described in Section3.3.Step III.
For better understanding, the data exploration analysis (EDA) was visualized through various libraries like Matplotlib and Seaborn, which have been used to plot correlation heatmap, typical distribution plots, and count plots.Step IV.
After this, the data were split into training and testing set in 75% and 25%.Step V.
The training data have been applied to twelve supervised machine learning models, and the four machine learning models trained well after adjustment in the hyperparameters.Step VI.
With the help of test data, all models were evaluated through a confusion matrix, mean accuracy, precision, recall, F1-score. The receiver operative characteristics (ROC) were only considered the best four models.Step VII.
At the last stage, the prediction of three classes was compared without class balancing and with the oversampling balancing of all twelve machine learning models.
Figure4 shows the overall proposed framework for the process and its septs.Figure 4
The proposed framework of our approach.


## 3.3. Handling Class Imbalanced Data

The class imbalance is a big problem in machine learning and image-related datasets [39]. It can handle undersampling [40], oversampling [41], and hybrid sampling techniques efficiently [42]. Our current dataset is an imbalance in nature, as shown in Figure 3. We applied the Scikit library and import resample [43]. Here, we are using oversampling in partial and ruptured tears classes. After applied oversampling, the ratios of the three categories are now equal, as shown in Figure 5.Figure 5
The percentage ratio in a pie chart.After oversampling, the data are shown with equal proportions that are 690 samples and 33.3% ratio of each sample percentage as shown in Figure6.Figure 6
After oversampling data ratio and percentage.


## 3.4. Data Exploration and Visualization

Data exploration and visualization are critical to evaluate machine learning models through the python libraries of Matplotlib [44] and Seaborn [45]. There are the following various plots after oversampling balanced datasets.


### 3.4.1. Heatmap Correlation Matrix

The correlation matrix indicates the highest correlation, namely roiWidth and roiHeight features for predicting a diagnosis of ACL tears. Figure7 shows the relationship covariance of each feature with the after-oversampling class balanced.(1)ρY1,Y2=CovarY1,Y2σY1σY2,where Covar means covariance measure and features Y1 and Y2 are computed for every pair in equations (1) and (2).(2)CovarY1,Y2=EY1−EY1Y2−EY2.Figure 7
Heatmap correlation matrix of a balanced dataset.


### 3.4.2. Normal Distribution of Data

Figure8 is related to various distribution plots of all components, and ROI height and ROI width are generally distributed for both cases.Figure 8
Distribution plot of balanced dataset plot.


### 3.4.3. Histogram Plots

Figure9 shows the histogram counts of each feature after oversampling.Figure 9
Histogram plot of balanced dataset plot.


### 3.4.4. Distribution of Class

Figure10 shows the distribution of three classes for every feature. Series 5 feature has contained healthy and partial tears much greater.Figure 10
Distribution plot of each feature in each class after a balanced dataset.


## 3.4.1. Heatmap Correlation Matrix

The correlation matrix indicates the highest correlation, namely roiWidth and roiHeight features for predicting a diagnosis of ACL tears. Figure7 shows the relationship covariance of each feature with the after-oversampling class balanced.(1)ρY1,Y2=CovarY1,Y2σY1σY2,where Covar means covariance measure and features Y1 and Y2 are computed for every pair in equations (1) and (2).(2)CovarY1,Y2=EY1−EY1Y2−EY2.Figure 7
Heatmap correlation matrix of a balanced dataset.


## 3.4.2. Normal Distribution of Data

Figure8 is related to various distribution plots of all components, and ROI height and ROI width are generally distributed for both cases.Figure 8
Distribution plot of balanced dataset plot.


## 3.4.3. Histogram Plots

Figure9 shows the histogram counts of each feature after oversampling.Figure 9
Histogram plot of balanced dataset plot.


## 3.4.4. Distribution of Class

Figure10 shows the distribution of three classes for every feature. Series 5 feature has contained healthy and partial tears much greater.Figure 10
Distribution plot of each feature in each class after a balanced dataset.


## 3.5. Machine Learning Approaches

We applied twelve various machine learning models out of eight classifier models, logistic regression [46], support vector machine [47], decision tree [48], k-nearest neighbour [49], Gaussian Naïve Bayes [50], AdaBoosting [51], gradient boosting [52], extreme gradient boosting [53] used for experiment results only. The following four proposed models are discussed in Section 3.5.1. Random Forest [54], Section 3.5.2. Extra Tree Classifier [55], Section 3.5.3. Categorical Boosting [56], and Section 3.5.4. LGBM Classifier. We have explained this because it performs better results for our datasets.


### 3.5.1. Random Forest

There areM Features and N Rows. In a random forest, it grows multiple trees such that each tree comprises the square root of the total number of features that are present. In our case, we have M features, so each tree would have a square root of M features to train on; additionally, it uses bootstrap samples or samples with replacement. Figure 11 shows the structure of a random forest tree [57].Figure 11
Random Forest structure ofN tree and three classes.The algorithm of random forest is shown in Table2.Table 2
The random forest classifier algorithm.
Input: Randomly selectm features from all number of parts where m << DTFor noded, calculate the best split point among them feature until n number of nodesSplit the node into two daughter nodes using the best splitEnd: Build your forest by repeating step in the loop for several trees constructed based on the highest votingThe final prediction (final Pred) is by taking the majority of the decision tree DT1 (m), DT2 (m) from m features(3)Final Pred=modeDT1m,DT2m,…,DTnm.Generally, it is written as(4)Final Pred=mode∑n=1nDTnm.


### 3.5.2. Extra Tree Classifier

An extremely randomized or extra tree classifier (ETC) is an ensemble algorithm that uses many unpruned decision trees from the training datasets [55]. The algorithm of ETC is described in Table 3.Table 3
Extra tree classifier algorithm.
Input: The local learning subsetk parameter corresponding to the number of splits to tryFor each of those splits is done on a randomly chosen feature, with a randomly chosen cut-pointFor an ordinal variable, pick uniformly in the range [min (xi, max (xj)] for a nominal variable, select one of the categories at randomEnd: Only optimize over theK random splitsThe extra tree is also a bootstrapping and bagging algorithm. Still, the big difference between ETC and RF is that a random forest is like a greedy algorithm that uses the best available parameter at each node for the split based on Gini or entropy. The process of ETC is random but not greedy. The extra used all the records of the samples [58].LetO be training samples with n possible classes (O = O1, O2,…, On).The entropy (En) is obtained by the following mathematical formula:(5)EnO=−∑j=1nPj.logPj.The entropy afterO samples were portioned in Oj with some features is obtained; M is given as follows:(6)EnO,M=∑j=1nPj.EnOj.The information gain (IG) in the equation is defined as follows:(7)I.G=EnO−EnO,M,(8)Gini=1−∑kPk2,where p is the probability number of samples of class k and a total number of samples.Extra tree classifier is much faster than random forest. There are three differences.(i)
The extra tree classifier is selected samples for every decision tree without replacement. All models are unique.(ii)
The total number of features selected remains the same, that is, the square root of the total number of features, in the case of the classification task.(iii)
The main difference between a random forest and an extra tree classifier is that instead of computing the locally optimal split for a feature combination, a random value is selected for the split for the extra tree. These are not the best split for features.The whole idea is rather than not spending time finding the best splitting point. The best criteria are randomly picking up a point and spit based on that; this leads to more diversified trees and fewer splitters to evaluate when training and extremely random forest. In the case of readily available datasets, if observed during testing with noisy features, the extra tree classifiers seemed to outperform the random forest.


### 3.5.3. Categorical Boost Classifier

A categorical boosting (CatBoost) method focuses on processing categorical features and boosting trees with some ordering principle without showing conversion error. A target leakage problem occurred in gradient boosting and the standard way of categorical features to numbers. The ordering principle can apply to target encoding, categorical features, and boosting trees [59].(1) Mean Target Encoding. It is an efficient way to deal with categorical variables to substitute them with numerical values. The mean target encoding can apply to categorical variables with the mean target value. Figure 12 explains the mean target encoding with a simple example. There are color features (red, blue, and green) in unique categories, and the target is either zero or one. Then, each type, red, blue, and green, is calculated by the target mean. The new feature column is named as encoded-color replaced with target mean value against each category. The advantage of target encoding was the explosion of the feature space compared with one-hot encoding, just adding one extra column at the end.Figure 12
Target means calculated using each color value and encoded to the color target.Target encoding could also smooth the calculation with a prior term as shown in the following formula.(9)mean_target=class_inclass+Priortotal_count+1,where, in the equation, count_inclass were the number of counts the label value equal to 1 for the objects against the categorical feature value, prior value can be assumed was determined the starting parameters, and total count means the total number of things with the categorical feature value.(2) Ordering Boosting. The ordered target encoding technique helps prevent overfitting due to target leakage.The encoded value estimates the expected target value against each feature category.(10)Estb|ai=aki.Boost implements an efficient modification of the ordered boosting on the basic decision tree. It was good for small datasets, support training with pairs, good quality with default parameters, extensive support of models formats, stable and model analysis tool. The classical boosting uses multiple trees and whole datasets with the residuals, which causes overfitting. The ordered boosting does not use the whole datasets to calculate residuals.Assuming modelMi was trained on the first data points, then calculating the residuals at each point i using model Mi − 1. The idea is that the tree did not see the data points as before, so it cannot overfit. Figure 13 shows the N separate trees with data point M4 [56].Figure 13
Ordering boosting to avoid overfitting problem on four data points.The model was trained on four data points,M4. The residuals are shown in equation (1).(11)rx5,y5=y5−M4x5,where N trees are not feasible, and it works with trees at location 2, where j = 1, 2, … log2 (n).


### 3.5.4. Light Gradient Boosting (LGBM)

LightGBM is a gradient boosting framework that uses a decision-tree-based learning algorithm fast, distributed, and reduces the memory usage designed by Microsoft Research Asia [60].(1) Gradient-based one-side Sampling (GOSS). This method focuses more on the under-trained part of the dataset, which tried to learn more aggressively. The slight gradient means that it contains minor errors, which means the data points are learned well. The large gradient implies significant errors, which means the data points are not known well. The algorithm is supported for large gradients, and it is much essential. The algorithm of GOSS in Table 4 first sorts the data points according to their absolute gradient value.Table 4
The gradient-based one-side sampling algorithm.
Input: [Tr = training data, iter = iterations, LGA = sample of large gradient data, SGD = sample of small gradient data, L = loss function, Lr = weak learner]Models ← [ ], fact ← 1 − LGA/SGDtopNumber ← LGA × len (Tr)randNumber ← SGD × len (Tr)Fori = 1 to iter dopreds ← models.predict (Tr) h ← L (Tr, preds),k ← [1, 1, ...] sorted ← GetSortedIndices (abs (h)) topSet ← sorted [1: topNumber] randSet ← RandomPick (sorted [topNumber:len (Tr)], randNumber) usedSet ← topSet + randSet k [randSet] × = fact (Assigning weight for small gradient) newModel ← Lr (I [usedSet], −h [usedSet], k [usedSet])models.append (newModel)Then, the top sampling ratio of the large gradient of data (LGD) points × 100% instances was considered. Then, it randomly samples the proportion of small gradient data (SGD) × 100% instances from the rest of the data points. In the end, GOSS amplified the sampled data with a small gradient by multiplying 1 − LGA/SGD when calculating the information gain. We focused more on the under-trained instances without changing the original data distribution by much.Figure14 explains the light GBM split tree leaf-wise.Figure 14
Light GBM classifier growth leaf-wise.(2) Exclusive Feature Bundling (FEB). It efficiently represents sparse features such as 100 encoded features, reducing the total number of features.It is designed to be a distributed, high-performance gradient boosting framework based on a decision tree algorithm with lower memory usage and capable of large-scale handling data [61].


## 3.5.1. Random Forest

There areM Features and N Rows. In a random forest, it grows multiple trees such that each tree comprises the square root of the total number of features that are present. In our case, we have M features, so each tree would have a square root of M features to train on; additionally, it uses bootstrap samples or samples with replacement. Figure 11 shows the structure of a random forest tree [57].Figure 11
Random Forest structure ofN tree and three classes.The algorithm of random forest is shown in Table2.Table 2
The random forest classifier algorithm.
Input: Randomly selectm features from all number of parts where m << DTFor noded, calculate the best split point among them feature until n number of nodesSplit the node into two daughter nodes using the best splitEnd: Build your forest by repeating step in the loop for several trees constructed based on the highest votingThe final prediction (final Pred) is by taking the majority of the decision tree DT1 (m), DT2 (m) from m features(3)Final Pred=modeDT1m,DT2m,…,DTnm.Generally, it is written as(4)Final Pred=mode∑n=1nDTnm.


## 3.5.2. Extra Tree Classifier

An extremely randomized or extra tree classifier (ETC) is an ensemble algorithm that uses many unpruned decision trees from the training datasets [55]. The algorithm of ETC is described in Table 3.Table 3
Extra tree classifier algorithm.
Input: The local learning subsetk parameter corresponding to the number of splits to tryFor each of those splits is done on a randomly chosen feature, with a randomly chosen cut-pointFor an ordinal variable, pick uniformly in the range [min (xi, max (xj)] for a nominal variable, select one of the categories at randomEnd: Only optimize over theK random splitsThe extra tree is also a bootstrapping and bagging algorithm. Still, the big difference between ETC and RF is that a random forest is like a greedy algorithm that uses the best available parameter at each node for the split based on Gini or entropy. The process of ETC is random but not greedy. The extra used all the records of the samples [58].LetO be training samples with n possible classes (O = O1, O2,…, On).The entropy (En) is obtained by the following mathematical formula:(5)EnO=−∑j=1nPj.logPj.The entropy afterO samples were portioned in Oj with some features is obtained; M is given as follows:(6)EnO,M=∑j=1nPj.EnOj.The information gain (IG) in the equation is defined as follows:(7)I.G=EnO−EnO,M,(8)Gini=1−∑kPk2,where p is the probability number of samples of class k and a total number of samples.Extra tree classifier is much faster than random forest. There are three differences.(i)
The extra tree classifier is selected samples for every decision tree without replacement. All models are unique.(ii)
The total number of features selected remains the same, that is, the square root of the total number of features, in the case of the classification task.(iii)
The main difference between a random forest and an extra tree classifier is that instead of computing the locally optimal split for a feature combination, a random value is selected for the split for the extra tree. These are not the best split for features.The whole idea is rather than not spending time finding the best splitting point. The best criteria are randomly picking up a point and spit based on that; this leads to more diversified trees and fewer splitters to evaluate when training and extremely random forest. In the case of readily available datasets, if observed during testing with noisy features, the extra tree classifiers seemed to outperform the random forest.


## 3.5.3. Categorical Boost Classifier

A categorical boosting (CatBoost) method focuses on processing categorical features and boosting trees with some ordering principle without showing conversion error. A target leakage problem occurred in gradient boosting and the standard way of categorical features to numbers. The ordering principle can apply to target encoding, categorical features, and boosting trees [59].(1) Mean Target Encoding. It is an efficient way to deal with categorical variables to substitute them with numerical values. The mean target encoding can apply to categorical variables with the mean target value. Figure 12 explains the mean target encoding with a simple example. There are color features (red, blue, and green) in unique categories, and the target is either zero or one. Then, each type, red, blue, and green, is calculated by the target mean. The new feature column is named as encoded-color replaced with target mean value against each category. The advantage of target encoding was the explosion of the feature space compared with one-hot encoding, just adding one extra column at the end.Figure 12
Target means calculated using each color value and encoded to the color target.Target encoding could also smooth the calculation with a prior term as shown in the following formula.(9)mean_target=class_inclass+Priortotal_count+1,where, in the equation, count_inclass were the number of counts the label value equal to 1 for the objects against the categorical feature value, prior value can be assumed was determined the starting parameters, and total count means the total number of things with the categorical feature value.(2) Ordering Boosting. The ordered target encoding technique helps prevent overfitting due to target leakage.The encoded value estimates the expected target value against each feature category.(10)Estb|ai=aki.Boost implements an efficient modification of the ordered boosting on the basic decision tree. It was good for small datasets, support training with pairs, good quality with default parameters, extensive support of models formats, stable and model analysis tool. The classical boosting uses multiple trees and whole datasets with the residuals, which causes overfitting. The ordered boosting does not use the whole datasets to calculate residuals.Assuming modelMi was trained on the first data points, then calculating the residuals at each point i using model Mi − 1. The idea is that the tree did not see the data points as before, so it cannot overfit. Figure 13 shows the N separate trees with data point M4 [56].Figure 13
Ordering boosting to avoid overfitting problem on four data points.The model was trained on four data points,M4. The residuals are shown in equation (1).(11)rx5,y5=y5−M4x5,where N trees are not feasible, and it works with trees at location 2, where j = 1, 2, … log2 (n).


## 3.5.4. Light Gradient Boosting (LGBM)

LightGBM is a gradient boosting framework that uses a decision-tree-based learning algorithm fast, distributed, and reduces the memory usage designed by Microsoft Research Asia [60].(1) Gradient-based one-side Sampling (GOSS). This method focuses more on the under-trained part of the dataset, which tried to learn more aggressively. The slight gradient means that it contains minor errors, which means the data points are learned well. The large gradient implies significant errors, which means the data points are not known well. The algorithm is supported for large gradients, and it is much essential. The algorithm of GOSS in Table 4 first sorts the data points according to their absolute gradient value.Table 4
The gradient-based one-side sampling algorithm.
Input: [Tr = training data, iter = iterations, LGA = sample of large gradient data, SGD = sample of small gradient data, L = loss function, Lr = weak learner]Models ← [ ], fact ← 1 − LGA/SGDtopNumber ← LGA × len (Tr)randNumber ← SGD × len (Tr)Fori = 1 to iter dopreds ← models.predict (Tr) h ← L (Tr, preds),k ← [1, 1, ...] sorted ← GetSortedIndices (abs (h)) topSet ← sorted [1: topNumber] randSet ← RandomPick (sorted [topNumber:len (Tr)], randNumber) usedSet ← topSet + randSet k [randSet] × = fact (Assigning weight for small gradient) newModel ← Lr (I [usedSet], −h [usedSet], k [usedSet])models.append (newModel)Then, the top sampling ratio of the large gradient of data (LGD) points × 100% instances was considered. Then, it randomly samples the proportion of small gradient data (SGD) × 100% instances from the rest of the data points. In the end, GOSS amplified the sampled data with a small gradient by multiplying 1 − LGA/SGD when calculating the information gain. We focused more on the under-trained instances without changing the original data distribution by much.Figure14 explains the light GBM split tree leaf-wise.Figure 14
Light GBM classifier growth leaf-wise.(2) Exclusive Feature Bundling (FEB). It efficiently represents sparse features such as 100 encoded features, reducing the total number of features.It is designed to be a distributed, high-performance gradient boosting framework based on a decision tree algorithm with lower memory usage and capable of large-scale handling data [61].


## 4. Experimental Setup and Hyperparameter Adjustments

The experiments were performed on Google Colab. The Python 3.8 language is used for our experiments. The original dataset splits with training samples is 687 for training data and 230 after 75 : 25 ratios without oversampling. After resampling, the division of datasets was 1552 518, respectively. Three healthy, partial, and ruptured classes for each test were divided into 170, 170, and 178, respectively. All machine learning models have used the machine learning library Scikit-learn with version 1.0.1 [62].Furthermore, we were trained our models on default parameters on all twelve machine learning models with and without oversampling class balancing. After a few adjustments in the parameter values of four models, random forest (RF), extra tree classifier (ETC), categorical boosting, and Light GBM, the results were performed very well during training. Table5 describes the parameters with descriptions and values against every four models. Some parameters have not applicable (NA) values in the table. For RF, ETC, the criteria performed well in the case of measures of Gini index entropy, respectively.Table 5
The parameters and values of four machine learning models.
ParametersDescriptionRF valuesETC valuesCatBoost valuesLGBM valuesn_estimatorsNumber of forest trees100100100200CriterionMeasure the quality of a splitGini indexEntropyNANAmin_samples_splitNumber of samples required to split an internal node24NANAn_jobsThe number of jobs to run in parallel11NA5num_iterationsNumber of boosting iterationsNANA200100Learning_rateThe learning rate used for trainingNANA0.50.2Max_depthThe maximum depth of the tree< min_sample_split< min_sample_split10-1num_leavesThe maximum number of leaves in one treeNANA3165


## 5. Results and Discussion

The final results and discussion are explained in this section for our best machine learning models and compared with the class imbalance and class balance. The performance of the proposed technique is evaluated through confusion matrix, accuracy, precision, recall, F1-score, an area under the curve (AUC), and receiver operative characteristics (ROC). The details of these evaluation metrics are as follows.


### 5.1. Confusion Matrix

The confusion matrix allows visualization of the performance of the models. The confusion matrix is based on theK × K matrix of the ratio of predicted categories or classes that were correctly predicted and not corrected predicted. The matrix gives the direct comparison of values such as true positive (TP), false positive (FP), true negative (TN), and false negative (FN).Figure15 shows the confusion matrix of four models before and after class balancing.Figure 15
The confusion matrix. (a) Random forest classifier 25% test split before class balanced. (b) Random forest classifier 25% test split after class balanced. (c) ETC 25% test split before class balanced. (d) ETC 25% test split after class balanced. (e) CatBoost classifier 25% test split before class balanced. (f) CatBoost classifier 25% test split after class balanced. (g) LGBM 25% test split before class balanced. (h) LGBM 25% test split after class balanced.
(a)(b)(c)(d)(e)(f)(g)(h)


### 5.2. Accuracy

The sum of the correct classification was divided by the total number of three ACL classifications. The accuracy of equation (2) is as follows:(12)accuacy=sum of correct classficationtotal number of three ACL classes.


### 5.3. Precision

The precision is the ratio between the true positive and the positive results. The precision is a valuable matrix when the false positives are more important than false negatives. Accuracy can be expressed as in equation (3).(13)precision=true positivetrue positive+false positive.


### 5.4. Recall

The proportion of actual positive cases was predicted correctly in the three classes. Equation (4) is expressed using the recall formula.(14)recall=true posistivetrue postive+false negative.


### 5.5. F1-Score

It is defined to be the harmonic mean between precision and recall. Equation (5) is the formula for F1-score.(15)F1−score=2∗precision∗recallprecision+recall.


### 5.6. Receiver Operating Characteristic Curve (ROC)

The receiver operating characteristic curve is the graph against the classification models for all class performance. The curve represents a comparison of the true positive rate (TPR) and the false positive rate (FPR) in the following equations:(16)false positive rate=false positivefalse positive+true negative,(17)true postive rate=true positivetrue positive+false negative.


### 5.7. Area under the Curve (AUC)

The last metric, AUC, is the quantitative index to describe accuracy. The AUC is computed as follows:(18)area under curve=1+true positive rate−false positive rate2.Table6 describes the result of three classes mean with accuracy, precision, recall, F1-score, and AUC of imbalanced and balanced datasets of our four machine learning models. The precision, recall, and F1-score results were lower than 40% in the case of without balanced classes. However, in the oversampled approach, the accuracy, recall, and F1-score were 94% to 98%.Table 6
The evaluation metrics of four machine learning models.
DatasetMachine learning modelsEvaluation metricsPrecisionRecallF1-scoreAccuracy (%)AUCImbalancedRandom forest classifier0.4030.3730.36375.650.596Extra tree classifier0.3360.3400.30375.210.597Cat Boost classifier0.3430.353033672.170.586Light GBM classifier0.3400.3600.34670.860.553Random forest classifier0.9600.9560.96095.750.997OversamplingExtra tree classifier0.9800.9830.98398.260.997Class-balanceCat Boost classifier0.9500.9530.94694.980.996Light GBM classifier0.9530.9530.95094.980.995Figure16 shows the comparison accuracy of twelve models in the case of imbalanced datasets. The accuracy of models logistic regression, support vector machine, random forest classifier, gradient boosting classifier, extra tree classifier achieved 75%. The XGB classifier, Naïve Bayes, k-nearest neighbours, AdaBoost classifier, Cat Boost classifier, and LGBM classifier remained accurate from 74% to 70%. The lowest accuracy, 63%, was the decision tree classifier.Figure 16
The accuracy comparison of the imbalanced dataset of twelve models.This study aims to achieve optimal performance through machine learning classifiers. For this, we were evaluated twelve machine learning models after balanced classes through oversampling. Figure17 shows the comparison accuracy of twelve models in balanced datasets.Figure 17
The accuracy comparison of the balanced dataset of twelve models.The accuracy of all models extra tree classifier, random forest classifier, Cat Boost classifier, LGBM classifier, gradient boosting classifier, decision tree classifier, XGB classifier, k-nearest neighbours, AdaBoost classifier, Naïve Bayes, logistic regression, support vector machine was achieved 98.26%, 95.75%, 94.98%, 94.98%, 82.04%, 77.79%, 75.48%, 75.09%, 54.44%, 42.08%, 32.81%, 31.85%, respectively. The accuracy was above 94% for extra tree classifiers, random forest classifier, Cat Boost classifier, and LGBM classifier. The worst accuracy was 31.85% in the case of support vector machines.Figure18 shows the plotting of receiver operating characteristic (ROC) and comparison of AUC on the best four models extra tree classifier, random forest classifier, Cat Boost classifier, LGBM classifier without class balancing.Figure 18
The ROC-AUC curve of four models without class balancing.In the end, Figure19 shows the plotting of receiver operating characteristic (ROC) and comparison of AUC on the best four models extra tree classifier, random forest classifier, Cat Boost classifier, LGBM classifier with oversampling class balancing. It is clearly shown that the AUC of these four models was 0.997, 0.997, 0.996, and 0.995, respectively, after oversampling technique, whereas, in the case of without class balancing, these remained 0.597, 0.595, 0.586, and 0.553, respectively.Figure 19
The ROC-AUC curve of four models with oversampling class balancing.Previously studies were performed on the author’s knee dataset on the MR images (unstructured) only. As per our knowledge, there was no such study available to diagnose ACL tears through structured data to resolve the imbalanced problem. Table7 shows the comparison of the proposed machine learning methods with oversampling with other benchmark techniques, machine learning, and deep learning approaches.Table 7
The benchmark studies comparison with four machine learning models.
StudiesDataset ACL nature/trained onModelsEvaluationClass balanceAccuracy (%)AUCStajduhar et al. [37]Unstructured: MR images 10-fold cross-validationHOG + Lin SVMNo—0.894HOG + RFNo—0.943Dunnhofer et al. [63]Unstructured: MR images 5-fold cross-validation train test split: 80 : 20MRNet with MRPyrNetNo83.40.914ELNet with MRPyrNetNo85.10.900Kapoor et al. [64]Unstructured: MR images train test split: 70 : 30CNNNo82.0—DNN82.0—RNN81.8—SVM88.20.910Javed Awan et al. [42]Unstructured: MR images train test split: 75 : 25ResNet-14Yes hybrid90.00.973Awan et al. [65]Unstructured: MR images train test split: 75 : 25CNNNo97.10.990Proposed machine learning modelsStructured: CSV train test split: 75 : 25RFYes oversampling95.750.997ETC98.260.997Boost94.980.996LGBM94.980.995It is clearly shown that the machine learning model extra tree classifier performed 98.26% accuracy result and AUC 0.997 among the best of all studies from structured and unstructured data.Our study has several limitations. First, the machine learning models tuned only four models. Second, the machine learning models have applied only class balancing techniques through oversampling. Third, the study is not evaluated through cross-validation and does not compute the processing time for the classification of ACL tears diagnosis. In the future, we can validate our models through big data approaches inspired by recent studies [66–72] after comparing all class balancing.


## 5.1. Confusion Matrix

The confusion matrix allows visualization of the performance of the models. The confusion matrix is based on theK × K matrix of the ratio of predicted categories or classes that were correctly predicted and not corrected predicted. The matrix gives the direct comparison of values such as true positive (TP), false positive (FP), true negative (TN), and false negative (FN).Figure15 shows the confusion matrix of four models before and after class balancing.Figure 15
The confusion matrix. (a) Random forest classifier 25% test split before class balanced. (b) Random forest classifier 25% test split after class balanced. (c) ETC 25% test split before class balanced. (d) ETC 25% test split after class balanced. (e) CatBoost classifier 25% test split before class balanced. (f) CatBoost classifier 25% test split after class balanced. (g) LGBM 25% test split before class balanced. (h) LGBM 25% test split after class balanced.
(a)(b)(c)(d)(e)(f)(g)(h)


## 5.2. Accuracy

The sum of the correct classification was divided by the total number of three ACL classifications. The accuracy of equation (2) is as follows:(12)accuacy=sum of correct classficationtotal number of three ACL classes.


## 5.3. Precision

The precision is the ratio between the true positive and the positive results. The precision is a valuable matrix when the false positives are more important than false negatives. Accuracy can be expressed as in equation (3).(13)precision=true positivetrue positive+false positive.


## 5.4. Recall

The proportion of actual positive cases was predicted correctly in the three classes. Equation (4) is expressed using the recall formula.(14)recall=true posistivetrue postive+false negative.


## 5.5. F1-Score

It is defined to be the harmonic mean between precision and recall. Equation (5) is the formula for F1-score.(15)F1−score=2∗precision∗recallprecision+recall.


## 5.6. Receiver Operating Characteristic Curve (ROC)

The receiver operating characteristic curve is the graph against the classification models for all class performance. The curve represents a comparison of the true positive rate (TPR) and the false positive rate (FPR) in the following equations:(16)false positive rate=false positivefalse positive+true negative,(17)true postive rate=true positivetrue positive+false negative.


## 5.7. Area under the Curve (AUC)

The last metric, AUC, is the quantitative index to describe accuracy. The AUC is computed as follows:(18)area under curve=1+true positive rate−false positive rate2.Table6 describes the result of three classes mean with accuracy, precision, recall, F1-score, and AUC of imbalanced and balanced datasets of our four machine learning models. The precision, recall, and F1-score results were lower than 40% in the case of without balanced classes. However, in the oversampled approach, the accuracy, recall, and F1-score were 94% to 98%.Table 6
The evaluation metrics of four machine learning models.
DatasetMachine learning modelsEvaluation metricsPrecisionRecallF1-scoreAccuracy (%)AUCImbalancedRandom forest classifier0.4030.3730.36375.650.596Extra tree classifier0.3360.3400.30375.210.597Cat Boost classifier0.3430.353033672.170.586Light GBM classifier0.3400.3600.34670.860.553Random forest classifier0.9600.9560.96095.750.997OversamplingExtra tree classifier0.9800.9830.98398.260.997Class-balanceCat Boost classifier0.9500.9530.94694.980.996Light GBM classifier0.9530.9530.95094.980.995Figure16 shows the comparison accuracy of twelve models in the case of imbalanced datasets. The accuracy of models logistic regression, support vector machine, random forest classifier, gradient boosting classifier, extra tree classifier achieved 75%. The XGB classifier, Naïve Bayes, k-nearest neighbours, AdaBoost classifier, Cat Boost classifier, and LGBM classifier remained accurate from 74% to 70%. The lowest accuracy, 63%, was the decision tree classifier.Figure 16
The accuracy comparison of the imbalanced dataset of twelve models.This study aims to achieve optimal performance through machine learning classifiers. For this, we were evaluated twelve machine learning models after balanced classes through oversampling. Figure17 shows the comparison accuracy of twelve models in balanced datasets.Figure 17
The accuracy comparison of the balanced dataset of twelve models.The accuracy of all models extra tree classifier, random forest classifier, Cat Boost classifier, LGBM classifier, gradient boosting classifier, decision tree classifier, XGB classifier, k-nearest neighbours, AdaBoost classifier, Naïve Bayes, logistic regression, support vector machine was achieved 98.26%, 95.75%, 94.98%, 94.98%, 82.04%, 77.79%, 75.48%, 75.09%, 54.44%, 42.08%, 32.81%, 31.85%, respectively. The accuracy was above 94% for extra tree classifiers, random forest classifier, Cat Boost classifier, and LGBM classifier. The worst accuracy was 31.85% in the case of support vector machines.Figure18 shows the plotting of receiver operating characteristic (ROC) and comparison of AUC on the best four models extra tree classifier, random forest classifier, Cat Boost classifier, LGBM classifier without class balancing.Figure 18
The ROC-AUC curve of four models without class balancing.In the end, Figure19 shows the plotting of receiver operating characteristic (ROC) and comparison of AUC on the best four models extra tree classifier, random forest classifier, Cat Boost classifier, LGBM classifier with oversampling class balancing. It is clearly shown that the AUC of these four models was 0.997, 0.997, 0.996, and 0.995, respectively, after oversampling technique, whereas, in the case of without class balancing, these remained 0.597, 0.595, 0.586, and 0.553, respectively.Figure 19
The ROC-AUC curve of four models with oversampling class balancing.Previously studies were performed on the author’s knee dataset on the MR images (unstructured) only. As per our knowledge, there was no such study available to diagnose ACL tears through structured data to resolve the imbalanced problem. Table7 shows the comparison of the proposed machine learning methods with oversampling with other benchmark techniques, machine learning, and deep learning approaches.Table 7
The benchmark studies comparison with four machine learning models.
StudiesDataset ACL nature/trained onModelsEvaluationClass balanceAccuracy (%)AUCStajduhar et al. [37]Unstructured: MR images 10-fold cross-validationHOG + Lin SVMNo—0.894HOG + RFNo—0.943Dunnhofer et al. [63]Unstructured: MR images 5-fold cross-validation train test split: 80 : 20MRNet with MRPyrNetNo83.40.914ELNet with MRPyrNetNo85.10.900Kapoor et al. [64]Unstructured: MR images train test split: 70 : 30CNNNo82.0—DNN82.0—RNN81.8—SVM88.20.910Javed Awan et al. [42]Unstructured: MR images train test split: 75 : 25ResNet-14Yes hybrid90.00.973Awan et al. [65]Unstructured: MR images train test split: 75 : 25CNNNo97.10.990Proposed machine learning modelsStructured: CSV train test split: 75 : 25RFYes oversampling95.750.997ETC98.260.997Boost94.980.996LGBM94.980.995It is clearly shown that the machine learning model extra tree classifier performed 98.26% accuracy result and AUC 0.997 among the best of all studies from structured and unstructured data.Our study has several limitations. First, the machine learning models tuned only four models. Second, the machine learning models have applied only class balancing techniques through oversampling. Third, the study is not evaluated through cross-validation and does not compute the processing time for the classification of ACL tears diagnosis. In the future, we can validate our models through big data approaches inspired by recent studies [66–72] after comparing all class balancing.


## 6. Conclusion

The anterior cruciate ligament is essential for evaluating osteoarthritis and osteoporosis. It is necessary to diagnose the ACL ruptured tears in the early stages to avoid the surgery procedure. The study fairly compared and evaluated four out of twelve machine learning classification models, namely, random forest (RF), extra tree classifier (ETC), categorical boosting (CatBoost), and light gradient boosting machines (LGBM). All models’ performance remained under 74% without class balancing. After adjusting hyperparameters and class balancing, the accuracy of the four models, RF, ETC, CatBoost, and LGBM, achieved 95.75%, 98.26%, 94.98%, and 94.98%, respectively. Moreover, the ROC-AUC score of the four models is 0.997. In the future, we can apply machine learning models through MR images.



---

*Source: 2550120-2022-04-11.xml*