<figure><img src="image_104.png" /><figcaption>**Figure 5.** Graphs of accuracy vs. epochs of multi-channel multi-scale deep CNN obtained using training and validation VCG instances for MI detection.</figcaption></figure>

<table><caption>**Table 3.** Classification results of two class multi-channel multi-scale deep CNN obtained for the detection of MI using hold-out validation.</caption><thead><tr><th>Accuracy (%)</th><th>Sensitivity (%)</th><th>Specificity (%)</th><th>Kappa</th></tr></thead><tbody><tr><td>99.58 ± 0.38</td><td>99.18 ± 0.90</td><td>99.87 ± 0.07</td><td>0.990 ± 0.01</td></tr></tbody></table>

<table><caption>**Table 4.** Confusion matrix for one-trail of hold-out CV for MI detection.</caption><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th colspan="2">Predicted</th></tr><tr><th>Healthy</th><th>MI</th></tr></thead><tbody><tr><th rowspan="2">Actual</th><th>Healthy</th><td>978</td><td>1</td></tr><tr><th>MI</th><td>1</td><td>1390</td></tr></tbody></table>

For MI detection, the classification results obtained for the proposed first stage MMDCNN using 10-fold CV are shown in Table 5. It can be observed from this table that, the accuracy values are more than 99.50% for each fold. Similar high percentages in the sensitivity and specificity are seen in each fold using the first stage MMDCNN method for MI detection. It can also be observed that the Cohen kappa score is more than 0.99 for each fold. From these 10-fold CV results, It can be noted that the proposed first stage deep CNN successfully detected MI using the modes of VCG beats.

<table><caption>**Table 5.** Results obtained using multi-channel multi-scale deep CNN with 10-fold CV.</caption><thead><tr><th>Folds</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th><th>9</th><th>10</th><th>Value (μ ± σ)</th></tr></thead><tbody><tr><th>Accuracy (%)</th><td>99.7</td><td>100</td><td>99.95</td><td>100</td><td>99.74</td><td>99.95</td><td>100</td><td>100</td><td>100</td><td>100</td><td>99.93 ± 0.11</td></tr><tr><th>Sensitivity (%)</th><td>100</td><td>100</td><td>100</td><td>100</td><td>99.38</td><td>100</td><td>100</td><td>100</td><td>100</td><td>100</td><td>99.93 ± 0.19</td></tr><tr><th>Specificity (%)</th><td>99.49</td><td>100</td><td>99.99</td><td>100</td><td>100</td><td>99.92</td><td>100</td><td>100</td><td>100</td><td>100</td><td>99.94 ± 0.16</td></tr><tr><th>Kappa</th><td>0.993</td><td>1</td><td>0.999</td><td>1</td><td>0.994</td><td>0.999</td><td>1</td><td>1</td><td>1</td><td>1</td><td>0.998 ± 0.002</td></tr></tbody></table>
The confusion matrix obtained using one random trial-based hold-out validation for MI localization with second stage MMDCNN is shown in Table 6. Similarly, we have shown the accuracy vs. epoch plots for training and validation of multi-scale VCG tensor instances in Figure 6. It can be observed from these plots that both training and validation accuracy values obtained are more than 99% after 10th epoch using the second stage MMDCNN model. It can be seen from Table 6 that the number of true positives for AMI, IMI, ALMI, ASMI, ILMI, and IPLMI classes are obtained as 162, 284, 185, 287, 301, and 201, respectively. Three multi-scale VCG tensor instances, which belong to IMI, are classified as ALMI class. Similarly, the classification results of the proposed second-stage MMDCNN obtained for MI localization using hold-out validation are shown in Table 7. It can be noted that the average IA values are more than 99% for AMI, IMI, ALMI, ILMI, and IPLMI classes. For ASMI class, the IA value is 94.38%. The OA and kappa values obtained are 98.77% and 0.982, respectively, using the proposed second-stage MMDCNN model.

<figure><img src="image_105.png" /><figcaption>**Figure 6.** Plots of accuracy vs. epochs of multi-channel multi-scale deep CNN obtained using training and validation VCG instances for MI localization.</figcaption></figure>

<table><caption>**Table 6.** Confusion matrix obtained using the proposed second stage multi-channel multi-scale deep CNN classifier for MI localization.</caption><thead><tr><th rowspan="2"></th><th colspan="6">Predicted Classes</th></tr><tr><th>AMI</th><th>ALMI</th><th>ASMI</th><th>IMI</th><th>ILMI</th><th>IPLMI</th></tr></thead><tbody><tr><th rowspan="6">Actual Classes</th><th>AMI</th><td>162</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><th>ALMI</th><td>0</td><td>185</td><td>0</td><td>0</td><td>0</td></tr><tr><th>ASMI</th><td>0</td><td>0</td><td>287</td><td>0</td><td>0</td></tr><tr><th>IMI</th><td>0</td><td>3</td><td>0</td><td>284</td><td>0</td></tr><tr><th>ILMI</th><td>0</td><td>0</td><td>0</td><td>0</td><td>301</td></tr><tr><th>IPLMI</th><td>0</td><td>0</td><td>0</td><td>0</td><td>200</td></tr></tbody></table>
<table><caption>**Table 7.** Classification results obtained for MI localization using proposed MMDCNN model with hold-out validation.</caption><thead><tr><th>Parameters</th><th>Value (μ ± σ)</th></tr></thead><tbody><tr><td>IA<sub>AMI</sub> (%)</td><td>99.79 ± 0.35</td></tr><tr><td>IA<sub>ALMI</sub> (%)</td><td>99.64 ± 0.31</td></tr><tr><td>IA<sub>ASMI</sub> (%)</td><td>94.38 ± 5.56</td></tr><tr><td>IA<sub>IMI</sub> (%)</td><td>99.53 ± 0.53</td></tr><tr><td>IA<sub>ILMI</sub> (%)</td><td>99.66 ± 0.00</td></tr><tr><td>IA<sub>IPLMI</sub> (%)</td><td>99.66 ± 0.57</td></tr><tr><td>OA (%)</td><td>98.77 ± 0.96</td></tr><tr><td>Kappa</td><td>0.982 ± 0.014</td></tr></tbody></table>

Moreover, we have shown the classification results of second-stage MMDCNN for MI localization using a 10-fold CV and these results are shown in Table 8. It can be observed from these results that for ILMI class, the accuracy value of each fold is more than 99%. Similarly, for IMI class, apart from 5th fold, more than 99% accuracy values are observed for other folds. For IPLMI and AMI classes, more than 98% accuracy values are obtained in each fold using second stage MMDCNN model. Similarly, more than 97% accuracy values are obtained using MMDCNN classifier for ASMI and ALMI classes. The overall accuracy (OA) values are obtained as more than 99% at each fold. The kappa value of more than 0.97 is observed for each fold using MMDCNN classifier.

<table><caption>**Table 8.** Classification results obtained for MI localization using proposed MMDCNN model with 10-fold CV.</caption><thead><tr><th>Folds</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th><th>9</th><th>10</th><th>μ ± σ</th></tr></thead><tbody><tr><td>IA<sub>AMI</sub> (%)</td><td>100</td><td>98.79</td><td>100</td><td>99.4</td><td>100</td><td>100</td><td>100</td><td>100</td><td>100</td><td>100</td><td>99.81 ± 0.40</td></tr><tr><td>IA<sub>ALMI</sub> (%)</td><td>99.43</td><td>97.75</td><td>97.75</td><td>96.57</td><td>99.43</td><td>98.31</td><td>97.19</td><td>99.43</td><td>97.75</td><td>98.3</td><td>98.19 ± 0.99</td></tr><tr><td>IA<sub>ASMI</sub> (%)</td><td>97.38</td><td>95.09</td><td>100</td><td>97.37</td><td>100</td><td>100</td><td>99.67</td><td>98.69</td><td>99.01</td><td>100</td><td>98.72 ± 1.64</td></tr><tr><td>IA<sub>IMI</sub> (%)</td><td>100</td><td>99.64</td><td>100</td><td>99.28</td><td>98.93</td><td>99.64</td><td>98.57</td><td>100</td><td>99.64</td><td>99.28</td><td>99.49 ± 0.48</td></tr><tr><td>IA<sub>ILMI</sub> (%)</td><td>99.67</td><td>99.66</td><td>99.34</td><td>99.67</td><td>99.01</td><td>99.67</td><td>99.01</td><td>99.67</td><td>100</td><td>100</td><td>99.57 ± 0.34</td></tr><tr><td>IA<sub>IPLMI</sub> (%)</td><td>99.49</td><td>100</td><td>98.98</td><td>98.48</td><td>99.49</td><td>99.49</td><td>98.48</td><td>100</td><td>98.98</td><td>98.48</td><td>99.18 ± 0.59</td></tr><tr><td>OA(%)</td><td>99.72</td><td>99.09</td><td>99.44</td><td>99.23</td><td>99.37</td><td>99.65</td><td>98.88</td><td>99.58</td><td>99.37</td><td>99.44</td><td>99.37 ± 0.25</td></tr><tr><td>Kappa</td><td>0.990</td><td>0.978</td><td>0.993</td><td>0.979</td><td>0.993</td><td>0.994</td><td>0.986</td><td>0.994</td><td>0.991</td><td>0.993</td><td>0.989 ± 0.006</td></tr></tbody></table>

The classification results of MMDCNN models evaluated using the selected modes of each lead VCG signal, and all modes of high-pass filtered VCG signals for MI detection with hold-out validation are shown in Table 9. It is observed that the average accuracy value of MMDCNN is 99.58% using mode 1 and mode 2 of each lead VCG signal. The average accuracy value remains the same as the accuracy of MMDCNN model using all modes of VCG signals for MI detection. Mode 1 and mode 2 capture the significant information of the VCG signal after decomposition using MVMD. Henceforth, the accuracy value remains the same for MI detection using selected modes and the MMDCNN classifier. Moreover, we have also evaluated the classification performance of the MMDCNN model using all modes of high-pass filtered VCG signal for MI detection. A high-pass Butterworth filter with a cut-off frequency of 0.5 Hz is applied to each lead VCG signal to remove baseline wondering artifacts [6,28]. It is observed from Table 9 that average accuracy, average kappa score, average sensitivity, and average specificity values are improved after the filtering of baseline wandering artifact from VCG signals. In Table 10, we have shown the individual accuracy value for each MI class, OA, and kappa scores of MMDCNN classifier for MI localization using mode 1 and mode 2 of each lead VCG signal and all modes of high-pass filtered VCG signals, respectively. It is observed that the OA value obtained using the MMDCNN model is less using mode 1 and mode 2 of VCG signals as compared to all modes of VCG signals. Similarly, the OA and kappa values are improved using the modes of high-pass filtered VCG signals with the MMDCNN classifier. For MI localization, the IA values for ASMI, IMI, and ILMI classes are also improved using the
modes of high-pass filtered VCG signals composed with the MMDCNN classifier. Moreover, we have also evaluated the classification results of the MMDCNN classifier using all modes of VCG signal with leave one out (LOO) CV strategy. The VCG beats of one recording are considered during testing of the MMDCNN model, whereas the VCG beats of all other VCG recordings are used to train the MMDCNN classifier. The same procedure is applied to all VCG recordings, and it can also be interpreted as a 171-fold CV strategy. The LOO CV or pre-recording-based MI detection results are shown in Figure 7. It is observed that out of 172 VCG recordings, 114 recordings are correctly classified with 100% accuracy. The OA value obtained using MMDCNN classifier with LOO CV strategy is 87.65%.

<figure><img src="image_106.png" /><figcaption>**Figure 7.** Accuracy values obtained using the proposed MMDCNN classifier for MI detection with leave one out CV strategy.</figcaption></figure>

<table><caption>**Table 9.** Classification results of MMDCNN obtained using selected modes and all modes for the detection of MI.</caption><thead><tr><th>Mode Selection</th><th>Accuracy (%)</th><th>Sensitivity (%)</th><th>Specificity (%)</th><th>Kappa</th></tr></thead><tbody><tr><td>Mode 1 and mode 2 of each lead VCG signals</td><td>99.58 ± 0.37</td><td>99.17 ± 0.89</td><td>99.86 ± 0.06</td><td>0.991 ± 0.007</td></tr><tr><td>All modes from high-pass filtered VCG signals</td><td>99.92 ± 0.02</td><td>99.84 ± 0.05</td><td>100 ± 0</td><td>0.998 ± 0.0005</td></tr></tbody></table>

<table><caption>**Table 10.** Classification results obtained for MI localization using proposed MMDCNN model with mode selection.</caption><thead><tr><th>Parameters</th><th>Mode 1 and Mode 2 from Each Lead VCG Signals</th><th>All Modes from High-Pass Filtered VCG Signals</th></tr></thead><tbody><tr><td>IA<sub>AMI</sub> (%)</td><td>100 ± 0</td><td>100 ± 0</td></tr><tr><td>IA<sub>ALMI</sub> (%)</td><td>99.17 ± 0.29</td><td>98.55 ± 2.22</td></tr><tr><td>IA<sub>ASMI</sub> (%)</td><td>93.09 ± 5.16</td><td>99.26 ± 0.27</td></tr><tr><td>IA<sub>IMI</sub> (%)</td><td>99.30 ± 0.28</td><td>100 ± 0</td></tr><tr><td>IA<sub>ILMI</sub> (%)</td><td>99.50 ± 0.19</td><td>99.86 ± 0.14</td></tr><tr><td>IA<sub>IPLMI</sub> (%)</td><td>99.25 ± 0.50</td><td>98.45 ± 2.45</td></tr><tr><td>OA (%)</td><td>98.37 ± 1.43</td><td>99.44 ± 0.56</td></tr><tr><td>Kappa</td><td>0.976 ± 0.014</td><td>0.992 ± 0.006</td></tr></tbody></table>

We have also formulated the seven-class classification scheme as (Healthy vs. AMI vs. ALMI vs. ASMI vs. IMI vs. ILMI vs. IPLMI) using MMDCNN classifier with all modes of VCG signals. The seven class classification results obtained using the MMDCNN model are shown in Table 11. It can be observed that for healthy, ALMI, IMI, and IPLMI classes, the IA values are 69.87%, 83.22%, 82.61%, and 41.03%, respectively. The OA value of MMDCNN classifier obtained is 81.48%, which is less than the proposed two-stage MMDCNN model for MI detection and localization.
<table><caption>**Table 11.** Classification results obtained using MMDCNN classifier for seven class classification scheme with hold-out validation.</caption><thead><tr><td>Parameters</td><td>Value (μ ± σ)</td></tr></thead><tbody><tr><td>Healthy (%)</td><td>69.87 ± 28.87</td></tr><tr><td>IA<sub>AMI</sub> (%)</td><td>99.46 ± 0.38</td></tr><tr><td>IA<sub>ALMI</sub> (%)</td><td>83.22 ± 10.14</td></tr><tr><td>IA<sub>ASMI</sub> (%)</td><td>99.63 ± 0.25</td></tr><tr><td>IA<sub>IMI</sub> (%)</td><td>82.61 ± 7.93</td></tr><tr><td>IA<sub>ILMI</sub> (%)</td><td>98.44 ± 2.28</td></tr><tr><td>IA<sub>IPLMI</sub> (%)</td><td>41.03 ± 18.18</td></tr><tr><td>OA (%)</td><td>81.48 ± 0.63</td></tr><tr><td>Kappa</td><td>0.777 ± 0.008</td></tr></tbody></table>

The classification performance of the proposed first stage MMDCNN classifier is compared with the existing techniques for MI detection using VCG signals with a 10-fold CV-based technique. The comparison results are shown in Table 12. The work reported in [15] has computed features from each lead of VCG signal using multi-scale recurrent quantification analysis (MRQA). The Gaussian discriminant analysis (GDA) based classification model has been used to detect MI using MRQA based VCG features. The sensitivity and specificity values of 96.50% and 75% have been obtained in their work. Similarly, in [14], the combination of octant and vector-based features have been obtained using VCG signal. The classification and regression tree (CART) based model has been used for the detection of MI. The classification performance, such as the sensitivity and specificity values of 97.28% and 96%, respectively, are reported. The complex wavelet sub-band features of VCG coupled with the RVM classifier have obtained the sensitivity and specificity values of 98.40%, and 98.66%, respectively, for MI detection [28]. The proposed MMDCNN model has obtained better classification performance than the existing machine learning-based methods for MI detection using VCG signals. The advantages of our proposed MMDCNN based approach are given as follows:

* A novel two-stage based MMDCNN model is proposed to detect and localize MI using VCG beats;

* The multi-scale analysis of VCG signal is performed using MVMD based multi-variate signal driven approach;

* The approach has demonstrated more than 99% accuracy for MI detection;

* The extraction of raw features from VCG signals are not required using the proposed approach for both detection and localization stages;

* The second stage MMDCNN model successfully classified six types of MI with an accuracy of more than 99%.

<table><caption>**Table 12.** Comparison of proposed MI detection approach with existing methods obtained using VCG signals (with 10-fold CV).</caption><thead><tr><td>Authors</td><td>Features Extracted</td><td>Classifiers Used</td><td>Sensitivity (%)</td><td>Specificity (%)</td></tr></thead><tbody><tr><td>Yang et al., 2012 [14]</td><td>Octant and vector features evaluated from VCG signal</td><td>CART</td><td>97.28</td><td>95</td></tr><tr><td>Yang, 2011 [15]</td><td>DWT domain RQA features from VCG Signal</td><td>GDA</td><td>96.50</td><td>75</td></tr><tr><td>Tripathy et al., 2017 [28]</td><td>Complex wavelet sub-band features from VCG signal</td><td>RVM</td><td>98.40</td><td>98.66</td></tr><tr><td>Proposed work</td><td>Multi-channel and multi-scale domain learnable features</td><td>CNN</td><td>99.93</td><td>99.94</td></tr></tbody></table>
In this work, the proposed approach has considered only 99 VCG recordings from different MI classes in the second stage for MI localization. The approach can be tested using VCG recordings from a huge database containing more subjects. The MVMD based multi-scale approach is used in this study to decompose the VCG signal. The other multi-scale analysis methods, such as multivariate empirical mode decomposition (MEMD) [40], multivariate projection based empirical wavelet transform (MPEWT) [41], and fast and adaptive based MEMD [42] can be used for the decomposition of VCG signals.

## 5. Conclusions

The multi-channel multi-scale two-stage deep CNN model is proposed to detect and localize MI using VCG signals. The MVMD is used to decompose the VCG beat into modes along with each orthogonal lead. The multi-channel multi-scale VCG tensor has been formulated and used as input to the deep CNN model to detect and localize MI. For MI detection, the proposed first-stage MMDCNN model obtained an average accuracy value of 99.93% with 10-fold CV. The second-stage MMDCNN model produced an average overall accuracy (OA) value of 99.37% for MI localization. The average OA values are more than 99% for AMI, IMI, ILMI, and ILMI classes. The proposed first-stage MMDCNN classifier obtained a higher accuracy value than the existing VCG based approaches for MI detection. The MMDCNN model can also be explored to detect other cardiac ailments, such as atrial fibrillation, hypertrophy, cardiomyopathy, ventricular arrhythmia, and bundle branch block using VCG signals.

**Author Contributions:** Conceptualization, J.K.; Data curation, R.K.T. and S.K.G.; Formal analysis, J.K. and U.R.A.; Methodology, P.G. and R.K.T.; Project administration, U.R.A.; Resources, R.K.T.; Supervision, R.K.T.; Writing—original draft, R.K.T., P.G.; Writing—review and editing, U.R.A. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** The codes of the work is available at (https://github.com/JayKarhade/MI_VCG_DL (accessed on 20 August 2021)).

**Conflicts of Interest:** The authors declare no conflict of interest.

## References

1. Antman, E.M.; Anbe, D.T.; Armstrong, P.W.; Bates, E.R.; Green, L.A.; Hand, M.; Hochman, J.S.; Krumholz, H.M.; Kushner, F.G.; Lamas, G.A.; et al. ACC/AHA guidelines for the management of patients with ST-elevation myocardial infarction: A report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Committee to Revise the 1999 Guidelines for the Management of Patients with Acute Myocardial Infarction). *J. Am. Coll. Cardiol.* **2004**, *44*, E1–E21.

2. Tripathy, R.K.; Bhattacharyya, A.; Pachori, R.B. Localization of myocardial infarction from multi-lead ECG signals using multiscale analysis and convolutional neural network. *IEEE Sens. J.* **2019**, *19*, 11437–11448. [CrossRef]

3. Tripathy, R.; Dandapat, S. Detection of myocardial infarction from vectorcardiogram using relevance vector machine. *Signal Image Video Process.* **2017**, *11*, 1139–1146. [CrossRef]

4. Hedén, B.; Ohlin, H.; Rittner, R.; Edenbrandt, L. Acute myocardial infarction detected in the 12-lead ECG by artificial neural networks. *Circulation* **1997**, *96*, 1798–1802. [CrossRef]

5. Boateng, S.; Sanborn, T. Acute myocardial infarction. *Disease-a-Month* **2013**, *59*, 83–96. [CrossRef]

6. Sharma, L.; Tripathy, R.; Dandapat, S. Multiscale energy and eigenspace approach to detection and localization of myocardial infarction. *IEEE Trans. Biomed. Eng.* **2015**, *62*, 1827–1837. [CrossRef]

7. Khan, J.N.; Chauhan, A.; Mozdiak, E.; Khan, J.M.; Varma, C. Posterior myocardial infarction: Are we failing to diagnose this? *Emerg. Med. J.* **2012**, *29*, 15–18. [CrossRef]

8. Parale, G.; Kulkarni, P.; Khade, S.; Athawale, S.; Vora, A. Importance of reciprocal leads in acute myocardial infarction. *JAPI* **2004**, *52*, 376–379.

9. Prabhakararao, E.; Dandapat, S. Automated detection of posterior myocardial infarction from VCG signals using stationary wavelet transform based features. *IEEE Sens. Lett.* **2020**, *4*, 1–4. [CrossRef]

10. Schreck, D.M.; Fishberg, R.D. Derivation of the 12-lead electrocardiogram and 3-lead vectorcardiogram. *Am. J. Emerg. Med.* **2013**, *31*, 1183–1190. [CrossRef]
11. Loperfido, F.; Digaetano, A.; Guccione, P.; Desantis, F.; Vigna, C.; Laurenzi, F.; Solfanelli, N.; Ferrazza, A.; Pennestri, F.; Manzoli, U. Assessment of left ventricular hypertrophy by ECG and VCG in patients with inferior and posterior myocardial infarction. A comparison with echocardiographic data. *J. Electrocardiol.* **1986**, *19*, 247–256. [CrossRef]

12. Bortolan, G.; Christov, I. Myocardial infarction and ischemia characterization from T-loop morphology in VCG. In Proceedings of the Computers in Cardiology 2001, Rotterdam, The Netherlands, 23–26 September 2001; Volume 28, pp. 633–636

13. Dehnavi, A.R.M.; Farahabadi, I.; Rabbani, H.; Farahabadi, A.; Mahjoob, M.P.; Dehnavi, N.R. Detection and classification of cardiac ischemia using vectorcardiogram signal via neural network. *J. Res. Med. Sci. Off. J. Isfahan Univ. Med Sci.* **2011**, *16*, 136.

14. Yang, H.; Bukkapatnam, S.T.; Le, T.; Komanduri, R. Identification of myocardial infarction (MI) using spatio-temporal heart dynamics. *Med. Eng. Phys.* **2012**, *34*, 485–497. [CrossRef] [PubMed]

15. Yang, H. Multiscale recurrence quantification analysis of spatial cardiac vectorcardiogram signals. *IEEE Trans. Biomed. Eng.* **2010**, *58*, 339–347. [CrossRef] [PubMed]

16. Correa, R.; Arini, P.D.; Correa, L.; Valentinuzzi, M.E.; Laciar, E. Acute myocardial ischemia monitoring before and during angioplasty by a novel vectorcardiographic parameter set. *J. Electrocardiol.* **2013**, *46*, 635–643. [CrossRef] [PubMed]

17. Correa, R.; Arini, P.D.; Valentinuzzi, M.E.; Laciar, E. Novel set of vectorcardiographic parameters for the identification of ischemic patients. *Med. Eng. Phys.* **2013**, *35*, 16–22. [CrossRef] [PubMed]

18. Correa, R.; Arini, P.D.; Correa, L.S.; Valentinuzzi, M.; Laciar, E. Novel technique for ST-T interval characterization in patients with acute myocardial ischemia. *Comput. Biol. Med.* **2014**, *50*, 49–55. [CrossRef]

19. Chen, C.; Qin, C.; Qiu, H.; Tarroni, G.; Duan, J.; Bai, W.; Rueckert, D. Deep learning for cardiac image segmentation: A review. *Front. Cardiovasc. Med.* **2020**, *7*, 25. [CrossRef]

20. Oktay, O.; Ferrante, E.; Kamnitsas, K.; Heinrich, M.; Bai, W.; Caballero, J.; Cook, S.A.; De Marvao, A.; Dawes, T.; O’Regan, D.P.; et al. Anatomically constrained neural networks (ACNNs): Application to cardiac image enhancement and segmentation. *IEEE Trans. Med. Imaging* **2017**, *37*, 384–395. [CrossRef]

21. Dahiya, N.; Yezzi, A.; Piccinelli, M.; Garcia, E. Integrated 3D anatomical model for automatic myocardial segmentation in cardiac CT imagery. *Comput. Methods Biomech. Biomed. Eng. Imaging Vis.* **2019**, *7*, 690–706. [CrossRef]

22. Militello, C.; Rundo, L.; Toia, P.; Conti, V.; Russo, G.; Filorizzo, C.; Maffei, E.; Cademartiri, F.; La Grutta, L.; Midiri, M.; et al. A semi-automatic approach for epicardial adipose tissue segmentation and quantification on cardiac CT scans. *Comput. Biol. Med.* **2019**, *114*, 103424. [CrossRef]

23. Tripathy, R.K.; Bhattacharyya, A.; Pachori, R.B. A novel approach for detection of myocardial infarction from ECG signals of multiple electrodes. *IEEE Sens. J.* **2019**, *19*, 4509–4517. [CrossRef]

24. Acharya, U.R.; Fujita, H.; Oh, S.L.; Hagiwara, Y.; Tan, J.H.; Adam, M. Application of deep convolutional neural network for automated detection of myocardial infarction using ECG signals. *Inf. Sci.* **2017**, *415*, 190–198. [CrossRef]

25. Liu, W.; Zhang, M.; Zhang, Y.; Liao, Y.; Huang, Q.; Chang, S.; Wang, H.; He, J. Real-time multilead convolutional neural network for myocardial infarction detection. *IEEE J. Biomed. Health Inform.* **2017**, *22*, 1434–1444. [CrossRef]

26. ur Rehman, N.; Aftab, H. Multivariate variational mode decomposition. *IEEE Trans. Signal Process.* **2019**, *67*, 6039–6052. [CrossRef]

27. Tripathy, R.; Sharma, L.; Dandapat, S. Detection of shockable ventricular arrhythmia using variational mode decomposition. *J. Med. Syst.* **2016**, *40*, 79. [CrossRef] [PubMed]

28. Tripathy, R.; Paternina, M.R.A.; Arrieta, J.G.; Pattanaik, P. Automated detection of atrial fibrillation ECG signals using two stage VMD and atrial fibrillation diagnosis index. *J. Mech. Med. Biol.* **2017**, *17*, 1740044. [CrossRef]

29. Panda, R.; Jain, S.; Tripathy, R.; Acharya, U.R. Detection of shockable ventricular cardiac arrhythmias from ECG signals using FFREWT filter-bank and deep convolutional neural network. *Comput. Biol. Med.* **2020**, *124*, 103939. [CrossRef] [PubMed]

30. Bousseljot, R.; Kreiseler, D.; Schnabel, A. Nutzung der EKG-Signaldatenbank CARDIODAT der PTB über das Internet. 1995. Available online: https://www.degruyter.com/document/doi/10.1515/bmte.1995.40.s1.317/html (accessed on 25 August 2021).

31. Goldberger, A.L.; Amaral, L.A.; Glass, L.; Hausdorff, J.M.; Ivanov, P.C.; Mark, R.G.; Mietus, J.E.; Moody, G.B.; Peng, C.K.; Stanley, H.E. PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals. *Circulation* **2000**, *101*, e215–e220. [CrossRef]

32. Hasan, M.A.; Abbott, D. A review of beat-to-beat vectorcardiographic (VCG) parameters for analyzing repolarization variability in ECG signals. *Biomed. Eng. (Biomedizinische Technik)* **2016**, *61*, 3–17. [CrossRef]

33. Correa, R.; Arini, P.; Correa, L.; Valentinuzzi, M.; Laciar, E. New VCG and ECG indexes for early identification of acute myocardial infarction in patients. In Proceedings of the VI Latin American Congress on Biomedical Engineering CLAIB 2014, Paraná, Argentina, 29–31 October 2014; pp. 369–372.

34. Yamauchi, K.; Segal, M.; Tatematsu, H.; Simonson, E. Analysis of discrepancies between VCG and ECG interpretation of anterior wall myocardial infarction. *J. Electrocardiol.* **1977**, *10*, 171–178. [CrossRef]

35. Zema, M.J. Electrocardiographic tall R waves in the right precordial leads: Comparison of recently proposed ECG and VCG criteria for distinguishing posterolateral myocardial infarction from prominent anterior forces in normal subjects. *J. Electrocardiol.* **1990**, *23*, 147–156. [CrossRef]

36. Jain, P.; Gajbhiye, P.; Tripathy, R.; Acharya, U.R. A two-stage Deep CNN Architecture for the Classification of Low-risk and High-risk Hypertension Classes using Multi-lead ECG Signals. *Inform. Med. Unlocked* **2020**, *21*, 100479. [CrossRef]
37. Maheshwari, D.; Ghosh, S.; Tripathy, R.; Sharma, M.; Acharya, U.R. Automated accurate emotion recognition system using rhythm-specific deep convolutional neural network technique with multi-channel EEG signals. *Comput. Biol. Med.* **2021**, *134*, 104428. [CrossRef] [PubMed]

38. Jani, V.P.; Ostovaneh, M.R.; Chamera, E.; Lima, J.A.; Ambale-Venkatesh, B. Automatic segmentation of left ventricular myocardium and scar from LGE-CMR images utilizing deep learning with weighted categorical cross entropy loss function weight initialization. *Circulation* **2019**, *140*, A15934–A15934.

39. Becker, G. Creating comparability among reliability coefficients: The case of Cronbach alpha and Cohen kappa. *Psychol. Rep.* **2000**, *87*, 1171E–1182E. [CrossRef] [PubMed]

40. Rehman, N.; Mandic, D.P. Multivariate empirical mode decomposition. *Proc. R. Soc. A Math. Phys. Eng. Sci.* **2010**, *466*, 1291–1302. [CrossRef]

41. Tripathy, R.K.; Ghosh, S.K.; Gajbhiye, P.; Acharya, U.R. Development of automated sleep stage classification system using multivariate projection-based fixed boundary empirical wavelet transform and entropy features extracted from multichannel EEG signals. *Entropy* **2020**, *22*, 1141. [CrossRef]

42. Tripathy, R.; Gajbhiye, P.; Acharya, U.R. Automated sleep apnea detection from cardio-pulmonary signal using bivariate fast and adaptive EMD coupled with cross time-frequency analysis. *Comput. Biol. Med.* **2020**, *120*, 103769. [CrossRef]
<figure><img src="image_107.png" /></figure>

Article

# Quantitative and Qualitative Image Analysis of In Vitro Co-Culture 3D Tumor Spheroid Model by Employing Image-Processing Techniques

Mukta Sharma <fnref n="1" />*, Venkanagouda S. Goudar <fnref n="2" />, Manohar Prasad Koduri <fnref n="3" />,<fnref n="4" />, Fan Gang Tseng <fnref n="2" />,<fnref n="5" />
and Mahua Bhattacharya <fnref n="1" />

**Citation:** Sharma, M.; Goudar, V.S.; Koduri, M.P.; Tseng, F.G.; Bhattacharya, M. Quantitative and Qualitative Image Analysis of *In-Vitro* Co-Culture 3D Tumor Spheroid Model by Employing Image-Processing Techniques. *Appl. Sci.* **2021**, *11*, 4636. https://doi.org/10.3390/app11104636

Academic Editors: Leonardo Rundo,
Carmelo Militello and Andrea
Tangherloni

Received: 24 February 2021

Accepted: 31 March 2021

Published: 19 May 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

<figure><img src="image_108.png" /></figure>

Copyright: © 2021 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).

**Abstract:** This work proposes a novel region-estimation (RE) algorithm using the quantification of colon-cancer (HCT-8) and fibroblasts (NIH3T3) cells to estimate the densest region of colon-cancer cells in *in vitro* 3D co-cultured spheroids. Cells were labelled with different cell tracker dyes to track the cells. The technique involves staining cells with cell trackers. The quantification of HCT-8 and NIH3T3 cells by the RE algorithm leads to distribution pattern analysis of cells from the core to the periphery, which ultimately estimates the densest region of HCT-8 cells in an *in vitro* 3D cell spheroid. Cell quantification by the RE algorithm was compared with the results of cell quantification by ImageJ software. Results demonstrated the distribution patterns of cells from the core to the peripheral region of the *in vitro* 3D cell spheroid. The overall experimentation showed that the proposed methodology outperformed state-of-the-art approaches in terms of segmentation, quantification, and reducing biasing error.

**Keywords:** distribution patterns; fibroblast cells; HCT-8 colon-cancer cells; nature-inspired techniques; quantification; segmentation

## 1. Introduction

In mimicking the structural and natural complexity of living tissue, current technology such as *in vitro* 3D spheroid cell culture models is evolving compared to the 2D cell culture model. *In vivo* cell growth and cell signaling are highly dependent on the extracellular matrix (ECM) and the interaction produced by different kinds of cells. *In vitro* 3D cell culture models aid in the study of molecular level tissue function by employing co-culture models and developing drugs for the cancer model in mitigating animal usage for drug testing. In oxygen and nutrients, the gradient can be closely mimicked by 3D cell culture techniques compared to 2D cell culture techniques [1]. However, 3D *in vitro* spheroid models seriously suffer from image acquisition and standalone image processing algorithms. As a result, user intervention during analysis might lead to heavier biases, thereby leading to erroneous results.

Such issues can be overcome by different automated computer-aided design (CAD) tasks such as segmentation, i.e., the extraction of the region of interest from the images. In the literature, several studies showed multiple usages of automatic segmentation approaches for variable cellular types [2,3]. Al-Kofahi et al. [4] stated that the segmentation

<footnote n="1">¹ Department of Information Technology, ABV-Indian Institute of Information Technology and Management, Gwalior, Madhya Pradesh 474015, India; mb@iiitm.ac.in</footnote>

<footnote n="2">² Department of Engineering and System Sciences, National Tsing Hua University, Hsinchu 30013, Taiwan; vsg@gapp.nthu.edu.tw (V.S.G.); fangang@ess.nthu.edu.tw (F.G.T.)</footnote>

<footnote n="3">³ Department of Mechanical, Materials and Aerospace, School of Engineering, University of Liverpool, Harrison Hughes Building, Liverpool L693GH, UK; mfgtjcjh@liverpool.ac.uk</footnote>

<footnote n="4">⁴ International Intercollegiate PhD Program, National Tsing Hua University, Hsinchu 30013, Taiwan</footnote>

<footnote n="5">5 Research Center for Applied Sciences, Academia Sinica, Taipei, Frontier Research Center on Fundamental and Applied Sciences of Matters, National Tsing Hua University, Hsinchu 30013, Taiwan</footnote>

<footnote>* Correspondence: mukta@iiitm.ac.in</footnote>
of cells through automation is an essential step in image cytometry and histometry. M. Sharma et al. [5] proposed a novel non-linear segmentation model to discriminate and quantify living or dead cells. Xing and Yang [6] focused on digital pathology and microscopy image analysis, and extensively reviewed the techniques. Considerable progress was achieved in the past, but algorithms still suffer from biasing and need to boost their accuracy and robustness, consume less time, and self act against the upcoming applications. Color-based segmentation using traditional clustering algorithms, on the other hand, is relatively easy, and complexity is lesser when compared to that of segmentation techniques [7]. It is likewise more relevant for biomedical image segmentation, as the count of clusters is known beforehand. However, traditional clustering techniques suffer from various issues such as being trapped in local optima, and having sensitivity to initial cluster centers and boundary-level constraints [8]. In recent advances, many nature-inspired algorithms arose to solve these clustering problems. The hybridization of nature-inspired algorithms with each other and with traditional clustering techniques was described by Krisna et al. [9], Rana et al. [10], and Chowdhury et al. [11] to resolve clustering issues. However, all of these clustering algorithms need much parameter initialization (Table S1), increasing their complexity and manual intervention. Moreover, incorrect parameter initialization affects the end outcome. To skip the overhead of parameter settings, a nature-inspired algorithm called the teacher learning-based optimization (TLBO) algorithm [12] was proposed.

## 2. Material Preparations

In this study, a poly-di-methyl-siloxane (PDMS) based microwell array chip was utilized to co-culture NIH3T3 and HCT-8 cells in *in vitro*. A PDMS based microwell array chip was used to construct the tumor spheroids. The well known soft lithography process was used to fabricate the microwells, and fabrication steps were followed as given by Patra et al. [13]. Images were obtained by scanning a horizontal cross-sectional view using fluorescence based confocal microscopy. To make the cells visible under confocal microscopy, NIH3T3 cells were labeled with CellTracker™ Green CMFDA (5-chloromethylfluorescein diacetate) dye (Thermo Fisher Scientific, China). HCT-8 cells were labeled with CellTracker™ Blue CMHC (4-chloromethyl-7-hydroxycoumarin) dye (Thermo Fisher Scientific, China). Dyes were functionalized as per the manufacturer's instructions. Briefly, both cell types were incubated with the respective cell tracker dye (50 µL) for 30 min at 37 °C. Cells were further washed 3 times with PBS, mixed, and seeded on the microwell array chip for spheroid formation.

HCT-8 and NIH3T3 cells (blue and green stained cells) were co-cultured in different ratios of 2:0.5; 2:1; 2:2; and 2:4, as shown in Figure 1; While preparing the *in vitro* 3D cell spheroid, the spheroid shaping capabilities expanded and became quicker by including more NIH3T3 cells with the HCT-8 cells. To analyze the cellular distribution in the spheroids, 3D images were captured by using confocal microscopy. Z-direction images were captured (using 10x objective) from the bottom of each spheroid with 6 µm step sizes of 50 slices, i.e., a total of 300 µm, which was equal to the spheroid size, as shown in Figure 2.

This study compares the different intra and inter-domain clustering techniques in the clustering of colored cells. The outcome of the best performing algorithms was used to quantify the cells and estimate the densest region of the colon-cancer cells over an *in vitro* 3D cell spheroid using a novel region estimation algorithm based on a distance transform (DT) technique.
<figure><img src="image_109.png" /><figcaption>**Figure 1.** Sample images of each ratio for horizontal cross-section of 3D cell spheroid: (a) 2:0.5; (b) 2:1; (c) 2:2; (d) 2:4.</figcaption></figure>

<figure><img src="image_110.png" /><figcaption>**Figure 2.** Partitioning of *in vitro* 3D cell spheroid into regions.</figcaption></figure>

**3. Problem Formulation**

The spheroid was divided into 50 slices ($z_{0--49}$) for each ratio (Figure 2). The width of the whole spheroid was 300 µm, which means that each slice had 6 µm thickness. Therefore, the whole *in vitro* 3D cell spheroid was partitioned into two regions: (1) the core region (from approx. slice nos. $z_{16--32}$) and (2) the peripheral region (from approx. slice nos. $z_{0--15}$ and $z_{33--49}$). The densest region was where the concentrations of the HCT-8 cells in the *in vitro* 3D cell spheroid were the maximum between the two regions. The estimation of the densest region of the colon-cancer cells in the *in vitro* 3D cell spheroid was
required for further analysis and treatment planning. With this, the technology of ultralow attachment surfaces of *in vitro* techniques is of the recent trend [14,15] recently became popular [16,17] in reducing the burden animal model. These microwell technologies have been used for the past few decades [18], and several cancer tumor models were constructed *in vitro*, which may provide a better environment to build *in vitro* 3D culturing of patient derived xenografts (PDX). Further improving image processing techniques may advance our understanding of stromal cell distribution in real tumor tissue *in vitro*, which helps in personalized medicine. Cells have different proliferation rates, but the environment depends on initial cell seeding density in the spheroid. We are currently experimenting on the aspect of the proliferation rate and monitoring them at different time points.

In order to understand the regional distribution of *in vitro* 3D cell spheroids, the total number $N_b$ of HCT-8 cells and $N_g$ NIH3T3 cells present in the *in vitro* 3D cell spheroid was considered. The cells were considered to be circular. The counts of HCT-8 and NIH3T3 cells were evaluated for horizontal cross sectional images of the *in vitro* 3D cell spheroid for each slice (slice nos. z₀—49). The identification of the densest region depends on the maximal concentrations of the HCT-8 cells in two different regions i.e., core ($B_c$) and periphery ($B_p$). Therefore, it was formulated as

$$R_{dense} = \begin{cases} CR, & if \ Count(B_c) > Count(G_p) \\ || Count(B_c) > Count(B_p) \\ PR, & if \ Count(B_p) > Count(G_c) \\ || Count(B_p) > Count(B_c) \end{cases} \quad (1)$$

where $G_c$ and $G_p$ are the count of NIH3T3 cells for the core and peripheral regions, respectively; $B_c$ and $B_p$ are the count of HCT-8 cells for the core and peripheral regions, respectively. The proposed region estimation algorithm finds the densest region $R_{dense}$ of the HCT-8 cells and gives the distribution patterns of the HCT-8 and NIH3T3 cells from the core to the peripheral region over the *in vitro* 3D cell spheroid.

## 4. Proposed Methodology

The overall proposed methodology comprises two main steps: (1) foreground cell clustering, and (2) region estimation and quantification, as shown in Figure 3. The original images of the *in vitro* 3D cell spheroid were first converted from the RGB color space to the $L^*a^*b^*$ color space. Unlike the RGB color model, the $L^*a^*b^*$ color model is approximately close to human vision. This color model provides uniformity in the range of perception [19]. Extracted $a^*b^*$ components from the RGB images were given as input to the TLBO clustering algorithm. The approach has two resultant phases, the teaching phase and the learner phase, as described in [12]. Initially, a K number of clusters were taken, each cluster datum or pixel datum was defined as a learner, and centroids that were selected randomly for each cluster are called teachers. After that, each learner’s Euclidean distance with the centroid for all k clusters and fitness value was evaluated. Using the learner-phase steps given in [12], each learner was modified. Likewise, the centroids or the best learners and the existing solution are modified. After reaching maximal iterations $I_{max}$, the foreground cells were separately clustered. The HCT-8 and NIH3T3 cluster cell images were then converted into a binary image using Ostu’s global thresholding technique [20].
<figure><img src="image_111.png" /><figcaption>Figure 3. Resultant view of proposed methodology.</figcaption></figure>

Binary images $B_b$ and $B_r$ were then given as input in Algorithm 1 to estimate the densest HCT-8 cell region by finding the cell count. The nearest feature voxel $NF_{v_i}$ of the $B_b$ and $B_r$ were calculated as follows:

$$NF_{v_i} = \begin{cases} 0 & v_i \in B_{v_i} \\ \min(\|v_{i,r} - v_j\|) & (v_{i,r}, v_j) \in F_{v_i} \end{cases} \quad (2)$$

where, $\|v_i, v_j\| = \|v_i - v_{i_0}, v_j - v_{j_0}\|, \forall (v_{i_0}, v_{i_0}) \in B_{v_i}$ is any distance metric.

The outcome of DT depends on the selection of distance metrics, which varies from application to application. However, among all other metrics, the Euclidean distance metric was the most generalized because its measurement corresponds to the way in which objects are measured in the real world, and is rotation-invariant. The metric uses the $L_2$ norm and is defined as $\|v_{i,r}, v_j\|_2 = \sqrt{v_{i,r}^2 + v_j^2}$. After obtaining the nearest feature voxel matrix, maxima propagation is applied using the inverted mask matrix of $NF_{v_i}$ and by limiting the propagation using some fraction ($c_1 = 0.5$) of the maximal distance of the nearest feature voxel matrix $NF_{v_i}$. To control the extent of propagation of $MP_{i,r}$, $MP_{i,r}$ is dilated with the 3 × 3 matrix of all ones. This operation transforms the finer distance image into coarser segments. After that, the different numbers of segments or levels as $Lvl_i$ are found using the “unique” function of MATLAB. Then, background voxels were removed from the $Lvl_i$ matrix. The area threshold value was calculated to impose the minimal area constraints for segments. After that, the 8-connected component (or blobs or cells) area and pixel indices were found using the “region props()” inbuilt function of MATLAB, which simply sums the pixels of a particular region (area) by using their assigned labels. Then, the area is thresholded by using the extracted areas and finding the count of blobs or cells encountered within that area region. This gave the count of HCT-8 and NIH3T3 cells in different regions of the in vitro 3D cell spheroid. On the basis of these counts, i.e., $B_c, B_p, G_c$ and $G_p$ the densest HCT-8 cell region ($R_{dense}$) using Equation (1) was estimated.
**Algorithm 1 Region-estimation algorithm.**

**Input:** Binary images ($B_b$ and $B_r$) of HCT-8 cluster and NIH3T3 cluster images, respectively.

**Output:** Count of cells ($B_c$, $B_p$, $G_c$ and $G_p$) and region of drug delivery ($R_{dense}$)

1. **Count** = 0, $[m, n] = size(B_b)$, where m belongs to number of rows, and n belongs to number of columns of $B_b$.

2. Calculate nearest feature voxel $NF_{v_i}$ of the $B_b$ and $B_r$ using Equation (2).

3. Calculate mask as:

4. **for** i=1 to n **do**

5.     mask = (1 - $NF_{v_i}$)

6. **end for**

7. $max_d = [\sum_{i=1}^{n} max(NF_{v_i})] \times c_1$, where $c_1$ is the constant.

8. Apply the maxima propagation on the distance-transformed image as:

9. **for** r=1 to $max_d$ **do**

10.     $MP_i = MP_i \oplus [1]_{3 \times 3}$

11.     $MP_i = MP_i \times mask$

12. **end for**

13. Extract unique values from the $MP_i$ matrix and save it as a $Lvl_i$ matrix in sorted order

14. Removes $B_{vi}$ from $Lvl_i$

15. Evaluate minimal area threshold as:

16. $TH_a = \pi \times (max_d)^2 \times c_2$, where $c_2$ is the constant.

17. **for** k=1 to length($Lvl_i$) **do**

18.     $L_i = MP_i == Lvl_i, \forall i = 1, 2, ..., n$

19. Calculate area and pixel indices for each 8-connected component (object) in the binary image $L_i$ as:

20. $P_r$ = regionprops($L_i', Area', Pixel_idx_list'$), where $P_r$ is the structure containing the specified properties values.

21. Extract areas of the objects or cells as:

22. $A_{cells_j} = [P_r AREA]$

23. Threshold area as:

24. $A'_{cells_j} = A_{cells} > TH_a$

25. Calculate count of cells that passes imposed area $A'_{cells}$ as:

26. $Count = Count + \sum_{j=1}^{n} A'_{cells_j}$, where n is the dimension of matrix $A'_{cells}$

27. Assign unique IDs to mark segments after area thresholding to create final segmented image as:

28. **for** p=1 to length($A'_{cells}$) **do**

29.     **if** 1 == $A'_{cells_p}$ **then**

30.         $idx = P_r(p).Pixel_idx_list$

31.         $seg(idx) = random(m * n)$

32. **end if**

33. **end for**

34. **end for**

35. $R_{dense} = \begin{cases} CR, & \text{if } Count(B_c) > Count(G_p) \\ || Count(B_c) > Count(B_p) \\ PR, & \text{if } Count(B_p) > Count(G_c) \\ || Count(B_p) > Count(B_c) \end{cases}$, where

CRandPR denote the core and peripheral region of the *in vitro* 3D cell spheroid, respectively (refer to Equation (1)).

**5. Experiment Analysis**

The proposed methodology was simulated using MATLAB R2017a on a system with Intel 7th generation 4770 @3.40 GHz, and validated using the dataset as mentioned in Section 1. The dataset was divided into 4 types of ratio images of NIH3T3 and HCT-8 cells. In the experiment analysis to evaluate the proposed methodology's performance, the capability of the foreground cell clustering algorithm (TLBO) was evaluated on the basis of qualitative and quantitative results as described in Section 5.1. Likewise, the region estimation algorithm's ability for cell quantification and in reducing the biasing error was evaluated by comparing it with ImageJ software as described in Section 5.2. All results are shown for 2:0.5 ratio images for reference, and the rest of the ratio images figures, graphs, and tables are provided as supplementary material.

**5.1. Comparative Analysis: Qualitative and Quantitative**

In this section, the reason is provided for selecting the TLBO nature-inspired clustering algorithm to extract the foreground cells of the *in vitro* 3D cell spheroid. However, comparing the ground truth images of colon-cancer cells was not possible due to the unavailability of segmentation masks to compare them with the TLBO clustering segmentation results. Therefore, the TLBO clustering algorithm results were compared with other nature-inspired clustering and traditional clustering approaches: Particle Swarm Optimization (PSO), Genetic Algorithm (GA), Invasive Weed Optimization (IWO), k-means (KM), k-medoids (KMed), and Fuzzy C-means (FCM). Results were compared on the basis of qualitative and
quantitative results. On 200 iterations, the value of the fitness metrics (clustering cost as shown in Equation (3)) converged, as shown in Figure 4. Thus, all clustering experiments were performed by running the algorithms on 200 iterations. The parameter values used for each algorithm are shown in Table S1 (provided in the supplementary material).

$$Cost_j = \frac{\sum_{k=1}^{n_{max}} (d_k)}{n_{max}} \qquad (3)$$

$$d_k = \sqrt{(x_s - y_t)^2 + (x_s - y_t)^2}$$

$\forall s, t \in 1, 2, \dots, C_n$, where $n_{max}$ is the number of distances $d_k$ calculated within the clusters.

<figure><img src="image_112.png" /><figcaption>Figure 4. Cost of clustering graph of each algorithm to show convergence up to 200 iterations.</figcaption></figure>

A qualitative comparison among clustering algorithms is shown in Tables 1–3 for the cellular ratio of 2:0.5 using three different images. Different cellular ratios of 2:1, 2:2, and 2:4 are shown in Tables S2–S4, respectively. The tables’ information contains labeled images, the three clusters, and the graphical representation of clusters and their centroids. In the clustering of HCT-8 cells, the TLBO algorithm had greater potential than that of other algorithms. A similar phenomenon was observed for other cellular ratios, as shown in Tables S2–S4. In the quantitative comparison, all clustering algorithms were compared on the basis of quantization error (QE) [21] and best cost (BC) (Equation (3)). The quantization error of 2:0.5 ratio images is shown in Figure 5a,b, and the best cost is shown in Figure 5c,d. The different cellular ratios of quantization error and best cost are shown in Figures S5–S10.

Qualitative and quantitative analysis suggested that TLBO clustering performance was better than that of other nature-inspired clustering approaches (PSO, GA, IWO) and traditional clustering approaches (k-means, k-medoids, FCM). Therefore, the TLBO clustering algorithm was selected for foreground cell clustering in the proposed methodology.
<figure><img src="image_113.png" /><figcaption>**Figure 5.** Comparative analysis between teacher learning based optimization (TLBO) and other existing nature-inspired clustering techniques using 2:0.5 ratio slices based on (a,b) quantization error (QE) and (c,d) best cost (BC).</figcaption></figure>

<figure><img src="image_114.png" /><figcaption>**Table 1.** Comparative Analysis of visual results of TLBO with other clustering algorithms for 2:0.5 ratio image of Figure 1a.</figcaption></figure>
<figure><img src="image_115.png" /><figcaption>**Table 3.** Comparative analysis of visual TLBO results with other clustering algorithms for 2:0.5 ratio image of Figure 1a.</figcaption></figure>

<figure><img src="image_116.png" /></figure>
## 5.2. *Quantitative Results of Region-Estimation Algorithm*

In cell quantification, a thresholding process was employed. During the start of processing or in the optimization threshold, images were inspected or visualized manually, and the best set of images (in this case, 2:1) that were visualized were taken in optimizing the threshold value. This might be because of variability in laser power excitation during the acquisition of images, and intensity based image variations had minimal or no effect on the quantification of cellular analysis. Employing one ratio threshold to other cellular ratios did not affect the results, as cellular sizes were uniform in all cellular ratios. Fibroblast (NIH3T3) cell-to-cell interactions are tight and they form tight junctions; therefore, cell boundaries may not be distinguishable. NIH3T3 cells were clustered, and various thresholding values were applied to find the optimal threshold values, as shown in Figure 6. Figure 6 indicates the segmented NIH3T3 cells overlain with binary masks to evaluate the most relevant threshold value. Values were obtained by multiplying a scalar quantity with maximal intensity value in the image, and passing the binary thresholding data. It was important to choose an optimal threshold value because, by choosing a higher threshold value, intensities start merging into another, and for small threshold values, tiny cells seem to appear that are oversegmented. By visual inspection, and from Figure 6 and 7, the value of T = 0.125 corresponded to the best possible results for all image slices.

In this way, clustered NIH3T3 cells were converted into their binary masks using the optimal threshold. The binarized mask was then fed into the RE algorithm 1 to count the NIH3T3 cells. The results of the overlain images of HCT-8 and NIH3T3 binary masks on its original images are shown in Figure 8.

The distribution of NIH3T3 and HCT-8 cells was as shown in Figure 9a,b for horizontal cross-sectional views. Figure 9a shows the blue (HCT-8) cell count (BCC), and green (NIH3T3) cell count (GCC); and Figure 9b shows the blue (HCT-8) cell area (BCA) and green (NIH3T3) cell area (GCA) of the *in vitro* 3D cell spheroid for 2:0.5 ratio images. For other cellular ratios, the distribution pattern is shown in Figures S11–S13. Cell-count distribution gave better observation for HCT-8 cells, whereas the area plot gave better distribution pattern analysis for NIH3T3 cells. The limitation in NIH3T3 cell count can be attributed to the high overlap and the staining procedure of the green cell tracker. After evaluating the count for HCT-8 and NIH3T3 cells (as shown in Figure 9a,b), further evaluation for the region-estimation algorithm (Algorithm 1) was performed for each ratio image. The concentration of HCT-8 cells was the maximum in the core region compared to the whole *in vitro* 3D cell spheroid. The densest region of HCT-8 cells in the *in vitro* 3D cell spheroid estimated by the RE algorithm 1 ($R_{dense}$) was the core region (CR), and the width of the core region was estimated to approximately be 100 µm.
<figure><img src="image_117.png" /><figcaption>**Figure 6.** Optimal threshold evaluation for NIH3T3 cells using various threshold segmentation results. (a) 2:0.5 (2_05_z12), (b) 2:1 (2_1_z14), (c) 2:2 (2_2_z17), and (d) 2:4 (2_4_z10) ratio images.</figcaption></figure>
<figure><img src="image_118.png" /><figcaption>**Figure 7.** NIH3T3 cells count for 2:1 ratio of horizontal cross-sectional images for various threshold values.</figcaption></figure>

Standalone image analysis software (ImageJ) was used in comparing the cellular quantification of the results. The procedure to segment and quantify HCT-8 and NIH3T3 cells for all cellular ratio images by ImageJ software was as shown in Figure 10, and the corresponding quantification data was as shown in Figure 11. The process steps during quantification using ImageJ introduce manual biasing: segmentation, thresholding, pixel size, and circularity bias. Briefly, the quantification process was as follows. Slices were input to the ImageJ software, and images were split into three channels: red, green, and blue. After splitting the channels, thresholding was adjusted. A watershed algorithm was then used for segmentation. After segmentation, images were analyzed by providing the pixel size (30–infinity) and circularity (0.30–1.00). Most biases were eliminated during the TLBO clustering and quantification approach except for thresholding bias. The ImageJ software separately clustered the HCT-8 and NIH3T3 cells along with background noise. Though it was required to perform the segmentation algorithm in both approaches, the watershed algorithm was applied in the ImageJ software, whereas TLBO clustering was applied in the proposed methodology. However, the watershed segmentation technique had the drawback of having excessive oversegmentation [22]. In contrast, TLBO separately clustered the HCT-8 and NIH3T3 cells without any background pixels.

Comparative analysis of GCC and BCC using the proposed methodology with the counts of both types of cells obtained from ImageJ software [14] from the peripheral to the core region (slice no. z₀—49) for 2:0.5 ratio images is shown in Figure 12. Other cellular ratios are presented in Figures S14–S16. Because of circularity and pixel-size bias, there was much observable difference between the BCC and GCC of ImageJ software, and the BCC and GCC obtained after applying the proposed methodology. The counting of cells by the proposed methodology showed that most HCT-8 cells were concentrated towards the core region, whereas NIH3T3 cells were more concentrated towards the peripheral region. The difference between manual cell counting and the proposed methodology was approximately 35%, 40%, 60%, and 80% for 2:0.5, 2:0.1, 2:2, and 2:4, respectively. Moreover, the proposed methodology accurately measured the physiological approximation of the cell count for all cell-ratio processes [15]. The difference percentage of the counting results of each ratio was variate, from 35% to 80%. This may be due to two reasons:

1. Figure 11 shows that the segmentation results of the ImageJ software for the HCT-8 and NIH3T3 cell clusters also included background pixels (noise). This happened for
each ratio image, which were further processed and detected (or counted) as blobs (or cells) by the software.

2. Moreover, the ImageJ software needed some parameter adjustment (threshold value, circularity, and size) for segmenting and counting the cells. Biasing error affected the final results.

<figure><img src="image_119.png" /><figcaption>**Figure 8.** Overlap cluster and original images for HCT-8 and NIH3T3 cells: (a) 2:0.5 (2_05_z12), (b) 2:1 (2_1_z14), (c) 2:2 (2_2_z17), and (d) 2:4 (2_4_z10) ratio images.</figcaption></figure>
<figure><img src="image_120.png" /><figcaption>Figure 9. (a) Blue cell count (BCC) and green cell count (GCC) plots. (b) Blue cell area (BCA) and green cell area (GCA) from periphery to core of in vitro 3D cell spheroid of 2:0.5 ratio slices.</figcaption></figure>

<figure><img src="image_121.png" /><figcaption>Figure 10. Schematic diagram of procedure followed by ImageJ software for segmentation and quantification of HCT-8 and NIH3T3 cells.</figcaption></figure>

The equation used to evaluate the percentage difference between manual cell counting and proposed-methodology counting is as follows:

$$Diff(\%) = \frac{\sum_{i=1}^{N} M_i - \sum_{i=1}^{N} P_i}{T_c} \times 100 \quad (4)$$

where $\sum_{i=1}^{N} M_i$ is the sum of the manual cell count $\forall i = 1 to N$, and N is the number of slices. Similarly, $\sum_{i=1}^{N} P_i$ is the sum of the proposed methodology cell count $\forall i = 1 to N$. $T_c$ is the total number of cells present in the in vitro 3D cell spheroid; N = 50.

Preliminary data derived from the current nature-inspired clustering algorithm (TLBO) help to understand 3D in vitro systems by the spectrometric location of the extracellular matrix generating protein. Thus, it aids biological scientists in further targeted molecular studies such as polymerase chain reaction (PCR) and Western blot techniques, which are highly selected for treatment planning and diagnostic procedures [23]. Hence, the proposed methodology offers better distribution analysis of HCT-8 and NIH3T3 cells compared to the ImageJ software.
<figure><img src="image_122.png" /><figcaption>**Figure 11.** Comparison of TLBO algorithm and ImageJ software based on segmentation of HCT-8 and NIH3T3 cell results.</figcaption></figure>
<figure><img src="image_123.png" /><figcaption>**Figure 12.** BCC and GCC comparison between ImageJ software and proposed methodology from periphery to core of 3D cell spheroid for 2:0.5 ratio images.</figcaption></figure>

**6. Conclusions**

The overall proposed methodology analyzes colon-cancer cells' distribution patterns and fibroblast cells in *in vitro* 3D cell spheroids. On the basis of comparative analysis, the TLBO clustering algorithm was best suited for the cells' color-based segmentation. The region estimation algorithm estimates the densest region of HCT-8 cells in *in vitro* 3D cell spheroids on the basis of cell quantification. Compared to manual segmentation and quantification by ImageJ software, the proposed methodology reduced the biasing error for cell quantification. The current acquisition methodology was based on and majorly limited by fluorescent trackers and confocal characterization techniques. The possibility of findings can be improved by employing transfecting cells using fluorescent proteins rather than mere cell trackers, and advanced microscopy techniques such as light-sheet microscopy. So, this results in a fast acquisition process and preserves cellular 3D co-cultural spheroids' dynamic nature. This enhances the efficient utilization of the distance transform technique and the nature-inspired clustering algorithm. Therefore, one can extend the work by removing the biasing problem of the proposed methodology in quantifying NIH3T3 cells and thresholding the biasing effect.

**Supplementary Materials:** The following are available online at https://www.mdpi.com/2076-3417/11/10/4636/s1.

**Author Contributions:** Conceptualization, methodology, validation, writing—original-draft preparation: M.S.; data curation, conceptualization, formal analysis, visualization, writing—review and editing: V.S.G. and M.P.K.; funding acquisition, supervision: F.G.T.; and supervision and resources: M.B. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the Ministry of Science and Technology (MOST) 108-2638-E-007-001-MY2.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Acknowledgments:** The authors would like to thank the ABV-Indian Institute of Information Technology and Management, India for providing the opportunity for the coordination of research work with researchers of National Tsing Hua University, Taiwan, ROC. The authors also acknowledge with thanks ABV-IIITM and NTHU for providing all facilities and expenditures related to the experiment setup, laboratory equipment, computational platform, and other requirements for the entire study and documentation of the work.

**Conflicts of Interest:** The authors declare no conflict of interest.
## References

1. Koduri, M.P.S.; Goudar, V.; Shao, Y.W.; Hunt, J.A.; Henstock, J.R.; Curran, J.; Tseng, F.G. Fluorescence-based nano-oxygen particles for spatiometric monitoring of cell physiological conditions. *ACS Appl. Mater. Interfaces* **2018**, *10*, 30163-30171. [CrossRef]

2. Yang, H.; Ahuja, N. Automatic segmentation of granular objects in images: Combining local density clustering and gradient-barrier watershed. *Pattern Recognit.* **2014**, *47*, 2266-2279. [CrossRef]

3. Sharma, M.; Bhattacharya, M. Segmentation of CA3 Hippocampal Region of Rat Brain Cells Images Based on Bio-inspired Clustering Technique. In *Proceedings of the 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)*, San Diego, CA, USA, 18-21 November 2019; pp. 2438-2445.

4. Al-Kofahi, Y.; Lassoued, W.; Lee, W.; Roysam, B. Improved automatic detection and segmentation of cell nuclei in histopathology images. *IEEE Trans. Biomed. Eng.* **2009**, *57*, 841-852. [CrossRef]

5. Sharma, M.; Bhattacharya, M. Discrimination and quantification of live/dead rat brain cells using a non-linear segmentation model. *Med. Biol. Eng. Comput.* **2020**, *1-20*. [CrossRef] [PubMed]

6. Xing, F.; Yang, L. Robust nucleus/cell detection and segmentation in digital pathology and microscopy images: A comprehensive review. *IEEE Rev. Biomed. Eng.* **2016**, *9*, 234-263. [CrossRef] [PubMed]

7. Mignotte, M. Segmentation by fusion of histogram-based *k*-means clusters in different color spaces. *IEEE Trans. Image Process.* **2008**, *17*, 780-787. [CrossRef] [PubMed]

8. Hathaway, R.J.; Bezdek, J.C. Local convergence of the fuzzy *c*-means algorithms. *Pattern Recognit.* **1986**, *19*, 477-480. [CrossRef]

9. Krishna, K.; Murty, M.N. Genetic *K*-means algorithm. *IEEE Trans. Syst. Man Cybern. Part B (Cybern.)* **1999**, *29*, 433-439. [CrossRef] [PubMed]

10. Rana, S.; Jasola, S.; Kumar, R. A review on particle swarm optimization algorithms and their applications to data clustering. *Artif. Intell. Rev.* **2011**, *35*, 211-222. [CrossRef]

11. Chowdhury, A.; Bose, S.; Das, S. Automatic clustering based on invasive weed optimization algorithm. In *International Conference on Swarm, Evolutionary, and Memetic Computing*; Springer: Berlin/Heidelberg, Germany, 2011; pp. 105-112.

12. Rao, R.V.; Savsani, V.J.; Vakharia, D.P. Teaching-learning-based optimization: An optimization method for continuous non-linear large scale problems. *Intf. Sci.* **2012**, *183*, 1-5. [CrossRef]

13. Patra, B.; Chen, Y.H.; Peng, C.C.; Lin, S.C.; Lee, C.H.; Tung, Y.C. A microfluidic device for Funiform-sized cell spheroids formation, culture, harvesting and flow cytometry analysis. *Biomicrofluidics* **2013**, *7*. [CrossRef]

14. Abrämoff, M.D.; Magalhães, P.J.; Ram, S.J. Image processing with ImageJ. *Biophotonics Int.* **2004**, *11*, 36-42.

15. Alkasalias, T.; Moyano-Galceran, L.; Arsenian-Henriksson, M.; Lehti, K. Fibroblasts in the tumor microenvironment: Shield or spear? *Int. J. Mol. Sci.* **2018**, *19*, 1532. [CrossRef]

16. Fang, Y.; Eglen, R.M. Three-Dimensional Cell Cultures in Drug Discovery and Development. *SLAS Discov.* **2017**, *22*, 456-472. [CrossRef]

17. Lv, D.; Hu, Z.; Lu, L.; Lu, H.; Xu, X. Three-dimensional cell culture: A powerful tool in tumor research and drug discovery. *Oncol. Lett.* **2017**, *14*, 6999-7010. [CrossRef] [PubMed]

18. Kim, S.H.; Lee, G.H.; Park, J.Y. Microwell fabrication methods and applications for cellular studies. *Biomed. Eng. Lett.* **2013**, *3*, 131-137. [CrossRef]

19. Schwarz, M.W.; Cowan, W.B.; Beatty, J.C. An experimental comparison of RGB, YIQ, LAB, HSV, and opponent color models. *ACM Trans. Graph. (TOG)* **1987**, *6*, 123-158. [CrossRef]

20. Vala, H.J.; Baxi, A. A review on Otsu image segmentation algorithm. *Int. J. Adv. Res. Comput. Eng. Technol. (IJARCET)* **2013**, *2*, 387-389.

21. Naik, A.; Satapathy, S.C.; Parvathi, K. Improvement of initial cluster center of *c*-means using teaching learning based optimization. *Procedia Technol. 2012*, *6*, 428-435. [CrossRef]

22. Grau, V.; Mewes, A.U.; Alcaniz, M.; Kikinis, R.; Warfield, S.K. Improved watershed transform for medical image segmentation using prior information. *IEEE Trans. Med. Imaging* **2004**, *23*, 447-458. [CrossRef] [PubMed]

23. Zanoni, M.; Piccinini, F.; Arienti, C.; Zamagni, A.; Santi, S.; Polico, R.; Bevilacqua, A.; Tesei, A. 3D tumor spheroid models for *in vitro* therapeutic screening: A systematic approach to enhance the biological relevance of data obtained. *Sci. Rep.* **2016**, *6*, 1. [CrossRef] [PubMed]

## Short Biography of Authors

Mukta Sharma received a bachelor's degree in computer science from Uttar Pradesh Technical University, India. She completed her master's degree from GLA University, India. She is currently pursuing her doctorate from the Department of Information Technology at Atal Bihari Vajpayee Indian Institute of Information Technology and Management, Gwalior, India. Under her Ph.D. studies, Mukta Sharma is actively working on the segmentation and classification of histopathological images of cells in biomedical imaging. She has published papers in several conferences and refereed journals such as MBEC, Journal of Supercomputing, Springer. Her areas of interest are image processing, computer vision, machine learning for biomedical applications, cell morphological analysis, deep learning, and medical imaging. She has a currently active IEEE membership. Contact her at mukta.24sharma@gmail.com.
**Venkanagouda S. Goudar** is presently a Ph.D. student under the guidance of Prof. Fan-Gang Tseng in engineering and system sciences, National Tsing Hua University, Hsinchu, Taiwan. He completed his M.Sc. degree in Microbiology from Karnataka University, Dharwad, India in 2009. He further worked as a project assistant at the Indian Institute of Science, Bangalore, India for 4 years. He later worked as a research trainee for 7 months at the Institute of Photonic Sciences (ICFO), Barcelona, Spain. During this period, he was involved in different research areas such as nanotechnology, biosensors, lab-on-a-chip, and antibacterial and antiviral surface modifications on transparent surfaces. Currently, he is working on rare cell 3D culturing and drug delivery. Overall, during his research, he was able to produce eight peer-reviewed journal publications (Sensors and Actuators, AMI, IEEE sensors, etc.) and seven conference abstracts, and to file three patents (Indian, Taiwan, and the U.S.) in the field of biosensors and drug delivery. Contact him at venkatesh.gdr23@gmail.com.

**Manohar Prasad Koduri** is a dual Ph.D. student of the National Tsing Hua University, Taiwan and the University of Liverpool, UK. Manohar completed his masters in integrated circuit technologies at the University of Hyderabad, India, and finished his bachelor's degree in electronics and communication engineering at Gayatri Vidya Parishad College of Engineering, Visakhapatnam, India. Manohar is currently working on developing nanosensors for 3D tissue engineering applications. His research interests include biosensors, drug delivery, electromagnetic theory, image processing, and tissue engineering. Contact him at manimanohar92@gmail.com.

**Prof. Tseng Fan-Gang** is presently the vice president for R&D, NTHU, leading a multidisciplinary team at national Tsing Hua University, Hsinchu, Taiwan. For the past 20 years, his research interest included BioMEMS, nano/microfluidics, biosensors, Microfuel cells, and hydrogen storage. His recent research in BioMEMS involves cancer marker diagnosis and drug delivery, single-cell diagnosis, CTC diagnosis, the study of single-cell protein and cell dynamics, and microbubble generation and the study of its applications in tissue engineering and cancer research. He was elected as a fellow of ASME in 2014. He has received 69 patents, written 9 book chapters, published more than 220 SCI/EI journal papers with an H index 41, and has more than 6774 citations and 360 conference technical papers in the related fields. Prof. Tseng co-organized or cochaired many conferences, including Micro TAS, ISMM, IEEE MEMS, IEEE NEMS, IEEE Transducers, IEEE Nano, and IEEE Nanomed. Contact him at fangangtseng@gmail.com.

**Prof. Mahua Bhattacharya** is currently working as a Full Professor of the ABV Indian Institute of Information Technology and Management, Gwalior. The research area of Prof. Bhattacharya is related to biomedical-image processing, the classification of tumor or cancer growth in the human brain using multimodality medical imaging, cell-image analysis under various types of environmental exposure, and the development of AI techniques for digital and smart farming. She is executing various Government of India-funded projects in collaboration with reputed institutes. She is the President of the International Neural Network Society, India Chapter. Prof. Bhattacharya has published more than 150 papers in refereed journals and international flagship conferences as book chapters. She is the reviewer of IEEE EMBC, Elsevier, Springer, and Taylor and Francis journals. She is an organizing committee member of international IEEE conferences in India and abroad. She has delivered expert lectures in different national and international academic forums. She is an editorial board member of Neural Computing and Applications, Springer. She was Indian liaison for IJCNN'19 in Budapest, Hungary. Contact her at mahuabhatta@gmail.com.
<figure><img src="image_124.png" /></figure>

Review

# Deep Learning for Orthopedic Disease Based on Medical Image Analysis: Present and Future

JiHwan Lee and Seok Won Chung *

Department of Orthopaedic Surgery, School of Medicine, Konkuk University,
120-1 Neungdong-ro (Hwayang-dong), Seoul 143-729, Korea; metaphoricusjh@gmail.com
* Correspondence: smilecsw@gmail.com

**Abstract:** Since its development, deep learning has been quickly incorporated into the field of medicine and has had a profound impact. Since 2017, many studies applying deep learning-based diagnostics in the field of orthopedics have demonstrated outstanding performance. However, most published papers have focused on disease detection or classification, leaving some unsatisfactory reports in areas such as segmentation and prediction. This review introduces research published in the field of orthopedics classified according to disease from the perspective of orthopedic surgeons, and areas of future research are discussed. This paper provides orthopedic surgeons with an overall understanding of artificial intelligence-based image analysis and the information that medical data should be treated with low prejudice, providing developers and researchers with insight into the real-world context in which clinicians are embracing medical artificial intelligence.

**Keywords:** artificial intelligence; orthopedics; neural network; deep learning

## 1. Introduction

A convolutional neural network (CNN) is a deep learning algorithm architecture created based on a 1962 study investigating the visual process of feline brains, and it has been applied in a wide range of areas, from autonomous vehicles to medical diagnoses [1].

A traditional CNN consists of an input layer that transmits input information, a hidden layer that modifies information (filtering) received from the input layer and amplifies the features (pooling) and an output layer that finally synthesizes and outputs the information.

According to the universal approximation theorem, it has been confirmed that various linear classifications are possible even if the neural network has a shallow hidden layer, and some pioneering studies have shown that classification and detection are improved as the layers constituting the neural network become deeper (deep neural network) [2]. Since 2012, the performance of deep learning has rapidly increased in medical image analysis with the use of deep neural networks, and this has led to a decrease in the classification error rate from approximately 25% in 2011 to 3.6% in 2015.

The CNN model was developed using a pipeline in terms of classification and detection [3], and the improved CNN shows excellent judgment, essentially giving the computer a new visual organ. A CNN has thus been expected to be used for medical diagnoses. However, a CNN does not provide any information on the basis of the decision. Therefore, even if a CNN shows an excellent diagnostic ability, it can only be discussed within a limited scope in medicine, where the basis for a judgment is important [4].

This has been pointed out as a technical limitation that reduces the effectiveness of a CNN in various fields other than medicine [5]. Researchers have dubbed this limitation "black box issues" and worked to develop "explainable artificial intelligence (XAI)" to look inside the problem [6]. The term "explainable" can be expressed as "understandability", "comprehensibility" or "interpretability" and has the same meaning. XAI should not degrade the classification or prediction performance of the model in any way and should improve the explainability. Various strategies and suitable CNN architectures have been

**Citation:** Lee, J.; Chung, S.W. Deep Learning for Orthopedic Disease Based on Medical Image Analysis: Present and Future. *Appl. Sci.* **2022**, *12*, 681. https://doi.org/10.3390/app12020681

Academic Editor: Fabio La Foresta

Received: 12 November 2021

Accepted: 4 January 2022

Published: 11 January 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

<figure><img src="image_125.png" /></figure>

Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
proposed to implement an appropriate XAI [7]. Unfortunately, the black box nature of deep
learning has not been completely resolved, but there are some notable achievements [8].
As one of these achievements, in 2016, Zhou et al. introduced a method explaining how a
CNN makes a decision through class activation mapping [9], and this method is widely
used in the field of medical artificial intelligence (Figure 1) [10].

<figure><img src="image_126.png" /><figcaption>**Figure 1.** Image highlighting the location and size of a rotator cuff tear through a class activation map (CAM). Figure obtained from a study performed by Chung et al. [10].</figcaption></figure>

In a similar context, there are attempts to improve the explainability by improving
the existing CNN architecture [11]. Kim et al. modified U-Net, a CNN architecture that
has strength in image segmentation, to appropriately increase the explainability. They
presented an interpretable version of U-Net (SAU-Net) using an attention module for the
decoder part [12].

Hence, studies introducing CNN models for diagnosing and classifying diseases using
deep learning have been published in various fields of medicine, including ophthalmology
and dermatology [13,14].

This trend is spreading rapidly in the field of orthopedics. Since 2017, when orthopedic
disease research using deep learning was first introduced, the number of related papers has
increased rapidly, and more than 300 papers in this area have been published. The search
was conducted using Pubmed, MEDLINE and Embase, and papers were screened from
1 January 2017 to 2 November 2021. The search query was (orthopedic OR orthopedic)
AND (deep learning). Among these studies, two orthopedic surgeons (S.W.C. and J.H.L.)
independently reviewed the full text of the retrieved papers. Among these studies, 48 stud-
ies which both authors judged to be interesting and practical within the clinical context of
orthopedic surgery are introduced and classified according to disease. This paper aims to
provide insight into how medical artificial intelligence can help orthopedic surgeons treat
patients vividly and in what context clinicians are accepting medical artificial intelligence
from developers and researchers.

The authors introduce the selected papers by classifying them into the following
sections: (1) Deep Learning for Fractures, (2) Deep Learning for Osteoarthritis and the
Prediction of Arthroplasty Implants, (3) Deep Learning for Joint-Specific Soft Tissue Disease,
(4) Miscellaneous and (5) Discussion.
## 2. Deep Learning for Fractures

Fractures are the most familiar ailments to orthopedists and the medical area in which deep learning methods were first applied. In 2018, Chung et al. published a CNN model for diagnosing and classifying proximal humerus fractures. Three specialists labeled 1891 anteroposterior shoulder radiographs as normal shoulders (n = 515) and 4 proximal humerus fracture types (greater tuberosity: 346; surgical neck: 514; 3-part: 269; and 4-part: 247) [15]. After labeling, a CNN model (ResNet-152) was trained with a training dataset created through augmentation of the labeled data. The CNN model recorded 96% accuracy for the normal shoulders and proximal humerus fractures, showing a higher accuracy than a general orthopedist (92.8% accuracy). This model showed a top-1 accuracy of 65–86% and an area under the curve (AUC) of 0.90–0.98 for classifying the fracture types. A recently published paper introduced a model with improved classification accuracy. In 2020, Demir et al. introduced a deep learning model to diagnose and classify humerus fractures using the exemplar pyramid method, a novel, stable feature extraction approach which showed a high classification accuracy of 99.12% [16].

Uurakawa et al. trained the VGG-16 CNN model using hip plain radiographs (1773 in-tertrochanteric hip fracture images and 1573 normal hip images) and showed an accuracy of 95.5% [17]. Yamada et al. trained the CNN model (Xception architectural) based on 3123 hip plain and lateral radiography images, and the trained model classified fractures with 98% accuracy, which is better than orthopedists (92.2% accuracy) [18].

For the hip, as with the shoulder, there has been an attempt to classify fractures by training the CNN model. Lee et al. introduced a CNN model for training 786 anteroposterior pelvic plain radiographs using GoogLeNet-inception v3 [19]. The model classified a proximal femur fracture into type A (trochanteric region), type B (femur neck) and type C (femoral head) according to AO/OTA classification with an overall accuracy of 86.8%, showing a reasonable result. Lind et al. trained a ResNet-based CNN with anteroposterior and lateral knee radiographs, amounting to 6768 images [20]. The trained CNN model classified knee radiographic images according to the AO/OTA classification system and classified proximal tibia fractures, patellar fractures and distal femur fractures with AUCs of 0.87, 0.89 and 0.89, respectively.

The trained CNN diagnosed and classified fractures at a relatively high level in the large appendices of the shoulder, knee and hip. By contrast, a CNN model trained to diagnose and classify fractures in small joints or axial joints showed a relatively low AUC and accuracy. Farda et al. trained a PCANet-based CNN model that classified calcaneal fractures according to Sanders classification using computer tomography with 5534 datasets [21]. The trained CNN model showed 72% accuracy. In addition, Ozkaya et al. trained a CNN model based on ResNet50 with 390 anteroposterior wrist radiographic images [22]. The AUC of the learned CNN was 0.84, showing a relatively satisfactory result, but it was lower than that of experienced orthopedists.

Langerhuizen et al. compared the scaphoid fracture diagnostic accuracy between a deep learning algorithm and an orthopedist [23]. They trained the VGG16 CNN model with 150 radiographic images of scaphoid fractures and 150 images of normal wrist radiography without a fracture. Of the 150 images with scaphoid fractures, 23 could not be judged by the radiographic images and could only be confirmed through magnetic resonance imaging (MRI). The accuracy of the trained CNN model was 72%, which was lower than that of an orthopedic surgeon (84%). However, five of six occult scaphoid fractures were missed by all human observers.

An attempt was also made to diagnose the compression fractures in the spine using a trained CNN. The results showed a significant difference depending on the type of data used for learning. Chen et al. trained a ResNet-based CNN model using plain spine X-rays, and the trained CNN showed an accuracy of 73.59% [24]. By contrast, Yabu et al. presented a CNN model using MRI images as the training data. This model showed a higher accuracy (88%) than that of the surgeons [25].
In summary, fracture diagnosis using artificial intelligence showed a high level of accuracy. The trained CNN model conducted fracture diagnosis (binary classification) with a higher accuracy than fracture classification (multiclass classification), and this gap is expected to decrease as more advanced CNN models are developed.

In classifying fractures, small and axial joints showed a lower accuracy than large joints (Table 1). This may be a limitation of a CNN-based approach, which makes judgments by recognizing the contrast information (e.g., normal margin of the cortical bone and the fracture line or normal joint line) and spatial information of the images. The authors believe that this limitation can be overcome using more powerful CNN models.

<table><caption>**Table 1.** Summary of diagnostic performance for detecting/classifying orthopedic fracture.</caption><thead><tr><th>Fracture Site</th><th>Image Used</th><th>Author. Year</th><th>CNN Used</th><th>Work</th><th>Dataset Size</th><th>Accuracy</th><th>AUC</th><th>Winner</th></tr></thead><tbody><tr><td>Hip (femur neck)</td><td>X-ray</td><td>Matthew et al. 2019</td><td>GooLeNet</td><td>Binary classification</td><td>805</td><td>94%</td><td>0.98</td><td></td></tr><tr><td>Hip</td><td>X-ray</td><td>Cheng et al. 2019</td><td>DenseNet</td><td>Binary classification</td><td>3605</td><td>91%</td><td>0.98</td><td>Orthopedist &gt; CNN</td></tr><tr><td>Hip</td><td>X-ray</td><td>Takaaki et al. 2019</td><td>VGG-16</td><td>Binary classification</td><td>3346</td><td></td><td></td><td>CNN &gt; Orthopedist</td></tr><tr><td>Hip</td><td>X-ray</td><td>Yamada et al. 2020</td><td>Xception, ImageNet</td><td>Binary classification</td><td>3123</td><td>98%</td><td></td><td>CNN &gt; Orthopedist</td></tr><tr><td>Hip</td><td>X-ray</td><td>Lee et al. 2020</td><td>GoogLeNet-inception v3</td><td>Classification</td><td>686</td><td>86.8%</td><td></td><td></td></tr><tr><td>Hip</td><td>X-ray</td><td>Tanzi et al. 2020</td><td>InceptionV3, VGG-16, ResNet50</td><td>Classification</td><td>2453</td><td>86% (3 class)<br>81% (5 class)</td><td></td><td></td></tr><tr><td>Hip (Atypical fracture)</td><td>X-ray</td><td>Zdolsek et al. 2021</td><td>VGG19, InceptionV3, ResNet</td><td>Binary classification</td><td>982</td><td>91% (ResNet50)<br>83% (VGG19)<br>89% (InceptionV3)</td><td></td><td></td></tr><tr><td>Shoulder (proximal humerus)</td><td>X-ray</td><td>Chung et al. 2018</td><td>ResNet</td><td>Binary classification<br>Classification</td><td>1891</td><td>95%</td><td>0.99</td><td>Orthopedist &gt; CNN (specialized in the shoulder)</td></tr><tr><td>Knee</td><td>X-ray</td><td>Lind et al. 2021</td><td>ResNet-based CNN</td><td>Classification</td><td>6768</td><td></td><td>0.87 (Proximal tibia)<br>0.89 (Patella)<br>0.89 (Distal femur)</td><td></td></tr><tr><td>Ankle</td><td>X-ray</td><td>Gene et al. 2019</td><td>Xception</td><td>Binary classification</td><td>596</td><td>75%</td><td></td><td></td></tr><tr><td>Ankle (Malleolar)</td><td>X-ray</td><td>Olczak et al. 2021</td><td>ResNet</td><td>Classification</td><td>5495</td><td></td><td>0.90</td><td></td></tr><tr><td>Ankle (Calcaneal)</td><td>CT</td><td>Farda et al. 2021</td><td>PCANet</td><td>Classification, Segmentation</td><td>5534</td><td>72%</td><td></td><td></td></tr><tr><td>Wrist</td><td>X-ray</td><td>Kim et al. 2017</td><td>Inception</td><td>Binary classification</td><td>1389</td><td></td><td>0.95</td><td></td></tr><tr><td>Wrist</td><td>X-ray</td><td>Thian et al. 2019</td><td>ResNet</td><td>Binary classification</td><td>7356</td><td>88.9%</td><td>0.90</td><td></td></tr></tbody></table>
<table><caption>Table 1. *Cont.*</caption><thead><tr><th>Fracture Site</th><th>Image Used</th><th>Author. Year</th><th>CNN Used</th><th>Work</th><th>Dataset Size</th><th>Accuracy</th><th>AUC</th><th>Winner</th></tr></thead><tbody><tr><td>Wrist (Scaphoid)</td><td>X-ray</td><td>Langerhuizen et al. 2020</td><td>VGG-16</td><td>Binary classification</td><td>300</td><td>72%</td><td>0.77</td><td>Orthopedist &gt; CNN</td></tr><tr><td>Wrist (Scaphoid)</td><td>X-ray</td><td>Ozkaya et al. 2020</td><td>ResNet50</td><td>Binary classification</td><td>390</td><td></td><td>084</td><td>Orthopedist &gt; CNN</td></tr><tr><td>Vertebra</td><td>X-ray</td><td>Chen et al. 2021</td><td>ImageNet, ResNeXt</td><td>Binary classification</td><td>1306</td><td>73.6%</td><td>0.72</td><td>Orthopedist &gt; CNN</td></tr><tr><td>Vertebra</td><td>MRI</td><td>Yabu et al. 2021</td><td>VGG-16,19, Inception V3, ResNet50</td><td>Binary classification</td><td>1624</td><td></td><td>0.95</td><td>CNN &gt; Orthopedist</td></tr></tbody></table>

Most of the diagnosis and classification of fractures using deep learning have focused on osteoporotic fractures, and studies on osteoporotic fracture joints with low frequencies are relatively poor [26]. This may be because the dataset for training the CNN model is sufficient because osteoporotic fractures account for a high proportion of the total fracture frequency, and the fracture pattern is relatively standardized, making it suitable for use in fracture classification.

### 3. Deep Learning for Osteoarthritis and Prediction of Arthroplasty Implants

Osteoarthritis is as familiar to orthopedists as fractures. Therefore, several attempts have been made to diagnose and classify osteoarthritis using deep learning algorithms. Xue et al. trained a CNN model based on VGG-16 with 420 plain hip X-rays [27]. This is one of the earliest studies to apply deep learning methods to the orthopedic field, and the trained model diagnosed hip osteoarthritis with an accuracy of 92.8%. Ureten et al. also presented a model for diagnosing hip osteoarthritis using a similar research design, showing an accuracy of 90.2% [28].

Tiulpin et al. trained a CNN model to classify knee osteoarthritis according to the Kellgren-Lawrence grading scale using a Siamese classification CNN [29]. The model trained using plain knee X-rays showed a multiclass accuracy of 66.7%. In addition, Swiecicki et al. trained a Faster R-CNN using plain and lateral knee X-rays from the Multicenter Osteoarthritis Study dataset [30]. The multiclass accuracy of this model was 71.9%, which showed improved performance compared with the previous study conducted by Tiulpin et al.

Pedoia et al. trained a DenseNet-based CNN based on MRI-T2 images rather than X-ray data, as used in previous studies, and this model showed a high AUC of 0.83 [31]. Kim et al. trained an SE-ResNet-based CNN model using 4366 knee anteroposterior X-rays as a dataset. Furthermore, they trained the model by adding demographic information (age, sex and body mass index), alignment and metabolic data information that can affect knee osteoarthritis, in addition to image information [32]. The diagnostic performance of the image data with additional patient information showed a significantly higher AUC (Table 2).

Advanced osteoarthritis of the hip or knee often requires arthroplasty. Several studies have introduced a model for classifying arthroplasty implants used by patients with deep learning algorithms. Karnuta et al. trained the InceptionV3 network-based CNN model using anteroposterior knee X-rays with nine different implant models inserted [33]. The trained model showed an accuracy of 99% and an AUC of 0.99, classifying the implant models at an almost perfect level. A similar attempt was made at the hip joint. In addition, Borjali et al. created a CNN model trained on 252 plain hip X-rays containing 3 different implant designs, and this model classified implants with 100% accuracy (Figure 2) [34]. Kang et al. also developed a CNN model trained on 170 plain hip X-rays containing 29 different implant designs. This model also showed a high level of performance, with an AUC of 0.99 [35].
<table><caption>**Table 2.** Summary of diagnostic performance for classifying osteoarthritis.</caption><thead><tr><th>Location</th><th>Image Used</th><th>Author. Year</th><th>CNN Used</th><th>Work</th><th>Dataset Size</th><th>Accuracy</th><th>AUC</th></tr></thead><tbody><tr><td>Knee</td><td>X-ray</td><td>Tiulpin et al. 2018</td><td>Siamese CNN</td><td>Classification</td><td>5960</td><td>66.7%</td><td></td></tr><tr><td>Knee</td><td>X-ray</td><td>Pedoia et al. 2019</td><td>DenseNet</td><td>Classification</td><td>5042</td><td>75%</td><td>0.83</td></tr><tr><td>Knee</td><td>X-ray</td><td>Kim et al. 2020</td><td>SE-ResNet</td><td>Classification</td><td>4366</td><td>61.6%<br>(with<br>additional<br>information)</td><td>0.75<br>(with<br>additional<br>information)</td></tr><tr><td>Knee</td><td>X-ray</td><td>Swiecicki et al. 2021</td><td>Faster R-CNN</td><td>Classification</td><td>2802</td><td>71.9%</td><td></td></tr><tr><td>Hip</td><td>X-ray</td><td>Xue et al. 2017</td><td>VGG-16</td><td>Binary<br>classification</td><td>420</td><td>92.8%</td><td></td></tr><tr><td>Hip</td><td>X-ray</td><td>Ureten et al. 2020</td><td>VGG-16</td><td>Binary<br>classification</td><td>434</td><td>90.2%</td><td></td></tr></tbody></table>

<figure><img src="image_127.png" /><figcaption>**Figure 2.** The figure shows how a trained convolutional neural network classifies total hip replacement implants of different designs in A, B and C. Figure obtained from a study performed by Borjali et al. [34].</figcaption></figure>

By contrast, the model classifying shoulder arthroplasty implants showed a relatively low AUC. Urban et al. developed a CNN model trained on 597 plain shoulder X-rays with 16 different implant designs, showing an accuracy of 80% [36]. In addition, Sultan et al. proposed a model for classifying the different designs of four manufacturers using modified ResNet and DenseNet, showing an accuracy of 85.9% [37].

In summary, as in the case of using deep learning for fractures, binary classification of osteoarthritis has a higher accuracy than multiclass classification. In particular, the CNN-based model for specifying arthroplasty implants of the hip or knee shows a high accuracy. This may be because, unlike human bone, the implant design is highly standardized, demonstrating a clear margin on X-rays and providing clear contrast information to the CNN model. However, the classification of shoulder arthroplasty implants shows a low
level of accuracy. This may be due to the fact that a shoulder anteroposterior X-ray can show a wider range of positions than an anteroposterior radiograph of the knee or hip.

## 4. Deep Learning for Joint-Specific Soft Tissue Disease

As for deep learning approaches, an algorithm specialized for detection based on learned images and an algorithm for segmentation by analyzing features have structural differences and have developed into different areas of application [3]. In particular, segmentation has technical difficulties in that it is necessary to preserve spastic information that is easily lost in the outer-layer process of synthesizing the results of the CNN model being trained [38]. Recent studies have attempted to overcome these limitations through techniques such as FCN-based semantic segmentation.

These differences in deep learning algorithms also affect the use of deep learning in the orthopedic field. The deep learning-based studies introduced above are cases of diagnosing and classifying diseases based on X-ray images, and a CNN model specialized for segmentation is not always required [39]. By contrast, for diseases that are diagnosed and classified based on images such as ultrasound or MRI, a satisfactory level of accuracy can be obtained using only a CNN model specialized for segmentation. For example, a CNN model for diagnosing rotator cuff tears is more appropriate for inferring such tears based on the outline of the normal rotator cuff (segmentation) than a method of diagnosis applied by specifying the location where the tear occurred (regional detection).

Therefore, CNN models for diagnosing soft tissue disease in the orthopedic field have mainly been published after 2018, which was when the segmentation technology began to mature. Kim et al. trained a CNN model using a shoulder MRI dataset of 240 patients. The trained model identified the muscle region of the rotator cuff with an accuracy of 99.9% and graded fatty infiltration at a high level [40]. Taghizadeh et al. also conducted a similar study using a shoulder computed tomography of 103 patients as a dataset. The trained CNN model measured fatty infiltration with an accuracy of 91% [41].

Medina et al. introduced a model for segmenting the rotator cuff muscle with 98% accuracy by applying a CNN model trained using the shoulder MRIs of 258 patients [42]. Furthermore, Shim and Chung et al. introduced a model for evaluating the presence of tears and their sizes in the rotator cuff by training a Voxception-ResNet (VRN)-based CNN with 2124 shoulder MRIs. The trained CNN model diagnosed and classified rotator cuff tears with accuracies of 92.5% and 76.5%, respectively [10]. In addition, Lee et al. developed a new deep learning architecture using an integrated positive loss function and a pretrained encoder. Using this, the location of the rotator cuff tear can be relatively accurately determined, even when imbalanced and noisy ultrasound images are provided [43].

Recent studies suggesting a CNN model for diagnosing meniscal tears, cartilage lesions and anterior cruciate ligament (ACL) ruptures in the knee joint have also been published. Couteaux et al. presented a model that trains a Mask-RCNN with 1828 T2-weighted 2D Fast Spin-Echo images to classify the torn part from the normal area of the meniscus and do so according to the location of the tear [44]. This model diagnosed and classified meniscal tears with an AUC of 0.91. Roblot et al. also proposed a model for diagnosing meniscal tears in a similar way, detecting meniscal tears with an AUC of 0.94 [45].

Chang et al. presented a model for diagnosing complete ACL tears by training a U-Net-based CNN using 320 coronal proton density-weighted 2D Fast Spin-Echo images, demonstrating an AUC of 0.97 [46]. In addition, Flannery et al. trained a modified U-Net-based CNN and evaluated the level of segmentation of the model. The segmentation level suggested by the trained model did not show a statistically significant difference from the ground truth (the value actually suggested by an expert) (Figure 3) [47].
<figure><img src="image_128.png" /><figcaption>**Figure 3.** Each row is the same MR slice, and each column is an unsegmented slice (MR Image), an expert measured value (Ground Truth), a trained CNN model predicted value (Prediction) and an overlay of manual and predicted segmentations ( contours Overlay). Figure obtained from a study performed by Flannery et al. [47].</figcaption></figure>

**5. Miscellaneous**

Concerning bone age, attempts to create a model that automatically predicts a bone's age through the learning of plain X-rays of carpal bones have been conducted since before the first deep learning algorithm was developed. Mahmoodi et al. presented a bone age prediction model with an accuracy of 82% in 2000, using a regression model and a Bayesian estimator [48]. A CNN model using a deep learning algorithm was developed, and it is now possible to predict the bone age with improved accuracy. In addition, Han et al. proposed a model with 97.6% accuracy by training the Inception ResNet v2 model with 5876 hand radiographs [39].

For pediatrics, developmental dysplasia of the hip is one of the most common hip joint disorders in infants and young children, and its diagnosis is difficult owing to the extensive variations in pediatric pelvic anatomy [49]. To create a deep learning algorithm that can diagnose developmental dysplasia of the hip, Zhang et al. trained a CNN model (based on ResNet-101) using 10,219 pelvic anteroposterior radiographs of children. The trained model showed a high AUC of 0.975 [50].

An acute pediatric elbow fracture is also difficult to diagnose, owing to the existence of multiple cartilaginous ossification centers and a highly variable appearance [51]. England et al. trained a CNN using 901 lateral elbow radiographs, and the trained model diagnoses elbow fractures with a high AUC of 0.985 [52].

Central dual-energy X-ray absorptiometry is the reference standard for diagnosing osteoporosis and osteopenia. A CNN model for diagnosing osteopenia and osteoporosis using plain radiography without dual-energy X-ray absorptiometry was recently introduced.

Zhang et al. trained a CNN model with 2564 lumbar X-ray images, and this model showed an AUC of 0.767 and 0.810 for osteoporosis and osteopenia, respectively [53]. Yamamoto et al. trained a CNN with 1131 hip X-rays, and this model diagnosed osteoporosis with an accuracy of 0.885 [54].

For alignment, Pei et al. published an interesting study using a deep learning algorithm to automatically measure the hip-knee-ankle angle. They trained a CNN model with
796 unilateral lower limb X-rays, showing a difference of 0.49° from the ground truth measured directly by orthopedic surgeons [55]. In addition, Rouzrokh and Pouria et al. trained a CNN model with 600 hip anteroposterior and 600 hip lateral X-rays taken after total hip arthroplasty and programmed this model to automatically derive the acetabular component inclination and version. Compared with the ground truth, this model showed a difference of 1.35° for the inclination and 1.39° for the anteversion [56].

Galbusera et al. presented a CNN model trained using biplanar radiographs of the spine. The model automatically calculated the T4-T12 kyphosis, L1-L5 lordosis, Cobb angle of scoliosis, pelvic incidence, sacral slope and pelvic tilt. Among them, the pelvic tilt showed a difference of 2.7° compared with the ground truth, whereas the L1-L5 lordosis showed a difference of 11.5° from the ground truth [56].

Concerning metastasis and infections in the spine, the spine is a joint that receives a high blood supply and is relatively easily exposed to metastasis compared with other joints [57]. Therefore, studies for diagnosing metastatic lesions using deep learning algorithms have mainly focused on the spine. Wang et al. reported that a CNN model trained with sagittal fat-suppressed T2 2D Fast Spin-Echo spine images localized metastatic lesions with a sensitivity of 90% [58]. In addition, Chmelik et al. trained a CNN with sagittal computed tomography images containing 1046 lytic lesions and 1135 sclerotic lesions, and the trained model detected lytic and sclerotic lesions with AUCs of 0.80 and 0.78, respectively [59].

Kim et al. published a CNN model to discriminate between tuberculous and pyogenic spondylitis. They trained the CNN using axial T2-weighted 2D Fast Spin-Echo images, and the trained CNN model divided the two conditions with an AUC of 0.80, with no significant difference from a human reader [60].

As for other applications, in addition to the previously introduced papers, studies using deep learning algorithms in the field of orthopedic surgery have been published. Won et al. introduced a model for grading spinal stenosis by training a Faster R-CNN [61]. Rouzrokh and Pouria et al. attempted to predict postoperative hip dislocation by training a CNN model with 92,584 hip X-rays taken after total hip arthroplasty. The trained model showed an AUC of 76.7% and an accuracy of 49.5% [62].

## 6. Discussion

Orthopedics, along with dermatology, ophthalmology and cardiology, is the medical field in which research into deep learning algorithms is most actively conducted. Related research has been explosively increasing since 2017, and this trend is expected to continue until the “new winter”, when the development of artificial intelligence will reach its limit.

To date, image analysis studies of orthopedic diseases using deep learning have shown excellent results overall. Several studies have reported that in fractures and osteoarthritis, a trained CNN model has a diagnostic accuracy comparable to that of an expert. The studies also presented satisfactory results for the classification of fractures and osteoarthritis. However, the accuracy of multiclass classification did not reach detection, and studies on small joints presented relatively poor results compared with studies on large joints.

Nevertheless, it is expected that this limitation can be overcome for two reasons. First, the CNN model for medical image analysis aims for accurate diagnosis and appropriate classification, and the types of classes required for this purpose are relatively small. When there are few class types, Basha et al. proved that the accuracy can be improved using a CNN model structured as a deeper layer [63]. Therefore, it is expected that the development of a CNN model with deep hyperparameters will increase the accuracy of multiclass classification through medical image analysis. Second, medical images are extremely refined data compared with images used to learn road traffic conditions or climate predictions; that is, researchers can relatively easily obtain appropriate image data without noise, such as different heights of traffic lights or flying birds. This means that even with simple data augmentation such as an affine transformation, an appropriate dataset for training the CNN model can be provided.
Therefore, the authors expect that the development of a CNN model and the accumulation of additional medical images will increase the classification accuracy of fractures and osteoarthritis, which are relatively weak compared with the accuracy of diagnosis. In the same context, it is also expected that the diagnosis and classification of joint-specific soft tissue will be improved, owing to the development of deep learning algorithms advantageous for segmentation. Indeed, there are several recent studies that have completed segmentation at a high level [64,65]. In particular, Hashimoto et al. and others segmented the psoas major muscle through a U-net-based CNN model, and the trained U-net-based CNN model showed an average of 86.6% intersection over union (IoU). U-net is one of the most important semantic segmentation frameworks of CNNs [66] and has the strength of having an architecture that can recognize structural edges. Therefore, U-net is expected to be widely used for segmentation of medical images [67]. Although not in the field of orthopedics, new CNN architectures based on U-Net are continuously being introduced and reporting notable results [68]. Rundo et al. performed prostate zonal segmentation with USE-Net, incorporating Squeeze-and-Excitation blocks (SE) into U-Net [69]. Yeung et al. showed that the model trained with a dual attention-gated CNN (Focus U-Net), which improved the U-Net, segmented the polyp of the colonoscopy image to a satisfactory level [70].

Studies published in the field of orthopedic surgery have thus far been unable to present a CNN model with a higher level of diagnosis and classification than experts. An in-depth discussion is needed as to whether these results are a problem that can be overcome through data accumulation or the development of a better CNN, or whether they are a natural limitation of a CNN model learned from image data.

The authors offer two approaches. First, experts do not solve problems with image data alone. Experts can utilize information other than images, such as the patient's demographic data, the degree of pain, the nature of the disease and a physical examination, which can affect the disease diagnosis and classification. Indeed, Kim et al. reported that a CNN model trained by adding demographic information (age, sex and body mass index), alignment and metabolic data that could affect knee osteoarthritis showed a statistically significantly higher AUC [32]. Therefore, even if an improved CNN model is developed and high-quality image data are accumulated, there is a possibility that the image analysis-based CNN model using a deep learning algorithm will not reach the level of experts.

Second, despite the opinions presented above, the possibility that CNN models will outperform experts in certain fields cannot be excluded, because the CNN model analyzes images from a different point of view than human beings. Among 150 images of scaphoid fractures, Langerhuizen et al. included 23 scaphoid fracture image data that could only be confirmed through an MRI. The trained CNN model showed a lower level of accuracy than orthopedic surgeons, but it detected five of six occult scaphoid fractures that were missed by all human observers [23]. It is therefore necessary to carefully discuss whether an image analysis model using deep learning can outperform experts.

It is clear that the present CNN models have room for improvement. However, this does not undermine the significance of the studies conducted to date. The currently developed CNN model can reduce the task intensity of the expert reader and can be used for the education of non-expert medical workers, such as medical students or specialists during training [71]. In addition, through a developed CNN model, a pediatrician can roughly estimate a patient's bone age using only X-rays without the help of an orthopedic surgeon.

A step away from the fate of clinical doctors and CNN's accuracy battle, there are interesting and more practical studies that give practical help to patients and doctors. Nie et al. converted native medical CT images to higher resolution images through generative adversarial networks (GANs) [72], and this study has the potential to be extended to MRI images [73]. Therefore, it can help a society that has no choice but to use low-quality MRI due to insufficient medical infrastructure or patients who have difficulty using high-quality MRI due to cost problems.
The authors reviewed deep learning approaches for orthopedic diseases applied through image analysis and found some limitations. First, there are no models approved by the Food and Drug Administration, other than a CNN model for predicting the bone age in children and a model for diagnosing wrist fractures [74]. In other medical departments, several models have been approved by the Food and Drug Administration, starting with a deep learning-based model for the automatic diagnosis of diabetic retinopathy in April 2018.

Second, no prospective studies have been conducted [75]. To improve the quality of research and continue applicable studies, a prospective and randomized trial according to the CONSORT-AI guidelines presented in 2020 will be necessary [76].

Third, recently described deep learning methods have mostly been designed to conduct a single task. To be useful in clinical practice, multiple deep learning algorithms will need to evaluate every possible abnormality. Some efforts have been made to overcome these limitations. For example, Grauhan et al. presented a CNN model for diagnosing fractures, joint dislocation and osteoarthritis through plain shoulder radiographs [77].

Finally, there is a need to reduce expert bias on a given dataset. Orthopedic surgeons have traditionally used ultrasound, computed tomography or MRIs to diagnose soft tissue diseases. However, deep learning algorithms often make appropriate judgments beyond human cognition. Kang et al. presented a model for diagnosing SSC tendon tears with a CNN model trained using axillary lateral radiographs, and the learned model showed an appropriate level of accuracy [78]. Thus, orthopedic surgeons may have the freedom to develop CNN models based on their imagination, free from prejudice.

In conclusion, image analysis using deep learning presents a clear milestone in the field of orthopedics and is experiencing explosive growth. The development of a CNN architecture and the accumulation of refined image data are expected to lead to the development of more sophisticated models. However, it is difficult to predict whether a deep learning model that exceeds the capability of experts can be created. Orthopedic surgeons who want to apply a deep learning algorithm to image analysis need to treat data with low prejudice, present research that meets the newly suggested guidelines and focus on developing models that can multitask.

**Author Contributions:** Conception and design of study, interpretation of data and approval of the version of the manuscript to be published, S.W.C.; interpretation of data, drafting the manuscript and approval of the version of the manuscript to be published, J.L. All authors have read and agreed to the published version of the manuscript.

**Funding:** This results was supported by “Regional Innovation Strategy (RIS)” through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (MOE). (2021RIS-001(1345341783)).

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Acknowledgments:** The authors would like to thank the authors for allowing the use of figures in this paper.

**Conflicts of Interest:** The authors declare no conflict of interest.

## References

1. Hubel, D.H.; Wiesel, T.N. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. *J. Physiol.* **1962**, *160*, 106-154. [CrossRef] [PubMed]

2. Hornik, K.; Stinchcombe, M.; White, H. Multilayer feedforward networks are universal approximators. *Neural Netw.* **1989**, *2*, 359-366. [CrossRef]

3. Zhao, Z.-Q.; Zheng, P.; Xu, S.-T.; Wu, X. Object detection with deep learning: A review. *IEEE Trans. Neural Netw. Learn. Syst.* **2019**, *30*, 3212-3232. [CrossRef] [PubMed]

4. Wang, F.; Casalino, L.P.; Khullar, D. Deep learning in medicine—Promise, progress, and challenges. *JAMA Intern. Med.* **2019**, *179*, 293-294. [CrossRef] [PubMed]

5. Budd, S.; Robinson, E.C.; Kainz, B. A survey on active learning and human-in-the-loop deep learning for medical image analysis. *Med. Image Anal.* **2021**, *71*, 102062. [CrossRef]
6. Fujita, H. AI-based computer-aided diagnosis (AI-CAD): The latest review to read first. *Radiol. Phys. Technol.* **2020**, *13*, 6–19.
[CrossRef] [PubMed]

7. Castiglioni, I.; Rundo, L.; Codari, M.; Leo, G.D.; Salvatore, C.; Interlenghi, M.; Gallivanone, F.; Cozzi, A.; D'Amico, N.C.; Sardanelli, F. AI applications to medical images: From machine learning to deep learning. *Phys. Med.* **2021**, *83*, 9–24. [CrossRef]

8. Tjoa, E.; Guan, C. A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI. *IEEE Trans. Neural Netw. Learn. Syst.* **2020**, *32*, 4793–4813. [CrossRef] [PubMed]

9. Zhou, B.; Khosla, A.; Lapedriza, A.; Oliva, A.; Torralba, A. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 30 June 2016; pp. 2921–2929.

10. Shim, E.; Kim, J.Y.; Yoon, J.P.; Ki, S.-Y.; Lho, T.; Kim, Y.; Chung, S.W. Automated rotator cuff tear classification using 3D convolutional neural network. *Sci. Rep.* **2020**, *10*, 1–9. [CrossRef]

11. Singh, A.; Sengupta, S.; Lakshminarayanan, V. Explainable Deep Learning Models in Medical Image Analysis. *J. Imaging* **2020**, *6*, 52. [CrossRef]

12. Kim, B.; Wattenberg, M.; Gilmer, J.; Cai, C.; Wexler, J.; Viegas, F. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In Proceedings of the 35th International Conference on Machine Learning, PMLR, Stockholm, Sweden, 10–15 July 2018; pp. 2668–2677.

13. Gulshan, V.; Peng, L.; Coram, M.; Stumpe, M.C.; Wu, D.; Narayanaswamy, A.; Venugopalan, S.; Widner, K.; Madams, T.; Cuadros, J.; et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. *JAMA* **2016**, *316*, 2402–2410. [CrossRef]

14. Bejnordi, B.E.; Veta, M.; Van Diest, P.J.; Van Ginneken, B.; Karssemeijer, N.; Litjens, G.; van der Laak, J.A.W.M.; The CAMELYON16 Consortium. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. *JAMA* **2017**, *318*, 2199–2210. [CrossRef]

15. Chung, S.W.; Han, S.S.; Lee, J.W.; Oh, K.-S.; Kim, N.R.; Yoon, J.P.; Kim, J.Y.; Moon, S.H.; Kwon, J.; Lee, H.-J.; et al. Automated detection and classification of the proximal humerus fracture by using deep learning algorithm. *Acta Orthop.* **2018**, *89*, 468–473. [CrossRef]

16. Demir, S.; Key, S.; Tuncer, T.; Dogan, S. An exemplar pyramid feature extraction based humerus fracture classification method. *Med. Hypotheses* **2020**, *140*, 109663. [CrossRef]

17. Urakawa, T.; Tanaka, Y.; Goto, S.; Matsuzawa, H.; Watanabe, K.; Endo, N. Detecting intertrochanteric hip fractures with orthopedist-level accuracy using a deep convolutional neural network. *Skelet. Radiol.* **2019**, *48*, 239–244. [CrossRef]

18. Yamada, Y.; Maki, S.; Kishida, S.; Nagai, H.; Arima, J.; Yamakawa, N.; Iijima, Y.; Shiko, Y.; Kawasaki, Y.; Kotani, T.; et al. Automated classification of hip fractures using deep convolutional neural networks with orthopedic surgeon-level accuracy: Ensemble decision-making with antero-posterior and lateral radiographs. *Acta Orthop.* **2020**, *91*, 699–704. [CrossRef]

19. Lee, C.; Jang, J.; Lee, S.; Kim, Y.S.; Jo, H.J.; Kim, Y. Classification of femur fracture in pelvic X-ray images using meta-learned deep neural network. *Sci. Rep.* **2020**, *10*, 1–12. [CrossRef] [PubMed]

20. Lind, A.; Akbarian, E.; Olsson, S.; Näsell, H.; Sköldenberg, O.; Razavian, A.S.; Gordon, M. Artificial intelligence for the classification of fractures around the knee in adults according to the 2018 AO/OTA classification system. *PLOS ONE* **2021**, *16*, e0248809. [CrossRef] [PubMed]

21. Farda, N.A.; Lai, J.-Y.; Wang, J.-C.; Lee, P.-Y.; Liu, J.-W.; Hsieh, I.-H. Sanders classification of calcaneal fractures in CT images with deep learning and differential data augmentation techniques. *Injury* **2020**, *52*, 616–624. [CrossRef] [PubMed]

22. Ozkaya, E.; Topal, F.E.; Bulut, T.; Gursoy, M.; Ozuysal, M.; Karakaya, Z. Evaluation of an artificial intelligence system for diagnosing scaphoid fracture on direct radiography. *Eur. J. Trauma Emerg. Surg.* **2020**, *1–8*. [CrossRef]

23. Langerhuizen, D.W.G.; Bulstra, A.E.J.; Janssen, S.J.; Ring, D.; Kerkhoffs, G.M.M.J.; Jaarsma, R.L.; Doornberg, J.N. Is deep learning on par with human observers for detection of radiographically visible and occult fractures of the scaphoid? *Clin. Orthop. Relat. Res.* **2020**, *478*, 2653–2659. [CrossRef] [PubMed]

24. Chen, H.-Y.; Hsu, B.W.-Y.; Yin, Y.-K.; Lin, F.-H.; Yang, T.-H.; Yang, R.-S.; Lee, C.-K.; Tseng, V.S. Application of deep learning algorithm to detect and visualize vertebral fractures on plain frontal radiographs. *PLOS ONE* **2021**, *16*, e0245992. [CrossRef] [PubMed]

25. Yabu, A.; Hoshino, M.; Tabuchi, H.; Takahashi, S.; Masumoto, H.; Akada, M.; Morita, S.; Maeno, T.; Iwamae, M.; Inose, H.; et al. Using artificial intelligence to diagnose fresh osteoporotic vertebral fractures on magnetic resonance images. *Spine J.* **2021**, *21*, 1652–1658. [CrossRef] [PubMed]

26. Moon, Y.L.; Jung, S.H.; Choi, G.Y. Ecaluation of focal bone mineral density using three-dimensional of Hounsfield units in the proximal humerus. *CiSE.* **2015**, *18*, 86–90.

27. Xue, Y.; Zhang, R.; Deng, Y.; Chen, K.; Jiang, T. A preliminary examination of the diagnostic value of deep learning in hip osteoarthritis. *PLOS ONE* **2017**, *12*, e0178992. [CrossRef]

28. Üreten, K.; Arslan, T.; Gültekin, K.E.; Demir, A.N.D.; Özer, H.F.; Bilgili, Y. Detection of hip osteoarthritis by using plain pelvic radiographs with deep learning methods. *Skelet. Radiol.* **2020**, *49*, 1369–1374. [CrossRef]

29. Tiulpin, A.; Thevenot, J.; Rahtu, E.; Lehenkari, P.; Saarakkala, S. Automatic Knee Osteoarthritis Diagnosis from Plain Radiographs: A Deep Learning-Based Approach. *Sci. Rep.* **2018**, *8*, 1–10. [CrossRef]
30. Swiecicki, A.; Li, N.; O'Donnell, J.; Said, N.; Yang, J.; Mather, R.C.; Jiranek, D.A.; Mazurowski, M.A. Deep learning-based algorithm for assessment of knee osteoarthritis severity in radiographs matches performance of radiologists. *Comput. Biol. Med.* **2021**, *133*, 104334. [CrossRef] [PubMed]

31. Pedoia, V.; Lee, J.; Norman, B.; Link, T.M.; Majumdar, S. Diagnosing osteoarthritis from T2 maps using deep learning: An analysis of the entire Osteoarthritis Initiative baseline cohort. *Osteoarthr. Cartil.* **2019**, *27*, 1002–1010. [CrossRef] [PubMed]

32. Kim, D.H.; Lee, K.J.; Choi, D.; Lee, J.I.; Choi, H.G.; Lee, Y.S. Can Additional Patient Information Improve the Diagnostic Performance of Deep Learning for the Interpretation of Knee Osteoarthritis Severity. *J. Clin. Med.* **2020**, *9*, 3341. [CrossRef]

33. Karnuta, J.M.; Luu, B.C.; Roth, A.L.; Haeberle, H.S.; Chen, A.F.; Iorio, R.; Schaffer, J.L.; Mont, M.A.; Patterson, B.M.; Krebs, V.E.; et al. Artificial Intelligence to Identify Arthroplasty Implants From Radiographs of the Knee. *J. Arthroplast.* **2021**, *36*, 935–940. [CrossRef] [PubMed]

34. Borjali, A.; Chen, A.F.; Muratoglu, O.K.; Morid, M.A.; Varadarajan, K.M. Detecting total hip replacement prosthesis design on plain radiographs using deep convolutional neural network. *J. Orthop. Res.* **2020**, *38*, 1465–1471. [CrossRef] [PubMed]

35. Kang, Y.-J.; Yoo, J.-L.; Cha, Y.-H.; Park, C.H.; Kim, J.-T. Machine learning-based identification of hip arthroplasty designs. *J. Orthop. Transl.* **2020**, *21*, 13–17. [CrossRef] [PubMed]

36. Urban, G.; Porhemmat, S.; Stark, M.; Feeley, B.; Okada, K.; Baldi, P. Classifying shoulder implants in X-ray images using deep learning. *Comput. Struct. Biotechnol. J.* **2020**, *18*, 967–972. [CrossRef] [PubMed]

37. Sultan, H.; Owais, M.; Park, C.; Mahmood, T.; Haider, A.; Park, K.R. Artificial Intelligence-Based Recognition of Different Types of Shoulder Implants in X-ray Scans Based on Dense Residual Ensemble-Network for Personalized Medicine. *J. Pers. Med.* **2021**, *11*, 482. [CrossRef]

38. Guo, Y.; Liu, Y.; Georgiou, T.; Lew, M.S. A review of semantic segmentation using deep neural networks. *Int. J. Multimed. Inf. Retr.* **2018**, *7*, 87–93. [CrossRef]

39. Han, Y.; Wang, G. Skeletal bone age prediction based on a deep residual network with spatial transformer. *Comput. Methods Programs Biomed.* **2020**, *197*, 105754. [CrossRef]

40. Kim, J.Y.; Ro, K.; You, S.; Nam, B.R.; Yook, S.; Park, H.S.; Yoo, J.C.; Park, E.; Cho, K.; Cho, B.H.; et al. Development of an automatic muscle atrophy measuring algorithm to calculate the ratio of supraspinatus in supraspinous fossa using deep learning. *Comput. Methods Programs Biomed.* **2019**, *182*, 105063. [CrossRef] [PubMed]

41. Taghizadeh, E.; Truffer, O.; Becce, F.; Eminian, S.; Gidoin, S.; Terrier, A.; Farron, A.; Büchler, P. Deep learning for the rapid automatic quantification and characterization of rotator cuff muscle degeneration from shoulder CT datasets. *Eur. Radiol.* **2021**, *31*, 181–190. [CrossRef]

42. Medina, G.; Buckless, C.G.; Thomasson, E.; Oh, L.S.; Torriani, M. Deep learning method for segmentation of rotator cuff muscles on MR images. *Skelet. Radiol. J.* **2021**, *50*, 683–692. [CrossRef]

43. Lee, K.; Kim, J.Y.; Lee, M.H.; Choi, C.-H.; Hwang, J.Y. Imbalanced Loss-Integrated Deep-Learning-Based Ultrasound Image Analysis for Diagnosis of Rotator-Cuff Tear. *Sensors* **2021**, *21*, 2214. [CrossRef]

44. Couteaux, V.; Si-Mohamed, S.; Nempont, O.; Lefevre, T.; Popoff, A.; Pizaine, G.; Villain, N.; Bloch, I.; Cotten, A.; Boussel, L. Automatic knee meniscus tear detection and orientation classification with Mask-RCNN. *Diagn. Interv. Imaging* **2019**, *100*, 235–242. [CrossRef]

45. Roblot, V.; Giret, Y.; Antoun, M.B.; Morillot, C.; Chassin, X.; Cotten, A.; Zerbib, J.; Fournier, L. Artificial intelligence to diagnose meniscus tears on MRI. *Diagn. Interv. Imaging* **2019**, *100*, 243–249. [CrossRef] [PubMed]

46. Chang, P.D.; Wong, T.T.; Rasiej, M.J. Deep Learning for Detection of Complete Anterior Cruciate Ligament Tear. *J. Digit. Imaging* **2019**, *32*, 980–986. [CrossRef] [PubMed]

47. Flannery, S.W.; Kiapour, A.M.; Edgar, D.J.; Murray, M.M.; Fleming, B.C. Automated magnetic resonance image segmentation of the anterior cruciate ligament. *J. Orthop. Res.* **2021**, *39*, 831–840. [CrossRef]

48. Mahmoodi, S.; Sharif, B.S.; Chester, E.G.; Owen, J.P.; Lee, R. Skeletal growth estimation using radiographic image processing and analysis. *IEEE Trans. Inf. Technol. Biomed.* **2000**, *4*, 292–297. [CrossRef]

49. Kyung, B.S.; Lee, S.H.; Jeong, W.K.; Park, S.Y. Disparity between clinical and ultrasound examinations in neonatal hip screening. *CiOS* **2016**, *8*, 203–209. [CrossRef]

50. Zhang, S.-C.; Sun, J.; Liu, C.-B.; Fang, J.-H.; Xie, H.-T.; Ning, B. Clinical application of artificial intelligence-assisted diagnosis using anteroposterior pelvic radiographs in children with developmental dysplasia of the hip. *Bone Jt. J.* **2020**, *102*, 1574–1581. [CrossRef]

51. Rhyou, I.H.; Lee, J.H.; Park, K.J.; Kang, H.S.; Kim, K.W. The ulnar collateral ligament is always torn in the posterolateral elbow dislocation: A suggestion on the new mechanism of dislocation using MRI findings. *CiSE* **2011**, *14*, 193–198. [CrossRef]

52. England, J.R.; Gross, J.S.; White, E.A.; Patel, D.B.; England, J.T.; Cheng, P.M. Detection of Traumatic Pediatric Elbow Joint Effusion Using a Deep Convolutional Neural Network. *Am. J. Roentgenol.* **2018**, *211*, 1361–1368. [CrossRef]

53. Zhang, B.; Yu, K.; Ning, Z.; Wang, K.; Dong, Y.; Liu, X.; Liu, S.; Wang, J.; Zhu, C.; Yu, Q.; et al. Deep learning of lumbar spine X-ray for osteopenia and osteoporosis screening: A multicenter retrospective cohort study. *Bone* **2020**, *140*, 115561. [CrossRef] [PubMed]

54. Yamamoto, N.; Sukegawa, S.; Kitamura, A.; Goto, R.; Noda, T.; Nakano, K.; Takabatake, K.; Kawai, H.; Nagatsuka, H.; Kawasaki, K.; et al. Deep Learning for Osteoporosis Classification Using Hip Radiographs and Patient Clinical Covariates. *Biomolecules* **2020**, *10*, 1534. [CrossRef] [PubMed]
55. Pei, Y.; Yang, W.; Wei, S.; Cai, R.; Li, J.; Guo, S.; Li, Q.; Wang, J.; Li, X. Automated measurement of hip-knee-ankle angle on the unilateral lower limb X-rays using deep learning. *Phys. Eng. Sci. Med.* **2021**, *44*, 53–62. [CrossRef]

56. Rouzrokh, P.; Wyles, C.C.; Philbrick, K.A.; Ramazanian, T.; Weston, A.D.; Cai, J.C.; Taunton, M.J.; Lewallen, D.G.; Berry, D.J.; Erickson, B.J.; et al. A Deep Learning Tool for Automated Radiographic Measurement of Acetabular Component Inclination and Version After Total Hip Arthroplasty. *J. Arthroplast.* **2021**, *36*, 2510–2517.e6. [CrossRef] [PubMed]

57. Lee, B.J.; Kim, S.T.; Yoon, M.G.; Kim, S.S.; Moon, M.S. Chronic osteomyelitis of the lumbar transverse process. *CiOS* **2011**, *3*, 254–257. [CrossRef] [PubMed]

58. Wang, J.; Fang, Z.; Lang, N.; Yuan, H.; Su, M.-Y.; Baldi, P. A multi-resolution approach for spinal metastasis detection using deep Siamese neural networks. *Comput. Biol. Med.* **2017**, *84*, 137–146. [CrossRef]

59. Chmelik, J.; Jakubicek, R.; Walek, P.; Jan, J.; Ourednicek, P.; Lambert, L.; Amadori, E.; Gavelli, G. Deep convolutional neural network-based segmentation and classification of difficult to define metastatic spinal lesions in 3D CT data. *Med. Image Anal.* **2018**, *49*, 76–88. [CrossRef]

60. Kim, K.; Kim, S.; Lee, Y.H.; Lee, S.H.; Lee, H.S.; Kim, S. Performance of the deep convolutional neural network based magnetic resonance image scoring algorithm for differentiating between tuberculous and pyogenic spondylitis. *Sci. Rep.* **2018**, *8*, 13124. [CrossRef]

61. Won, D.; Lee, H.-J.; Lee, S.-J.; Park, S.H. Spinal Stenosis Grading in Magnetic Resonance Imaging Using Deep Convolutional Neural Networks. *Spine* **2020**, *45*, 804–812. [CrossRef] [PubMed]

62. Rouzrokh, P.; Ramazanian, T.; Wyles, C.C.; Philbrick, K.A.; Cai, J.C.; Taunton, M.J.; Kremers, H.M.; Lewallen, D.G.; Erickson, B.J. Deep Learning Artificial Intelligence Model for Assessment of Hip Dislocation Risk Following Primary Total Hip Arthroplasty From Postoperative Radiographs. *J. Arthroplast.* **2021**, *36*, 2197–2203.e3. [CrossRef]

63. Huang, W.; Zhou, F. DA-CapsNet: Dual attention mechanism capsule network. *Sci. Rep.* **2020**, *10*, 11383. [CrossRef] [PubMed]

64. Hashimoto, F.; Kakimoto, A.; Ota, N.; Ito, S.; Nishizawa, S. Automated segmentation of 2D low-dose CT images of the psoas-major muscle using deep convolutional neural networks. *Radiol. Phys. Technol.* **2019**, *12*, 210–215. [CrossRef] [PubMed]

65. Kamiya, N.; Li, J.; Kume, M.; Fujita, H.; Shen, D.; Zheng, G. Fully automatic segmentation of paraspinal muscles from 3D torso CT images via multi-scale iterative random forest classifications. *Int. J. Comput. Assist. Radiol. Surg.* **2018**, *13*, 1697–1706. [CrossRef] [PubMed]

66. Du, G.; Cao, X.; Liang, J.; Chen, X.; Zhan, Y. Medical Image Segmentation based on U-Net: A Review. *J. Imaging Sci. Technol.* **2020**, *64*, 20508. [CrossRef]

67. Hiasa, Y.; Otake, Y.; Takao, M.; Ogawa, T.; Sugano, N.; Sato, Y. Automated Muscle Segmentation from Clinical CT Using Bayesian U-Net for Personalized Musculoskeletal Modeling. *IEEE Trans. Med. Imaging* **2020**, *39*, 1030–1040. [CrossRef]

68. Schlemper, J.; Oktay, O.; Schaap, M.; Heinrich, M.; Kainz, B.; Glocker, B.; Rueckert, D. Attention gated networks: Learning to leverage salient regions in medical images. *Med. Image Anal.* **2019**, *53*, 197–207. [CrossRef] [PubMed]

69. Rundo, L.; Han, C.; Nagano, Y.; Zhang, J.; Hataya, R.; Militello, C.; Tangherloni, A.; Nobile, M.S.; Ferretti, C.; Besozzi, D.; et al. USE-Net: Incorporating Squeeze-and-Excitation blocks into U-Net for prostate zonal segmentation of multi-institutional MRI datasets. *Neurocomputing* **2019**, *365*, 31–43. [CrossRef]

70. Yeung, M.; Sala, E.; Schönlieb, C.B.; Rundo, L. Focus U-Net: A novel dual attention-gated CNN for polyp segmentation during colonoscopy. *Comput. Biol. Med.* **2021**, *137*, 104815. [CrossRef] [PubMed]

71. Kolachalama, V.B.; Garg, P.S. Machine learning and medical education. *NPJ Digit. Med.* **2018**, *1*, 54. [CrossRef]

72. Nie, D.; Trullo, R.; Lian, J.; Petitjean, C.; Ruan, S.; Wang, Q.; Shen, D. Medical image synthesis with context-aware generative adversarial networks. In *Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Quebec City, QC, Canada, 11–13 September 2017*; Springer: Cham, Switzerland; pp. 417–425.

73. Ker, J.; Wang, L.; Rao, J.; Lim, C.T. Deep Learning Applications in Medical Image Analysis. *IEEE Access* **2017**, *6*, 9375–9389. [CrossRef]

74. Benjamens, S.; Dhunnoo, P.; Meskó, B. The state of artificial intelligence-based FDA-approved medical devices and algorithms: An online database. *NPJ Digit. Med.* **2020**, *3*, 118. [CrossRef] [PubMed]

75. Topol, E.J. Welcoming new guidelines for AI clinical research. *Nat. Med.* **2020**, *26*, 1318–1320. [CrossRef]

76. Liu, X.; Rivera, S.C.; Moher, D.; Calvert, M.J.; Denniston, A.K. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: The CONSORT-AI Extension. *BMJ* **2020**, *370*, m3164. [CrossRef] [PubMed]

77. Grauhan, N.F.; Niehues, S.M.; Gaudin, R.A.; Keller, S.; Vahldiek, J.L.; Adams, L.C.; Bressem, K.K. Deep learning for accurately recognizing common causes of shoulder pain on radiographs. *Skelet. Radiol.* **2021**, *51*, 355–362. [CrossRef]

78. Kang, Y.; Choi, D.; Lee, K.J.; Oh, J.H.; Kim, B.R.; Ahn, J.M. Evaluating subscapularis tendon tears on axillary lateral radiographs using deep learning. *Eur. Radiol.* **2021**, *31*, 9408–9417. [CrossRef] [PubMed]
MDPI

St. Alban-Anlage 66

4052 Basel

Switzerland

Tel. +41 61 683 77 34

Fax +41 61 302 89 18

www.mdpi.com

*Applied Sciences* Editorial Office

E-mail: applsci@mdpi.com

www.mdpi.com/journal/applsci

<figure><img src="image_129.png" /></figure>
<figure><img src="image_130.png" /></figure>
MDPI

St. Alban-Anlage 66
4052 Basel
Switzerland

Tel: +41 61 683 77 34

www.mdpi.com

<figure><img src="image_131.png" /></figure>

ISBN 978-3-0365-6488-3