# LungVision: X-ray Imagery Classification for On-Edge Diagnosis Applications
## 4 Experimental Results and Discussion
### Classification Model Performance  
In terms of classification model performance, as expected, the original models without quantization achieved the best performance, with the EfficientNetB2 model achieving a 98.58% F1-Score with 98.60% specificity and 98.59% sensitivity scores. The V2 series of the same model showed a similar performance, with an F1-Score of almost 98%. This is followed by the classification results of ResNet50, which achieved a 96.60% F1-Score. Comparing our results with existing literature revealed that our proposed solution exceeded the performance of the existing work. It is important to note that the existing work used a subset of the lung disease classes; some used as few as two classes and others used up to six classes. In our work, on the other hand, we have used all nine classes of the dataset. We believe that the main reason behind the low performance of previous works arises from the training methodologies followed. This is expected, as these works used off-the-shelf solutions. Common drawbacks include the lack of the use of transfer learning, severely reducing the image sizes and resulting in a loss of information, and over-training the model, risking overfitting. In this work, we believe that we followed a more appropriate training methodology, by using transfer learning and finetuning the models instead of training from scratch. Additionally, given the detailed features of X-ray images, resizing the images to a reasonable (384 \(\times\) 384) resolution ensures that the loss of information is minimal and that it is in line with the inputs expected by the architectures used. Lastly, to avoid overfitting, we utilized an early stopping criteria during the training of our models. The early stopping monitored the model performance on a 10% validation subset of the training dataset, with a patience of five epochs. Additionally, the architecture included a dropout layer before the fully connected dense layer, to further assure that the model does not fit closely to its training data. Table 3 compares this work and previous approaches tackling the same lung disease classification problem. A comparison is carried out in terms of the number of classes used, the F1-Score before quantization, whether quantization has been utilized or not, and whether the trained model has been deployed and tested on a mobile phone or not. What is interesting about our obtained results is that the classification performance of the models optimized using QAT had a moderate impact on the classification accuracy. In fact, aside from the NASNetMobile model--which was severely degraded--the average loss in performance across all other models optimized with QAT was 4.1% in terms of the F1-Score. The EfficientNetV2B2 combined with QAT achieved the best F1-Score, with a result of 97.51%. In terms of FP16 quantization, the average loss in classification performance was 5.9% across all models. However, QAT models seem to have a higher performance to size ratio \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Technique/Model** & **ResNet50** & **MobileNetV2** & **DenseNet21** & **NasNetMobile** & **EfficientNetB2** & **EfficientNetV2B2** \\ \hline Original (FP32) & 102.4 & 11.5 & 33.7 & 23.9 & 38.5 & 42.4 \\ FP16 & 93.6 & 11 & 28.6 & 18.4 & 32.1 & 35.9 \\ INT-REP & 24.1 & 3.2 & **7.5** & 5.6 & 9.4 & 10.6 \\ QAT & **23.8** & **3** & 7.6 & **5.3** & **8.8** & **9.9** \\ \%Maximum Opt. & 76.76\% & 73.91\% & 77.74\% & 77.82\% & 77.14\% & 76.65\% \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Impact of quantization techniques on different lung disease model sizes (MB). \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Work** & **\# Dataset Classes** & **F1-Score** & **Quantization** & **Deployment** \\ \hline
[19], 2019 & 2 & 98.39\% & No & Yes \\
[20], 2023 & 2 & 97.39\% & No & No \\
[16], 2020 & 3 & 94.40\% & Yes & Yes \\
[17], 2022 & 6 & 84.66\% & Yes & Yes \\ Proposed solution & 9 & 98.58\% & Yes & Yes \\ \hline \hline \end{tabular}
\end{table}