# Accelerating Detection of Lung Pathologies with Explainable Ultrasound Image Analysis
## 3 Classification of Lung Ultrasound Data
### Results
#### 3.2.3 Evaluation on Independent Test Data  
To the best of our knowledge, the only other publicly-available lung ultrasound database was released by the ICLUS initiative [57], comprising 60 lung US recordings from Italian patients suspected for COVID-19 (39 convex + 21 linear probes). The data was annotated by medical experts by observation, with severity scores from 0 to 3 ([PERSON] et al. [57] \"Score 0 indicates the presence of a continuous pleural-line accompanied by horizontal artifacts called A-lines [33], which characterize a healthy lung surface. In contrast, score 1 indicates the first) signs of abnormality, that is, the appearance of alterations in the pleural-line in conjunction with vertical artifacts. Scores 2 and 3 are representative of a more advanced pathological state, with the presence of small or large consolidations, respectively. Finally score 3 is associated with the presence of a wider hyperechogenic area below the pleural surface, which can be referred to as white lung\". We evaluated the performance of the VGG model on all 39 convex probes, predicting each frame as the average of the five VGG models trained in cross-validation. Since many frames in a raw lung US video are not informative, [PERSON] et al. [57] train an aggregation layer to combine frame-based scores into a video prediction. Instead, we propose to use confidence estimates as explained in Section 4.2 and discard frames with low confidence. Discarding the 5% frames with lowest certainty, and assuming that a severity score of 0 corresponds to a healthy lung and severity of 1, 2 or 3 are patients infected with COVID-19, the ensemble achieves a frame-prediction accuracy of 72.9%, and a video classification accuracy of 82.1%. The sensitivity for COVID-19 is 0.806 and the specificity 0.962.  
We also compare the predicted probability of COVID-19 to the severity scores in Figure 4. Although, in contrast to their model, our model was not trained to assess severity,  
\begin{table}
\begin{tabular}{l l l l l l} \hline \hline  & **Class** & **Recall** & **Precision** & **F1-Score** & **Specificity** \\ \hline
**VGG** & & & & & \\
**Accuracy: 90\%** & COVID-19 & \(0.90\pm 0.08\) & \(0.92\pm 0.07\) & \(0.91\pm 0.06\) & \(0.96\pm 0.04\) \\ Balanced: 90\% & Pneumonia & \(0.93\pm 0.10\) & \(0.88\pm 0.08\) & \(0.90\pm 0.08\) & \(0.95\pm 0.04\) \\ \#Param.: 14.7 M & Healthy & \(0.88\pm 0.06\) & \(0.91\pm 0.08\) & \(0.89\pm 0.06\) & \(0.95\pm 0.05\) \\ \hline
**Models Genesis** & & & & & \\
**Accuracy: 78\%** & COVID-19 & \(0.74\pm 0.17\) & \(0.77\pm 0.15\) & \(0.75\pm 0.15\) & \(0.87\pm 0.11\) \\ Balanced: 77\% & Pneumonia & \(0.79\pm 0.14\) & \(0.80\pm 0.15\) & \(0.78\pm 0.11\) & \(0.91\pm 0.07\) \\ \#Param.: 7.6 M & Healthy & \(0.78\pm 0.28\) & \(0.79\pm 0.12\) & \(0.77\pm 0.22\) & \(0.88\pm 0.08\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Video classification results. The frame-based model VGG-CAM outperforms the 3D CNN Models Genesis in all shown metrics, displaying high accuracy (94%), recall, precision for COVID-19 and pneumonia detection.  
there is a clear correlation between the severity and the probability of our model ([PERSON]'s \(r=0.36\)). At this point, we can safely conclude that test data performance is encouraging, but requires further validation with labeled data including healthy patients and patients infected with other pneumonias.