# Machine Learning in Recognition of Basic Pulmonary Pathologies
## 3 Considerations on Quality of the Models  
The previous section described the process of training models which support the diagnosis. However, they were firstly trained on original images and then on segmented images. The purpose was to check whether model accuracy would improve after the segmentation process. The results that the networks achieved are briefly presented in Table 2; this section will describe them more precisely and will show how the quality of models was determined.  
All models were tested with test sets that accounted for 10% of the total data. For each tested case, the models returned a number ranging from 0 to 1, which can be interpreted as the probability of the appearance of a given disease. In order to determine the quality of the prediction, it has been decided that a certain threshold will be set that will separate the predictions into positive and negative ones, creating a binary prediction vector. By comparing this vector with the label vector (Grand Truth), the number of true positives, true negatives, false positives, and false negatives was calculated. These numbers also allowed the sensitivity and specificity of each model to be calculated and the ROC curve to be drawn. The area under this curve (ROC-AUC) defines the quality of the model.  
In order to counteract the phenomenon of network overfitting during training, a function for creating checkpoints was used in the form of model saves every epoch. Then, using the training graphs and validation, the model quality was analyzed depending on the training duration (number of epochs). The analysis was performed for different threshold values separating positive and negative cases. Results had shown that the model was not overfitted, and with training lasting 40 epochs, it achieved the best result. This analysis method was performed for all models, and the best one was selected.  
Figures 4-7 show the confusion matrices and the corresponding ROC curves. All those plots describe the quality of the models. Each of them, during testing, had a tendency to return small values, which means that output could not be interpreted as a probability of the presence of pathology. It resulted in the necessity of using a threshold that could divide predictions into positive and negative ones. Models were tested, and the best threshold was chosen. Results are presented in Table 3.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Model** & **Pneumothorax** & **Pneumonia** & \
\begin{tabular}{c} **Lung** \\ **Consolidation** \\ \end{tabular} &
\begin{tabular}{c} **Lung** \\ **Lesions** \\ \end{tabular} \\ \hline Trained on & 57\% & 69\% & 80\% & 67\% \\ original images & 78\% & 78\% & 91\% & 77\% \\ segmented images & +21\% & +9\% & +11\% & +10\% \\ \hline \hline \end{tabular}
\end{table}
Table 2: Difference in accuracies depending on training model on segmented or original images.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Model** & **Pneumothorax** & **Pneumonia** & \
\begin{tabular}{c} **Lung** \\ **Consolidation** \\ \end{tabular} &
\begin{tabular}{c} **Lung** \\ **Lesions** \\ \end{tabular} \\ \hline Optimal & 0.3 & 0.2 & 0.2 & 0.3 \\ threshold & +0.3 & 0.2 & 0.2 & 0.3 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Best thresholds for each model.  
Figure 4: Model quality—Pneumothorax. (**a**) Confusion matrix. (**b**) ROC curve.  
Figure 5: Model quality—Pneumonia. (**a**) Confusion matrix. (**b**) ROC curve.  
Additionally, it was decided to count one more measure. For each model, three independent testing sets were created; it was checked how the quality of the model would differ between them. Thanks to this, it is possible to determine the level of precision of a given model, which allows defining the boundary that helps to compare models with each other. The results are shown in Table 4.  
Figure 6: Model quality—Pulmonary Consolidation. (**a**) Confusion matrix. (**b**) ROC curve.  
Figure 7: Model quality—Lung Lesions. (**a**) Confusion matrix. (**b**) ROC curve.