## 4 Results
### Hot-Spot Detection  
The classification metrics were calculated for a test set including 25,264 tiles using a one-shot experiment. The statistical results are shown in Table 2 and were calculated as follows:  
\[Acc=\frac{TP+TN}{P+N} \tag{2}\]  
\[F1 score=2\frac{PPV*TPR}{PPV+TPR} \tag{3}\]  
where \(TP\) is true positive prediction, \(TN\) is true negative prediction, \(P\) is positive (ground true), \(N\) is negative (ground true), \(PPV\) is the positive prediction value (precision), and \(TPR\) is the true negative rate (sensitivity).  
\begin{table}
\begin{tabular}{l l l l l l} \hline  & **Accuracy** & **Sensitivity (TPR)** & **Specificity** & **Precision (PPV)** & **F1-Score** \\ \hline Method 1 & 96\% & 47\% & 98\% & 47\% & 47\% \\ Method 2 & 95\% & 39\% & 96\% & 44\% & 41\% \\ Method 3 (HEKI) & 95\% & 77\% & 98\% & 70\% & 73\% \\ \hline \end{tabular}
\end{table}
Table 2: Average values for the statistical results of tile classification (N = 25,264).  
Figure 7: Automatic hot-spot area detection and selection for two WSIs: (**A**) a WSI with the Ki67 LI equal to 50% and (**B**) a WSI with the Ki67 LI equal to 5%. The length bars shown in the WSIs are 2 mm and 1 mm in the zoomed areas.  
The results confirm the correct tile classification using each method. However, incorrect classifications between classes must also be considered since they are crucially important to the classification quality. A correct hot-spot classification must accomplish the following:  
* Data from classes I (no tumor) and II (lymphocytes) must not be classified as tumor areas (hot-spots).
* Data with a low proliferation index must not be classified as data with high tumor proliferation.  
Figure 8 shows the confusion matrices for each method of classification within various tumor groups (low, medium, and high tumor proliferation). The analysis of the confusion matrix for each method shows that Method 3 (HEKI) provides the best performance in discriminating classes. Ultimately, Method 3 achieved correct lymphocyte classification, while Methods 1 and 2 incorrectly classified lymphocytes as tumor areas with a high value of index proliferation. This is important because the presence of lymphocytes means that the patient is reacting against the tumor, which must not be incorrectly classified as a false positive hot-spot.  
Figure 8: Confusion matrix for the tile classification (N = 25,264) under each method: (**A**) Method 1, (**B**) Method 2, and (**C**) Method 3 (HEKI).  
Contrary to our initial assumptions, Method 2 provided a worse confusion matrix than Method 1, even though the system combined the Ki67 and H&E stain information in a more complex framework. In Method 2, we can effectively observe the incorrect classification of many hot-spot areas as non-tumor areas. The reason for this result was found after an analysis of the results, as small tumor proliferation appeared in these places, leading to incorrectly classified non-tumor areas, thereby avoiding analysis by the Ki67 CNN.  
The goal of medical doctors is to select only a few hot-spot areas in WSIs, while automatic evaluation can detect a significantly higher number of such regions. The proposed methods allow one to create a proliferation index map. Figure 9 shows the results of automatic hot-spot detection.  
The results presented in this section show that Method 3 (HEKI) provides both correct lymphocyte classification and the correct detection of a significant number of gold-standard hot-spot areas. It also performs better in terms of the confusion between classes. Hence, we focus on Method 3 (HEKI) to compute the quantitative results for the Ki67 proliferation index estimation.  
Figure 9: Automatic tumor proliferation map examples: (**A,D**) maps for Method 1, (**B,E**) maps for Method 2, and (**C,F**) maps for Method 3 (HEKI). High tumor proliferation areas are marked in red (dark red), whereas areas with low tumor proliferation are marked in yellow. The length bars shown in the WSIs are 3 mm.