SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 13: Loss - focal loss (single-scale feature calculation). The blue curve represents the loss curve of the training data set, while the red curve represents the loss curve of the validation data set.
Figure 14: Loss - rebalanced loss (single-scale feature calculation), blue curve - training data set, red curve - validation data set.
In Figure 12, Figure 14, Figure 15, Figure 16 and Figure 17, we can clearly observe the presence of "sharp fluctuations", which are a result of the operation mentioned on main paper and Section B, where we increased the batch size by a factor of two approximately every 1,000 epochs. We followed this practice proposed by Smith et al. [36], which involves increasing the batch size instead of decaying the learning rate.
E Single-scale vs Multiple-scale
In this section, we mainly showcase multiple-scale geometric features calculation outperform single-scale at our task. After applying the rebalanced loss as the loss function, we observed that computing geometric features at multiple scales ultimately improved the performance of SOUL. When comparing the evolution of specificity with the number of epochs between single-scale and multiple-scale (see Figure 16 and Figure 17), multiple-scales not only exhibit higher values but also converge more effectively over time, same for loss convergence (see Figure 14 and Figure 15).
The Matthews Correlation Coefficient (MCC) (see Yao & Shepperd [37]) and AUROC are both effective metrics to evaluate the model performance under class imbalance. Therefore, besides the specificity, we provide the MCC and AUROC values for both single-scale and multiple-scale cases
Figure 15: Loss - rebalanced loss (multiple-scale feature calculation), blue curve - training data set, red curve - validation data set.
Figure 16: Specificity - Single-scale geometric features calculation, cyan curve - training data set, gray curve - validation data set.
Figure 17: Specificity - Multiple-scale geometric features calculation, cyan curve - training data set, gray curve - validation data set.
during the training process. Upon analyzing the comparative results of MCC in Figure 18 and Figure 19, and AUROC in Figure 20 and Figure 21, we consistently observe that the multiple-scale computation of geometric features output higher values compared to the single-scale. This strongly supports the superior performance of multiple-scale feature computation over single-scale.
Figure 18: During the training process, the MCC (single-scale) values change with the number of epochs.
Figure 19: MCC (multiple-scale) values change with the number of epochs.
Figure 20: AUROC (single-scale) values change with the number of epochs.
Figure 21: AUROC (single-scale) values change with the number of epochs.
MCC and AUROC provide valuable insights for selecting the final model state (checkpoint). For example, up to now, the best-performing model state is the one obtained at the 3161st epoch with the multiple-scale geometric features computation, as mentioned in the main paper. The model achieves an MCC value of 0.605 and an AUROC value of 0.888, these results generally outperform all single-scale metrics. As MCC is a discrete case of Pearson correlation coefficient, we can conclude that there exists a strong positive relationship exists between the model's predictions and the ground truth by using the interpretation of Pearson correlation coefficient.