# From Classic to Cutting-Edge: A Near-Perfect Global Thresholding Approach with Machine Learning
## 2 Materials and Methods
### Dataset and Metrics  
Our dataset is composed of all the images from the following datasets: the DIBCO datasets [36; 37; 38; 39; 40; 41; 42; 43; 44; 45], a synthetic dataset from [46], the CMATERdb dataset [47], a subset of 10 pages from the Einsielden Stiftsbibliothek [48], a subset of 10 pages from the Salzinnes Antiphonal Manuscript [49], a subset of 15 images from the LiveMemory project [50], the Nabuco dataset [51], the Noisy Office dataset [52], the LRDE-DBD dataset [53], the palm leaf dataset [54], the PHIBD 2012 dataset [55], the [PERSON] dataset [56], and a multispectral dataset [57]. To enrich the dataset, 15 variants were generated for each image by adjusting the gamma value with values ranging from 0.5 to 2. For each image, 48 features were computed: the thresholds resulting from the 15 algorithms presented in Section 2.1; the confidence values for the 7 algorithms that provide them, Otsu, Lloyd, Ridler, Huang, Ramesh, Li and Lee, and Li and Tam (which represent the normalized values of their respective optimization functions for their selected thresholds); the normalized between-class variances associated with the computed thresholds; the pixel average, standard deviation, and normalized standardized moments up to the eighth order; [PERSON]'s bimodality coefficient; and the generalized bimodality coefficients up to the third order. Higher-order moments require significantly better precision than that offered by the standard 64-bit double floating-point representation. In order to ensure the required precision, the Boost Multiprecision C++ library [58] was employed using 256 bits for the cpp_bin_float back-end. All features were normalized using min-max normalization, which ensures all values are in the \([0,1]\) range by subtracting \(min\) from all values and then dividing them by \(max-min\). The min value was considered 0 for all features that could not have negative values. The max value was considered the maximum possible value for all features with values that, in practice, reach values close to their theoretical upper limit (255 for thresholds, 127.5 for the standard deviation, 127.5\({}^{2}\) for all variances, and 1 for all bimodality coefficients). For the features that do not have a finite or obvious lower/upper limit (both limits for standardized moments of odd orders and upper limits for standardized moments of even orders), quantiles were used instead of min and max values (0.001 quantile for the lower limit, and 0.999 quantile for the upper limit) to clamp outlier values to a more compact interval, ensuring a less sparse distribution of feature values. A popular metric used to evaluate binarized images is the F-measure (FM), defined as  
\[FM=\frac{2 TP}{2 TP+F}, \tag{30}\]  
where \(TP\) is the number of pixels correctly classified as pixels containing text, and \(F\) is the number of pixels incorrectly classified. \(FM\) takes values between 0 and 1, with 1 indicating a perfect classifier. However, because we are using global thresholding, the maximum value of \(FM\) can differ from image to image. The maximum possible value of \(FM\) for an image is not necessarily provided by a single threshold. If two thresholds \(T_{1}\) and \(T_{2}\) provide the maximum possible \(FM\), then all thresholds in the interval \(\overline{T_{1},T_{2}}\) provide the maximum possible \(FM\). The ideal threshold of an image, in the global thresholding scenario, was defined as the middle of the largest interval of thresholds that provide the maximum possible FM for that image. The ideal threshold was computed for each image in the dataset and used as the column to be predicted by the ML models. The goal of the experiment is to find a regression model instance that approximates the ideal threshold as closely as possible.