# An Introduction to Machine and Deep Learning Methods for Cloud Masking Applications
## 4 Deep Learning Methods
### Datasets and Performance Metrics  
It is common to assess the result of a masking method by measuring its agreement with reference ground truths, which can be manually or tool-supported annotated masks, the most popular of which are summarized in Table 1, or validated cloud masks released as official satellite products. Training Deep Learning models requires sets of annotated data, not only for training purposes but also for evaluating and comparing their performance. However, the datasets currently available have inconsistent characteristics. They are often created without standard protocols, and experts annotate images either by visual inspection or with the aid of tools (e.g., eCognition, Active Learning for Cloud Detection (ALCD) [38], Intelligently Reinforced Image Segmentation (IRIS) graphical user interface [126]). Moreover, there is no universal definition of a cloud, leading to variations in handling ambiguous scenarios like cloud boundaries, thin clouds, and cloud shadows across different teams. Factors like the unit of labeled sample, geographic coverage, number of annotated classes, and spatial resolution vary, impacting the evaluation and comparison of models or masks with different datasets. CloudSEN12 [122] represents a significant advancement in addressing these issues. It is a recent, global, multitemporal dataset designed for Sentinel data. Its strengths lie in its extensive data volume, variety, inclusion of different cloud types including thin clouds and shadows, and a meticulous annotation process with quality control. Additionally, it provides temporal images of the same locations under varying weather conditions, enhancing its utility and overcoming many limitations of existing datasets. An extensive overview of current annotated data collections, in addition to those mentioned in Table 1, can be found in [3,39]. A different approach to the creation of training datasets is presented in [127]. The authors introduce an open-source tool for the generation of various synthetic cloud types that is useful for different tasks, such as masking and cloud removal, without annotation costs. It has been shown that, in terms of performance, a good strategy is to perform training and evaluation using both simulated and real data. To evaluate, validate and compare the performance of predicted masks and segmentation models, a framework of metrics is needed in addition to ground truth masks. The most popular metrics to quantify the accuracy of the segmentation models are summarized in Table 2. They are derived from the Confusion Matrix, determined on all pixels of the images in the dataset. \begin{table}
\begin{tabular}{l l l} \hline
**Dataset** & **Geo-Distribution** & **Number of Scenes** \\ \hline \multicolumn{3}{c}{Landsat 8 (30 m)} \\ \hline L8\_SPARCS [55] & Worldwide & 80 \\ L8\_Biome [27] & Worldwide & 96 \\ L8-38 Cloud [100] & USA & 38 \\ L8-95 Cloud [120] & USA & 95 \\ \hline \multicolumn{3}{c}{Sentinel-2 (10 m)} \\ \hline S2-Hollstein [54] & Europe & 59 \\ S2-BaetensHagolle [121] & Worldwide & 35 \\ CESBIO [38] & Europe & 30 \\ S2-CloudCatalogue (20 m) [104] & Worldwide & 513 \\ KappaZeta [103] & Northern Europe & 150 \\ CloudSEN12 [122] & Worldwide & 49,400 \\ WHUS2-CD [123] & China & 32 \\ \hline \multicolumn{3}{c}{Gaofen1 (16 m)} \\ \hline GF1\_WHU [124] & Worldwide & 108 \\ Levir\_CS[125] & Worldwide & 4168 \\ \hline \end{tabular}
\end{table} 
Table 1: Summary of the most used datasets. OA might be biased when the dataset is not balanced; in this case BOA or mIoU are preferred. BOA is the average of true positive rate and true negative rate. mIoU is computed by the average of the IoU (also Jaccard index) on the dataset images. This metric takes into account false positives and false negatives contrary to the OA. Recall PA (also Producer's Accuracy) is a complement to the Omission Error, while precision UA (also User's Accuracy) is a complement to the Commission Error.