SEGMENTATION OF CLOUD PATTERNS FROM SATELLITE IMAGES TO IMPROVE CLIMATE MODELS Kirill Vishnyakov† and Mukharbek Organokov‡ †Peter the Great St.Petersburg Polytechnic University, ‡Universit´e de Strasbourg kirill.a.vishnyakov@gmail.com, mukharbek.organokov@gmail.com Introduction Climate change has been at the top of our minds and at the forefront of important political decision-making for many years [1]. Classiﬁcation of different types of clouds is substantial for understanding climate change. Human ability to identify patterns is limited and murky boundaries between different forms of clouds lead to obstacles in traditional rule-based al- gorithms cloud features separation [1]. In these situations, machine learning techniques, particularly deep learning, have demonstrated their ability to mimic the human capacity for identifying patterns in the clouds using satellite images [2]. This work focuses on seg- mentation of four subjective patterns of clouds organization [2] (see Fig. 1): Sugar, Flower, Fish, Gravel. (a) Sugar (b) Flower (c) Fish (d) Gravel Figure 1: Canonical examples of the four cloud organization patterns. Method Dataset The dataset used in the analysis is released on the Kaggle platform [1]. The images with three regions, spanning 21 degrees longitude and 14 degrees latitude, were taken from NASA Worldview [3]. The labels were created on a crowd-sourcing platform [4]. On cloud labeling days at two institutes, 68 scientists screened satellite images, identiﬁed areas of cloud patterns in each image, and each image was labeled by approximately 3 different sci- entists [1]. Ground truth was determined by the union of the areas marked by all labelers for that image, after removing any black band area from the areas [1]. The ground-truth masks presented in the dataset are quite noisy meaning they include a lot of areas that ac- tually do not contain clouds at all. Also, the masks of different classes can overlap. These two facts signiﬁcantly increase problem complexity. The resolution of original images is 1400×2100, in our experiments we use downscaled images, namely we work with three different image resolutions: 352×512, 512×768, 768×1152. Example of images containing different number of clouds patterns is shown in Fig. 2. Figure 2: Images of clouds with corresponding masks. Metric The mean Dice coefﬁcient (DC) [5] is chosen as the metric to measure the performance of the model by pixel-wise comparison of the predicted segmentation (Ypred) output and the corresponding ground truth (Ygt). If Ypred = ∅and Ygt = ∅, then DC = 1; otherwise, it is deﬁned as: 2|Ypred ∩Ygt| |Ypred| + |Ygt| (1) The objective of this research is to develop a DL model that ensures higher accuracy. Thus, the goal is to maximize the model on the mean DC so that a perfect model achieves an accuracy of 1. Architecture and Implementation This problem was treated as a semantic segmentation task with four different classes. During the past years numerous architectures (see Fig. 3) have been developed to tackle the problem of semantic segmentation including the most famous U-Net [6], LinkNet [7], FPN [8], PAN [9]. In this work, we used U-Net architecture. Figure 3: Simpliﬁed schematic view of the architecture. We developed a two-stage (S1 and S2) semantic segmentation pipeline [10] (see Fig. 4) that accepts images with corresponding ground-truth masks and outputs raw (non- normalized) predictions for each of four classes. Then during the post-processing proce- dure predictions are normalized with sigmoid function to eliminate masks of insufﬁcient size. Thus, ﬁnal processed predictions were generated. Figure 4: Pipeline. Experiments Our main experiments with data included training on two stages with different image res- olutions and different set of pseudo-labels (PLs). Recently, a semi-supervised approach became popular and provided improvements in training procedure. We used approach similar to [11] in which authors ﬁrst trained an EfﬁcientNet model