dataesr/scientific-paragraphs-categorization huggingface dataset:
Cloud reference datasets Intercomparison of algorithms within CMIX is performed using existing Sentinel-2 and Landsat 8 cloud reference datasets (Table  1 ), which include Hollstein  (Hollstein et al. , 2016) , PixBox  (Paperin et al. , 2021a (Paperin et al. , , 2021b)) , L8Biome  (Foga et al. , 2017) , CESBIO  (Baetens et al. , 2019)  and GSFC  (Skakun et al. , 2021). These datasets were collected/ generated for different purposes using different methodologies and cloud class nomenclatures. Some of the datasets are single-pixel collections (where a minimum mapping unit is a pixel), while others are the collections of connected pixel areas (polygons) or correspond to whole images. For the majority of datasets, pixels were classified manually through photointerpretation by an expert or a group of experts; in others, the labelling process was semi-automatic with extensive manual checking during classification and post-processing.
CESBIO dataset (Sentinel-2) The CESBIO dataset was generated using an active learning method  (Baetens et al. , 2019)    total number of labeled pixels 85,782,723 (at 60 m spatial resolution). The scenes were acquired from ten sites around the world, five mainly vegetated and five arid sites. The detailed description of the CESBIO dataset is given in  Baetens et al. (2019).
GSFC dataset (Landsat 8, Sentinel-2) GSFC cloud reference data were collected over the NASA Goddard Space Flight Center (GSFC)  (Skakun et al. , 2021). The area is quite heterogeneous with major land cover classes being forest (~52%) and impervious surfaces (31%) with patches of natural vegetation and cultivated areas (totaling 17%) (Fig. 3 ). NASA GSFC also has an AER-ONET station  (Holben et al. , 1998) , which provides aerosol optical thickness (AOT) and water vapor. Ground-based images of the sky were collected from 2017 through 2019 using a smartphone camera with a fisheye lens. These data were collected manually during the Landsat 8 and Sentinel-2 overpasses. Reference data were collected for 6 Landsat 8 and 28 Sentinel-2 scenes. The objective was to capture various cloud conditions and seasonal variability.
Labeling of satellite imagery was performed into cloud, thin cloud (semi-transparent), shadows, and clear classes (Fig. 3 ). Regions within cloud boundaries were excluded from the reference data due to large uncertainties regarding the exact boundaries of clouds, especially on Sentinel-2 imagery  (Skakun et al. , 2021). In order to facilitate the labelling process, Sentinel-2 and Landsat 8 images were presented in various spectral combinations including true color (red-green-blue) and false color (NIR-red-green, SWIR1-NIR-red), and using a cirrus band (at 1. 38 μm). The detailed description of the GSFC dataset is given in  Skakun et al. (2021).
Hollstein dataset (Sentinel-2) The "S2 Hollstein dataset" is a database of manually labeled Sentinel-2A spectra of clouds  (Hollstein et al. , 2016). By means of different spectral tools, pixels were selected and classified into one of the following six classes (Fig. 4 ): cloud (opaque clouds), cirrus (cirrus, semitransparent clouds and vapor trails), snow (snow and ice), shadow (shadows from clouds, cirrus, mountains, buildings, etc. ), water (lakes, rivers, seas), and clear-sky (other remaining areas). Spectral tools include false-color composites of Sentinel-2 images, image enhancements and graphical visualization of spectra. The aim was to create highly heterogeneous classes with a balanced number of pixels. There were 59 total Sentinel-2 scenes and 1,593,911 reference (labeled) pixels.
L8Biome dataset (Landsat 8) The "L8 Biome" cloud validation dataset consisted of 96 Landsat scenes, which were selected using a semi-random sampling by biome  (Foga et al. , 2017). These biomes included barren, forest, grass/crops, shrubland, snow/ice, urban, water, and wetlands. For each biome Landsat 8 scenes were selected, and each scene was manually classified by an expert into the following classes (Fig. 5 ): clear, thin cloud, cloud, and cloud shadow. It should be noted that no specific threshold was used to detect thin (semi-transparent) clouds, which were primarily determined by the analyst. Also, the cloud shadow class in the validation dataset was not provided for all the Landsat 8 scenes. The detailed description of the L8Biome dataset is provided in  Foga et al. (2017).
PixBox dataset (Landsat 8, Sentinel-2) The overarching goal of the so called "PixBox" is to enable a quantitative assessment of the quality of a pixel classification produced by an automated algorithm/procedure. Pixel classification is defined as assigning a certain number of attributes to an image pixel, such as cloud, clear sky, water, land, inland water, flooded, snow etc. These pixel classification attributes are typically used to further guide higher level processing. PixBox is not only a dataset but also includes a method comprising a procedure to define the best thematic, spatial and temporal distribution for each collection purpose, a dedicated software for collecting pixels, the analysis, comparing the collected reference against an automatic classification, as well as the generation of a report.
For the PixBox Reference Dataset, a trained expert(s) manually labels pixels of an image sensor into a detailed set of pre-defined classes. These are typically different cloud transparencies, cloud shadow, and condition of the underlying surface ("semi-transparent clouds over snow", "clouds over bright scattering water"). The collected dataset includes 10's of thousands of pixels because it necessitates representation for all classes, and for various observation and environmental conditions such as climate zones, solar illumination, viewing angles, etc. Prior to the collection process the expert is provided with a detailed list of distribution of categories and classes that needs to be fulfilled. During the collection process the growing database is constantly checked against this reference.
Quality control of the collected pixels is important in order to detect misclassifications and systematic errors. PixBox is a commercially sold product/service of Brockmann Consult GmbH. The following two PixBox datasets have been made freely available to be used for CMIX  (Paperin et al. , 2021a; Paperin et al. , 2021b). The Sentinel-2 PixBox dataset contained 17,351 pixels (at 10 m) manually collected from 29 Sentinel-2A/B Level 1C products (top-ofatmosphere reflectance-TOA reflectance). The Landsat 8 PixBox dataset contained 20,500 pixels (at 30 m) manually collected from 11 Landsat-8 Level 1 products (TOA reflectance). The Sentinel-2 PixBox dataset is spatially, temporally, and thematically evenly distributed, while the Landsat 8 dataset has a strong spatial focus on the Northern European coastal areas.
Summary of strengths and limitations of cloud reference datasets Table  2  summarizes the strengths and limitations of cloud reference datasets used in this study. Reference data incorporating global coverage and a wide range of image conditions (L8Biome, PixBox, Hollstein) are based on the photointerpretation of images by an expert or a group of experts. This can introduce some subjectivity in labelling clouds, especially for thin/semi-transparent clouds that can be wavelengthdependent and fog  (Scaramuzza et al. , 2012)  (Fig. 7 ), and it is usually difficult to draw the exact boundary between this type of clouds and clear pixels. Another approach is to use high-quality pixels (with no uncertainties in cloud detection) and subsequently apply machine learning algorithms to extrapolate classification for the whole scene through an iterative process until the classification results assessed by an expert are deemed to be satisfactory (CESBIO) (Fig.
8 ). The quality of the resulting map, however, can still depend on the training data and classification method used. A third approach (GSFC dataset) is to utilize ground-based imagery of the sky to produce a training/validation cloud dataset, either through manual or automatic labelling (Fig. 8 ). While such an approach would potentially decrease subjectivity in identifying clouds, a network of such sites with sky cameras would be required (similar to the Aeronet network) in order to capture various geographical conditions. Table  A1  (Appendix A) provides a list of classes from the reference datasets that were used to define cloud and non-cloud pixels in the CMIX. Most of the datasets were balanced in terms of cloud and noncloud pixels, except of CESBIO, which had 24% of cloud pixels (Fig.
ATCOR ATCOR is a generic atmospheric correction algorithm for monotemporal multi-/hyper-spectral satellite imagery in the solar reflective region (400-2500 nm) and thermal region (8-13 μm)  (Richter and Schläpfer, 2019b). The code uses MODTRAN5 look-up tables for the radiative transfer functions. Separate codes exist for the processing of flat and rugged terrain imagery. A preprocessing step calculates different masks (water cloud, cirrus cloud, shadow, water) based on spectral tests. The cloud masking uses a buffer of 100 m. For Landsat-8 and Sentinel-2 data the TOA reflectance threshold of the cirrus band is set to 0. 01 (reflectance units). The lower threshold for thin cirrus detection was used to prevent scenes with very thin cirrus being classified as (thin) cirrus because other classes (e.
g. , water, shadow) are generally of more interest than very thin cirrus. Cloud detection in ATCOR was aimed to have a balance between commission and omission errors. In CMIX, ATCOR version 9. 3. 0 (2019) was used. CMIX processing of ATCOR did not use a Digital Elevation Model (DEM) or any other auxiliary data. Some scenes from reference datasets were not processed by ATCOR, since they were acquired with Sun elevation angle values less than 20 •.
CD-FCNN The cloud detection approach based on deep learning, proposed by the Image and Signal Processing (ISP) group of the University of Valencia, is applicable to multispectral images from moderate spatial resolution satellites, including Landsat 8 and Sentinel-2. Training  accurate global cloud detection models based on deep learning requires large datasets of annotated images, which must reflect the high variability of clouds, surface, and atmospheric conditions. This is a major difficulty since high-quality labeled datasets usually do not exist or are not publicly available for most satellite sensors. For Landsat 8, the L8Biome dataset matches these requirements  (Jeppesen et al. , 2019). However, similar global datasets do not exist for Sentinel-2 yet. (Sentinel-2 Cloud Mask Catalog  (Francis et al.
, 2020)  was made available after CMIX was initiated). Therefore, Landsat 8 datasets (L8Biome, 80%, and L8SPARCS, 20%) were used to train fully convolutional neural networks (FCNN) that may be transferred to perform cloud detection in Sentinel-2 images. L8SPARCS (Spatial Procedures for Automated Removal of Cloud and Shadow) (U. S. Geological Survey, 2016) was created for the validation of the cloud detection approach proposed by  Hughes and Hayes (2014). It consists of 80 Landsat-8 sub-scenes manually labeled in five different classes: cloud, cloud-shadow, snow/ ice, water, flooded, and clear-sky. The size of each sub-scene is 1000 × 1000 pixels. After a minimum adaptation of Sentinel-2 data, in terms of band selection and spatial resolution, the models trained on Landsat 8 data are directly applied to Sentinel-2 images.
g. over snow, urban areas or coastal lines. Adjusting this threshold for a specific dataset may improve the tradeoff between omission and commission errors depending on the requirements of the application, i. e. cloud or cloud-free conservative applications. The resulting cloud mask is spatially resampled from the native Landsat 8 resolution of 30 m to the corresponding Sentinel-2 resolutions of 10, 20 and 60 m. Throughout the entire process the work is done at a pixel level, and no spatial dilation of the cloud mask is considered at any stage.
Fmask 4. 0 CCA Function of Mask (Fmask) 4. 0 is a cloud assessment algorithm used with Landsat and Sentinel-2 imagery  (Qiu et al. , 2019 ). An earlier version, Fmask 3. 3, is applied operationally to create cloud masks for USGS Landsat products. The algorithm provisionally identifies cloud pixels using spectral tests, then matches those pixels to provisional cloud shadow pixels using sensor geometry, the Digital Elevation Model (DEM) of the terrain, and an iterative search of altitudes (in Landsat imagery). Fmask was designed to provide a balance between cloud commission and omission errors. Fmask 4. 0 is available under an MIT license at https://github. com/GERSL/Fmask.
FORCE FORCE (Framework for Operational Radiometric Correction for Environmental monitoring, https://github. com/davidfrantz/force) is developed as an 'all-in-one' open-source software solution for the massprocessing and analysis of Landsat and Sentinel-2 image archives  (Frantz, 2019). FORCE includes a mono-temporal Level 2 processing system for Analysis Ready Data (ARD) generation which includes: radiometric correction, cloud masking, and data cube generation  (Frantz et al. , 2016). The cloud masking has branched from Fmask version 1. 6. 3 (Zhu and Woodcock, 2012) , and since then has been developed in parallel  (Frantz et al. , 2015; Frantz et al. , 2016; Frantz et al. , 2018). Parts of the updates in  Zhu et al. (2015)  were incorporated. A darkness filter was implemented to mitigate false positives in bifidly structured dryland areas, where the scene-based temperature distribution tests for Landsat can result in commission errors of cold image parts  (Frantz et al.
, 2015). Cirrus masking is based on an elevation-dependent equation  (Baetens et al. , 2019). The most notable difference to the original Fmask, however, is the complete replacement of the cloud probability module for Sentinel-2 with a new algorithm that makes use of the Cloud Displacement Index, which is formulated to enhance parallax effects in highly correlated NIR bands  (Frantz et al. , 2018). The FORCE cloud masking aims to aggressively detect clouds and cloud shadows to increase cloud producer's accuracy at the deliberate expense of cloud commission for its safe operation in time-series applications. Circular buffers are used to reduce false negatives (300 m for opaque clouds). FORCE provides quality bits whereby 12 quality indicators with respect to atmospheric conditions are provided  (Frantz, 2019).
S2cloudless The s2cloudless is an automated cloud-detection algorithm for Sentinel-2 imagery  (Zupanc, 2017)  based on a gradient boosting algorithm. It was developed by the EO Research team at Sinergise and is published under the MIT License on https://github. com/sentinel-hub/ sentinel2-cloud-detector. The model was trained on a large training dataset with a global coverage. The algorithm is monotemporal, does not consider any spatial context, and therefore can be executed at any resolution. The s2cloudless algorithm can, unlike many other algorithms, be executed also on averaged Sentinel-2 reflectance values over arbitrary user-defined geometries and still provide meaningful results. The input features are Sentinel-2 Level-1C TOA reflectance values of the following ten bands: B01, B02, B04, B05, B08, B8A, B09, B10, B11, B12 and output of the algorithm is a cloud probability map.
Users of the algorithm can convert the cloud probability map to a cloud mask by thresholding the cloud probability map. The recommended value for the threshold is 0. 4 to minimize cloud omission errors. Users can optionally apply additional morphological operations during the conversion of the cloud probability map to the cloud mask. These operations are as follows: convolution of the probability map and dilation of the binary cloud mask with a disk. We recommend convolving cloud probability maps at 10 m (160 m) resolution with a disk with a radius of 22 (2) px and dilate cloud masks with a disk with radius 11 (1) px. Sentinel Hub (https://www. sentinel-hub. com, details in EO Research Team, 2020) and Google Earth Engine (https://developers. google. com/earth-engine/ datasets/catalog/COPERNICUS_S2_CLOUD_PROBABILITY) provide precomputed s2cloudless cloud probability maps and masks to their users for the entire Sentinel-2 archive.
InterSSIM The InterSSIM cloud detection algorithm is a multi-temporal extension of the s2cloudless algorithm (Section 2. 2. 6), but unlike s2cloudless, the InterSSIM algorithm takes temporal and spatial contexts into account. The algorithm was developed by the EO Research Team at Sinergise  (Puc and Žust, 2019)  and integrated into the eo-learn Python library published under the MIT License on https://github. com/sentine l-hub/eo-learn. The input data and parameters for the InterSSIM are the same as in s2cloudless (see Section 2. 2. 6) with the addition of prior satellite observations. The algorithm works on the ten Sentinel-2 TOA bands, and in addition to cloud probabilities from the s2cloudless model incorporates additional features: spatially averaged reflectance values, minimum and mean reflectance values over all prior observations, and maximum, mean, and standard deviation of structural similarity indices computed between the observation for which cloud mask is being predicted and every other prior observations.
MAJA MAJA is applicable to satellites which perform repetitive observations at similar viewing angles, such as Sentinel-2. It was developed by CNES with methods designed by CESBIO with a few modules provided by DLR. MAJA is an open-source software and available at https://gitla b. orfeo-toolbox. org/maja/maja. MAJA's cloud and shadow detection methods include several tests, which use the multi-spectral and multi-temporal properties of surfaces, clouds, and shadows to classify different types of pixels. The methods are described in  Hagolle et al. (2010)  and  Hagolle et al. (2017). The main cloud test detects the pixels for which the surface reflectance in the blue band increases sharply. The cloud masks obtained with MAJA are dilated by 240 m, firstly to account for the parallax effects due to differences in observation angles between spectral bands, and secondly for the adjacency effects of clouds and for their 'fuzzy' borders.
Sen2Cor Sen2Cor is a processor for Sentinel-2 Level 2A product generation; it performs the atmospheric correction of the TOA Level 1C input data. It is composed of two main modules: an atmospheric correction module and a scene classification module that provides a "Scene Classification Map" (SCL), which is used internally in the atmospheric correction module to distinguish between cloudy, clear and water pixels. The Sen2Cor processor is used by the European Space Agency to generate Sentinel-2 Level-2A products within the Sentinel-2 ground segment. Sen2Cor software is available for download at https://step. esa. int/main/th ird-party-plugins-2/sen2cor/. The cloud screening and classification part of Sen2Cor is available as source code within the distributed packages. The Sen2Cor version 2. 8 cloud screening algorithm  (Louis et al.
Performance metrics A standard set of classification metrics derived from confusion matrices (Table  4 ) was used to compare cloud masking algorithms and included (Table  5 ) overall accuracy (OA) and balanced OA (BOA), producer's (PA) and user's accuracies (UA). BOA  (Brodersen et al. , 2010)  was used in addition to OA since some of the reference datasets were imbalanced in terms of cloud/clear pixels and therefore BOA would be a better indicator of algorithms performance. Performance metrics were estimated from confusion matrices that incorporated all valid pixels over all scenes available in the dataset. PA is complementary to the omission error, which shows a fraction of missed clouds; UA is complementary to the commission error, which shows a fraction of over detected clouds. High PA (cloud-free, non-cloud or clear conservative) means that after elimination of clouds, the users results will be minimally affected by remaining clouds, while high UA (cloud conservative) means that the cloud masks will not discard supernumerary valid pixels.
CESBIO reference dataset Table  6  and Fig. 9  show performance metrics when applying cloud masking algorithms on the Sentinel-2 CESBIO dataset. Several observations can be made when analyzing these results. The number of n cloud_as_cloud + n ncloud_as_ncloud n cloud_as_cloud + n ncloud_as_ncloud + n ncloud_as_cloud + n cloud_as_ncloud (1) Balanced OA (BOA) 0. 5 ( n cloud_as_cloud n cloud_as_cloud + n cloud_as_ncloud + n ncloud_as_ncloud n ncloud_as_cloud + n ncloud_as_ncloud ) (2) PA (for clouds) n cloud_as_cloud n cloud_as_cloud + n cloud_as_ncloud (3) UA (for clouds) n cloud_as_cloud n cloud_as_cloud + n ncloud_as_cloud (4) reference pixels varied, since the CESBIO dataset was generated at 60 m spatial resolution, and processors produced masks at various spatial resolution: 10 m (FORCE, InterSSIM, LaSRC and S2cloudless), 20 m (ATCOR, Idepix, Fmask 4.
0 CCA, Sen2Cor), 60 m (CD-FCNN, interpolated from 30 m), and 240 m (MAJA). Cloud and non-cloud classes were imbalanced in the reference dataset (of all labeled pixels 24. 3% were clouds), therefore it results in the OA to be biased towards the non-cloud (dominant) class. Therefore, the balanced OA (BOA) is a more appropriate metric. Overall, BOA varied from 79. 5% to 90. 5%, an average of 85. 9 ± 3. 7%. When not considering MAJA (whose developers generated the CESBIO dataset), the highest cloud PA was 85. 6%, with the average being 75. 9 ± 8. 7%, meaning that most algorithms missed almost 24% of clouds identified in the CESBIO dataset. Average cloud UA without MAJA was 85. 1 ± 10. 6%, meaning an average of 15% over detection of clouds, which may lie in the dilated parts of the cloud masks (FORCE, MAJA), or be associated with a stricter detection of cirrus clouds (LaSRC).
GSFC S2 reference dataset Table  7  and Fig. 10  show the results of comparing algorithm outcomes against the Sentinel-2 GSFC dataset. MAJA provided only 10 images out of 28 images. In the S2 GSFC dataset, cloud and non-cloud are almost balanced (approx. 61% of reference pixels are identified as clouds), therefore there is minimal difference between OA and BOA. BOA varied from 80. 7% to 96. 8% with LaSRC being the outlier (developers of LaSRC produced the GSFC data), with average being 85. 7 ± 2. 8% (not considering LaSRC). Average values of cloud PA and UA not considering LaSRC were 73. 7 ± 5. 6% and 98. 2 ± 2. 7%, respectively, meaning large omission errors. It is worth noting that FORCE and MAJA, whose PA was better than the UA for the other reference datasets, have the opposite result for the GSFC reference, due to the strict classification of very thin clouds as clouds in the GSFC reference data set.
Hollstein reference dataset Table  8  and Fig. 11  show algorithms performance for the Hollstein data depending on the opaque and semi-transparent/cirrus clouds. BOA varied from 84. 2% to 92. 3% (average 89. 4 ± 2. 4%) for all cloud types and 86. 2% to 97. 8% (93. 4 ± 3. 8%) for opaque clouds only. Not considering semi-transparent/cirrus clouds improved algorithms performance, especially for cloud PA: an average gain +8. 0 ± 8. 1%. Variation of performance was comparable to the GSFC results with 8% (5% for opaque only) for PA and 4% (7%) for UA. Note that the Hollstein dataset was used to set radii of disks with which the cloud probability mask and binary cloud mask are convoluted and dilated, respectively, by the s2cloudless algorithm. MAJA was not evaluated against the Hollstein data set, as the images were acquired before Sentinel-2B launch.