dataesr/scientific-paragraphs-categorization huggingface dataset:
The backbone of most of the multidecadal climate datasets of cloud properties has been the National Oceanic and Atmospheric Administration (NOAA) Polar Operational Environmental Satellites (POES) series. The Advanced Very High Resolution Radiometer (AVHRR) has been on board the NOAA satellites since the end of the 1970s (i. e. NOAA-5 and beyond). AVHRR is a passive imaging sensor, where the source of measured radiation is not emitted by the instrument. Instead, the upwards reflected solar and emitted thermal radiation is measured at the top of the atmosphere (TOA). This is done in abutting pixels that assemble a seamless image. With its four to six spectral channels, AVHRR allows the retrieval of key cloud properties. AVHRR has been a significant contributor to many global cloud climatologies, e.
g. the International Satellite Cloud Climatology Project (ISCCP;  Schiffer and Rossow, 1983; Rossow and Schiffer, 1999) , the Pathfinder extended dataset (PATMOS-x;  Heidinger et al. , 2014)  and the Climate Monitoring Satellite Application Facility's (CM SAF) cloud, albedo and radiation dataset (CLARA-A1/A2;  Karlsson et al. , 2013 Karlsson et al. , , 2016)). Since the 1990s the National Aeronautics and Space Administration (NASA) and the European Space Agency  (ESA)  have launched research satellite missions, e. g. Terra, Aqua, the European Remote Sensing Satellite (ERS-1/2) and the Environmental Satellite (Envisat), that carry AVHRR heritage sensors. These are the Moderate Resolution Imaging Spectroradiometer (MODIS), the Along-Track Scanning Radiometers (ATSR-1/2) and the Advanced Along-Track Scanning Radiometer (AATSR), which provide an increased number of spectral channels as well as higher spatial resolution (≤ 1 km footprint size) than AVHRR.
The cloud datasets derived from these measurement records cover more than one decade and are thus becoming useful for climate studies. Examples of related cloud property datasets are the Global Retrieval of ATSR cloud Parameters and Evaluation (GRAPE;  Sayer et al. , 2011)  for ATSR/AATSR, the NASA MODIS Collection 5  (Platnick et al. , 2003; King et al. , 2003)  and Collection 6  (Baum et al. , 2012; Platnick et al. , 2015 Platnick et al. , , 2017;; Marchant et al. , 2016). The MODIS and ATSR/AATSR sensors include the spectral channels of AVHRR but have additional ones in the visible, near-infrared and, in the case of MODIS, also in the thermal infrared. However, even when restricted to the AVHRR heritage channels, their increased spatial resolution as well as their contribution to increasing the observation frequency motivates their consideration in climate research, in particular in conjunction with AVHRR.
Most of the aforementioned cloud property datasets have improved over the years and have now reached quality levels that facilitate qualitative and quantitative assessments of clouds in the Earth's climate system (e. g. Norris et al. , 2016; Sun et al. , 2015; Enriquez-Alonso et al. , 2015; Carro-Calvo et al. , 2016; Terai et al. , 2016) , including studies to understand cloud processes and the evaluation of atmospheric models. However, there is still potential for advancing such datasets. A common shortcoming of existing datasets is the absence of uncertainty information for pixel-level retrievals (Level-2 data) as well as for daily and monthly averages. These uncertainties should be derived using a mathematically sound framework with uncertainty propagation. Another improvement to cloud property datasets is to ensure that the properties retrieved using mainly shortwave mea-surements are radiatively consistent with those mainly based on thermal infrared measurements.
This is known as spectral consistency and is important to ensure that subsequent simulations of TOA radiances using these retrieved cloud properties match the measured radiances in all spectral bands. The same can be inferred for TOA broad-band fluxes produced using the retrieved parameters. Spectral consistency is not maintained in existing cloud retrievals (e. g. Ham et al. , 2009)  despite being of particular importance to, for example, studies investigating the impact of cloud properties and their change on TOA broadband fluxes and latent heating rates. The ESA Cloud_cci project covers the cloud component within ESA's Climate Change Initiative  (Hollmann et al. , 2013). The overarching aim of the ESA Cloud_cci project has been the generation of state-of-the-art cloud property datasets based on European and non-European satellite missions including the investigation of their synergistic capabilities.
This was achieved by characterizing and advancing measurement records of passive sensors of ESA and non-ESA satellite missions  (Karlsson and Johansson, 2014) ; developing physical retrieval systems for cloud properties with spectral consistency over all utilized spectral bands (see above for definition of spectral consistency and see Sect. 2. 1 for the set of the spectral bands that have been utilized for each sensor considered); generating multi-decadal global cloud datasets, based on both single sensors and on a synergistic use of multiple sensors, including uncertainty estimates which are propagated through all processing levels. The retrieval systems presented in this paper are based on the optimal estimation (OE) technique (e. g. Rodgers, 2000)  and are used to derive a set of cloud variables simultaneously using the visible, near-infrared and thermal infrared measurements.
The retrieval systems were used to generate cloud property datasets spanning the entire available measurement record from 1982 until 2014. In the first phase of Cloud_cci project, prototype versions of the datasets (version 1. 0) were generated. In this paper, version 2. 0 of the Cloud_cci datasets is introduced by presenting a concise overview of the most important technical and scientific aspects. Section 2 gives an overview of the Cloud_cci datasets. This includes a description of the underlying measurement records, the retrieval systems used, the cloud variables produced at different processing levels, and the propagation of the Level-2 uncertainties. In Sect. 3 selected examples of the datasets are shown and discussed, and Sect. 4 reports the most important validation results. Section 6 summarizes the paper.
Composition of the Cloud_cci datasets The following satellites and sensors were used in Cloud_cci: -AVHRR on board the NOAA POES satellites   the Medium Resolution Imaging Spectrometer (MERIS), also on board Envisat. Considering imaging and orbital characteristics of the sensors processed, six datasets were compiled as given in Table. For all datasets digital object identifiers (DOIs) have been established (also given in Table ). Figure  1  reports the local Equator-crossing times of all sensors considered.
The measurement records used Measurements from the passive imaging sensors AVHRR, MODIS, ATSR2, AATSR and MERIS sensors were used to produce the Cloud_cci datasets. Each sensor has different imaging characteristics, such as differences in swath width, which leads to varying observation frequency for any given position on Earth. All sensors operate in a sun-synchronous polar orbit. Each individual orbit has an ascending and descending part, which roughly corresponds to either daytime or night-time conditions, which are thus also referred to as daytime and night-time node. For the MERIS+AATSR dataset, the night-time observations are ignored. Individual AVHRR and MODIS sensors cover the globe nearly twice a day with the daytime and night-time nodes of their orbits. Due to their narrow swath width, ATSR2 and AATSR need 3 days to cover the full globe with daytime and night-time observations (Fig.
AVHRR The second and third generations of the AVHRR sensor (AVHRR/2 and AVHRR/3) provide measurements in two visible, one near-infrared and two thermal infrared channels with the following (approximate) central wavelengths: 0. Table  1. List of Cloud_cci datasets together with the corresponding retrieval scheme, the sensor(s), satellite(s) used and the time period covered as well as the digital object identifiers (DOIs) issued.
Cloud_cci dataset Retrieval used  7, 9, 11, 14, 16, 18, 19 1982-2014 doi:10. 5676 The two visible channels and the 1. 6 µm channel (if present) were intercalibrated using MODIS Collection 6 measurements  (Heidinger et al. , 2017; an update of Heidinger et al. , 2010). For the IR channels no further calibration was performed beyond the on-board blackbody calibration. Sensor(s) Satellite(s) Temporal coverage DOI AVHRR-PM CC4CL AVHRR N-.
Retrieval systems Based on a comprehensive comparison to existing cloud property retrieval systems applicable to passive imaging sensors  (Stengel et al. , 2015) , the two Cloud_cci algorithms were further developed and then used to generate the Cloud_cci climate records. For datasets derived from AVHRR and from the AVHRR heritage channels of MODIS, ATSR-2 and AATSR, the Community Cloud retrieval for CLimate (CC4CL;  Sus et al. , 2017; McGarragh et al. , 2017b)  retrieval system was employed. For the MERIS+AATSR dataset, the Freie Universität Berlin AATSR MERIS Cloud (FAME-C; Carbajal Henken et al. , 2014) retrieval system was employed. Common to both systems is the OE technique, which uses a Levenberg-Marquardt  (Levenberg, 1944; Marquardt, 1963; Rodgers, 2000)  non-linear inversion method to iteratively fit simulated TOA radiances to the measured TOA radiances.
, 2005). Clear-sky transmittance and radiance profiles are computed using the Radiative Transfer for Television Infrared Observation Satellite Program (TIROS) Operational Vertical Sounder (TOVS) (RTTOV;  Eyre, 1991; Saunders et al. , 1999 ) model version 11. 3 (Hocking et al. , 2014). For each iteration the above-cloud and below-cloud clear-sky transmittances and radiances are interpolated from the corresponding RTTOV profiles as a function of CTP. From the derived state vector variables, the properties CTT, CTH, CLA, LWP and IWP are inferred with LWP and IWP calculations being based on  (Stephens, 1978)  for all cloudy conditions. A full list of retrieved cloud variables is given in Table  2. The reader is referred to  (Sus et al. , 2017) and (McGarragh et al. , 2017b)  for more details on CC4CL.
Propagating the uncertainties Different metrics can be used to represent the uncertainty of monthly mean Level-3C products. A simple and often used metric is the standard deviation σ SD (Eq. 1) calculated over the same set of retrieved pixels (x i ) that is used for the calculation of the mean ( x ): σ 2 SD = 1 N N i=1 (x i -x ) 2 , (1) with N being the number of pixels. The OE approach provides pixel-based retrieval uncertainties (σ i ) that are based on mathematically consistent propagation of the uncertainties of the input data (e. g. auxiliary data, measurement data, background errors) into the Level-2 product space (see for example  McGarragh et al. , 2017b, and Sus et al. , 2017). For the Cloud_cci datasets, the Level-2 uncertainties were further propagated into Level-3C products by two measures: the mean of the pixel-based uncertainties ( σ i , Eq.
Hence, the propagated uncertainties are meant to be experimental for these dataset versions. Assuming Gaussian and bias-free distributions, the estimated natural variability represents the standard deviation (thus the inner 68 % percentile) of the distribution around the true mean. Furthermore, the estimated uncertainty of the monthly mean represents the 68 % confidence interval around the calculated monthly mean. The given framework was applied to all cloud properties and their OE-based uncertainties in the generation of the    , 90, 180, 245, 310, 375, 440, 500, 560, 620, 680, 740, 800, 875, 950, 1100 CTT (K) 200, 210, 220, 230, 235, 240, 245, 250, 255, 260, 265, 270, 280, 290, 300, 310, 350 COT 0, 0. 3, 0. 6, 1. 3, 2. 2, 3. 6, 5. 8, 9. 4, 15, 23, 41, 60, 80, 99. 99, 1000 CER (µm) 0,  3, 6, 9, 12, 15, 20, 25, 30, 40, 60 , 80 CWP (g m -2 ) 0,  5, 10, 20, 35, 50, 75, 100, 150, 200, 300, 500, 1000, 2000 , 100 000 CLA_vis006/008 * 0, 0.
1, 0. 2, 0. 3, 0. 4, 0. 5, 0. 55, 0. 6, 0. 65, 0. 7, 0. 75, 0. 8, 0. 9, 1 JCH See COT and CTP bins * For FAME-C, the cloud albedo is derived at 1. 6 µm instead of 0. 87 µm. Level-3C products. The results are discussed using the example of COT from the AVHRR-PM dataset for June 2008. Figure  4  shows global maps of monthly mean COT and the corresponding monthly standard deviation (panels a and b) as calculated from the retrieved Level-2 values. The estimate of the true variability is shown along with the estimated uncertainty of the calculated mean (panels c and d) for an uncertainty correlation (c) of 0. 1. Panels (e) and (f) show the results for an uncertainty correlation of 1. 0. The exact correlation is not known and it is likely to have spatial and temporal variability. Due to this, two very different values have been chosen to illustrate the sensitivity.
Product examples In this section most Cloud_cci products are discussed using the example of the AVHRR-PM dataset, i. e. (1) Level-3U data of NOAA-18/AVHRR for 22 June 2008 and (2) Level-3C data for the month of June 2008. Figure  5  shows CMA, CFC, CPH, liquid cloud fraction, COT and CER. Figure  6  shows CTP, LWP, IWP and CLA. In Fig. 7  the JCH is presented in two ways:  (1)  shown as a global COT-CTP histogram aggregated over all grid cells and (  2 ) the relative fraction of a certain subset of clouds, in this case cumulus clouds according to the ISCCP definition given in  (Rossow and Schiffer, 1999) : clouds with CTP larger than 680 hPa and with COT lower than 3. 6. In Fig. 8  time series of selected cloud properties are shown for monthly, latitude-weighted averages (within a 60 • S-60 • N latitude band) of Cloud_cci AVHRR-AM, MODIS-Terra, ATSR2-AATSR and MERIS+AATSR datasets.
All datasets are relatively stable. However, the time series exhibit some systematic offsets between the datasets. Though these offsets have not been investigated in detail yet, it is currently assumed that they are caused by the following three reasons. (  1 ) Differences in the spectral characteristics of the AVHRR heritage channels used, i. e. the position and shape of the spectral response functions, which is only accounted for empirically in cloud detection and cloud typing schemes. (2) The applied, but maybe still imperfect, calibration of the measurement records. (  3 ) Differences in the spatial resolution of AVHRR (footprint size of 1 km × 4 km) compared with MODIS and AATSR (1 km × 1 km footprint size) may have a significant impact. Figure  9  shows the time series for Cloud_cci AVHRR-PM and MODIS-Aqua datasets.
Considering the time series of all datasets, some inhomogeneities are found for AVHRR-AM, AVHRR-PM. These are mainly due to differences in local observation time of the individual satellites. A significant portion of this is due to a growing delay in local observation time with satellite lifetime caused by the drift of the satellite orbit. For AVHRR-AM, there is an additional jump in local observation time between the early morning orbits of NOAA-12 and NOAA15 and the mid-morning orbits of NOAA-17 and Metop-A. Variability in local observation time means that different parts of a diurnal cycle of clouds are sampled. Also, the solar zenith angle and the relative azimuth angle between satellite and sun change with local observation time and can also lead to inhomogeneities in a time series. Statistical correction methods for mentioned effects exist (e.
g. Foster and Heidinger, 2013)   and should be applied precedent to any trend analysis. The impact of spectral deviation among the individual AVHRR sensors of a dataset is assumed to have a smaller impact compared to the satellite drift effect. For ATSR2-AATSR a small jump in the time series of some cloud properties is found at the sensor transition. This is primarily driven by differences in the dynamic range of the 3. 7 µm channel. The channel saturates more often for ATSR-2. This is particularly evident for CFC, LWP and IWP. Another feature in the AVHRR-PM CFC time series (Fig. 9 ) is worth mentioning. In 1982 (and onwards) and in 1991 (and onwards) positive anomalies are found which are related to the major eruptions of the El Chichón (Mexico) and Pinatubo (Philippines) volcanoes. A first analysis revealed that heavy aerosol loadings are sometimes detected as clouds.
Validation summary Cloud_cci datasets between 2006 and 2014 were collocated in space and time with observations from the CALIOP instrument mounted on board the CALIPSO satellite. The active measurements of CALIOP observations can be considered as an accurate reference for CMA, CPH and CTH. However, it is important to note that cloud properties from CALIOP are physically different to that given by the Cloud_cci products. For CTH, for example, the active sensor detects where the density of particles sharply increases -the physical cloud-top edge. Passive sensors observe the entire atmospheric column simultaneously. The CTH retrieved from passive sensors is an effective average through the cloud-top layer. As such, it is expected that Cloud_cci CTHs are lower than those observed by CALIOP, in particular for clouds containing fuzzy (semi-transparent) cloud layers.
To account for this, various metrics have been considered for the validation of Cloud_cci cloud properties, where the CALIOP properties have been adjusted with respect to the optical depth profile. The CAL_LID_L2_05kmCLay-Prov product has been used, which was downloaded from ICARE Data and Service Center (http://www. icare. univ-lille1. fr). All collocations are based on searching for the nearest neighbour in the Cloud_cci Level-3U data to each CALIOP observation. Due to the similar orbital characteristics of NOAA-18, NOAA-19 (both are part of AVHRR-PM), Aqua and CALIPSO, a very large set of collocations between these passive imagers and CALIOP was found with only short temporal mismatches. A time window of ±3 min was used in these cases. The orbital characteristics of NOAA-17, Metop-A (both are part of the AVHRR-AM dataset), Envisat (part of the ATSR2- AATSR and MERIS+AATSR datasets) and Terra deviate significantly from CALIPSO.
The biases range from -13 to -1 % indicating a slight underestimation of cloudiness, despite the fact that the POD cloudy is significantly higher than POD clear for all  datasets, which can be explained by the higher frequency of cloudy scenes compared to clear-sky scenes. Removing optically very thin clouds from the CALIOP data (i. e. classifying all clouds with optical thickness lower than 0. 15 as clear sky in the CALIOP data) significantly improves the agreement of Cloud_cci data with CALIOP (lower part of Table  6 ). The hit rates are mostly increased (except for MODIS-Terra) and the biases are reduced (except MODIS-Terra and ATSR2-AATSR). In Table  7  the validation results for CPH are presented, i. e. the probabilities of correctly detecting liquid and ice clouds (Hit rate, POD liquid , POD ice ; see Appendix A for the definition of these terms), the bias and the number of considered cloudy pixels.
As for CMA, two validation set-ups have been used: in the first the CALIOP cloud phase from the uppermost reported cloud layer was used (upper part of Ta-ble 7), while in the second set-up the CALIOP cloud phase was taken at that level in the cloud at which the top-down cloud optical depth is 0. 15 or higher (COD lev = 0. 15; lower part of Table  7 ). For the first set-up, the probabilities of detecting the correct phase range from 72 to 78 %, with highest values for AVHRR-PM and ATSR2-AATSR. All datasets except MERIS+AATSR and ATSR2-AATSR show a clear liquid bias, meaning an overestimation of the occurrence of liquid clouds at the cost of ice clouds. A liquid bias can be explained by a lack of sensitivity of the passive imager retrievals to optically very thin ice cloud layers above liquid cloud layers.
This is supported by the lower POD ice values compared to POD liquid. The hit rates for phase agreement between Cloud_cci datasets and CALIOP increase for COD lev = 0. 15, which is mainly driven by a better detection of ice phase (increased POD ice ) while at the same time the detection efficiency for liquid phase (POD liq ) decreases only slightly. The biases change towards ice (reduced liquid bias or ice bias instead of liquid bias). The results show that the correct cloud phase determination for passive imager retrieval is very sensitive to phase changes in the uppermost cloud layers (e. g. between the physical cloud top and 1 optical depth into the cloud). In addition to the CPH comparisons presented, the scores were again calculated including only conditions for which no phase change occurred in the CALIOP data between the physical cloud top and 1 optical depth into the cloud (not shown).
The agreement between Cloud_cci datasets and CALIOP data further improves significantly. Hit rates increase by 2 to 10 %, mainly driven by a much better detection probability for ice clouds. In Table  8  the validation results for CTH are presented, i. e. standard and mean deviations. The comparisons were limited to those collocated pixels for which both CALIOP and the Cloud_cci dataset report clouds and valid retrievals of cloud phase and cloud-top height. In addition, the data were restricted to cases where the phase assignment between CALIOP and the Cloud_cci dataset is in agreement. Removing this uncertainty in the phase assignment gives a clearer picture of the actual cloud-top height retrieval, since the first guess of COT, CER and CTP in the Cloud_cci retrieval systems is a function of the prior determined cloud phase.
The comparisons were separated into liquid and ice cloud conditions and carried out twice using different top-down cloud optical depth levels (COD lev ) at which the reference CTH was selected from CALIOP profiles. For liquid clouds the standard deviation between Cloud_cci datasets and CALIOP is around 1 km and the bias is generally below 0. 2 km (except MERIS+AATSR with a bias of 0. 79 km). These values do not change significantly when selecting the reference CALIOP CTH at COD lev = 0. 15 (bottom part of Table 8), which indicates that water clouds usually do not have small optical thicknesses. For ice clouds the agreement with CALIOP is lower as expected. Standard deviations range from 1. 9 to 2. 8 km and the bias is generally negative (thus an underestimation of CTH for ice clouds in Cloud_cci datasets) between -2.
5 and -3. 6 km. These negative biases are reduced to -1. 9 to -2. 8 km when the CALIOP CTH is taken at COD lev = 0. 15. Standard deviations are also reduced by about 0. 2 km for this setting. The agreement of the Cloud_cci ice cloud CTH to CALIOP further improves with increasing COD lev (not shown). For example, at COD lev = 1. 0 the biases for ice cloud CTHs are decreased to -0. 78 to -1. 99 km. LWP retrievals were validated against LWP data derived from satellite-based, passive microwave (MW) data  (O'Dell et al. , 2008). Microwave radiation can normally fully penetrate clouds. Thus, MW measurements can provide a direct   sidering the given uncertainty estimates for the reference data (15-30 %), one can still conclude that there is agreement between all Cloud_cci datasets and MW in nearly all regions.
It is worth mentioning that the agreement with MW reduces when considering the time period before 2003 for AVHRR-AM, AVHRR-PM and ATSR2-AATSR. This is mainly due to problems with the earlier satellites, which can also be seen in the LWP time series plots of Figs. 8  and 9. Beyond the limited validation results presented in this paper, a comprehensive effort has been carried out to compare the Cloud_cci datasets with other, well-established datasets such as PATMOS-x, CLARA-A2 and MODIS Collection 6  (Stapelberg et al. , 2017). Their results prove the quality of the Cloud_cci datasets.