dataesr/scientific-paragraphs-categorization huggingface dataset:
This paper revisits the data collected by early sounders Nimbus-4 IRIS (1970)  and Nimbus-6 HIRS (1975), after recovery of ageing tapes by NASA GES DISC. New quality controls are proposed to screen out erroneous or suspicious mission data, based on instrument health status data records and other inspection of the data. Radiative transfer coefficients are derived for the fast computation of clearsky radiative transfer simulations. Atmospheric profiles from ERA-40 and ERA-20C reanalyses are used in input. These spatiotemporally complete datasets are interpolated to each sounding location, using the closest estimate in time. A modern cloud detection method derived for current hyperspectral sounders is applied to IRIS and yields maps of cloud cover that are in line with current knowledge of cloud climatology.
For clear scenes, the standard deviation of brightness temperature differences between IRIS observations and simulations from ERA-20C is around 1 K for the lower-peaking temperature channels of the 15 lm CO 2 band, and lower than 1 K for simulations from ERA-40. The IRIS and HIRS instrumental data records are projected in a common sub-space to alleviate issues with different field-of-view resolutions and spectral resolutions. A proxy cloud detection scheme screens out clouds in the same manner in both data records. Considering the month of August, common to both missions, a detailed analysis of the departures from observations suggests that ERA-40 suffers from spurious tropospheric warming, possibly caused by changes in the observation input during the 1970s including a known error in ERA-40 radiance assimilation bias correction.
, 2013). This enhanced understanding has opened the way to new techniques for data processing and exploitation such as radiance data assimilation (e. g. ,  Andersson et al. , 1994). Among these advances, and relevant to infrared sounding, are in particular improved techniques to detect clouds that may dominate the infrared signal (e. g. ,  McNally and Watts, 2003; Joiner et al. , 2004). Curiosity suggests trying out these new methods onto the initial missions, and assessing whether new insight can be gained from old data using current advanced methods, unavailable at the time of the early missions. This seemingly simple concept is not straightforward however, because one has today, with present satellite missions, much more documentation of the satellite data quality, including about instrument pre-and post-launch calibration; one is also assisted by many other ancillary datasets that can help characterize a given atmospheric scene or background (surface state), and thereby better inform a priori the satellite data quality control.
In addition, the atmosphere may have changed in subtle ways that one may not fully understand. For example, there is a dependence of surface emissivity, aerosols, and methane in infrared sounding (e. g. ,  Prabhakara et al. , 1974) , but our knowledge of the past for these variables is too insufficient to use anything else other than near-stationarity or global trends. This fuzzy past, along with the retirement of mission experts, explains partially why historical satellite datasets have received gradually less attention, as new datasets have come along, offering more numerous (with several collocated instruments) views of the same scene, with reduced errors (improved precision, calibration, resolution). Nevertheless, the past decade has witnessed attempts to reproduce accurately the weather and climate evolutions of the past century (e.
g. ,  Compo et al. , 2011). The reanalyses combine observations with the latest advances in atmospheric modelling (e. g. ,  Dee et al. , 2014). They represent both an opportunity and an impetus to reconsider these early data records: an opportunity, for they provide ancillary data to better understand the satellite data collected then, and an impetus, for their exploitation could eventually help improve or better qualify the reanalysis record. In addition, outside the reanalysis context, these ancient data can also help tackle the problem of climate change monitoring (e. g. ,  Brindley and Bantges, 2016) , for they offer high-resolution information collected more than forty years ago. There is also raising awareness that these past missions, to be accessed through satellite data rescue when needed, offer information of interest to advance our understanding  (Poli et al.
, 2017). The present article attempts to answer the following two questions: can new insight be gained about early mission IRIS and HIRS data, and what can be learnt about the quality of reanalyses by a comparison exercise with data from these early sounder. The outline of the paper is as follows. Section 2 presents the data used in the study, Section 3 explains the methodology, Section 4 presents results of comparisons to IRIS, and Section 5 discusses findings obtained by comparing the reanalyses to IRIS and HIRS instrumental data records in a common sub-space. Conclusions and perspectives for future work are given in Section 6.
The data from Nimbus-4 IRIS used in this study are freely available from NASA Goddard Earth Sciences Data and Information Services Center (GES DISC). Table  2  indicates the number of spectra collected per month between April 1970 and January 1971. Fig. 1a  shows the spatial coverage achieved by the mission over its duration. Some gaps are visible around the South Atlantic Anomaly; today satellites carry improved radiation shields that protect against electromagnetic radiation. Large data gaps above Fairbanks, Alaska and Orroral, Australia may be in proportion to the respective orbit data transmissions to these ground stations, when parallel data acquisition onboard the satellite may have been limited. The area covered by IRIS extends up to latitude about 80°in both hemispheres due to the satellite orbit inclination and the absence of cross-track scanning.
7 lm and in the visible at 0. 7 lm. The usage of shortwave channels is generally restricted to night-time, as the data are difficult to simulate during daytime due to non-local thermal equilibrium, though advances have been made for high spectral resolution sounders  (De Souza-Machado et al. , 2007). Nimbus-6 HIRS was the first of a long series. Second-, third-, and fourth-generation instruments (HIRS/2, /3, and /4, respectively) were mounted, respectively, on: TIROS-N (launched in 1979), National Oceanic and Atmospheric Administration (NOAA) satellites-6 to -14; NOAA-15 to -17; NOAA-18 and -19 and Metop-A and -B. Some of these instruments are still operating at time of writing. As compared to the first-generation Nimbus-6 HIRS, HIRS/2 featured similar channels, except for Nimbus-6 HIRS channel 16 (3.
One may then consider ERA-20C to feature lower precision than ERA-40. The word precision bears here a different meaning than the definition of the Joint Committee for Guides in Metrology Working Group 1 (  2008 ), but refers instead to the accuracy of representation of synoptic weather timescales (or intra-month variability). Yet, ERA-20C is interesting to consider for it offers a century-long dataset and it uses a more recent version of the atmospheric model than ERA-40. Indeed, any conclusion about the reanalysis data quality, even on short time periods like those considered in this paper  (around 1970 and around 1975) , may help to qualify eventually the overall data record. The reanalysis data used here (ERA-40 and ERA-20C) are freely available from ECMWF. The data are used at their native respective spatio-temporal resolutions, namely horizontal spectral truncation T159 in ERA-40 and ERA-20C (approximately 125 km), 60 levels in the vertical for ERA-40 (91 for ERA-20C), and 6-hourly temporal resolution for ERA-40 (3-hourly for ERA-20C).
Radiative transfer simulations The RTTOV model is updated as follows as compared to the default version. Both instruments' temperature sounding capabilities are located in a carbon dioxide band. However, the atmospheric concentration of this constituent has changed substantially since the 1970s. Consequently, the study has to assume concentrations that correspond to that time. This is achieved by using the capability of RTTOV to use as input predictor a profile of carbon dioxide. One issue is that the current minimum carbon dioxide concentration acceptable by RTTOV at the surface is 372 ppmv, which is larger than the concentration in 1970s. However, RTTOV is able to manage user input profiles with concentrations lower than the training dataset. The present study assumes a CO 2 concentration of 325 ppmv for IRIS in  1970 -1971 and 331 ppmv for HIRS in 1975 -1976 (NOAA/ESRL, 2018).
The apodization function is calculated for 51,200 samples, with a resolution of 0. 195 lm in order to obtain a final spectral resolution of 0. 01 cm À1. The ISRF-1C is defined over the range [À16, 16] cm À1 , with a full width at half maximum of 2. 50 cm À1. The same ISRF-1C is used for all 862 IRIS channels. The ISRF-1C is applied to the line-by-line transmittances of the RTTOV training set with variable carbon dioxide as indicated above. The Line-By-Line Radiative Transfer Model (LBLRTM,  Clough et al. , 2005)  v12. 2 and the Atmospheric and Environmental Research (AER) v3. 2 spectroscopic database are used to build the line-by-line transmittance database. The training set is designed for the thermal infrared in the range 3-50 lm, with margins in the line-by-line calculations (wavenumber range 150-3350 cm À1 ), well outside of the spectral range of IRIS.
Proposed common sub-space in the temporal domain In the temporal domain, the two missions did not overlap in time (see Tables  2  and 3 ). One solution is to consider the month of August, which was well observed by both missions, albeit for two different years 1970 for Nimbus-4, and 1975 for Nimbus-6. Some data for the months of January and September were also collected by both missions, but in unequal amounts, as Nimbus-6 HIRS only operated for a very short time in each of these two months.
However, the differences in instantaneous FOV (IFOV) angles can be corrected for. The IRIS IFOV angle is 5. 00°whereas the HIRS IFOV angle is 1. 24°. Fig. 4  shows a simulated IRIS FOV at the center of 4 HIRS FOV. Consequently, IRIS-equivalent FOV can be formed as follows, from HIRS observation (or simulation) data: the (observation or radiative transfer simulation) data from 4 HIRS central FOVs, i. e. , scan positions 20 and 21 in two consecutive scan lines, are averaged to form the equivalent of one (observation or radiative transfer simulation) IRIS FOV. In summary, a common sub-space is used to compare IRIS and HIRS, by projecting to the common denominator in the spectral, temporal, and horizontal domains. In that sub-space: 1. There are only 7 HIRS-equivalent channels (numbers 2 to 8).
The IRIS (high-spectral resolution) data are averaged to form HIRS-equivalent channels. 2. There are only observations for the month of August, for days of the month between 17 and 31. 3. There are only nadir-pointing observations, at latitudes between 80°S and 80°N. Four HIRS (high-horizontal resolution) FOVs at nadir are averaged to form one IRIS-equivalent FOV.
Cloud detection for IRIS Fig. 5a  shows differences (departures) between Nimbus-4 IRIS brightness temperatures and the simulations from ERA-20C. Near the center of the 15 lm CO 2 band, the departures are more clustered around zero, but otherwise for most channels one observes the effect of clouds, with negative brightness temperature departures down to À100 K. The NWP-SAF Aerosol and Cloud Detection Software  (Eresmaa, 2015)  is applied to detect scenes where the infrared observations are affected by clouds. This software implements the cloud detection scheme of  McNally and Watts (2003). Channels are first ranked in the vertical, according to transmission. The scheme processes departures of the observations from brightness temperatures simulated from ERA-20C. For this study, the cloud detection code is modified to handle the IRIS instrument as follows.
Mean of reanalysis departures from IRIS Using the results returned by the cloud detection based on ERA-40, Fig. 7a  shows the mean of reanalysis departures from IRIS (differences between IRIS observations and reanalysis simulations). The bias mentioned in Fig. 5b  is now more visible, and the observations appear cooler by about 1 K than the radiative transfer simulations from both reanalyses for the window channels. The agreement between the two reanalyses, coupled with the fact that they are both much constrained for temperature near the surface by forcing sea-surface temperature datasets (based on observations), points to a perfectible cloud detection, or a perfectible calibration of the IRIS observation data, or both. Another possibility is that the wavenumbers actually covered by IRIS are not exactly as specified, and this could generate a small bias when converting from radiances to brightness temperatures.
An empirically-derived multiplicative correction for IRIS wavenumber scales was applied for example by  Harries et al. (2001) , when comparing IRIS to IMG, even if it is unclear which of the two instruments actually presented incorrect wavenumbers. The mean departures for the temperature sounding and window channels (15-13 lm) are generally small and agree rather well for both reanalyses. For the lower-tropospheric temperature sounding channels, the bias is closer to zero for both reanalyses. The two reanalyses agree well for these channels and for the window channels, even though the average departures are then negative (around À1 K). This agreement is to be expected as both reanalyses are constrained near the surface by similar Sea-Surface Temperature (SST) forcing datasets. Except near the center of the 15 lm band, the mean ERA-20C departures from IRIS are generally warmer than the mean ERA-40 departures from IRIS.
This is consistent  with the finding that ERA-20C presents cooler temperatures than ERA-40 in the upper troposphere. This is consistent with the cloud detection results discussed earlier. The differences between ERA-20C and ERA-40 departures for the higher-peaking channels suggest that the temperature bias of ERA-20C in the troposphere, as compared to ERA-40, is negative, and under 1 K. This result is consistent with those obtained by  Hersbach et al. (2017). They place the origin of this bias in the assimilating model, and postulate that the reanalysis upper-air temperature bias is reduced when sufficient amounts of upper-air and satellite observations are assimilated. Such data assimilation occurred in ERA-40 and would explain the results found in the present study, by comparing with IRIS.
Cloud detection for HIRS The cloud detection method for IRIS cannot be applied to HIRS because this latter instrument features too few channels. Conversely, the advanced cloud detection methods for HIRS are not available for IRIS. For example,  Wylie et al. (2005)  use the method of  Jackson and Bates (2001)  to form a HIRS cloud mask; the method relies on the spatio-temporal variance in the observations (within a given time window and within a given area, i. e. over several FOVs). This requires dense high spatial resolution observations, unavailable from IRIS. One looks however for a single method for cloud detection, in order to continue with the overall approach of analysis methods and criteria equally applicable to both IRIS and HIRS data records. One can compare a predefined threshold with the brightness temperature (BT) difference between a window channel (in the region of HIRS channel 8) and a background surface temperature, where surface emissivity is assumed to be rather constant, such as over open ocean  (Harries et al.
One solution consists in projecting the IRIS data and cloud detection results into the HIRS spectral space, so the threshold for cloud detection from HIRS-equivalent channel 8 BT differences can be found empirically based on the IRIS cloud detection. Another approach, which could be considered in future work, would be to tune the proxy cloud detection based on the results obtained by applying the HIRS cloud detection method of  Jackson and Bates (2001). First, the IRIS observations and simulations are projected to form HIRS-equivalent channels using Eq. (1). Second, the cloud detection results from IRIS are also projected into HIRS-equivalent channels, using the same equation, but replacing brightness temperatures by the cloud detection results (where 0 indicates clear and 1 indicates cloudy).
g. , unlike a monthly climatology). As mentioned earlier, a symmetric approach could be to assess the results with respect to a HIRSbased cloud detection method, by working with the HIRS data instead of the IRIS data. Fig. 8b  shows that the profiles considered as clear by the proxy cloud detection feature statistics comparable with the population of cases that are effectively free of cloud. This type of analysis is actually the basis used by  Harries et al. (2001)  to derive their maximum BT threshold for cloud detection; in the present case, we use such analysis instead as an indirect assessment of the proxy cloud detection method. This gives some confidence that the settings chosen for that proxy method do not affect adversely the departure statistics, to be interpreted later.
Projection in the common sub-space The IRIS and HIRS observations and radiative transfer simulations from reanalyses are projected into the common sub-space defined in Section 3. 4. This sub-space is a common geographical domain (single-track nadir, between latitudes 80°S and 80°N), at the IRIS FOV resolution, in a common spectral domain (HIRS channels 2 to 8), and at HIRS spectral resolution. The temporal restriction is not applied yet, in order to consider the largest amount possi-ble of data and investigate the percentage of clear scenes at each location. One considers only the scenes that were found to be clear by the proxy cloud detection defined in the earlier section. This proxy cloud detection is applied equally to HIRS data (projected at IRIS FOV resolution) and to IRIS data (projected into HIRS-equivalent channels).
Consequently, only extrapolar and clear ocean regions may be compared (latitudes 60°S to 60°N). The restriction to the temporal domain (to the month of August) is applied hereafter, to avoid aliasing of seasonal effects. The standard deviations within the observations for the selected scenes are shown in Fig. 10. These scenes are sea-ice-free ocean areas at latitudes between 60°S and 60°N in August 1970 and August 1975, found clear by the proxy cloud detection. The HIRS-equivalent channels are ordered from top-peaking (channel 2) to window (channel 8). For each channel, the variability is very similar in both data records, although the number of profiles differ (1363 for IRIS and 8938 for HIRS).
This seems to suggest that this assimilation of satellite and upper-air observations in ERA-40 is the cause for the change in the respective mean departures. The interpretations above can be verified using another source of information. When the new strategy for climate reanalysis was laid out when designing the ERA-CLIM project, the plan included the production of a climate model integration without data assimilation. This climate model integration, ERA-20CM  (Hersbach et al. , 2015) , was archived at low temporal resolution with the aim of supporting low-frequency data analysis (only monthly means). It is hence not possible to project the ERA-20CM into radiative transfer simulations without incurring a large error in the time representation. However, the ERA-20CM monthly means can still be useful to compare with the ERA-20C and ERA-40 monthly means.
Fig. 12  confirms that ERA-40 is quite warmer than ERA-20C in August 1970 in the troposphere. However, the differences change sign around the 20 hPa pressure level, which could explain the agreement for departures for channel 2 (cancelling effects). In August 1975, Fig. 12  the troposphere in ERA-40 is much warmer than ERA-20C. This warming is in line with earlier findings obtained from the analysis of IRIS and HIRS departures. These results provide confidence that it is possible to interpret mean differences between reanalysis and satellite data records, even if they are short, provided sufficient effort is made to restrict the comparison to a comparable sub-space.
This appears consistent with the instrument specifications  (Nimbus project, 1975). Conversely, when a given instrumental record is compared to two reanalyses which have similar resolutions, the latter two contributions to the brightness temperature departure standard deviations (instrument noise and representativeness error) should be similar between the two reanalyses. The comparison of departure standard deviations allows one to propose a ranking of the reanalysis datasets in terms of precision. The results in Fig. 13  suggest that ERA-40 has higher precision than ERA-20C, as standard deviations for ERA-40 departures are lower than for ERA-20 departures, both for IRIS (in line with Fig. 7 ) and for HIRS.
Conclusions Early satellite sounders such as Nimbus-4 IRIS and Nimbus-6 HIRS have collected unique data records of our atmosphere, at times when no other comparable resources are available. In this paper, advanced methods, unavailable at the time these instruments were flown, have been applied to simulate the observations in clear sky and to screen out cloudy scenes. The results shown in the paper demonstrate that new insight can be gained from these data by submitting them to detailed comparisons with radiative transfer simulations from reanalysis. Refined quality controls are indeed proposed, which should assist with future analysis of these data records. In addition, the paper proposes a method to project the observations into a common sub-space, to alleviate differences in FOV and spectral resolution.
It is then possible to derive meaningful conclusions about the systematic departures that are observed. Considering a continuous dataset as reference (a climate model integration) validates the results, with ERA-40 suspected to suffer from a spurious warming in the troposphere between 1970 and 1975 due to the introduction of more upper-air observations, including VTPR NOAA-4 affected by an erroneous bias correction  (Uppala et al. , 2005). The methods presented in the paper can be easily extended to other early data records. The pre-requisites are the availability of the data records in digital form, and the possibility to simulate the observations via knowledge of the instrument response functions and updates of fast radiative transfer codes. The method proposed for projection in a common subspace requires that the instruments observe overlapping regions of the spectrum, and that a common denominator can be found behind the viewing geometries (the effect of slant viewing would introduce differences difficult to analyze).
Acknowledgments The  NASA GES DISC  and  ECMWF  are thanked for providing data used in the present study. The authors would like to thank  James Johnson  ( ADNET Systems Inc. ) at  NASA GES DISC  for providing software for reading the Nimbus data,  Ronald Glumb  for providing the Nimbus-6 HIRS weighting functions, and  Reima Eresmaa ,  Anthony McNally , and  Marco Matricardi  ( ECMWF ) for providing guidance in using the  NWP-SAF Aerosol and Cloud Detection Software  with  IRIS  and for fruitful discussions. Meteo-France,  ECMWF , and the  NWP-SAF  are acknowledged for providing technical support that enabled this work to be conducted and finalized. Three anonymous reviewers and the journal editor are thanked for their comments that helped improve the manuscript.