dataesr/scientific-paragraphs-categorization huggingface dataset:
Data Scientists have long struggled with making long-term SSI observations. As  Schmidtke (2014)  describes in his historical overview of SSI measurements, the first attempts at obtaining SSI were not suitable for long-term recording of the variable solar spectrum. While E. Regener and V. H. Regener, in 1932, used balloons flying up to an altitude of 33 km to obtain a spectrum of the visible to the far ultraviolet (UVC, 200 nm-280 nm) region,  Baum et al. (1946, cited by Schmidtke)  used V2 rockets to obtain the first SSI measurements in the extreme ultraviolet (EUV) band. Though innovative, their methods, based on photographic film, were restricted by the necessity of instrumental retrieval. Therefore, they represented only a snapshot of solar activity. Del Zanna & Andretta (2011) collected an exhaustive list of rocket flights, measuring the EUV.
It was the development of photoelectric diodes in the 1960s that allowed for satellite-based long-term measurements. Using these diodes, the EUVS experiment aboard the AE-E satellite showed, for the first time in history, the variability of the EUV from 20 to 185 nm over the course of 3 years from a solar minimum in 1976 to a solar maximum in 1979  (Hinteregger 1981). This was the beginning of long-term SSI measurements. The first instrument observing a full solar cycle was UARS/SU-SIM (Upper atmosphere research satellite/Solar Ultraviolet Spectral Irradiance Monitor), 1991-2005  (Rottman et al. 1993). Starting with data from the AEE/EUVS experiment in 1967, we have collected and processed data from 26 instruments over a wavelength range from 4 nm to 2. 4 lm. Our space-based observations cover the time range of 1977-2015.
We also provide nine reference spectra, with our first spectrum by  Arvesen et al. (1969)  and our latest one by  Woods et al. (2009) , as well as 14 proxies of solar activity. We utilize these proxies to enhance instrumental data by detection of possible outliers  (Sect. 4. 4)  and by interpolation of missing data (Sect. 4. 5). Figure  1  presents an overview of all instruments, reference spectra, and proxies available in the SOLID database. Not all existing data have been included, either due to bad quality or missing availability. Notably, the Airglow Solar Spectrometer Instrument (ASSI) data from the San Marco mission and the EUVS from the Orbiting Solar Observatory (OSO) III mission are not included. Other data, such as the broadband filter radiometer LYRA on PROBA2, will be used to verify the composite.
To establish a common data format, we follow the Net Common Data Format (NetCDF) Climate Forecast standard, version 1. 6, 2  which provides standardized variable names, well-defined field names for global meta-information such as author, data source, and creation date, meta-information for each variable such as units, flag description for masks, and cross-references to other variables. In Appendix A, we present an overview of all the data together with references, sources, and key information such as time spectral ranges below in tabular form. Table  A. 1 lists all instruments included in the SOLID database, Table  A. 3 contains all nine reference spectra, Table  A. 4 displays all proxies, and Table A. 5 presents the three composite time series in the database. Two of the composite time series, the  Woods et al.
(2000)  Lyman a composite and the  Fröhlich (2006)  TSI composite, are also used as proxies for SSI. The third composite, the aforementioned DeLand SSI composite has been included in the database, but will not be used for the construction of our composite. Instead we will use the original instrumental data. Altogether, we currently have 53 datasets in the database. Any new dataset incorporated into SOLID will also be made available at the SOLID homepage.
Stability, here, describes a time-dependent estimate   A. 1 (Instruments), A. 3 (References), A. 4 (Proxies), and A. 5 (Composite datasets) representing almost all the datasets incorporated in SOLID versus time and wavelength. The upper panel presents all available proxies, together with two spectral composites. The lower blue line, labeled Radio, represents the 3. 2, 8, 10. 7, 15, and 30 cm radio fluxes from the Nobeyama and Penticton radioobservatories. The time range shown corresponds to the available times in the SOLID database, which truncates all proxies but the radio fluxes at 1969. The lower panel presents the temporal and spectral ranges for all instruments in the database. Vertical lines represent the time point and spectral range of the reference spectra. Instruments with fewer than 20 channels are shown as light boxes with a horizontal line at the wavelength of the central frequency of each channel.
While the 21st century is well covered both in temporal and spectral dimensions, previous data are only available for selected wavelengths. Only the UV between 170 nm and 400 nm has been covered continuously since 8 November 1978. Fig. 2. Instrument uncertainties for different TSI instruments. They differ by up to three orders of magnitude with the highest uncertainties for a modern instrument, TIM. This is due to different definitions used for what an instrumental uncertainty is. For that reason, these values cannot be meaningfully compared. of the instrumental drift, in other words, a trend; accuracy stands for the offset from the ''true zero'' at the beginning of the mission, i. e. the bias of the absolute value. As the same concept often comes with different names, we provide our own naming convention in Table  1.
Since uncertainty estimates given by data providers vary enormously in definition and quality, it is necessary to provide an independent and homogeneous uncertainty estimate, which can be meaningfully compared between instruments. We provide two such estimates for precision and stability. There is no way to independently assess accuracy without resorting to external data, except as to refer to the absolute ground and on-board calibration. Therefore, only instrument teams can provide these estimates and we do not attempt to estimate them independently. If they are provided, we include them in the SOLID database. As discussed above, uncertainty estimates provided by the PI (principal investigator) cannot be directly compared with each other. However, they do contain valuable information, such as satellite off-pointing or different integration time, and as such, they are also provided.
Each PI presents these estimates differently: They can be provided as absolute or relative values for each data-point, as time-or wavelength-independent fractional values or percentages or as a global estimate. We considered two possibilities: either the uncertainty estimates are provided for each data-point of the time series, in which case we convert them (if needed) into absolute values, or the provided uncertainties are time-or wavelengths-independent, in which case they are expressed in fraction of the data. In the latter case, we did not multiply the uncertainty by each data-point (which would have provided an uncertainty estimate in the same format as in the first case) to reduce the file size. This results in the somewhat cumbersome situation that the uncertainties may differ in units and dimensions from one dataset to the other: in some cases, they have the same units as the timeseries, in other cases, they are unit-less fractions.
The weights are determined empirically such that they provide a best fit over b 2 ½0; 2. This does not only estimate the white noise level correctly, but also blue and violet noise levels which may be present in the case of a positive feedback loop. However, we are not aware of a reliable method to determine Brownian noise level without resorting to external data. As such, we cannot take into account possible Brownian contribution, despite the fact that most observations are likely to have a negative b. Therefore, for consistency, and although we know that this is not optimal, we assume our data to be free from Brownian noise and use our adapted Donoho estimator as a systematic means for comparison. Our stability estimator does use external data in the form of proxies. Hence, the red noise can be estimated as part of our stability estimate.
Name Synonyms As for stability, that is the uncertainty on the drift, the problem is a bit different. Instruments that have on-board monitoring systems (using calibration lamps or star observations) can provide an estimate of the drift with its uncertainty. Not all datasets, however, contain such stability estimates. Furthermore, the stability estimates for some instruments rely on assumptions based on the correlation between irradiance and solar proxies (e. g. the SBUV datasets). Although we cannot provide a completely independent estimate of the stability, there is a need for a common and homogeneous stability estimate for each dataset. The estimation of the stability (i. e. longterm uncertainties) is a very challenging task and can only be formally done through instrument intercomparison.
Methods Here we present the pipeline for the data preparation for each individual dataset, providing methods that can be applied to all datasets and uncertainty estimations that allow different datasets to be compared in a meaningful way. Our pipeline approach entails the following steps: pre-processing, interpolation of missing data, re-gridding in time, outlier detection, and, finally, a proxy-based interpolation of erroneous or missing data. We provide uncertainty estimates for each step in the processing pipeline. Figure  5  presents a graphical overview of all steps to construct the final composite, marking all steps discussed in this paper. Appendix B includes a complete and formal description of the data format, while Appendix C provides the formal description of all methods in meta-code.
This work discusses the construction of the homogeneous individual datasets (green, numbers link to the corresponding sections), while the stability estimation (magenta) is discussed in a separate paper. The combination of all these datasets will be provided in the future (blue). binary flags. The numeric value of each flag is a unique power of two -1, 2, 4, etc. -and the entry on the quality map is the sum of all the values of the present flags. For example, a value of 12 represents two flags, corresponding to the values 4 and 8. Any processing applied to the data is indicated in the quality map.
Interpolation of missing data Most instruments incorporated in SOLID feature missing data with notable exceptions being SME/UV and SNOE/SXP. Missing data result from various problems, including faulty transmissions, eclipses, temporary device failures, and outages when the satellite is put in safe mode. Whatever its origin, we need to systematically deal with it for each instrument. To reconstruct missing data, we rely on the high coherency of spectral solar variability. This coherency implies a strong redundancy of the observed variability, in the sense that temporal variations in a given spectral band can be adequately reconstructed by linear combination of typically 1-5 time series. This property has been exploited by Dudok de Wit (2011) to build an interpolation scheme that relies on Singular Value Decomposition (SVD,  Kalman 1996).
While most data gaps can easily be interpolated due to neighboring channels providing data, gaps due to instrumental outage, which affect the whole instrument, are also flagged as missing and temporarily interpolated linearly. For the final data product, we use proxies to provide the temporal information. However at this step we refrain from inducing any external data into the dataset. The error induced by this method is estimated via bootstrapping. Table  A. 2 presents an overview of all data adjustments, including the interpolation of data. These adjustments result in a dataset without any missing value on a two-dimensional time-wavelength grid with accompanying uncertainties.
Re-gridding in time This section describes our regridding of the datasets onto a regular time grid. This procedure is necessary for several reasons. First, the autoregressive model utilized in our outlier detection, described in Section 4. 4, requires a regular time grid. Also, a regular time grid is helpful to directly compare data with each other and to apply multiscale decomposition methods, as planned for future work. Luckily, most datasets are already available with daily resolution, centered at noon. Only three datasets, namely AEE/ EUV, ENVISAT/SCIAMACHY, and ISS/SolACES, need to be adjusted in time. Another dataset, NIMBUS7/SBUV, has multiple data values for the same time. We take the average of these and adjust the uncertainties by calculating the standard deviation. Table  2  provides the statistics of these datasets.
3 e re-grid these datasets to a constant time grid of daily resolution centered at 12:00 UTC via a linear interpolation scheme and estimate the interpolation-induced uncertainty by a linear error regression of the variability. For this, we assume an independent error propagation, and, since the wavelet noise estimator is an accurate short-term noise estimate, we use the wavelet estimator of the original nearest neighbor as the nonweighted starting point. This estimate is weighted by the distance of the re-gridding. A proxy-based interpolation scheme is used for the final interpolation (Sect. 4. 5); linear interpolation is used in intermediate steps for its cheap computational costs and simple statistical properties. This procedure presents a classic chicken-and-egg problem. While the autoregressive model, utilized in our outlier detection, requires a regular time grid, the regularization of this time grid, based on interpolation, dilutes precisely these outliers.
Outlier detection The following section describes our outlier detection routine based on an autoregressive model (AR,  Chatfield 2003). Figure  8  provides a quick overview of this procedure, consisting of the aforementioned AR model together with a proxybased quality estimator to differentiate physical outliers from instrumental artifacts. Before addressing the procedure above in greater detail, it is necessary to discuss the nature of outliers. Having removed Fig. 6. UARS/SUSIM at 239. 5 nm as provided by the instrument team and the 16 r thresholds (dashed). All points outside of the thresholds are marked as outliers, replaced by linear interpolation. Further possible outliers are selected and handled in Section 4. 4. We define their uncertainties as the difference from the original data to the interpolated points.
In the case of SUSIM, these outliers occur directly after instrumental data gaps, giving further credence to the possibility of instrumental, non-physical, outliers. the most extreme outliers during pre-treatment (Sect. 4. 1), we now aim at distinguishing instrumental artifacts from peculiar data corresponding to actual physical phenomena such as solar flares and huge sunspots. As they may have the same magnitude, they cannot be distinguished by a simple threshold. We also want to include sunspots but flag solar flares due to their short time spans, and, as such, their inclusion into each instrumental dataset strongly depends on internal characteristics like integration time and data selection, i. e. whether, for example, averaged daily data or a single daily measurement has been provided.
Yet solar flares, as another type of physical outliers, are removed from our datasets, as our solar proxies do not contain Fig. 7. Coherency interpolation at different points in time for the 144 nm channel of SORCE SOLSTICE A, which contains 65 channels in total (115 nm-179 nm). Each color represents a reconstruction of the 144 nm channel using a sequence of datasets of which some contain gaps for the days marked by a black horizontal line that were filled using the SVD interpolation. We show the original data (solid thick black), interpolated data where only this channel has been removed and re-interpolated (long-short dashed, blue), data where not only the 144 nm channel has been removed, but also its lower and upper 10 channels, i. e. everything from 134 nm to 154 nm (short-dashed green), and data where everything but the outermost channels (the two channels at 115 nm and 179 nm) have been removed for the selected days (long-dashed red).
Second, to account for the possibility of outliers being of physical origin, the parameter n is increased by 2 if at least 70% of our solar proxies are also flagged as outliers. Third, all such flagged data, as well as data that have been modified in a previous method, are temporarily replaced by linear interpolation. This procedure is repeated until convergence is reached. The linear interpolation is replaced by a more advanced scheme in a later step. Typically, this process converges within three steps, but the number of steps necessary for convergence increases when long data gaps have been interpolated in a previous step. We have empirically determined the required parameters, that is n, the increase by 2, and the percentage threshold of the proxies. Finally, we convert the replaced data into uncertainties while keeping the original data in place, with the final step, coherency interpolation, only applied after all processing is finished.
This is described below in Section 4. 5. The estimated uncertainty induced by outliers is defined as the absolute difference between the original and predicted data. Applying the outlier detection routine as discussed above to our datasets yields results which we found to be physically consistent. We present an application of our routine to SOHO-SEM in Figure  10. As this figure shows, after the removal of the most extreme outliers, our method still detects not only several obvious instrumental artifacts, but also data-points that are below any meaningful direct threshold. In some cases, as in the case of SUSIM (Fig. 6 ), they appear before a data gap, giving further credence that they are due to instrumental artifacts. Our routine also correctly flags all linearly interpolated data as non-physical.
Final interpolation of missing data Finally, we interpolate all data that have been flagged using the coherency interpolation scheme as described above in Section 4. 2, this time including proxies as additional channels. While this interweaves instrumental data with proxy data for all data-points that are missing from the original dataset, it attains temporal variability during times of instrumental outages when no channel has data. 9. A sample of the three provided levels of UARS/SUSIM, the original data, named level 0 (a), re-gridded and missing data interpolated as level 1 (b), and our best-estimate, level 2, where outliers have been removed and re-interpolated (c). One solar proxy, Mg II, is shown for comparison (black dots). The error bars represent our estimate of the precision, including the induced uncertainty of each method.
Also shown are the values of all set flags. For example, a value of 24 means that bits 3 (2 3 ) and 4 (2 4 ) are set, marking this data-point as both being detected as an outlier (8) and re-interpolated in the final interpolation (16). Fig. 8. Flowchart of the outlier detection with the possibility of physical outliers. First, we construct an autoregressive model of the time series. Then we flag all values whose difference to the model is greater than a given threshold n as possible outlier candidates. To account for the possibility of physical outliers, we relax this condition if proxies flag the same data as outliers.
Results Based upon these methods, we provide three kinds of datasets: first, the original data converted into the common data format; second, the original data accompanied by uncertainty estimates and interpolated data; and, third, a ''best-estimate''data product with possible outliers replaced by proxy interpolation. Figure  9  displays these three datasets, together with the accompanying flags and precision estimation. As our database now contains three different types of datasets, the question ''Which one is best for a given purpose?'' naturally arises. This depends largely on the chosen method, as the subsequent examples illustrate. As a simple least-square fit can be dominated by outliers, it is of vital importance that instrumental artifacts be removed. Thus, the third type of dataset should be used for least-square fitting.
4 However, if a weighted leastsquare fit or, more generally, any method that takes into account the uncertainties of data is used, the second type of dataset is more appropriate. The first type of dataset serves primarily as a reference and starting point, as it does not include independent uncertainty estimates. All datasets are available at ftp://www. pmodwrc. ch/pub/ projects/solid/database. The database will be updated regularly as new data become available, including modeled data, which will complete temporal and spectral coverage. Using standardized methods to homogenize the data makes the inclusion of new data, or new versions of already available data, easily achievable.
Conclusion As part of SOLID, we provide homogeneous datasets of available space-based SSI measurements with uncertainty estimates and accompanying meta-data in a unified format, following standard conventions. Several challenges, however, remain. Even though we have put much effort into developing an independent and homogeneous uncertainty estimate, we do recognize that we have not yet solved the problem of combining these with the PI provided uncertainty estimates. As for now, we provide both sets of estimates, and it is up to the user to combine them. Furthermore, our methods are designed to be generic and applicable to all instruments incorporated in SOLID, but consequently, they do not reflect all properties specific to each instrument. Some datasets, for instance, are accompanied by flags describing the origin, the processing method, and the quality of the data or the number of measurements per datapoint.
Though provided in our database, these flags are not part of our processing. Notwithstanding these challenges, our datasets are a starting point to merge all available data into one single homogeneous dataset. A follow-up paper will present a new approach to create such an SSI composite, which will also eventually be available at the above address. In any case, the present database will help to provide an SSI reconstruction of the satellite era, and, thereby, it will foster diverse research in solar science, space weather, and climate studies. Acknowledgements. The authors acknowledge that the research leading to these results has received funding from the European Community's Seventh Framework Programme (FP7 2012) under Grant Agreement No. 313188 (SOLID, http://projects. pmodwrc. ch/solid).
The authors would like to thank all data providers who provided their data on-line, David Bolsée for the SOLSPEC data, Robert Schäfer for SolACES data, Gérard Thuillier for the ATLAS reference spectra, and Marty Snow for the helpful discussions during the SOLID meetings in Bremen and London. This research has made use of NASA's Astrophysics Data System. The editor thanks two anonymous referees for their assistance in evaluating this paper. Rottman et al. (1993); Floyd et al. (2003) , http://wwwsolar. nrl. navy. mil/uars/v22/ Table  A2. An overview of the amount of data adjusted by each method. Each flag corresponds to a specific method: interpolation of missing data (1, Sect. 4. 2), averaging multiple data-points (2, Sect. 4. 3), outlier detection (3, Sect. 4. 4), and proxy interpolation (4, Sect.
B. 2. Dimensions All our datasets have two dimensions, time and wavelength. For ease of use and generic application, the same scheme has also been applied to proxy data. In these cases, the wavelength dimension is the number of proxies in the data file and the wavelength entries are negative indices, that is À1 for the first proxy, À2 for the second, and so on. The names of the proxies are stored in the field proxy_names. time The number of distinct time points. wavelength The number of distinct wavelengths in the data file.