dataesr/scientific-paragraphs-categorization huggingface dataset:
, 2017)  with 0. 53 • beam width. The properties and settings of both radars are summarized in Table 2. As preprocessing steps, the radars are cross-calibrated and an attenuation correction is implemented at the W band (similarly to  Kneifel et al. , 2015) , as detailed in Appendix A. Then, the spectrograms of both radars are remapped to a common grid, by averaging in time (with a resolution of 20 s), interpolating in range (resolution of 50 m), and average-binning the velocity to the same bins as the synthetic dataset, i. e. , with 256 bins and a velocity cutoff v Nyq = 6. 92 m s -1. Only time frames with detectable signal in both frequencies are used, leading to a total of ∼ 9000 profiles corresponding to around 50 h of measurements, collected between 16 and 28 January.
Aircraft in situ measurements In addition to the ground-based measurements, the ICE GENESIS campaign included scientific aircraft overpasses with remote-sensing and in situ instruments. The airborne in situ data are particularly valuable for the quantitative evaluation of the microphysical retrieval, presented in Sect. 5. Airborne measurements used in this work were collected during three flights of the Safire ATR 42 (22, 23, and 27 January) as the aircraft was performing overpasses over the ground site, with data from several probes. First, the counterflow virtual impactor (CVI,  Anderson et al. , 1994; Schwarzenboeck et al. , 2000)  provides a measurement of total water content (TWC), and the Cloud Droplet Probe (CDP) provides a measurement of liquid water content (LWC); from those measurements, an estimate of IWC can be obtained as IWC = TWC -LWC.
The decoder: a differentiable emulator of PAMTRA The first part of the framework consists in developing a differentiable emulator of PAMTRA by designing a deeplearning model and training it on the synthetic dataset (Sect. 3. 1). If viewed in perspective with the technique of auto-encoders, this consists in learning the decoder, which maps the latent space -containing the physical variablesto the high-dimensional measurement space -the spectra. It was chosen to train the X-band and W-band decoders separately rather than use a single algorithm emulating both simultaneously; indeed, the two frequencies may have slightly different smoothness or amplitude features, which justifies the use of distinct architectures. Each decoder takes as input a vector of dimension 10 containing IWC, D 0 , b m , a m , β a , α a , A r , Turb F , Wind F , and Lnoise F , where F is either X or W.
Decoder training The synthetic dataset described in Sect. 3. 1 is split into training, validation, and testing sets (80 %-10 %-10 %). The NN is trained on the training set, while the validation set is used to tune the architecture of the NN, and the testing set is used for a final assessment of its performance (Sect. 5. 1. 1). The input is normalized using the statistics of the training dataset, with the mean and standard deviation of each variable. For certain variables, the natural logarithm is used instead of the original value, in order to have more homogeneously spread distributions: this is the case for IWC, a m , and α a. The network is trained using the Adam optimizer  (Kingma and Ba, 2015) , with mean square error (MSE) as a loss metric and with Xavier normal initialization of weights and biases (Glorot and Bengio, 2010); in addition, the learning rate is periodically decreased with a scheduler.
Encoder training The encoder is trained using the radar data presented in Sect. 3. 2. The spectrograms are normalized using the same statistics as in the decoder part (means and standard deviations of the synthetic spectra). Rather than using the entire Doppler spectrograms (N rg = 100) as one training item, chunks of the spectrograms are used with N rg = 25 and are sampled in the following way: the first chunk corresponds to i rg = 0. 24, the second chunk to i rg = 5. 29, the third to i rg = 10. 34, etc. , with i rg being the range gate index. The dataset is then randomly shuffled at each epoch during training. Rearranging the dataset in this way makes training both more tractable, thanks to the smaller size of each item, and more robust, due to the data augmentation, which helps avoid local minima during the training process.
Decoder The training of the decoder networks was successful, with the loss function decreasing with the number of epochs until it plateaus. It was verified that increasing the training set size did not result in a change of this plateau value, meaning that the training dataset was large enough for the chosen NN complexity. Examples of model outputs on the synthetic testing set are shown in Fig. 3. Since the loss function (MSE on normalized spectra) is not easily interpretable to assess the model's performance, the , where S * = Smin(S) and S * = Smin(S); O(S, S) is equal to 1 (or 100 %) when the spectra are perfectly identical and to 0 if they are disjoint. A detailed illustration of this metric is provided in Appendix D. On the synthetic testing set, the overlap for the X band (respectively W band) is of 90.
When looking only at spectrograms with moderate apparent turbulence (e. g. , 0. 5-1. 5 km) and strict unimodality, the overlap metrics are similar to the decoder. Finally, it is also empirically observed that in some cases (Fig. 4e ) the model can slightly underestimate the peak of the spectrograms, which is a propagation of the decoder behavior. As a safety check, it was also verified that running PAMTRA on the latent variables also led to spectrograms close to the original ones, as displayed in Fig. 4c. Let us point out at this stage that unlike most machinelearning models, which are trained on a dedicated dataset and then implemented on independent data, the encoder is https://doi. org/10. 5194/amt-16-911-2023 Atmos. Meas. Tech. , 16, 911-940, 2023 here trained directly on the data of interest.
Indeed, the aim is not to create a generic model that can be used to retrieve microphysical variables from any dual-frequency spectrogram: the aim is to find the latent variables which minimize the reconstruction error on specific measurements; the encoder can learn any relevant feature from the input data to achieve this goal. In that sense, overfitting the data, which can be an issue in usual machine-learning problems, is not a concern when training the encoder. It is, however, preferable to train the model on a large enough dataset rather than just a few spectrograms: this reduces the risk of converging toward local minima that would correspond to non-physical combinations of microphysical parameters. 5. 2 Qualitative assessment of the retrieval.
Sensitivity to miscalibration and differential attenuation One limitation of our framework is that it requires a good calibration of the radars -both absolute and relative -as well as an independent correction of attenuation. As detailed in Appendix A, the issue of attenuation was here tackled by implementing a correction of W-band reflectivity based on estimates of gaseous, snowfall, and liquid water attenuation. This correction method is, however, error-prone, and we cannot exclude that reflectivity biases are present in the measurement dataset. The presence of supercooled liquid water cloud layers or wet snow can be particularly difficult to identify and diagnose, while it can strongly attenuate the millimeterwavelength signal (with, e. g. , path-integrated attenuation up to 5 dB for liquid water paths of 500 g m -2 ;  Kneifel et al.
Training set limitations Another aspect of our framework which could cause a bias in the retrieval is if the training set is too narrow. While special attention was paid to this potential issue as the microphysical parameters were sampled from the MASC database, there is likely still room for improvement. In particular, the size cutoff for good-quality images in the MASC is quite high, and very few particles with a diameter below 0. 5 mm are accurately captured. For reference, Fig. 13  illustrates the histogram of D 0 derived from MASC measurements  (Grazioli et al. , 2022)  and by the aircraft 2D-S and PIP probes during the ICE GENESIS campaign. The limitation is apparent: the aircraft is able to capture much smaller particles but not beyond a certain size, while the MASC can detect large snowflakes but very few small particles.
Scattering model Surely one of the strongest hypotheses on which the pipeline was built is the parameterization of the scattering model in forward simulations. As explained in Sect. online documentation;  Mech et al. , 2023). Several studies suggest, however, that more parameters are needed and moreover that their values can vary significantly (e. g. ,  Leinonen et al. , 2018a; Ori et al. , 2020)  depending on particle type and shape, among others. To get an empirical sense of how this could affect the retrieval, the approach described hereafter was followed. First, a few time and range gates were randomly selected from the dataset. The corresponding retrieved values were then modified by adding a D 0 offset ranging from -1. 5 to +2 mm, and PAMTRA simulations were run on the microphysical parameters obtained, using the same settings as in Sect.
3. 1. 1. Parallel to that, slight modifications of the PAMTRA code were conducted to allow for modification of the four literature coefficients of the self-similar Rayleigh-Gans approximation (κ SSRG , β SSRG , γ SSRG , ζ SSRG ); new simulations were run for the selected time and range gates, keeping the retrieved microphysics unchanged but randomly changing the SSRGA parameters within ±10 % of their original values. As seen in Appendix B2, this is well within the typical variability of the coefficients calculated from simulating various types of particles (e. g. ,  Leinonen et al. , 2018a, Fig. 5). A visual inspection suggests that changes in D 0 and changes in SSRGA coefficients mostly affected the amplitude of the spectra: hence, the influence of these changes was measured by the change in the scalar total reflectivity at the X and W band and then in the dual-frequency ratio.
In addition to these important hypotheses -SSRGA scattering model and assumption of exponential size distributions -we recall that other modeling choices were made during the design of the synthetic dataset and the underlying physical framework, such as assumptions on particle orientation and velocity-size relation, among others (see  Sect. 3. 1. 3) , which are inevitably a simplification of the physical reality and may thus also influence the retrieval. Another point should be briefly mentioned regarding small particles, which are Rayleigh scatterers at both the X band and W band. This means that if a population is composed entirely of small particles, the influence of particle size and number concentration is hardly distinguishable in the spectrograms. The ill-posedness of the problem is reinforced, and the retrieval could be expected to have a reduced accuracy, even if the training set and scattering model were improved.
59 (with this direct inversion). Similar behaviors are observed for the other retrieved variables. Let us mention an alternative approach that could be used, which lies halfway between classical OE and the proposed method. The notations used here are those of Sect. 2. Once a differentiable approximation of the forward model is known ( f ), another way to look for X r is to find the minimizing argument of F (X) -Y r 2 using gradient descent; a regularizing term can be added to ensure, for instance, the spatial continuity of X or to enforce some degree of spatiotemporal smoothness. This requires only one deep-learning model instead of two and could thus seem more appealing, but the first approach was preferred. Indeed, by actually learning an approximation of the inverse mapping G and doing so on a large dataset, the risk of reaching a local minimum in X is reduced.
A2 W-band attenuation correction Once the X-band data are considered reliable enough, we focus on the correction of W-band attenuation by following the method described in  Kneifel et al. (2015). -Gaseous attenuation. Atmospheric profiles are taken from COSMO-1 analyses (Consortium for Small-scale Modeling, 2017), and the corresponding profile of gaseous attenuation is computed using PAMTRA. -Snowfall attenuation. We use a baseline Z e,X -IWC relation (IWC = 0. 015 Z 0. 44 e,X ; linear units;  Kneifel et al. , 2015; Boudala et al. , 2006)  to estimate the profile of snowfall content, and the corresponding attenuation profile is obtained considering that ice attenuates around 0. 9 dB km -1 (g m -3 ) -1. -Supercooled liquid water attenuation. This is the mosterror-prone step, considering that no measurements of the profile of liquid water content (LWC) are available.
We make use of liquid water path (LWP) estimates based on radiometer measurements (Billault-Roux and Berne, 2021) and assume a uniform LWC profile in the cloud/precipitation column. The corresponding attenuation profile is then computed with PAMTRA. The mean path-integrated attenuation for each of these categories is respectively 0. 3, 0. 4, and 1. 7 dB on the entire dataset. W-band reflectivity and Doppler spectrograms are then corrected using these attenuation profiles. cloud top and with low reflectivity and by correcting WProf with the mean reflectivity offset in these regions (regions within 1 km of cloud top and Z e,X < -3 dBZ were used; if lower Z e values were to be used, not enough points would be available). This relies on the assumption that near cloud top, small ice crystals (with low reflectivity) are Rayleigh scatterers at both frequencies, for which the DFR is 0 dB.
A further note should be added regarding one event in the dataset (22 January) which featured rain at ground level during a few hours before a transition to snowfall in the evening. Here, step 3 of the method described could not be conducted because LWP retrieval is dominated by the lowerlevel rain. The other steps were performed similarly, and only data above the melting layer were used for the retrieval. Overall, this allows us to mitigate attenuation-related issues but cannot eliminate them. In particular, the presence of supercooled liquid water is difficult to assess and correct accurately. This should be considered as a limitation of our method at its current stage, and it motivated the discussion in Sect. 6. 1 about the sensitivity of the retrieval to reflectivity offsets.
Acknowledgements. Airborne data were obtained using the aircraft managed by Safire, the  French facility  for airborne research, an infrastructure of the  French National Centre for Scientific Research (CNRS) ,  Météo-France , and the  French National Centre for Space Studies (CNES). Most of the microphysical in situ data were collected using instruments from the  French Airborne Measurement Platform , a facility partially funded by the  National Institute of Sciences of the Universe (INSU)  of  CNRS  and  CNES. We thank  Davide Ori  for his help in the initial parameterization of PAMTRA, and we are grateful to  Julien Delanoë  and  Susana Jorquera  for providing the ice water content retrieved from airborne RASTA radar measurements. Finally, we thank  Stefan Kneifel  and  Florian Ewald  for their constructive and helpful comments on the manuscript.
Code availability. The code developed in this study for generating the training set using PAMTRA and for training the decoder and encoder models is available at https://doi. org/10. 5281/zenodo. 7638756 (Billault-Roux, 2023; latest version is available at: https://github. com/annecbroux/ DeepSpectralRetrieval, last access: 15 February 2023). Data availability. The data of the ICE GENESIS campaign are available on the AERIS platform (https://ice-genesis. aeris-data. fr/ catalogue/, last access: 13 February 2023;  Billault-Roux et al. , 2023).
Introduction The availability of data used in research is one of the cornerstones of open science, which contributes to improving the quality, reproducibility, and impact of the findings. In addition, data sharing increases openness and transparent and collaborative scientific practices. The global push for open science is exemplified by the recent publication of UNESCO guidelines  [1] , the engagement of many research institutions, and the requirements of some scientific journals to make data available upon publication. Finally, the sharing and re-use of data also maximizes the return on investment of the agencies (e. g. , states, charities, associations) that fund the data collection. In light of this, our chapter aims at providing a broad (albeit partial) overview of some of the human datasets publicly available to researchers.
To assist researchers and data managers, we first describe the different file formats and the size of the different data types (see Table  1 ). As many of these data are high-dimensional, the size of the data can cause storage and computational challenges, which need to be anticipated before download and analysis. Of note, some datasets cannot be downloaded or analyzed outside of a dedicated system/server. This is the case of the UK Biobank (UKB) exome and whole genome sequencing, whose sheer size has led to the creation of a dedicated Research Analysis Platform, accessible (at some cost) by UKB-approved researchers. In addition, the Swedish registry data is only accessible via national dedicated servers due to the extreme sensitive nature of the data. This chapter breaks down into sections that focus on each data type, although the same dataset may be mentioned in several sections.
Beyond a practical writing advantage (each author or group of authors contributed a section), this also reflects the fact that most datasets are organized around a central data type. For example, the ADNI (Alzheimer's disease Neuroimaging Initiative) focuses on brain imaging and later included genotyping information. Another example is the UKB, which released genotyping data of the 500 K participants in 2017, is now collecting brain MRI (as well as cardiac and abdominal MRI, whole-body DXA, and carotid ultrasound), and has recently made available sequencing data. The different sections also discuss and present the specific data sharing tools and portals (e. g. , LONI for brain imaging, GTEx for gene expression) or organization of the different fields (e. g. , consortia in  We hope this overview could be useful to the readers wanting to replicate findings, maximize sample size and statistical power, develop and apply methods that utilize multi-level data, or even select the most relevant dataset to tackle a research question.
Functional MRI measures BOLD (blood oxygen level dependent) signal, which is thought to measure dynamic oxygen consumption in the different brain regions. Of note, fMRI consists of a series of 3D images acquired over time (typically 5-10 min). Brain MRI is available as a series of DICOM files (brain slices, traditional format of the MRI machines) or a single NIfTI (single 3D image) format (see Table  1 ). The two formats are roughly equivalent, and most image processing pipelines allow both data sources as input. MR images are composed of voxels (3D pixel), and their size (e. g. , 1 × 1 × 1 mm) corresponds to the image resolution. In practice, most MR images are archived and shared via web-based applications and more rarely using specific software (e. g. , UKB). The two major web platforms are XNAT (eXtensible Neuroimaging Archive Toolkit)  [9] , an open-source platform developed by the Neuroinformatics Research Group of the Washington University School of Medicine (Missouri, (1, 2)), and IDA (Image and Data Archive) created by the Laboratory of Neu-roImaging of the University of South California (LONI, https:// loni.
usc. edu/). Of note, XNAT also allows to perform some image processing  [9]. The neuroimaging community has developed BIDS (Brain Imaging Data Structure), a standard for MR image organization to accommodate multimodal acquisitions and facilitate processing. In practice, few datasets come in BIDS format, and tools have been developed to assist with download and conversion (e. g. , https:// clinica. run)  [10]. We have listed a handful of datasets (see Table  1 ), which is far from being exhaustive but aims at summarizing some of the largest and/or most used samples. Our selection aims at presenting diverse and complementary samples in terms of age range, populations, and country of origin. First, we have described three clinical elderly samples from the USA and Australia, with a focus on Alzheimer's disease and cognitive disorders.
The Alzheimer's Disease Neuroimaging Initiative (ADNI) was launched in 2004 and funded by a partnership between private companies, foundations, the National Institute of Health, and the National Institute for Aging. ADNI is a longitudinal study, with data collected across 63 sites in the USA and Canada. To date, four phases of the study have been funded, which makes ADNI one of the largest clinical neuroimaging samples to study Alzheimer's disease and cognitive impairment in aging. ADNI collected a wide range of clinical, neuropsychological, cognitive scales as well as biomarkers, in addition to multimodal imaging and genotyping data  [11]. Sites contribute data to the LONI, which is automatically shared with approved researchers without embargo. The breadth of data available and its accessibility have made ADNI one of the most used neuroimaging samples, with more than 1000 scientific articles published to date.
The Australian Imaging Biomarkers and Lifestyle Study of Ageing (AIBL) started in 2006 and has since recruited about 1100 participants over 60 years of age, who have been followed over several years (see Table  1 )  [12]. AIBL collected data across the different Australian states and, similar to ADNI, consisted in an in-depth assessment of individual cognition, clinical status, genetics, genomics, as well as multimodal brain imaging  [12]. In 2010, AIBL partnered with ADNI to release the AIBL imaging subset and selected clinical data via the LONI platform. Having the same MRI protocols and similar fields collected, AIBL represents a great addition to the ADNI study, by boosting statistical power or allowing for replication. The full clinical information as well as genetics, genomics, and wearable data (actigraphy watches) are not available via the LONI and require a direct application to the Commonwealth Scientific and Industrial Research Organisation (CSIRO) (see Table  1 ).
The Open Access Series of Imaging Studies v3 (OASIS3) is another longitudinal sample comprising almost 1100 adult participants (see Table  1 )  [13]. Its main focus is around aging and neurological disorders, and the application/approval process is extremely fast (typically a couple of days). OASIS3 is hosted on XNAT and is the third dataset to be made available by the Washington University in Saint Louis (WUSTL) Knight Alzheimer's Disease Research Center (ADRC), although the three datasets are not independent and cannot be analyzed together. Contrary to ADNI and AIBL, OASIS3 is a retrospective study that aggregates several research studies conducted by the WUSTL over the past 30 years. As a result, the data collected may vary from one individual to the next, with a variable time window between visits.
Main Existing Datasets 771 The UK Biobank (UKB) imaging study  [14]  is the largest brain imaging study to date, with around 50,000 individuals already imaged (target of 100,000). The imaging wave complements the wealth of data already collected in the previous waves (see Table  1 ; see also Subheading 5 for a description of the full dataset). Considering the sheer size of the data, the biobank shares raw and processed images as well as structured data (measurements of regions of interest)  [15]. Data is accessible upon request by all bona fide researchers, with certified profiles. Data access requires payment of a fee, which only aims to cover the biobank functioning costs. The UKB has developed proprietary tools for a secure download and data management (https://biobank. ndph. ox. ac. uk/showcase/ download.