dataesr/scientific-paragraphs-categorization huggingface dataset:
Lidar network Starting in 2016, Meteo-France has deployed a network of 6 automatic backscatter lidars to help the Volcanic Ash Advisory Center (VAAC) of Toulouse characterize volcanic ash and aerosols layers. One sensor can be quickly redeployed on a more suitable geographic location depending on the transport event to follow. The network, fully operational since April 2017, is functioning continuously and has been able to detect aerosol events up to an altitude of 17 km. It is part of the wider Automatic Lidars and Ceilometers (ALC) network of the E-PROFILE program described in  Haefele et al. (2016). Two sampling sites of this network have been selected:  Brest (48. 444 N, 4. 412 W, 94 m a. s. l) and Trappes (48. 773 N, 2. 0124 E, 166 m a. s. l). Each site is equipped with a Mini Micro Pulse LiDAR (MiniMPL), built by Sigma Space Corporation; the exterior casing was provided by Envicontrol.
Data processing The miniMPL acquires profiles of atmospheric backscattering at high frequency (2500 Hz) using a low energy pulse (3. 5 µJ) emitted by a Nd:YAG laser at 532 nm. The profiles are acquired in photon-counting mode and, in our present configuration, averaged over 5 minutes and 30 meters vertical resolution bins. The instrument uses a monostatic coaxial design: the laser beam and the receiver optics share the same axis. Due to geometrical limitations, only a fraction of the signal can be recovered in the near field. Therefore the first usable data are provided at 120 meters above ground level on our system. The instrument has polarization capabilities with the collection of photons on two different channels (more details  in Flynn et al. (2007) ): the measured raw signals on the "copolarized" and "crosspolarized" channel, respectively suffixed co and cr.
These raw signals are then processed to obtain the quantities of interest: the range corrected signal RCS, also called normalized relative backscatter, N RB. This industrial processing comprises several procedures such as background, overlap, afterpulse and dead-time corrections. A comprehensive description of this processing is given in  Campbell et al. (2002). Finally the "copolarized" and "crosspolarized" range corrected signal as delivered by the industrial software, respectively RCS co and RCS cr will be used as predictors for the machine learning algorithms described in part 3. Raw data type and format depends on the instrumental device used. To make the algorithms usable on other devices, we used as input of the algorithms the files processed to a normalised format by the raw2l1 routine.
Ancillary data Ancillary data have been used for the description of the meteorological situation. They are not used by machine learning algorithms. All the instruments are collocated with lidar and radiosoundings. -Rain gauges were used to detect rain events. -Vaisala CL31 ceilometers were used to detect cloud base height, and distinguish cases with cloud on top or inside the boundary layer. Although the MiniMPL is perfectly capable of detecting clouds, we chose not to use the industrial algorithm. The algorithm has shown in our experience to make some false positives. -Scatterometers were used to estimate visibility and detect fog cases.
In this study, weak learners in AdaBoost are trees with a maximum depth of 5 (maximum 5 forks between root an leaves). AdaBoost is based on decision tree classifiers. It aggregates them in order to have the most accurate border. The idea of AdaBoost is illustrated by figure 3. First, a shallow decision tree is fitted on a random subsample of the dataset. Some points of the dataset are misclassified: the error of the classifier is the number of misclassified points. Another shallow decision tree is fitted on a subsample of the dataset where the previously misclassified points are over-represented. This new tree has new misclassified points, that will be over-represented in the training of the next tree, and so on, up the specified number of tree (m = 200 in our case). The classification given in output of AdaBoost is the average ŷ of all the predicted class by the trees {ŷ m } m∈[[1,M ]] , weighted by their performance {α m } m∈[[1,M ]] : ŷ = sign( m m=1 αm ŷm ).
KABL flowchart The simplified flowchart of KABL is shown in figure  6. A netCDF file generated by the raw2l1 software must be provided as input data of the KABL code. The data, namely: the vector of altitude z (size N z ), the vector of time t (size N t ), the range-corrected signals RCS co and RCS cr (matrices of shape N t × N z ), are extracted from this file. Such data are prepared  The parameters of this computer code were in bold font in the text and they are detailed here: -algo: the machine learning algorithm that will be applied. Possible values are -'gmm', for the EM algorithm (Gaussian mixture) -'kmeans', for the K-means algorithm -classif_score: internal score used to automatically choose the number of clusters (only used when n_clusters='auto'). See section 3. 4 for a description of these scores.
Sensitivity analysis of KABL algorithm A sensitivity analysis was carried out on KABL code in order to find the "best" configuration. Various configuration of KABL have been tested extensively on a single day: the 2nd of August 2018 at Trappes, for which we have a hand-made reference (see figure  4 -top). The more relevant configuration is then retained and tested on the two years dataset. There are 8 parameters in the KABL code (see section 3. 3 for their description). To assess the sensitivity of KABL to these parameters, the performance metrics (given in section 3. 4) were estimated with the hand-made BLH as Z ref and with KABL's output as Ẑ for different combination of input parameters. The tested values for the input parameters are given in table  2  and the output metrics are given in table  1.
We call a configuration a set of values for KABL parameters. Screening all the possible values listed in table 2 would take 3240 different configurations. To look into these 3240 configurations at a glance, we started by estimating the influence of the code parameters (listed in table 2) onto the different metrics (listed in table  1 ). Their influence is quantified by first order Sobol indices  (Sobol, 2001; Iooss and Lemaître, 2015; Rieutord, 2017) , that is to say the ratio of the variance of the metric when the parameter is fixed over the total variance of the metric. If we denote by Y the metric and by X the vector of parameters, all considered as random, the first order Sobol index of the i-th parameter is defined by S i = V (E [Y |X i ]) /V (Y ) (with V (•) denoting variance and E [•] expectation).
3. 1. 3 Retained configurationFour predictors are used: the two lidar channels, the time (number of seconds since midnight) and the altitude (meters above ground level). The current configuration of ADABL is thus the following:-Weak learner: decision tree of depth 5-Number of weak learners: 200 -Predictors: time, altitude, RCS co , RCS cr It was chosen by a trade-off between accuracy and computing time. The accuracy is the percentage of individuals well classified. It is estimated by cross-validation (random split, 80% training set, 20% testing set) and reaches 99. 5% of the testing test. The computing is still reasonable: it takes 23 seconds to train on the full dataset and 3. 7 seconds to predict the BLH for a full day.
Rain (rain gauge measures RR>0) -Fog (scatterometer measures visibility <1000m) -Low level cloud (ceilometer measures cloud base height <3000m) -RS estimation below 120m (blind zone for lidar) -Nighttime (launch of 23:15 UTC) This selection rejects a large part of the dataset, but it ensures to keep only well-defined boundary layer. Meteorological conditions are measured by ancillary instruments presented in section 2. 3. The results of the comparison are shown in figure 9.
https://doi. org/10. 5194/amt-2020-78 Preprint. Discussion started: 7 April 2020 c Author(s) 2020. CC BY 4. 0 License. The other is based on AdaBoost algorithm: it is named ADABL (AdaBoost for Atmospheric Boundary Layer). Both take the same input: one day of data generated by raw2l1 routine; and give the same output: the time series of BLH for this day. KABL is a non-supervised algorithm. It will look for a natural separation in the basckscatter signals between the boundary layer and the free atmosphere. ADABL is a supervised algorithm. It fits a large number of decisions trees on a labelled dataset and aggregates them smartly to give a good prediction. KABL, ADABL and the lidar manufacturer's algorithm (unknown) have been tested on a 2-years dataset taken from Meteo-France operational network of lidars.
Code availability. The source code of KABL was made to be available and usable to all users, including commercial ones. It is freely available under an open-source license at the following link: https://github. com/ThomasRieutord/kabl It is made in Python 3. 7 with regular packages in statistics and machine learning, namely: Scikit-learn 0. 20  (Pedregosa et al. , 2011)  and SALib 1. 3. 7  (Herman and Usher, 2017) , which are all open-source and under free licences. The repository contains all necessary features to run the code on raw2l1 outputs. Several days of data are also provided as examples.
- Rain (rain gauge measures RR>0) -Fog (scatterometer measures visibility <1000m) -Low level cloud (ceilometer measures cloud base height <3000m) -RS estimation below 120m (blind zone for lidar) -Nighttime (launch of 23:15 UTC) This selection rejects a large part of the dataset, but it ensures to keep only well-defined boundary layer. Meteorological conditions are measured by ancillary instruments presented in section 2. 3. The results of the comparison are shown in figure 9.
Recent studies have massively expanded the known species repertoire of the human gut, making available unprecedented numbers of new cultured and uncultured genomes  16, [18] [19] [20] [21]. Two culturing efforts isolated and sequenced over 500 human-gut-associated bacterial genomes each  19, 21  , while three independent studies  16, 18, 20  reconstructed 60,000-150,000 MAGs from public human microbiome data, most of which belong to species lacking cultured representatives. Combining these individual efforts and establishing a unified nonredundant dataset of human gut genomes is essential for driving future microbiome studies. To accomplish this, we compiled and analyzed 204,938 genomes and 170,602,708 genes from human gut microbiome datasets to generate the Unified Human Gastrointestinal Genome (UHGG) and Protein (UHGP) catalogs, the most comprehensive sequence resources of the human gut microbiome established thus far.
1 ). These 573 genomes (535 from isolates and 38 from MAGs) satisfy the 'high quality' criteria set for MAGs by the Genomic Standards Consortium  28. The rRNA operon has previously been shown to be a problematic region to assemble from short-read metagenomic datasets  13, 16, 18, 20  , which might explain the low number of high-quality MAGs. Thereafter, we classified each species representative using the Genome Taxonomy Database c, Overview of the methods used to generate the genome (UHGG) and protein sequence (UHGP) catalogs. Genomes retrieved from public datasets first underwent quality control by CheckM. Filtered genomes were clustered at an estimated species level (95% ANI), and their intraspecies diversity was assessed (genes from conspecific genomes were clustered at 90% protein identity).
In parallel, a nonredundant protein catalog was generated from all coding sequences of the 286,997 genomes at 100% (UHGP-100, n = 170,602,708), 95% (UHGP-95, n = 20,239,340), 90% (UHGP-90, n = 13,907,849) and 50% (UHGP-50, n = 4,735,546) protein identity. Toolkit  29, 30  (GTDB-Tk; Extended Data Fig. 2 ), a standardized taxonomic framework based on a concatenated protein phylogeny representing >140,000 public prokaryote genomes, fully resolved to the species level (see Methods for details on the taxonomy nomenclature used). However, over 60% of the gut genomes could not be assigned to an existing species, confirming that the majority of the UHGG species lack representation in current reference databases.
2a ). Each of the large MAG studies used a different assembly and binning approach: the CIBIO study used metaSPAdes  31  and MetaBAT 2 (ref. 32 ) for assembling and binning sequencing runs previously merged by sample; the HGM study used MEGAHIT  33  to assemble runs merged by sample and applied a combination of MaxBin 2 (ref. 34 ), MetaBAT 2 (ref. 32 ), CONCOCT  35  and DAS Tool  36  for binning and refinement; and the EBI study used metaSPAdes  31  and MetaBAT 2 (ref. 32 ) for assembling and binning individual runs without merging by sample. Despite these methodological differences, the largest intersection found was between these collections of MAGs, with the same 1,081 species detected independently in the CIBIO, EBI and HGM datasets, but not in any of the cultured genome studies. By restricting the analysis to genomes recovered from 1,554 samples common to all three MAG studies, we found that 93-97% of the species from each set were detected in at least one other MAG collection and 79-86% were detected across all three (Extended Data Fig.
2b ). However, these most likely represent rarer members of the human gut microbiome, as the number of species is closer to saturating when only considering those with at least two conspecific genomes. We also investigated the intersection between the three large culture-based datasets: the HBC, CGR and NCBI (which contains gut genomes from the HMP 4 ). Unlike the MAGs, the majority of cultured species were unique within a single collection (486/698; 70%), with only 70 (10%) common to all three collections (Extended Data Fig. 3d ). This may be due to varied geographic sampling between the collections (Asia, Europe and North America) or might highlight the stochastic nature of culture-based studies. Most gut microbial species lack isolate genomes. We found that 3,750 (81%) of the species in the UHGG catalog did not have a representative in any of the human gut culture databases.
2c  and Extended Data Fig. 5b ). We determined how representative the UHGG catalog is of the human gut microbial diversity by mapping 1,005 independent metagenomic datasets against the 4,644 UHGG species (Fig. 2d  and Supplementary Table  4 ). Using Kraken 2 (ref. 38 ), we obtained a median classification rate of 85. 9% (IQR = 83. 5-88. 1%). Notably, this corresponded to a median improvement of 155% over the standard RefSeq database. The increase in classification rate was most pronounced in non-Western samples from Cameroon, Ethiopia, Ghana and Tanzania, highlighting the potential of the UHGG catalog to improve the study of microbiome diversity from these understudied populations. The phylogenetic distribution of the 4,616 bacterial (Fig. 3a ) and 28 archaeal (Extended Data Fig. 6 ) species revealed that uncultured species exclusively represented 66% and 31% of the phylogenetic diversity of Bacteria and Archaea, respectively, with several phyla lacking cultured representatives (Fig.
4a ), in line with previous estimates  6. To determine how comprehensive the UHGP is when compared to existing human gut gene catalogs, we combined the UHGP-90 (n = 13,910,025 protein clusters) with the IGC 5 , a collection of 9. 9 million genes from 1,267 gut metagenome assemblies, which we grouped into 7,063,981 protein clusters at 90% protein identity (referred to as IGC-90). Nearly all samples used to generate the IGC were also included in the UHGP catalog (except for 59 transcriptome datasets), but the latter was generated from a larger and more geographically diverse metagenomic dataset (including samples from Africa, South America and Oceania). Combining the UHGP-90 and IGC-90 resulted in a set of 15. 2 million protein clusters, with an overlap of 5. 8 million sequences (Fig. 4b ). This revealed that 81% of the IGC is represented in the UHGP catalog, with the missing 19% likely representing fragments of prokaryotic genomes that are <50% complete and viral or eukaryotic sequences, plasmids or other sequences not binned into MAGs.
Functional capacity of the human gut microbiota. We used the eggNOG  41  , InterPro  42  , COG  43  and KEGG  44  annotation schemes to capture the full breadth of functions within the UHGP. However, we found that 41. 5% of UHGP-100 was poorly characterized, as 27. 3% lacked a match to any database and a further 14. 2% only had a match to a COG with no known function (Fig. 4c ). On the basis of the distribution of COG functions, the most highly represented categories were related to amino acid transport and metabolism, cell wall/membrane/envelope biogenesis and transcription. We further leveraged the set of 171 million proteins derived from the human gut genomes to explore the functional diversity within each of the UHGG species. Protein sequences from all conspecific genomes were clustered at 90% amino acid identity to generate a pan-genome for each species.
5a ). There was wide variation in the proportion of core genes between species even among clades with more than 1,000 genomes (Fig. 5b ), with a median core genome proportion (percentage of core genes among all genes in the representative genome) estimated at 66% (IQR = 59. 6-73. 9%). To distinguish the functions encoded in the core and accessory genes, we analyzed their associated annotations. Core genes were well covered, with a median of 96%, 94%, 92% and 69% of the genes assigned with an eggNOG, InterPro, COG and KEGG annotation, respectively (Fig. 5c ). In contrast, the accessory genes had a significantly higher proportion of unknown functions (P < 0. 001), with a median of 21% of the genes (IQR = 16. 7-27. 3%) lacking a match in any of the databases considered. Thereafter, we investigated the functions encoded by the core and accessory genes on the basis of the COG functional categories.
Discussion We have generated a unified sequence catalog representing over 200,000 genomes and 171 million protein sequences of the human gut microbiome. Of the 4,644 species contained in the UHGG   49  , which will be incorporated in future versions of the resource. As 96% of these genomes were reported to have a species representative in the culture collections included here, we do not anticipate that this dataset will provide a substantial increase in the number of species discovered. Nevertheless, our analyses suggest that additional uncultured species from the human gut microbiome are yet to be discovered, highlighting the importance and need for culture-based studies. Furthermore, given the sampling bias toward populations from China, Europe and the United States, we expect that many under-represented regions still contain substantial uncultured diversity.
By comparing recently published large datasets of uncultured genomes  16, 18, 20  , we were able to assess the reproducibility of the results from each study. We show that, despite the different assembly, binning and refinement procedures used in the three studies, almost all of the same species and similar strains were recovered independently when using a consistent sample set. Although these results increase confidence in the use of MAGs, new methods for metagenome assembly, binning and quality control continue to be developed to overcome existing limitations, meaning that improved versions of the MAGs included here will likely be generated in the future. With the establishment of this massive sequence catalog, it is evident that a large portion of the species and functional diversity within the human gut microbiome remains uncharacterized.
Methods Genome collection. We compiled all the prokaryotic genomes publicly available as of March 2019 that were sampled from the human gut. To retrieve isolate genomes, we surveyed the IMG  24  , NCBI  22  and PATRIC  23  databases for genome sequences annotated as having been isolated from the human gastrointestinal tract. We complemented this set with bacterial genomes belonging to two recent culture collections: the HBC  19  and CGR  21. To avoid including duplicate entries due to redundancy between reference databases, we combined genomes obtained from the PATRIC and IMG repositories and added only those without an identical genome in the sets extracted from NCBI, HBC and CGR. This was determined by comparing isolate genomes between different databases using Mash v2. 1 (ref. 26 ; 'mash dist' function) and only selecting one genome among those estimated to be identical (Mash distance of 0).
Assessing genome quality. Assembly statistics were calculated with the 'stats. sh' script from BBMap v38. 75 (https://sourceforge. net/projects/bbmap/). Genome quality (completeness and contamination) was estimated with CheckM v1. 0. 11 (ref. 53 ) using the 'lineage_wf ' workflow to select only genomes that passed the following criteria: >50% genome completeness, <5% contamination and an estimated quality score (completeness -5 × contamination) > 50. We also searched for the presence of rRNAs in each genome with the 'cmsearch' function of INFERNAL v1. 1. 2 (ref. 54 ; options '-Z 1000 --hmmonly --cut_ga --noali --tblout') against the Rfam 55 covariance models for the 5S, 16S and 23S rRNAs. tRNAs of the standard 20 amino acids were identified with tRNAScan-SE v2. 0 (ref. 56 ) with options '-A -Q' for archaeal species and '-B -Q' for species belonging to bacterial lineages.
To investigate the level of strain heterogeneity represented within each MAG, we used the CMseq tool (https://github. com/SegataLab/cmseq) as previously described  20. Briefly, metagenomic reads from the sample used to generate the MAG were aligned to the respective MAG using bowtie v2. 2. 3 (ref. 57 ), with the resulting alignment file indexed and sorted with samtools v1. 5 (ref. 58 ). The level of strain heterogeneity was estimated with the 'polymut. py' script from the CMseq package by calculating the number of nonsynonymous substitutions detected out of all positions mapped with a depth of coverage of at least 10 reads and base quality of at least 30 (a minimum of 100 positions were needed to estimate strain heterogeneity). Species clustering. We clustered the total set of 286,997 genomes at an estimated species level (ANI ≥ 95%; ref.
The genome with the highest score was chosen as the species representative, with cultured genomes prioritized over uncultured genomes (that is, if a MAG had a higher score than an isolate genome, the latter would still be chosen as the representative). To further investigate the within-species population diversity, we calculated pairwise distances for all conspecific genomes using Mash v2. 1 (ref. 26 ; default sketch size). From these results, we generated individual distance trees for each species using the 'complete' hierarchical clustering method implemented in the Fastcluster R package  60. We calculated the number of clusters recovered using a distance cutoff of 0. 03 (97% ANI) and 0. 01 (99% ANI). Evaluating reproducibility of the methods. The species clusters inferred here were compared with those previously generated in human gut MAG studies  16, 18, 20  from a common set of genomes.
Similarity between species clusterings was estimated using the adjusted Rand index (ARI) computed in the Scikit-learn Python package  61. This metric considers both the number of clusters and cluster membership to compute a similarity score ranging from 0 to 1. Conspecific genomes recovered in the same metagenomic samples but in different studies were compared with FastANI v1. 1 (ref. 27 ) with default parameters to obtain both the maximum AF and ANI for each pairwise comparison. Inferring cultured status. To determine cultured status, the UHGG species representatives were searched against NCBI RefSeq release 93 after excluding uncultured genomes (that is, metagenome-assembled or single-cell amplified genomes). Genome alignments were performed in two stages: (1) Mash v2. 1 (ref. 26 ) was used as an initial screen (using the function 'mash dist') to identify the most similar RefSeq genome to each of the UHGG species and (2) 'dnadiff ' from MUMmer v4.