dataesr/scientific-paragraphs-categorization huggingface dataset:
UPARSE_q3 removed roughly 4% of the reads and reported improved results on a par with those of Swarm, SUMACLUST, UCLUST, and USEARCH61. The highest F measure (0. 83 to 0. 84) and number of OTUs closest to the expected one (1,076) were reported for software using input files from QIIME's method of sequence filtering (Swarm, SUMACLUST, UCLUST, and USEARCH61). All tools except UPARSE_q16 reported comparable alpha diversity phylogenetic diversity (PD) whole-tree (35) (a measure of diversity which considers the phylogenetic differences between species) values (mean of 109. 21 with a standard deviation of 2. 16 [  Table 2] ). Among the closed-reference methods, SortMeRNA yields the fewest OTUs while achieving comparable or higher F measures for assigned taxonomy (F1 tax in Table  2 ) and OTUs (F1 OTUs in Table  2 ) and reported a phylogenetic diversity (121.
89) closest to TABLE 2 Benchmark summary a a OTU counts do not include singletons. F measure (F1) is for assigned taxonomies at the genus level. The phylogenetic diversity (PD) whole-tree column for Bokulich_2 and Bokulich_3 represent PD intervals across various sampling depths. Procrustes M 2 (the sum of the squared deviations or the dissimilarity of two datasets for UniFrac PCoA) and rho (Pearson's correlation coefficient for taxonomies at genus level) values are with respect to UCLUST (default for QIIME versions 1. 0. 0 to 1. 9. 1). Monte Carlo P values were not included, since all values were Ͻ0. 05 except for de novo usearch52 versus uclust (P ϭ 0. 09). The darkest blue shades represent the highest F1 scores, while the darkest red shades represent results closest to those obtained with UCLUST.
the ground truth (123. 75) in comparison to UCLUST (126. 02), USEARCH 5. 2 (113. 39), and USEARCH 6. 1 (127. 50). This is a result of SortMeRNA's more exhaustive search for better alignments, which can increase run time but becomes imperative when short reads are aligned against a highly conservative set of sequences, such as the rRNA gene. The complete Greengenes database contains over a million rRNAs, and almost 73% of all full-length V4 regions (~250 nucleotides) are not unique. This emphasizes the highly conservative nature of rRNA, even in this hypervariable region, and suggests the need for thorough searches to ensure higher-quality alignments (especially for read lengths that do not cover the entire region). At the genus level of taxonomy, all tools report F measures of 0. 80 to 0. 83 (sim_even) and 0.
78 to 0. 84 (sim_staggered), which can be attributed to the many-to-one relationship between OTUs and taxonomy strings for the Greengenes 97% database. For open-reference clustering, QIIME's subsampling pipeline combining SortMeRNA and SUMACLUST reports the fewest OTUs in comparison to UCLUST and USEARCH 6. 1. The F measure is 0. 82 to 0. 83 (sim_even) and 0. 81 to 0. 83 (sim_staggered) for all tools, which is in agreement with de novo and closed-reference results. These results are expected given the nature of open-reference clustering, which combines the closedreference approach with the de novo approach. (ii) Mock communities. Results for Bokulich_2 (and Bokulich_3 for UPARSE) are unavailable for UPARSE, OTUCLUST, and mothur due to significant memory, run time, and disk space requirements, respectively.
However, false-positive taxonomies can also arise from OTUs formed by chimeric sequences (sequences from two organisms that bind together during PCR and are subsequently sequenced as a single read) or incorrect assignment by taxonomy assignment tools. To investigate the origins of false-positive taxonomies reported by the tools, we checked all OTUs for chimeras using UCHIME  (20)  and mapped the nonchimeric OTUs against BLAST's NT database using MEGABLAST. Most of the false-positive taxa were not wholly comprised of chimeric OTUs (meaning that the collection of OTUs mapping to the same taxa was composed of chimeric and genuine sequences), and the majority of such nonchimeric taxa consisted of OTUs mapping with an E value of Ͻ1eϪ50 to BLAST's NT database (e. g. , in Table  3  there are 57 false-positive taxa reported by SUMACLUST, but only 4 of those taxa are fully comprised of chimeric OTUs [FP-chimeric], and 99% of OTUs representing the remaining 53 nonchimeric taxa mapped with high similarity to BLAST's NT database).
However, for the much smaller data set Bokulich_6, UPARSE_q16 reported only half of the expected genera relative to all other tools, and the relative abundance of some of the genera significantly decreased (Fig. 1 ; PD values in Table  2 ). OTUCLUST, mothur_nearest, mothur_average, Swarm, and SUMACLUST reported significantly fewer OTUs than UCLUST and USEARCH 6. 1, as well as a lower alpha diversity (Tables  2  and 3 ). Thus, tools with lower false-positive rates accomplish this by more stringent quality control, but they are less suitable for finding lowerabundance genera. For closed-reference clustering, SortMeRNA reported up to 60% fewer OTUs and a PD of about half that of UCLUST and USEARCH 6. 1 (Table  2 ). On the genus taxonomy level, USEARCH 5. 2 reported a high F measure (due to a lower number of false-positive genera), but unlike all other tools, the majority of false-positive genera are composed of reads mapping with Յ97% identity and coverage to BLAST's NT database (FP-other).
In fact, all other tools filtered out a large portion of these false-positive reads due to insufficient identity matches to the reference database. The difference appears to be caused by USEARCH 5. 2's identity definition (which does not consider insertions or deletions), which scores alignments higher than other tools. SortMeRNA generates taxonomic profiles similar to those obtained with other tools, with a Pearson's correlation coefficient of Ͼ0. 93 (Fig. 1 ). As expected for open-reference clustering, SortMeRNA combined with SUMACLUST reported fewer OTUs than UCLUST and USEARCH 6. 1 while preserving high accuracy and lower alpha diversity for both the number of observed OTUs and the phylogenetic diversity. Specific details can be found in the supplemental material. The Pearson coefficient for comparisons between all tools and methods remained relatively stable, from ~0.
Procrustes analysis  (36)  was used to compare unweighted UniFrac  (37, 38)  principal coordinates analysis (PCoA)  (22)  generated by all methods versus UCLUST (the current default OTU picker in QIIME). The Procrustes M 2 metric for body_sites and canadian_soil was Ͻ0. 3 for most software (Table  2 ), indicating that beta diversity patterns are similar irrespective of the OTU clustering method used. Neither recommended nor relaxed quality filtering parameters for UPARSE worked well for the body_sites data set, where 98. 5% and 99. 2% of reads were filtered out for UPARSE_q3 and UPARSE_q16 (with a trim length of 250 bp), respectively, resulting in very few remaining samples and high M 2 values (Table  2 ; also, see Fig. S4  in the supplemental material). Although read quality filtering is an important preprocessing step, more work is required to regulate these parameters (perhaps by an automated estimation of optimal truncation length and quality), as they can be very sensitive to different types of data.
DISCUSSION We evaluated the performance of four recently published open-source sequence clustering tools against the widely used mothur, UCLUST, and USEARCH tools using simulated data, mock communities, and natural microbial communities. We found that Swarm, SUMACLUST, UCLUST, and UPARSE (with relaxed parameters) performed equally well on simulated datasets where the ground truth was well established, with mothur_average and OTUCLUST closely behind. Despite this controlled chimera-free environment, UPARSE with recommended parameters reported the lowest accuracy for the sim_staggered data set, implying that stringent quality filtering can cause a significant underestimation of species abundance and diversity and lead to incorrect biological results. For the mock communities, most tools were able to correctly detect the expected number and identity of genera, but only UPARSE reported significantly fewer false-positive taxa (followed by OTUCLUST and USEARCH).
For UPARSE, this was expected, as a large proportion of reads was filtered out prior to clustering, leaving evidence of only the most abundant taxa (OTUs comprised of hundreds of thousands of reads). The majority of false-positive taxa reported by other tools were low-abundance OTUs that could be mapped to BLAST's NT database with very high similarity (E value, Ͻ1eϪ50). If the user's primary goal is to focus on the most abundant microbial profiles, low-abundance OTUs may be filtered out postclustering, but care should be taken, as such low-abundance OTUs can be important members of communities  (39). In terms of accurately predicted taxonomic composition for de novo tools, Swarm performed well across all simulated and mock datasets, followed closely by SUMACLUST and UCLUST. However, both Swarm and SUMACLUST reported significantly fewer OTUs and lower alpha diversities than UCLUST.
The performance of other de novo methods, such as mothur and OTUCLUST, showed more variation across datasets; however, these results were largely influenced by the preliminary sequence filtering step, where both tools removed more data than QIIME's method. We found that QIIME's filtering approach worked well across all datasets, rendering the most data for clustering tools to work with. For closed-reference tools, SortMeRNA significantly outperformed UCLUST and USEARCH for predicting OTUs and performed as well or better in terms of predicted taxonomic composition. Several studies could not be processed with mothur, OTUCLUST, or the free academic distribution of UPARSE due to their large sizes, either because of an unreasonable disk space requirement in the case of mothur, unreasonable run time in the case of OTUCLUST (no multithreading support), or a relatively small Although most open-source tools report an increased run time in comparison to UCLUST and USEARCH (Fig.
5 ), they provide the benefit of finding significantly fewer OTUs. In the case of SortMeRNA, longer reads (~150 bp) are quicker to align than the same number of shorter reads (~100 bp) due to many fewer high-scoring candidate reference sequences to analyze. Moreover, all of these tools support multilevel multi- threading and can easily scale to modern big-data processing demands. An alternative to reducing run time is to filter out a substantial number of reads, as done by UPARSE; unfortunately, the filtering parameters are sensitive to different data, and choosing them manually by trial and error can be a time-consuming task with unpredictable outcomes in diversity. The three open-source software products, Swarm, SUMACLUST, and SortMeRNA, are now accessible through the widely used QIIME software package (release 1.
9. 0). Swarm 2 was released in reference 14 and reported to be faster and more memory efficient than Swarm 1; however, as of this writing, only Swarm 1 has been integrated into QIIME. Ongoing work to improve the QIIME OTU clustering workflows that use these tools includes adding a targeted gene prefilter for de novo clustering to remove (prior to clustering) any sequences not matching a specific gene model (e. g. , 16S rRNA) and a refined reference database for targeted hypervariable regions (e. g. , V4 at 97% identity) to improve alignment quality  (41). Furthermore, research is in progress to introduce an open-source implementation of chimera detection directly within QIIME. Both of these improvements will further reduce the number of unrelated or erroneous reads recruited into OTUs, a known problem with both the UCLUST-and USEARCH-based OTU clustering illustrated here, without underestimating diversity.
MATERIALS AND METHODS All steps taken to generate the analyses presented in this article are documented and implemented as shell or python scripts, available at https://github. com/ekopylova/OTU-clustering. Performance benchmarks. Open-source with multilevel parallelization tools tested in this paper-Swarm, SUMACLUST, and SortMeRNA-have been integrated into QIIME 1. 9. 0. For these tools, all benchmarks were launched through QIIME. For UPARSE, the recommended workflow (http:// www. drive5. com/usearch/manual/uparse_cmds. html) was run. For OTUCLUST, the script micca-preproc was used for sequence filtering and the command otuclust for clustering. For mothur, the MiSeq SOP (42) (website accessed 27 October 2015) and 454 SOP (43) (website accessed 27 October 2015) were run. The shell scripts commands_16S.
sh and commands_18S. sh were used to launch all tools, and the opensource project (https://github. com/josenavas/QIIME-Scaling) was used for measuring their run time performance. All run time performance tests were performed using 1 to 32 threads on Intel Xeon CPU E5-2640 v3 at 2. 60 GHz. Precision and recall. For simulated and mock datasets, false-positive (FP; taxonomy/OTU string exists in observed but not expected), false-negative (FN; taxonomy/OTU string exists in expected but not observed), and true-positive (TP; taxonomy/OTU string exists in both observed and expected) measures were computed between the pickers' results (observed) and the ground truth or expected taxonomic composition (expected). The following definitions were used: precision ϭ TP/(TP ϩ FP); recall ϭ TP/(TP ϩ FN); F measure ϭ 2 ϫ precision ϫ recall/(precision ϩ recall).
The python script run_compute_precision_recall. py was used to compute TP, FP, FN, precision, recall, F measure, the number of false-positive taxa whose complete set of OTUs are identified as chimeric (FP-chimeric) by UCHIME, the number of false-positive taxa whose complete set of OTUs map with Ն97% identity and coverage to BLAST's NT database (FP-known), and the number of false-positive taxa whose complete set of OTUs map with Ͻ97% identity and coverage to BLAST's NT database (FP-other). The script plot_tp_fp_distribution. py was used to generate Fig. S1, S2 , and S3 in the supplemental material. Simulating reads (even and staggered). All of the following steps can be executed using the shell script simulate_reads. sh. Reads were simulated using PrimerProspector  (23)  and the ART simulator  (24).
For the even data set, the following steps were taken. (i) Use PrimerProspector to extract V4 regions from the Greengenes 97% representative database (version 13. 8); (ii) subsample 0. 011% of the sequences from the resulting V4 region database; and (iii) simulate even abundance reads with ART simulator using the subsampled V4 sequences. Amplicon sequencing simulation in ART (version VanillaIceCream-03-11-2014) could generate only evenly distributed communities. To simulate the staggered data set, a staggered distribution of template sequences was passed (for example, 3 duplicates of OTU1, 10 duplicates of OTU2, etc. ). To simulate the staggered data set, the following steps were taken. (i) Generate a random staggered distribution FASTA file of template V4 sequences using the list of OTU identifications from the even data set and the V4 subsampled sequences and (ii) simulate staggered abundance reads with ART using the staggered subsampled V4 sequences.
For both even and staggered reads, QIIME's split_libraries_fastq. py script was run to filter simulated reads based on quality scores and format FASTA labels to be compatible with QIIME (reads for UPARSE, mothur, and OTUCLUST were not filtered; only FASTA labels were reformatted). Building ground-truth BIOM tables. Ground-truth OTU maps and BIOM tables were constructed using the simulate_reads. sh script that was used for simulating reads. OTU maps were generated using the reads' origin information stored in the FASTA labels of ART-simulated reads. BIOM tables were generated using QIIME's make_otu_table. py script together with Greengenes 97% taxonomy strings.
FUNDING INFORMATION HHS | National Institutes of Health (NIH) provided funding to Evguenia Kopylova, José Antonio Navas-Molina, and Rob Knight under grant number 1S10OD012300. Deutsche Forschungsgemeinschaft (DFG) provided funding to Frédéric Mahé under grant number DU1319/1-1. This work was partially supported by the Howard Hughes Medical Institute and the Alfred P. Sloan Foundation.
(Of course, certification does not necessarily mean formal proof!) In this context, we can cite the work done on the certification of the SCADE compiler  [15] , a tool-suite based on the synchronous language Lustre that integrates a model-checking engine. Nonetheless, only the code-generation part of the compiler is certified and not the verification part. Concerning observer-based model-checking, most of the works rely on an automatic way to synthesize observers from a formal definition of the properties. For instance, Aceto et al. [2]  propose a method to verify properties based on the use of test automata. In this framework, verification is limited to safety and bounded liveness properties since the authors focus on properties that can be reduced to reachability checking. In the context of Time Petri Net, Toussaint et al.
g. [14] ), but also multi-scale problems (e. g. [15] ). Arlequin method relies on a partition of models, each model being valid in the domain on which it is defined. Furthermore, Arlequin method introduces coupling operators and partitions of energy between models. Generally, there is a hierarchy between the two models, one model being a simplified version of the other. As compared with the classical method proposed by Ben Dhia and Rateau  [15] , we introduce a relevant coupling operator and a simple procedure to build varying meshes (see Fig. 3 ). Thus, the 3D model and the shell model are applied in sub-domains of variable sizes. By performing this coupling, we consider the same spatial domain in LAM3 code and in the present buckling code. Thus, we avoid boundary conditions problems that are necessary within Abdelkhalek's model described above.
Numerical applications The coupling procedure presented previously will be applied to the numerical simulation of flatness defects generated by a rolling process, i. e. of deformed shapes of the sheet due to residual stress field. We are interested in very thin sheets where flatness defects are of great interest. This procedure is non-iterative: firstly a rolling calculation is performed using LAM3 code; secondly, the residual stress field resulting from this calculation is used as a datum in the present code 3D/shell/Arlequin. The results are compared with Counhaye's  [10]  and Abdelkhalek's  [4]  models and with experimental measurements. Counhaye's model is similar as Roddeman's approach to account for the influence of wrinkling on membrane behaviour  [30]  in which the constitutive law is modified to limit the in-plane compressive stress.
However, the challenge is to choose appropriately the ingredients of the method, in particular the size of the gluing zone and the weighting functions. In the proposed study, we limit ourselves to the effectiveness and validation of the Arlequin coupling between 3D and shell models in the framework of buckling computation under residual stresses. An originality of this work lies in the coupling area which varies during the process simulation. Since the resulting problem is nonlinear, it is solved using Asymptotic Numerical Method which is an efficient tool for solving problems involving instabilities. We have proposed first a simplified model by introducing residual stresses by analytical formulae. Moreover, we performed an industrial rolling test case taking into account all the components of the residual stresses obtained by the help of the rolling code LAM3.
Acknowledgments The authors wish to thank the  French National Research Agency (ANR)   ANR-11-LABX-0008-01 ,  LabEx DAMAS , for its financial support and the partners of the  ANR PLAT-FORM , under contract no. 2012-RNMP-019-07 , ( ArcelorMittal, CEA, CEMEFMINES-PARISTECH, Constellium, Ecole Centrale of Paris, INSA Lyon and Paul Verlaine university of Metz ) for the authorization to publish this work.
Request Flow Performance Indicator K, λ1, λ2,. , λK K, n1, n2,. , n K C → HR(C) Figure  1 : Schematic view of a performance model (left); example in the IRM case (right) #yt and #vod respectively. Dataset #yt comes from the YouTube traffic delivered for three months in 2013 by the Orange Network in Tunisia, while #vod comes from the Videoon-Demand Orange service in France for 3. 5 years. The traffic consists in 46 000 000 (resp. 3 400 000) requests to 6 300 000 (resp. 120 000) videos in the #yt (resp. #vod) set. More details on the collection and processing of these two datasets can be found in  [17]. We also use two synthetic datasets, called #prt and #delta. This allows us to highlight in a more clear way some of our findings and, more importantly, to validate the results with controlled experiments when the ground-truth is not available.
In this model, the number N k of requests for document k in a time window W is an independent Poisson random variable P(λ k W ) of mean λ k W. Up to a time normalization, we assume in the following that W = 1. Figure  1  illustrates those two stages, both for an arbitrary performance model and the IRM case. The first stage consists in mapping the model parameters to a request flow (or a request flow distribution). The second step of the model computes the performance indicator, based on this request flow. In order to keep this paper concise, we now limit ourselves to the IRM model (see Section 7. 1 for extensions). Assume now that an observer has access to a sample of the actual request flow, e. g. , a trace dataset or server logs. In the case of IRM, a sufficient statistics of the request process is the request counts n1, n2,.
Second, each document popularity λ k is estimated by a single sample n k of the random count N k. This last limitation is well illustrated in the case of the #delta dataset. By definition, the ground-truth (real) popularities are λ k = 4. In the dataset, however, the counts of document requests are Poisson random variables of mean 4, hence λnv k = P(4) and the naive estimation "dilutes" the mass of popularities over the set of positive integers. In Figure  2.
Thus, we can first obtain an estimator ĝ of the mixing distribution g, and then approximate K by Kml = K0 E ĝ [1 -e -λ ] (2) which is asymptotically close to the ML estimator. We now proceed with the detailed form of the likelihood function for the parametric and non-parametric estimation procedures. In both approaches, we numerically solve the problems with a generic non-linear optimization solver in MATLAB based on an interior point algorithm. Our code is freely available online. 1 We discuss the use of specialized algorithms in Section 7.
Estimation of popularity distribution First, we start with the most general question, that is, the estimation of the mixing distribution. Such an inverse problem is known to be ill-posed. For the NP estimation, we obtain an estimate ĝnp of the popularity density by applying the NP method, using a support with 0. 01 as lower bound, exponentially increasing spacings and an upper bound slightly larger than the maximum of observed requests (e. g. , 2 400 for #prt and 16 for #delta). The naive fitting corresponds to the empirical measure of the request counts, that is, the mixture of Dirac measures 1 K 0 K 0 k=1 δn k (. ). We observe in Figure  3  the NP estimator of the mixing distribution for the #delta and #prt datasets. In the #delta case, the ground-truth is a Dirac measure at λ = 4, and the naive method fails at correctly estimating its shape, whereas the ML estimator concentrates its mass around the In the #prt case, the estimated distribution is irregular, tending to accumulate mass at certain points (see Section 7.