dataesr/scientific-paragraphs-categorization huggingface dataset:
Final scores and multiple candidate SMILES were reported for 199 challenges for submission to CASMI 2016. Nine challenges could not be processed due to time constraints (Challenges 13, 61, 72, 78, 80, 106, 120, 133, 203). Full details on this entry, MS-FINDER and file modifications required are given in Additional files 1 and 2. Team Verdegem (Dries Verdegem and Bart Ghesquière) participated in Category 2 with MAGMa+  [42] , which is a wrapper script for the identification engine MAGMa  [43]. For any given challenge, MAGMa+ runs MAGMa twice with two different parameter sets. A total of four optimized parameter sets exist (two for positive and two for negative ionization mode), which all differ from the original MAGMa parameters. Within one ionization mode, both corresponding parameter sets were each optimized for a unique latent molecular class.
Following the outcome of both MAGMa runs, MAGMa+ determines the molecular class of the top ranked candidates returned by each run using a trained two-class random forest classifier. Depending on the most prevalent molecular class, one outcome (the one from the run with the parameters corresponding to the most prevalent class) is returned to the user. The candidate lists provided were used as a structure database without any prefiltering. MAGMa determines the score by adding an intensity-weighted term for each experimental peak. If a peak is explained by the in silico fragmentation process, the added term reflects the difficulty with which the corresponding fragment was generated. Otherwise, an "unexplained peak penalty" is added. Consequently, MAGMa returns smaller scores for better matches, and therefore the reciprocal of the scoring values was submitted to the contest.
Separating the challenges into positive and negative modes revealed that Team Dührkop clearly led the positive mode predictions (82 wins/gold medals and 70 Top 1 candidates, versus 66 wins and 53 Top 1 candidates for Team Brouard). Both MS-FINDER (14 Top 1) and CFM_orig (12 Top 1) outperformed Team Brouard for negative mode (9 Top 1), showing that a greater amount of training data for negative spectra would likely improve the CSI methods in the future. The training set used by Team Brouard contained 7300 spectra for positive mode and only 816 negative mode spectra. The difference between positive and negative mode was less dramatic for the other approaches. The results of Category 2 were dominated by the methods that use machine learning on large spectral databases (GNPS  [32] , MassBank  [33] , METLIN  [19]  and NIST  [20] ), namely Teams Brouard and Dührkop (CSI) and Allen (CFM).
These also outperformed the MS library (MoNA) entry. The retention time prediction alone performed poorly, because this does not contain sufficient structural information to distinguish candidates, as demonstrated in Additional file 1: Figure  S2. The lowest identifier strategy, which was used as a "gut feeling" decision criteria commonly in environmental studies before retrieval of reference information could be automated, takes advantage of the fact that well known substances were added to ChemSpider earlier and thus have lower identifiers. Surprisingly this still outperformed the combined fragmenters-but again this is highly dependent on the dataset. The references outperformed all individual metadata categories and even the combined fragmenters clearly. The influence of the metadata is discussed further in "Metadata and consensus identification" section.
Machine learning approaches and training data The CASMI2016 results show very clearly how the training data influences the performance of different approaches. The difference in Top 1 positive mode ranks between CSI:IOKR_A, 62 and CSI:FID, 70 (see Table  2 ) were due to the different training sets used, the CSI:IOKR_AR results (retrained on the same data as CSI:FID) had 69 Top 1 ranks. The results for CSI:IOKR in negative mode were also generally worse than all other approaches, which shows that the decision of Team Dührkop not to submit entries due to a lack of training data was quite well justified (even though it likely cost them the overall contest "win" for Category 2). Team Dührkop noted that there was a large overlap between the challenges and their training set and investigated this with the CSI:FID_leaveout entry (described in the methods).
The general observations made on the full contest data are supported by this reduced dataset as well, despite the unsurprising fact that the results on this reduced dataset were generally worse than the official contest results (see Table  2 ). This demonstrates that, as expected, machine learning methods do better on compounds from within their training sets (for example, the percentage of maximum Top 1 ranks dropped from 34 to 18%). Although the median ranks were worse, the Top 10 ranks still remained around 40-50% for most methods. Cluster plots on this reduced dataset for negative and positive mode, given in the supporting information (Additional file 1: Figures  S5, S6 ), show similar patterns to the cluster plots on the full dataset. Interestingly, these results show that the CSI:FID_ leaveout entry outperformed CSI:FID, while CSI:IOKR_A also outperformed CSI:IOKR_AR, the retrained dataset, also for some different scores-similar observations could be made for CFM_orig versus CFM_retrain.
While this could be a potential sign for overfitting, this is a small dataset and some or all of these observations could be due to fluctuations in the data. Overfitting is a potential problem that developers, especially of non-standard machine learning methods should test for, e. g. by checking if their performance decreases significantly for compounds which are structural dissimilar to compounds in the training data. These results highlight just one means by which the choice of training set can influence the performance of automated methods. The training set can also impact challenge results in a range of other ways that are harder to disambiguate. One training set may be more or less compatible with the challenge set, even after common compounds are removed. This suggests the importance of assessing automated methods using the same training set, where at all possible.
Metadata and consensus identification The dataset for CASMI 2016 was predominantly wellknown anthropogenic substances and as a result there are many distinct and highly referenced substances in the candidate lists. This is shown in the huge improvement that the metadata made to the ranking performance (Tables  4, 5 ). Figure  3  shows clearly that the vast majority of substances were either ranked first or second based purely on the reference count, with most other candidates having much lower counts. Figure  4  gives an overview of the contribution the metadata made to each approach based on the CASMI 2016 entries, merging team results in the case of MS-FINDER. In the environmental context, it is quite common to search an exact mass or formula in databases such as ChemSpider, where e. g. the highest reference count as well as the substance with the "lowest CSID" are often picked as the most promising hit in many cases, discussed e.
g. in  [49]. The success with these strategies would have been quite considerable with this dataset. However, for new (emerging) anthropogenic substances and transformation products of known chemicals, these strategies would not work so well as they would have neither a high reference count nor a low database identifier. This situation is also likely to be drastically different for natural products and metabolites, where many more closely-related substances or even isomers could be expected. The metadata results in Category 3 show that the importance of the sample context cannot be ignored during identification, especially for studies looking to find well-known substances. This is also highlighted by the comparison with the approaches used in Category 1, where also manual and semi-automatic approaches were considered.
The current reality is that most automated approaches still depend on retrieving candidates from compound databases containing known structures-i. e. the situation replicated in this CASMI contest. Compound databases such as the Metabolic In Silico Network Expansion Databases (MINEs)  [50]  could be used as alternative sources of candidates for predicted metabolites in the metabolomics context, but would have had limited relevance in this contest. While metadata, the way it was used here, will not help in the case of true unknowns, there are two cases to consider for automated approaches at this stage. For "unknowns" that happen to be in a database almost accidentally (e. g. a to-date unknown transformation product), the automated fragmentation approaches are very useful, because these structures can be retrieved from substance databases.
However for true "unknown unknowns" that are not in any database, fragmenters could only be used in combination with structure generation, which is still impractical with the quality of data and methods at this stage unless candidate numbers can be restrained sufficiently. These cases are often extremely difficult to elucidate using MS n alone and the informa- tion from additional analysis such as NMR will usually be necessary. Stereoisomerism is another aspect of identification that was not covered in this contest. None of the current approaches are able to distinguish stereoisomers (even cis/trans isomers) using only MS/MS information for known unknowns. The evaluation of this contest addressed this by taking the best scoring stereoisomer and eliminating others (see "Methods") to reduce the influence of stereoisomers on the ranking results.
4  The influence of Metadata on CASMI 2016 first seven groups-light green MS/MS information only, i. e. Category 2. Dark green with metadata, i. e. Category 3 participants. Note these are plotted according to the number 1 ranks, not wins. Next 4 groups: dark green metadata only; Last group: light green is the equally-weighted combination of the six individual Category 2 entries and dark green is this plus metadata as shown in Table  5  this, as a machine-learning approach with similar training set coverage to CSI, which allowed the complementary approach of MS-FINDER to claim third place ahead of CFM. An alternative approach could be to look at this in terms of overall success and say that if a team had the correct structure as the 20th hit and other teams were even worse, none of the approaches were really sufficient to the task and nobody should then earn a 'win'.
Participant perspectives Team Allen submitted two alternative versions of CFM, the main difference being that for CFM_retrain version, additional training data was added from the 2014 NIST MS/MS database. While the addition of extra training data may have been expected to improve the results, this appears not to have been the case for this competition. One possible reason for this is that the additional data were generally of poorer (often integer) mass accuracy as compared to that used to train the original CFM model. This required a wider mass tolerance (0. 5 Da) to be used during the retraining (compared to 0. 01 Da previously), which may have hindered the training algorithm from accurately assigning explanations to peaks, and so modeling their likelihoods. This highlights that while the production of larger, more comprehensive data sets is likely crucial for better training of automated methods, the quality of these data sets is also very important.
Most automated methods would likely benefit from training on cleaner data with better mass accuracies. Team Dührkop investigated how CSI:FingerId compared with a direct spectral library search. A spectral library containing all structures and spectra used to train CSI:FingerId was created and searched with a 10 ppm precursor mass deviation. The resulting spectra were sorted via cosine similarity (normalized dot product), again with 10 ppm mass accuracy. Candidates were returned for 91 of the 127 (positive mode) challenges; the correct answer was contained in the library for 69 of these. The spectral library search correctly identified 63 of the 69 structures in total, 40 of these were "trivial" (the correct answer was the only candidate). On average, candidate lists for the spectral library search contained only 2.
Team Vaniya The CASMI Category 2 contest was a reshuffling contest: potential structures were given to all participants, listing one to over 8000 potential structures for each challenge. These structures were within 5 ppm mass accuracy and often included different elemental formulas. Therefore, Category 2 was a 'structure dereplication' contest, finding the best structure within a pre-defined list of structures, not a completely open in silico test on all exhaustive structures in the chemosphere. In practical terms, it is important to note that an in silico software does not eliminate the time consuming aspects of data preparation, formatting, and interpretation. Counting the computing power and manual effort between two people, it took about 24 h to complete the 208 challenges for the MS-FINDER submission.
From Table  2 , one could say that MS-FINDER was best based on the mean rank (19. 75), but ranks lower than 10 are less relevant in reality. While MS-FINDER had almost 50% of the challenges within the top 10 ranks, so did every other software (or team). In reality, no chemist would use a software without any database or mass spectral library behind it. The importance of using a priori knowledge is seen by Team Allen's submission that improved the Top 1 correct structure hits from 39 to 120 challenges in Category 3, a bit more than 50% of the challenges. Hence, we conclude that the glass is half full: if only in silico methods are used, some 50% of the challenges are within the top 10 hits within the structures given by the CASMI organizers. However, many challenges would score much higher if other metadata are used, e.
g. constraining the search database to particular classes of compounds that can be expected for a specific study. Which parameters need to be optimized, and which a priori metadata should be used? Those questions may be answered in a more tailored future CASMI contest. Team Verdegem participated in Category 2 of the CASMI 2016 contest with MAGMa+, which is a fast, plugand-play method relying on combinatorial fragmentation without requiring a preliminary training phase for improved performance. The entire submission, including scripting for automation and single core calculations, took less than 1 day. MAGMa+ outperformed MAGMa, showing the use of the parameter optimization performed to improve several second and third ranked candidates to first place. MAGMa+ shared the best ranking for 44 of 208 challenges (see Table  2 ) and performed considerably better than other contestants for nine of those challenges  (21, 32, 36, 40, 52, 61, 121 , 157 and 189), indicating the relevance of the underlying algorithm.
Since MAGMa+ outperformed MAGMa according to some (e. g. number of gold medals, Top 1 and 3 ranks) but not all metrics, further more advanced parameter optimizations are planned to achieve a more global performance improvement. However, further improvements to the performance of MAGMa/MAGMa+ will require interventions of a different kind. The performance of MAGMa+ decreases with increasing candidate numbers (in this contest 1116 on average after the removal of duplicate stereoisomers), however, in case of smaller numbers, it starts to outperform some of the other methods  [25, 42]. For untargeted metabolite identification in biological/biomedical setups, it is arguably more suitable to restrict the candidate structure database to those metabolites known to exist in the organism under study, e.
g. using only the ≈42,000 metabolites currently present in the HMDB  [21]  for samples of human origin. This was noted also in previous CASMI contests  [2]. Many candidate structures had identical scores with MAGMa+, resulting in the correct matches being given lower ranks according to the evaluation rules. Whereas on average 1098 structures were retained from the structure database based on the parent mass match, only 616 different score values were observed (on average). Team Verdegem will investigate more discriminative scoring options for MAGMa+ in the future.
Conclusions This was the first CASMI contest to use a large set of challenges, targeted especially at the automated methods. This decision was taken on the basis of feedback from several representatives at the 2015 Dagstuhl seminar in Computational Metabolomics  [51] , to allow a statistically more robust comparison of the methods. The decision to provide candidates this year was also on the basis of Dagstuhl discussions, to eliminate the data source as an influence on the contest outcomes and thus focus more on the role of the in silico fragmentation approaches themselves. From the perspective of the organizers, it was a great success to have participants contribute from each of the major different approaches; MetFrag was added internally for the sake of completion as this was not otherwise represented and allows this paper to complement the work in  [25]  on a different dataset.
The role of the metadata and comparison with Category 1 shows that sample context cannot be ignored during identification. In this contest, few participants used the CASMI training set provided, which was also a suggestion from Dagstuhl. In the end this was too "big" for pure parameter optimization (where a few spectra may suffice), but too small for serious method training. Team Brouard added it to their other training data in their original submissions, while it was used to determine the score weights in the MetFrag entries. Team Vaniya did not use this for MS-FINDER to avoid over-training; Team Allen due to a lack of time. One conclusion from the post-contest evaluation is that future CASMIs could consider providing an extensive, open training dataset (e. g. the GNPS/ MassBank collection used by CSI:FID) and ensure all CASMI challenges are absent from this set.
This would, however, force all machine-learning approaches to retrain their methods prior to submission. Another option is that the organizers would have to ensure that all challenges are outside all available datasets-which is possible but also difficult with the number of private and closed collections available. A compromise could be to ensure that a sufficient majority of the candidates are outside the "major" mass spectral resources, with some overlap to ensure sufficient challenges are available (finding data sources for CASMI is a challenging task!) and require participants to submit InChIKey lists of their training sets with their submissions; as done with Teams Allen, Brouard and Dührkop post-contest here. Challenges for future contests remain true unknowns, i. e. substances that are not present in compound databases.
This would currently be feasible for manual approaches and was attempted already once in CASMI 2014, Challenges 43-48  [54] , albeit with limited success. Automated approaches would need either a metabolite database such as MINEs  [50]  or structure generation  [55] , but finding sufficient appropriate data for an automated category will also be a challenge for the contest organizers, let alone the participants! The ability to distinguish stereoisomers using MS/MS alone also remains a challenge for the future that is not yet ripe enough for a CASMI contest; distinguishing (positional) isomers is likely sufficient challenge for the next few years. The huge improvements in machine learning approaches will continue as more training data becomes available-the more high quality data with likewise high quality annotations that becomes available in the open data domain will ensure that the best computational people can work on the best identification methods.
Acknowledgements CASMI is a joint effort and we wish to thank all organizers and participants of previous CASMIs as well as other interested parties for valuable discussions. For this manuscript, AV wishes to acknowledge the contributions of  Stephanie N. Samra ,  Sajjan S. Mehta ,  Diego Pedrosa  and  Hiroshi Tsugawa. KD  and team acknowledge the contributions of  Marvin Meusel. FA and TS acknowledge the contributions of  Russ Greiner  and  David Wishart. MK acknowledges the help of  Janek P. Dann  with spectral acquisition. We thank the reviewers for their comments and suggestions.
These observations were confirmed by biometrical data showing that atltp2-1 hypocotyl diameters became significantly smaller than the WT ones a few minutes after their transfer to normal atmospheric conditions (Figure  5B ). Moreover, non-functional cuticle often showed an increased permeability to water-soluble molecules that can be monitored with toluidine blue (TB) hydrophilic dye uptake  (Tanaka et al. , 2004). The atltp2-1 hypocotyls showed an intense TB coloration (Figure  5C ) as compared to those of the WT and of the two selected complemented lines which remained mostly white. Toluidine blue was extracted for quantification from stained aerial organs. Significant 6-to 10-fold higher stain intensity was revealed for atltp2-1 seedlings as compared to WT and complemented lines, in agreement with the macroscopic observations (Figure  5C ).
Hypocotyl diameters were measured using the ImageJ software. The results were analyzed by ANOVA and Tukey HSD test (α = 0. 01) using R software. Different letters indicate significant differences at p-value < 0. 01. (C) Permeability of etiolated hypocotyls was examined using the TB staining test. TB uptake was quantified spectrophotometrically at OD 626 nm. Results are presented as mean values ± SD (n = 5). Significant differences from WT were assessed by a Student's t-test ( * * p-value < 0. 01). contained substantial amounts of very long-chain fatty acids with less than 28 carbon atoms (mainly C24) and very longchain fatty alcohols with less than 26 carbon atoms (mainly C22; Figure  6A ). Such compounds were found as very minor components of A. thaliana leaf and stem waxes, which usually contained similar compounds (i.
DISCUSSION In Etiolated Seedlings, the Abundance of AtLTP2 Gene Products Is Restricted to Cuticle-Carrying Epidermal Cells The rationale to study AtLTP2 was primarily linked to the fact that it was the unique and abundant member of the nsLTP multigene family present in the cell wall proteome of etiolated hypocotyls  (Irshad et al. , 2008). In agreement with the AtLTP2 expression patterns described in the Arabidopsis eFP browser database  (Winter et al. , 2007) , we found that the expression of AtLTP2 was 15-30 times increased over a 7 days period in the dark. In addition, we have demonstrated here that in etiolated seedlings, the AtLTP2 strong expression was restricted to epidermal cells of hypocotyls and to a lesser extent of cotyledons. In contrast, when atltp2-1 plants were grown under a 16 h light/8 h dark photoperiod in soil in a greenhouse to produce seeds, we did not notice any obvious change in germination rate, early vegetative growth, flowering time, and seed production in comparison to WT (data not shown), suggesting low expression under normal conditions.
Water-Loss and Toluidine Blue Assays Five-day-old etiolated seedlings from WT and atltp2-1 (n = 11) were aligned on a microscopy slide and imaged with an Axiozoom V 16 14 during a 25 min kinetics to follow their degree of desiccation. Hypocotyl diameters were measured using the ImageJ software 15. The results were analyzed by 11 http://www. riken. jp/en/ 12 www. thermofisher. com 13 www. leica-microsystems. com 14 www. zeiss. fr/microscopy 15 https://imagej. nih. gov/ij/ ANOVA and Tukey HSD test (α = 0. 01) using the R software  (R Core Team, 2016). In parallel, 500 mg of WT, atltp2-1, and complemented lines (5 2 and 14 3) seedlings were put in Petri dishes. They were placed in a 25 • C oven and weighted with a precision balance during a 3 h kinetics to monitor waterloss. Dry mass was measured after 3 days at 25 • C.