dataesr/scientific-paragraphs-categorization huggingface dataset:
The annotations are directly linked to this graph. Figure  4 (b) shows the syntax for this example. Singleton properties. The singleton property model  [8]  proposes creating a unique property for every triple that has associated metadata. In this model, a new node is created for each triple that needs to be reified. This node is connected directly to the original property using singletonPropertyOf. The created node is linked to all metadata associated with a statement. An example of RDF triples using this model is shown in Figure  4(c). RDF-Star. [9] 10  proposed RDF-star and SPARQL-star as extensions to RDF and SPARQL to enable graph nesting and simplify the representation of reified statements. RDF-star and SPARQL-star allow for the recursive nesting of graphs, eliminating the need for declaring edge identifiers that are linked with metadata.
Flexibility. All of these reification models are flexible when it comes to adding new annotations to an already reified statement. Adding new annotations only requires adding one additional triple for each approach. Additionally, all of these models cause no issues with multi-valued properties. Also, one advantage specific to named graphs is that reification can be defined also for a group of triples or even a dataset. Syntax support. Standard reification and singleton properties conform to the core RDF model proposed in 2004. Named graphs represent an extension to the triple RDF model and is part of the standard RDF1. 1, which was published in 2014. RDF-star proposes to extend the RDF specification further. Concerning the query language, all of these models are supported in the SPARQL standard, except for RDF-star which proposes SPARQL-star as a query language.
Experimental setup Experiments were run on a virtual machine with 128GB of RAM, 2GHz with 32 cores, on a Debian GNU/Linux 11 (Bullseye). All tests were run using docker images of the triplestores 14 15 16 17. All triplestores were parameterized with a query timeout of 30 minutes and given access to 16 GB of RAM. Only Oxigraph was not parameterized as we did not find the way to do it. GraphDB was also parameterized to use a context index when dealing with the named graphs version of the graph. All four triplestores were evaluated with the four reification models except for Virtuoso which does not support RDF-star, and GraphDB which does not support singleton properties in an efficient way. This is because GraphDB makes the assumption that there will be only a small number of properties in the graph.
Given a concept, it returns the associated ERs whose PageRank score is greater than the cosine similarity using the FILTER keyword. Q3 is a join query. It returns the set of ERs associated to three given concepts. Q4 is similar to Q3 but with a FILTER that specifies how the three PageRank scores must relate together. 12 https://w3c. github. io/rdf-star/implementations. html 13 https://w3c. github. io/rdf-star/cg-spec/editors_draft. html#sa-mode-and-pg-mode Methodology. Queries were instantiated with a number of groundings depending on the query as some queries need three different concepts (Q3, Q4, Q6). In that case only three grounding were given, each composed of three concepts. Instantiated queries were executed sequentially in increasing order of the size of the corresponding multivalued property.
Experiments used Wikidata and DBpedia datasets on the triplestores Blazegraph, Stardog and Virtuoso. As DBpedia does not have singificant metadata, authors build a dataset with the Wikipedia revision history focusing on a company dataset. The experiments show that when the granularity of metadata is not by statement, companion properties and named graphs outperform. Concerning statement-level metadata, while standard reification results in the highest number of triples, it consumes the least storage in the database files and named graphs the most. This is because additional index structures for the graph identifiers are maintained. Concerning query performance, metadata characteristics have an impact on the reification models. Named graphs and RDF-star support queries against meta-metadata much better than the other models.
Authors compared experimentally LKG with Singleton Property, RDF Reification, Named Graph and PaCE  [17]. Used datasets were (with and without meta-knowledge): SPARQL Performance Benchmark (SP 2 Bench), the Biomedical Knowledge Repository (BKR), and the Gov-track. Results highlighed that LKG outperforms these methods by generating fewer statements, having a smaller graph size, avoiding resource redundancy, and achieving faster query response time. [12]  proposes a benchmark (dataset and set of queries) to analyse reification models. To illustrate the utility of the benchmark, authors analysed querying performance, storage efficiency and usability on the Stardog triplestore using three reification models: standard reification, singleton properties and RDF-star. Authors used the Biomedical Knowledge Repository (BKR) dataset  20  in order to make their results comparable with  [8]  that compares singleton property against standard reification.
Twelve queries are used to evaluate performance. Five of these queries were proposed in this work to focus on SPARQL-star. Experimental results suggest that singleton property seems to have the worst performance. Probably because of the high number of unique properties and because indexes are usually not optimised with that in mind. Authors also observed that for simple queries, standard reification performs better than RDF-star. For complex queries, clearly, RDF-star outperforms standard reification. [18] presents a novel approach for representing metadata, which outperforms existing reification models such as Singleton Property, Named Graph, PaCE, Companion Property, N-ary relations, RDF-star, and RDF-star  [19]. The authors employed various datasets for their study, including a BKR dataset, a Gov-track dataset, a Synthetic dataset, and a dataset obtained from  [20].
Acknowledgments This work has received a French government support granted to the  Labex Cominlabs excellence laboratory  and managed by the  National Research Agency   in the "Investing for the Future" program  under reference  ANR-10-LABX-07-01. The authors thank  Corentin Follenfant  for his valuable contribution during the initial stages of this work. Additionally, the authors would like to acknowledge the Master's students in  Computer Science at Nantes University  for their involvement in various facets of this project.
Imaging mass spectrometry enables spatial metabolomics, yet metabolites can be assigned only to a fraction of the data generated. METASPACE-ML is a machine learning-based approach addressing this challenge. Integrating new scores and computationally-efficient False Discovery Rate estimation, trained and evaluated on 444 representative datasets from 44 groups, METASPACE-ML surpasses the rule-based predecessor in precision, coverage, and computational efficiency. Our work helps illuminate dark matter in spatial metabolomics with machine learning.
Introduction Imaging mass spectrometry (imaging MS) has emerged as a leading technology in spatial metabolomics, finding applications in diverse fields such as biology, medicine, and pharmacology  [1] [2] [3]. However, a key challenge in this field is the accurate and confident annotation of metabolites, primarily due to limitations in data collection, existing algorithms and software  4. Similar to bulk metabolomics  5  , the vast majority of the imaging MS data, so-called "dark matter", cannot be molecularly annotated with existing tools. We previously developed METASPACE, an engine for metabolite annotation  6  and a community-populated knowledge base  7. METASPACE utilizes a False Discovery Rate (FDR)-controlled approach, where metabolite ions are reported at a given confidence level by ranking them against implausible generated decoy ions.
However, a limitation of METASPACE is its rule-based scoring system, namely Metabolite Signal Matching (MSM), which assigns equal weights to the features and lacks adaptability to data variations. Previous attempts to employ a data-driven rescoring approach did not consistently increase the number of annotations across diverse datasets  8. Here, we introduce METASPACE-ML, a machine learning model for FDR-controlled metabolite annotation for imaging MS, incorporating novel features and trained on 165 representative datasets from 40 groups. By evaluating METASPACE-ML on 279 datasets from 20 groups, we demonstrate its superiority over the traditional MSM rule-based approach, delivering more annotations, particularly at low FDR thresholds. We offer an efficient implementation using the Lithops serverless framework  9  This advancement enables richer molecular information from imaging MS data to be extracted with enhanced likelihood of correct assignment.
Comprehensive set of representative test datasets For training and evaluation, we selected two extensive sets from 6,478 public METASPACE datasets (as of July 12th, 2022) and EMBL datasets. The training and testing sets comprised 165 and 279 datasets (see Methods), respectively, selected to be representative of METASPACE datasets (Figure  2A ). With a balanced distribution of positive and negative polarity, MALDI was the most prevalent ionization source (119 out of 165), often coupled with Orbitrap or FTICR analyzers. Approximately 65% represented common mammalian model organisms (human, mouse, and rat). Human derived samples were predominantly analyzed via FTICR analyzers, with over 90% having a high resolving power (≥ 200k @ 200 m/z). The majority of the datasets (155 out of 165) were acquired with a high upper m/z limit (maximum m/z > 500).
METASPACE-ML outperforms the rule-based approach particularly for highly-confident annotations To assess the performance of the METASPACE-ML model, we conducted a comparative evaluation against the rule-based MSM approach  6. The mean average precision (MAP) was used to evaluate the quality of target vs decoy ion rankings. When using cross-validation on the training datasets (see Supp Figure  S1  for the cross-validation splits), METASPACE-ML demonstrated a significantly superior performance (p-value < 0. 0001), with higher MAP scores (Figure  2B ) and a higher median MAP value (0. 36 vs 0. 28). The evaluation on the 279 testing datasets highlighted the applicability and consistent improvement achieved by METASPACE-ML in annotating both metabolites and lipids, with particularly promising results for the novel CoreMetabolome database (Figure  2C , Supp Figure  S2 ).
METASPACE-ML consistently outperformed the MSM approach, enabling the discovery of substantially more annotations: At the default FDR 10%, METASPACE-ML, on average, identified approximately 15 more annotations and at least 30 more annotations in half of the datasets across all databases (Figure  2D ). At FDR 5%, METASPACE-ML revealed approximately a two-fold increase in annotations compared to the MSM approach in half of the datasets across all databases (Figure  2E ), indicating a greater improvement for most-confident annotations at lower FDR. This trend was consistently observed for each database, with higher relative improvements for LipidMaps and SwissLipids at lower FDRs compared to CoreMetabolome (Supp Figure  S3 ). METASPACE-ML also performed significantly better for data with medium-high resolving power (Wilcoxon p-value < 0.
METASPACE-ML helps annotate low-intensity ions Having established the improved performance of METASPACE-ML in delivering more annotations, we examined the impact of ion intensities. In nearly all test datasets with annotations at FDR 10% (221/233), compared to MSM, newly-found annotations by METASPACE-ML had significantly (Wilcoxon p-value<0. 01) lower intensities (Figure  2I ; Figure  S6  for a detailed example). This property may be especially useful in annotating biologically relevant ions corresponding to low-concentration metabolites, and thus turning acquired data into actionable hypotheses. Among molecular classes newly-found by METASPACE-ML carbonyl compounds, estrane steroids, and carboxylic acid derivatives were the most and significantly (Fisher p-value<0. 05) overrepresented (Figure  2J ).
METASPACE-ML outperforms MSM in runtime We evaluated the computational performance of METASPACE-ML open-source implementation using the Lithops serverless framework 9 , available at https://github. com/metaspace2020. Compared to MSM on the considered subset of the 50 training datasets, on average, METASPACE-ML exhibited reduced run time (Supp Figure  S7 ) likely due to the novel way of decoys selection (see Methods). This demonstrates the readiness of METASPACE-ML for public use on the METASPACE platform, where it is already accessible to early adopters.
Discussion Compared to our previously proposed adaptive Support Vector Machine classification  8  , the Gradient Boosting Decision Trees (GBDT) approach, formulated as a ranking model, demonstrated enhanced performance across a broader range of datasets. This proved particularly valuable as GBDT is known to be robust to noisy outliers, which can significantly impact the separation of target and decoy ions. We defined ways to calculate absolute and relative m/z error for centroided data. As expected, this helped improve the accuracy of the model predictions. Given the increasing use of centroided data, these scores hold potential for wider application beyond METASPACE-ML. We introduced a novel expert-curated metabolome database, CoreMetabolome, which resulted in better improvements compared to more general databases.
Despite the notable performance and annotation coverage achieved by METASPACE-ML compared to the rule-based MSM approach, we acknowledge its limitations. The quality of the training datasets affects the performance. We have realized it by observing an improvement when excluding non-centroided datasets submitted by mistake to METASPACE by setting a cut-off on the number of peaks (50. 000). In terms of scores, rho_chaos 6 had only a minor impact on the model performance (Figure  2H ) which demands re-defining the quantification of spatial informativeness. As discussed earlier  6  , the way of producing decoy ions is key and new data shows their heterogeneity in terms of similarity to the target ions (Figure  2G ). Further investigations are needed into how this affects FDR estimation and in finding the most reliable way of decoy selection.
Lastly, although the training datasets were selected to be representative for a large number of public datasets from 44 labs, this selection is biased towards datasets represented on METASPACE. This warrants further work for evaluating METASPACE-ML for less common data e. g. from cultured cells in single-cell metabolomics, or data from industrial labs, where the ability to publically deposit data may be limited. Finally, while the introduced CoreMetabolome database does offer some degree of improved performance, more rigorous and automated methods for generating such databases are desired. Due to its flexible architecture, METASPACE-ML can include additional features. Of particular interest is the integration of Collisional Cross Section (CCS) values, enabling automated use of ion mobility separation able to resolve molecular isomers and isobars  11.
Selection of training datasets In order to train the model, we selected datasets from METASPACE in an unbiased way to balance maximizing diversity of exploited technologies, acquisition parameters and labs while also staying relatively representative to all datasets submitted to METASPACE. The training datasets were selected from 6,478 public METASPACE datasets (as of July 12th, 2022) as well as 4,358 private EMBL datasets with the METASPACE API (https://metaspace2020. readthedocs. io). Each dataset was given a weight based on the values of the corresponding parameters from the metadata (polarity, ionization source, analyzer, resolving power range, submitter, group). Those parameters are ordered to highlight their respective impact on the selection, with polarity having the highest impact and group having the lowest.
Datasets from the same METASPACE group with similar names (e. g. "SlideA", "SlideB") had proportionally decreased selection weights compared to the rest of the datasets to reduce the grouping effect. In addition, the following exclusion criteria have been applied: datasets submitted by METASPACE developers, datasets with no submitter information, and datasets with fewer than 10 HMDB annotations at 10% FDR according to the rule-based approach. After applying the exclusion criteria, 200 training datasets have been randomly selected based on their assigned weights (Table  S2 ). The datasets were further cleaned up to match the required input of the employed ranking model (CatBoost; see below): we considered only ions with non-zero MSM scores. Furthermore, for a dataset we considered only target adducts which had 10 or more rule-based annotations at FDR 20% and decoy adducts which had 10 or more rule-based annotations at FDR 50%.
Processing training data using the rule-based METASPACE and CoreMetabolome The selected training datasets were then reprocessed on the METASPACE server using the rule-based approach as previously described  6  with minor modifications to their configuration file (Section S1). The datasets were annotated against the CoreMetabolome database. The CoreMetabolome database is a molecular database specially designed and developed by us for METASPACE annotation. CoreMetabolome was designed to be large enough for untargeted spatial metabolomics, include only chemically plausible molecules, prioritize endogenous molecules over exogenous molecules, and include the molecules from primary metabolic pathways that are commonly abundant in different types of samples. HMDB (version 4) and KEGG were used as input databases and curated manually by an experienced chemist and mass spectrometrist.
Training and cross-validating the model We employed a ranking-based model using gradient boosting decision trees implemented using the CatBoost framework  12. The model was trained on 200 processed training datasets. As input features, we used the original MSM features (spatial isotope, spectral isotope and spatial chaos) in addition to the newly introduced features (relative and absolute m/z error), five features in total. The CatBoost model was first initialized using the "CatBoost'' method using PairLogit as the loss function and fitted on the 165 training datasets for 1000 iterations. In each iteration, the decision trees are built in such a way that it improves the previous trees' output based on a loss function. For each dataset, a decision tree at a specific iteration scores all decoys and target ions based on their feature scores, using combinations of pairwise objects where one is considered a winner (target ion) and the other a loser (decoy ion).
The pairlogit loss function  13  selects the best tree that maximizes the positive difference between the tree score for the target vs decoy ion. To evaluate the model and ensure that it was not overfitting, we performed cross validation where the training datasets were split into 5 splits, where 80% of the dataset were used for training and the remaining 20% were used for evaluation (see Evaluation metrics and benchmarking). The final prediction score of the model was scaled [0,1] so that scores closer denote high confidence annotations and vice versa.
Mean average precision (MAP). In order to evaluate the performance of METASPACE-ML compared to the rule-based approach, we compared how well their respective scores were able to rank target ions relative to decoy ions. We used Mean Average Precision (MAP), a commonly used metric that provides a comprehensive evaluation of the ranking accuracy and precision. Precision is defined as the number of targets in the top "k" ions of a ranked list, divided by k. Then, the average precision for each ion in the ranked list is calculated followed by taking the mean of these average precision over all datasets. MAP scores were calculated for each cross-validated dataset for each of the 5 splits (see Training and cross validation) and for the testing datasets.
Selection of testing datasets and database comparison. In order to appropriately evaluate the performance of METASPACE-ML and ensure its ability to generalize to new unseen data, we have selected 300 datasets using the same selection criteria as the training datasets (see selection and pre-processing of training datasets) which is also annotated using different databases (CoreMetabolome, SwissLipids, LipidMaps) (Supp Figures  8 9 ). In addition, we also filtered out datasets with a median number of peaks higher than 50. 000 yielding a total of 279 datasets (Table  S3 ) for downstream evaluation. Separation of target-decoy and feature importance An important aspect of the ideal annotation score is its ability to optimally differentiate between target and decoys. In order to assess it, we have developed a visualization strategy where for each ion, we calculated three original rule-based scores plus the new m/z error features and projected both target and decoy ions onto a two-dimensional space using UMAP implemented in the M3C R package  14.
Testing for difference in intensities of annotations To further investigate the characteristics of the newly captured ions by the METASPACE-ML model, we compared the intensity distributions of those ions to the ones captured by both METASPACE-ML and the rule-based approach across multiple datasets. Using METASPACE API (https://metaspace2020. readthedocs. io) we retrieved the ion images for each target ion in a given dataset and calculated the 99% percentile intensity across all pixels, followed by Log 10 transformation and taking the median across all ions captured at specific FDR threshold (default 10%). The distribution of the median-transformed intensities per dataset for each approach (METASPACE-ML and MSM) were compared using the Wilcoxon test. In addition, we used the 'ggbetweenstats' function in the ggstatsplot R package  16  to perform the pairwise Mann-Whitney test between the intensity distributions of ions only captured by METASPACE-ML compared to either all ions captured by METASPACE-ML or only ions captured by the rule-based approach for a given dataset.
Enrichment analysis To learn more about the types of metabolites that were only picked up by METASPACE-ML, we performed a hypergeometric test to identify the molecular classes that were enriched in those metabolites. Accordingly, we retrieved the class and subclass information for all annotated metabolites from the HMDB database (version 4) and used the HMDB "subclass" as background for enrichment. Then, for each dataset and subclass, we first filtered annotations with 10% FDR and then performed a two-tailed Fisher exact test where we consider the log fold enrichment as described in  19  as a proxy for the enrichment score. Finally, we filtered significantly enriched terms (p-value < 0. 05) and only terms that were enriched in at least 10% of the total number of input datasets were considered for visualization purposes.
Profiling of computational performance To assess the performance impact of METASPACE-ML we reprocessed a random selection of 50 training datasets with performance profiling. The average cloud compute resources required per dataset decreased from 3898 GB-sec to 2875 GB-sec (26% cost reduction) and the average wall-clock processing time decreased from 8. 1 minutes to 5. 8 minutes (28% faster). This was in part also due to more granular job subdivision affording greater parallelization, with an average dataset increasing from 91 to 117 parallel jobs and peak memory usage increasing from 20. 6GB to 21. 8GB.