dataesr/scientific-paragraphs-categorization huggingface dataset:
After reparameterization, the results are expressed as a microbial signature consisting of two taxa groups that are associated with the phenotype. coda4microbiome's signatures are more versatile than selbal's, as they allow different weights for taxa in each group, while selbal assigns the same weight to all taxa in each group. Coda4microbiome has also been implemented for both crosssectional and longitudinal studies. The website of the project contains several tutorials. 3 Other log-ratio based approaches for analyzing microbiome data include CodaCore  (Gordon-Rodriguez et al. , 2021)  and the R package amalgam  (Quinn and Erb, 2020) , which aim to identify predictive balances or amalgams in a stepwise additive fashion. Some log-ratio based approaches in microbiome data analysis try to improve predictive accuracy by considering log-ratios that can contain several original features.
However, many methods rely on pairwise log-ratios or additive log-ratios, which only involve two features. For example, the easyCoda R package includes three options for choosing pairwise log-ratios in a regression setting  (Coenders and Greenacre, 2022) , while the logratiolasso R package proposes a log-ratio LASSO model that aims to produce a sparse model from the all-pairs log-ratio model  (Bates and Tibshirani, 2019). DMMM/DBMC is a Dirichlet Multinomial Mixture Model (DMMM) tool that can be used in both unsupervised and supervised settings to identify clusters in microbiome datasets and act as a Bayes classifier. It is implemented in the R package DirichletMultinomial  (Holmes et al. , 2012)  and was extended by  Gao et al. (2017)  to include automatic feature selection, resulting in better classification accuracy than DMMM and random forest.
mikropml is an R package that follows best practices for machine learning, producing trained models, performance metrics, and feature importances  (Topçuoğlu et al. , 2021). It includes data preprocessing, model training, and selection, as well as hyperparameter tuning. The package has been used to classify colorectal cancer patients and identify variables associated with bacterial infections  (Topçuoğlu et al. , 2021). The tool has also been applied to test ML models for associations between microbiome composition and diseases like Clostridium difficile infections, producing significant results in  10. 3389/fmicb. 2023. 1250806  Frontiers in Microbiology 12 frontiersin. org multiple studies  (Lapp et al. , 2021; Armour et al. , 2022; Lesniak et al. , 2022).
, 2008)  extends this approach by integrating speciesspecific patterns of ribosome binding sites to improve translation start site prediction. Model-based methods are commonly used in gene prediction, and there are several notable examples. MetaGeneMark  (Zhu et al. , 2010)  is based on Hidden Markov models that are applicable to short DNA fragments. It uses training prokaryotic genomes to estimate polynomial and logistic approximations of oligonucleotide frequencies as a function of GC content. FragGeneScan  (Rho et al. , 2010)  and Glimmer-MG  (Kelley et al. , 2012)  both use Interpolated Markov Models to distinguish coding areas from non-coding DNA. Orphelia  (Hoff et al. , 2008 (Hoff et al. , , 2009) )  instead uses linear discriminants for mono-codon usage, di-codon usage, and translation initiation sites to extract characteristics from sequences, and also incorporates a neural network trained on random sub-sequences of genomes from the reference database to classify ORFs as proteincoding or not.
, 2011) , originally developed for a marine metagenome, predicts metabolite consumption or production based on the enzymatic activities present in a metagenome. Briefly, it leverages information from KEGG and MG-RAST (reactions and EC numbers, respectively) to generate an environmental metabolomic matrix (EMM), estimates enzymatic activity based on number of sequences, and calculates a PRMT-score for each metabolite in the EMM  (Larsen et al. , 2011). MIMOSA adapts this methodology in a multi-omic framework that combines taxonomic and metabolomic profiles in the context of the human microbiome  (Noecker et al. , 2016). This framework first infers community gene content based on taxonomic data and available and inferred genomic information. Then, making use of the PRMT method, it predicts the communitywide uptake or production of each metabolite, and estimates how species and genes might be contributing to these activities.
Similarly to MIMOSA, Mangosteen is a metabolome prediction pipeline that relies on relationships between KEGG/BioCyc reactions and their associated molecular compounds  (Yin et al. , 2020). However, with the increasing availability of both metagenomic and metabolomic data, numerous ML models have been developed to map metagenomic features to metabolites. These methods overcome the main limitation of reference-based methods, which are dependent on the quality of the queried databases. For instance, MelonnPan uses Elastic net regularization to predict community metabolomes from taxonomic profiles  (Mallick et al. , 2019). This approach has been used to predict metabolites in new microbial communities based on metagenomic data, shedding light on the functional role of microbiota in cardiovascular diseases  (Liu et al.
, 2020). Another ML-based approach, MiMeNet, is a multi-layer perceptron neural network that models microbe-metabolite relationships and the metabolomic profile of microbial communities from metagenomic taxonomic or functional features. This approach allows for scalability in handling large amounts of metagenomic and metabolomic features and leads to more robust predictive models by  10. 3389/fmicb. 2023. 1250806  Frontiers in Microbiology 13 frontiersin. org simultaneously learning metabolites and enhancing the transfer of information  (Reiman et al. , 2021). Metage2Metabo (M2M) is another software tool that simulates the metabolism of the gut microbiota and describes the metabolic relationships between the species' metabolic genes to establish how they complement each other in metabolic terms.
, 2019). It is reliant on the COBRA method which is a welldescribed set of strategies to employ when using metabolic modeling  (Heirendt et al. , 2019). Currently, the COBRA Toolbox is in its third edition and aims to simulate the relationship between genotype and phenotype through mathematical modeling  (Heirendt et al. , 2019). The Python COBRApy was developed as a framework allowing to model complex biological processes using COBRA methods  (Ebrahim et al. , 2013). COBRA modeling has been used to create personalized human microbiome models and stratify them based on structure and function, which has been used to treat conditions such as inflammatory bowel disease and colorectal cancer  (Heinken et al. , 2021). It also supports other computational methods used for metabolome predictions with microbial data.
Data integration The complexity and heterogeneity of the metagenomics datasets, which include various types, scales, and distributions, make it challenging to extract useful information from them in the context of omics data mining. One of the main obstacles to the successful use of ML techniques in metagenomics analysis is the integration of such a wide variety of heterogeneous data. Picard et al. (2021)  classified integration approaches into horizontal and vertical categories. Within the vertical integration strategies, further divisions include early, mixed, intermediate, late, and hierarchical approaches. Early and intermediate integration strategies enable the analysis of datasets within the context of their relationships with other datasets, leading to additional insights. However, early integration is challenging for most ML models, while intermediate integration often relies on unsupervised matrix factorization, which lacks the incorporation of pre-existing biological knowledge.
Late integration involves applying ML models separately to each dataset and then combining their predictions. Hierarchical integration considers the interaction between different layers of omics data explicitly, but its implementation is currently in its early stages. Most of the integration approaches implemented in software packages are based on mixed integration, which typically first modifies and transforms each dataset using different ML models. This enables  10. 3389/fmicb. 2023. 1250806  Frontiers in Microbiology 14 frontiersin. org them to reduce data complexity and heterogeneity, as well as to facilitate subsequent integration and analysis of datasets. Here we collect some of the ML software used for metagenomics data integration: There are several software packages available for metagenomics data integration.
mixOmics  (Rohart et al. , 2017a)  for example, is an R package that provides a wide range of multivariate methods for data exploration, sizing, and visualization, including integration platforms that investigate relationships between heterogeneous omics data (in terms of types, scales and distributions). Its multivariate projectionbased methods are computationally efficient for processing large omics datasets and provide flexibility in analyzing biological datasets by using relaxed assumptions about the distribution of the data. MixOmics R includes both supervised and unsupervised frameworks as well as feature selection. Other frameworks, like DIABLO  (Singh et al. , 2019)  and MINT  (Rohart et al. , 2017b) , enable the integration of datasets to identify relevant relationships and significant patterns in heterogeneous data for better exploration of complex metagenomic data.
Kernel methods allow data scientists to model non-linear relationships between the data points with low computational complexity, thanks to the so-called 'kernel trick'. These have already been used to extend well-known algorithms such as PCA, linear DA and ridge regression  (Cabassi and Kirk, 2020). A consensus multiple kernels is based on ideas similar to STATIS as an exploratory method designed to integrate multi-block datasets when the blocks are measured on the same samples  (Mariette and Villa-Vialaneix, 2018). MixKernel  (Mariette and Villa-Vialaneix, 2018)  is another R package that offers methods for integrating heterogeneous types of data, focusing on kernel fusion methods for unsupervised exploratory analysis. Its kernel methods allow data scientists to model non-linear relationships between the data points with low computational complexity, thanks to the so-called kernel trick.
General software for machine learning applications A variety of ML software tools are available, with the majority being open source. Goodswen et al. (2021)  and co-authors have compiled a brief list of general ML software tools to be applied in microbiome data. We have here extended this list in Supplementary Table  2  to include additional relevant general ML software for microbiology data analysis. These tools are primarily based on Python and R frameworks that contain collections of software libraries (packages) and require some basic programming knowledge for optimal use. However, some ML tools like WEKA, KNIME Analytics Platform, and Orange Data Mining, can be used through a GUI without extensive coding or programming expertise.
Commercial approaches and solutions We identified more than 240 companies (in >350 locations) worldwide based on an online database of companies applying or offering microbiome analysis (Microbiome Employers, 2022) complemented with search engine results. The companies' activities ranged from clinical research and the study of diagnostic and therapeutic effects in healthcare to the implementation of microbiome data analysis in agriculture, nutritional supplements and pharmaceuticals. The majority of these address microbiome analysis for therapeutics/pharmacy. Three typical examples are the discovery of novel molecules for therapeutics, agriculture, and nutrition (Adapsyn Bioscience, 2022), the prediction of viable biomarkers and therapeutic candidates against immunologic disorders (Pragmabio, 2022) and microbiome tests as a diagnostic application in medicine and cosmetics  (Atlas Biomed, 2022).
Impact of dataset size on the model accuracy In general, the availability of large amounts of data in available repositories such as NCBI, 7 METAHIT, 8 Human Microbiome Project, 9 ExperimentHub, 10 etc. , increases the chance of learning accurate ML models, and the impact of the dataset size on the model accuracy depends on the data source. However, it varies on the basis of the specific problem at hand. For example, fewer data are required if there are clear patterns within the data, if they are easily separable (in the case of classification tasks), or if simple (e. g. , linear) relationships can be identified between descriptive and target attributes (in the case of regression tasks). In addition, some ML algorithms inherently require huge amounts of data due to their complexity (e. g. , the number of parameters to optimize): simpler methods, such as linear regression and decision trees, typically need less training examples than solutions based on deep learning.
(2021)  and  Moreno-Indias et al. (2021) , along with the fragmentation and dispersion of the ML software and microbiome data require further efforts to create federated infrastructures and services, as stated by the European Open Science Cloud (European Commission Directorate General for  Research and Innovation. and EOSC Executive Board, 2021)  or ELIXIR  (Balech et al. , 2022) , to exploit complex human microbiome data accelerating innovation, and ensuring that the benefits of ML are distributed more broadly across society, these tools can help drive progress and create a more equitable and sustainable future. Hence, ML4Microbiome contributes to this aim with a very valuable resource to microbiologists and biomedical scientists identifying and cataloguing the ML software available, facilitating and supporting the analysis and interpretation of large human microbiome datasets.
Funding This study was supported by  COST Action   CA18131  " Statistical and machine learning techniques in human microbiome studies. " LM-Z is supported by  Spanish State Research Agency Juan de la Cierva  Grant  IJC2019-042188-I  (LM -Z ). MB is supported by  Metagenopolis  grant  ANR-11-DPBS-0001. MLC was partially supported by the  Spanish Ministry of Economy, Industry and Competitiveness , Reference  PID2019-104830RB-I00.
Measures the overall correctness of the predictions made by a model. It is the ratio of the correctly predicted instances to the total number of instances in the dataset. of actual positive instances that are correctly identified as positive by the model. It is the ratio of true positive predictions to the sum of true positives and false negatives. SpecificityRepresents the ability of a model to identify negative instances correctly. It is the ratio of true negative predictions to the sum of true negatives and false positives. PrecisionIndicates the proportion of correctly predicted positive instances out of the total instances predicted as positive by the model. F1 scoreIs the harmonic mean of precision and sensitivity and provides a balanced evaluation of a model's performance. AUC (AreaUnder theROC Curve)The ROC curve plots the true positive rate against the false positive rate at various classification thresholds.
ANR-18-CE23-0020). 1 dcase. community/ sult, captioning-specific alternatives CIDEr  [8]  and SPICE  [9]  were proposed. CIDEr retains the approach of n-gram comparison, but integrates reference consensus information and Term Frequency Inverse Document Frequency (TF-IDF) to focus on relevant terms in the description. SPICE instead parses reference captions to a graph containing semantic elements, their attributes, and relations to one another, and evaluates the candidate graph via synonym lemma matching. SPI-DEr  [10] , the combination of SPICE and CIDEr, was found to outperform both in correlation to human assessments, and is now the main metric for evaluating AAC systems. The properties indicative of acceptable captions are not well defined in the current formulation of the AAC task. Examples from Clotho  [11]  and AudioCaps  [12]  associate several human reference captions to examples, each describing events relevant to the annotator instead of the entire scene.
Likewise, the same acoustic content can be described with different sentence structure, which reference captions do not cover exhaustively. Evaluation through n-gram overlap, suited for language generation with specific wording, thus seems less relevant for audio captioning. Labbé et al. [13]  further note that SPIDEr shows high variability to small differences in caption formulation. As a workaround, they propose SPIDEr-max, where the maximum SPIDEr score out of several candidates produced by the system is retained. In  [14] , the authors propose a benchmark to evaluate the accuracy of metrics in choosing the best caption in candidate pairs, against human preference. Most n-gram metrics as well as SPICE perform poorly, cementing the need for further research in caption evaluation. The authors propose FENSE, a metric based on Sentence-BERT  [15]  embeddings similarity and penalized by fluency error detection.
Although this method achieves high correlation with human judgements, Martín-Morató et al. [16]  argue that sentence-wise semantic similarity does not necessarily reflect the correspondence of described sounds. Instead, they present CBscore, a summarization-oriented metric that extracts AudioSet tags  [17]  from captions, then derives a relevance-weighted precision score from the consensus in the reference set. However, the approach is limited to sound classes, and discards their qualifiers and relationships in the description. In this paper, we propose to combine the semantic graph parsing paradigm of SPICE with the strengths of deep language models. The metric uses a pre-trained language model to produce caption annotations, which are then parsed to semantic graphs with a broad set of linguistic rules.
PROPOSED METHOD The proposed metric evaluates a candidate caption against an arbitrarily large set of references, following the same process as SPICE  [9]. First, the caption is annotated with partof-speech tagging, lemmatization, and dependency parsing, which produces a tree-like representation of grammatical relations in the sentence. The annotation is then parsed into semantic graphs with a set of linguistic rules. Graphs are composed of nodes (nouns), attributes (adjective, verbs) and relations. SPICE views graphs as sets of node, node-attribute, and node-relation-node tuples. The candidate graph is evaluated against a single graph comprising all reference captions by comparing each tuple pair independently. SPICE+ alters the three processing steps, by performing dependency annotation with a pre-trained language model, adding new linguistic rules and post-processing to semantic graph construction, and evaluating candidates on deep embedding comparison 3.
[18]  using the Stanford probabilistic context-free grammar (PCFG) parser  [19]. From empirical observations on Clotho examples, this parser has a high error rate, which propagates to poor rule-based parsing and leads to incomplete or incorrect semantic graphs. Since then, better parsers have been developed in the natural language processing community  [20]. Thus, SPICE+ replaces the original PCFG parser with a language model trained on Universal Dependencies parsing.
SPICE annotates captions on Universal Dependencies A set of linguistic rules is then matched to the dependency annotation to obtain a semantic graph. Five rules are added compared to SPICE to handle formulations often found in audio captions, such as open clausal complements (e. g. "rain continues to pour"), passive nominal subjects (e. g. "a radio is being played"), and clausal noun modifiers (e. g. "peo- 2 Code is available at: github. com/felixgontier/spice-audio. 3 Examples on the companion page: felixgontier. github. io/spice-audio ple talking"). In addition, two post-processing steps are applied on semantic graphs, that led to better results in empirical experiments. First, coordinating conjunctions systematically duplicate attributes and relations. For instance, "men and women are talking and laughing" is parsed to two nodes man and woman, each with both attributes talk and laugh.
Candidate evaluation The original implementation of SPICE compares candidate and reference graphs as sets of tuples through synonym comparison. The process outputs a binary matrix, from which true positives, false positives, and false negatives are derive to compute an F-score. This approach discards related but nonsynonymous words, which results in a prominent spike at 0 in score distributions  [16]. At the same time, in audio captioning there may be several valid ways to describe a sound, e. g. a metal tube clanging, something hitting a pipe, that are not exhaustively represented in the reference set. SPICE+ retains a tuple comparison approach, but replaces binary synonym matching with a continuous similarity measure. First, a pre-trained language model extracts sentence embeddings for each tuple in the candidate and reference graphs.
Experiments consider the general captioning task setting of current datasets. As such, the evaluated metric is the pseudo F-score with default precision-recall balance. Furthermore, no decision threshold or tuple weighting is applied to the similarity matrix S. The dependency and part-of-speech annotations are done with UDify  [21]. The Stanford parser  [19]  is retained for lemma annotations following empirical experimentation. Tuple embeddings are computed with Sentence-BERT  [15]  (checkpoint paraphrase-TinyBERT-L6-v2) to follow FENSE  [14]. However, similar results are obtained using MPNet  [22].
Ranking capabilities The ranking performance of SPICE+ is first evaluated on the FENSE benchmark against current metrics CIDEr, SPICE, and SPIDEr, as well as the FENSE metric based on S-BERT. CB-score  [16]  is omitted as it is precision-oriented, and is not intended for ranking in the FENSE setting. results with or without adding a penalty on fluency using the FENSE error detection model  [14]. SPICE+ slightly improves on SPICE, indicating that the pre-trained dependency annotator and additional graph parsing rules are beneficial. However, its absolute performance is still poor with less than 50% MM ranking accuracy on both datasets without accounting for fluency. Adding embedding similarity significantly increases performance, confirming the limitations of synonym tuple matching. SPICE+emb also significantly outperforms CIDEr and SPIDEr on MM and total accuracy, the current accepted metrics for audio captioning.
Combining SPICE+emb with CIDEr to obtain a SPIDEr equivalent did not improve performance in our experiments. Overall, SPICE+emb is worse than Sentence-BERT on total accuracy, with a smaller difference when penalizing fluency through error detection. The metric evaluates semantic elements rather than the entire caption, and thus cannot account for fluency. However, this approach provides more information on system performance by analyzing the contributions of elements to the final score, which is not possible with full sentence embedding comparison. The currently used SPIDEr score, as well as Sentence-BERT and SPICE+emb, are shown in Table  2  for systems of the DCASE2022 challenge on the Clotho-testing dataset 4. SPICE+emb appears correlated to Sentence-BERT, and significantly alters system ranking compared to SPIDEr.