Assessment of single cell RNA-seq statistical methods on microbiome data

The correct identification of differentially abundant microbial taxa between experimental conditions is a methodological and computational challenge. Recent work has shown that commonly used methods do not control the false discovery rate due to the peculiarity of these data (e.g. high sparsity), leading to an abundance of false positive results. Since single-cell RNA-seq shares some of these peculiarities, we apply methods developed for single cell differential expression to microbiome data. We compare these approaches to methods developed for bulk RNA-seq and microbiome data, in terms of suitability of distributional assumptions, ability to control false discoveries, consistency, replicability, and power. We benchmark these methods using 100 manually curated datasets from 16S and whole metagenome shotgun sequencing. A simulation framework is developed to assess the impact of experimental design in power analysis. Our analyses suggest that DESeq2 and limma-voom show the best performance. We recommend a careful exploratory data analysis prior to application of any inferential model and we present a framework to help scientists make an informed choice of analysis methods in a dataset-specific manner. Introduction Study of the microbiome, the uncultured collection of microbes present in most environments, is a novel application of high-throughput sequencing that shares certain similarities but important differences from other applications of DNA and RNA sequencing. Common approaches for the microbiome studies are based on the deep sequencing of amplicons of universal marker-genes, such as the 16S rRNA genes, or on whole metagenome shotgun sequencing (WMS). Community taxonomic composition can be estimated from microbiome data by assigning each read to the most plausible microbial lineage using a reference annotated database, with a higher taxonomic resolution in WMS than in 16S. The final output of such analyses usually consists of a large, highly sparse taxa per samples count table. Differential abundance (DA) analysis is one of the primary approaches to identify differences in the microbial community composition between samples and to understand the structures of the microbial communities and the associations between the microbial compositions and the environment. DA analysis has commonly been performed using methods adapted from RNA-seq analysis; however, the peculiar characteristics of microbiome data make differential abundance analysis challenging. Compared to other high-throughput sequencing techniques such as RNA sequencing (RNA-seq), metagenomic data are sparse, i.e., the taxa count matrix contains many zeros. This sparsity can be explained by both biological and technical reasons: some taxa are very rare and present only in a few samples, while others are very lowly represented and cannot be detected because of an insufficient sequencing depth or other technical reasons. In recent years, single-cell RNA-seq (scRNA-seq) has revolutionized the field of transcriptomics, providing new insight on the transcriptional program of individual cells, shading light on complex, heterogeneous tissues, and revealing rare cell populations with distinct gene expression profiles. However, due to the relatively inefficient mRNA capture rate, scRNA-seq data are characterized by dropout events, which leads to an excess of zero read counts compared to bulk RNA-seq data. Thus, with the advent of this technology new statistical models accounting for dropout events have been proposed. The similarities with respect to sparsity observed in both scRNA-seq and metagenomics data led us to pose the question of whether statistical methods developed for the differential expression of scRNA-seq data perform well on metagenomic DA analysis. Some benchmarking efforts have compared the performance of methods both adapted from bulk RNA-seq and developed for microbiome DA. While some tools exist to guide researchers, a general consensus on the best approach is still missing, especially regarding the methods’ capability of controlling false discoveries. In this study, we benchmark several statistical models and methods developed for metagenomics, bulk RNA-seq and, for the first time, single-cell RNA-seq on a collection of manually curated 16S and WMS real data as well as on a comprehensive set of simulations. We also consider use of the Dirichlet Multinomial Distribution for explicit compositional analysis (i.e., reflecting the fact that counts represent a proportion of the total rather than absolute counts), and the use of geometric mean normalization for reducing the impact of compositionality. The novelty of our benchmarking efforts are twofold. First, we include in the comparison novel methods recently developed in the scRNA-seq literature; second, unlike previous efforts, our conclusions are based on several performance metrics that range from type I error control and goodness of fit to replicability across datasets and internal consistency among methods.