Deep Multiple Instance Learning for Taxonomic Classification of Metagenomic read sets
Authors: Andreas Georgiou
Vincent Fortuin
Harun Mustafa
Gunnar Rätsch
Venue:  (2020)

Abstract: Metagenomic studies have increasingly utilized sequencing technologies in order to analyze DNA fragments found in environmental samples. It can provide useful insights for studying the interactions between hosts and microbes, infectious disease proliferation, and novel species discovery. One important step in this analysis is the taxonomic classification of those DNA fragments. Of particular interest is the determination of the distribution of the taxa of microbes in metagenomic samples. Recent attempts using deep learning focus on architectures that classify single DNA reads independently from each other. In this work, we attempt to solve the task of directly predicting the distribution over the taxa of whole metagenomic read sets. We formulate this task as a Multiple Instance Learning (MIL) problem. We extend architectures used in single-read taxonomic classification with two different types of permutation-invariant MIL pooling layers: a) deepsets and b) attention-based pooling. We illustrate that our architecture can exploit the co-occurrence of species in metagenomic read sets and outperforms the single-read architectures in predicting the distribution over the taxa at higher taxonomic ranks.

Under review as a conference paper at ICLR 2020
DEEP MULTIPLE INSTANCE LEARNING FOR TAXONOMIC CLASSIFICATION OF METAGENOMIC READ SETS
Anonymous authors Paper under double-blind review
ABSTRACT
Metagenomic studies have increasingly utilized sequencing technologies in order to analyze DNA fragments found in environmental samples. can provide useful insights for studying the interactions between hosts and microbes (Methé et al. 2012: Qin et al. 2010), infectious disease proliferation (Chiu & Miller 2019), and novel species discovery (Nayfach et al. (201)). One important step in this analysis is the taxonomic classification of those DNA fragments. Of particular interest is the determination of the distribution of the taxa of microbes in metagenomic samples. Recent attempts using deep learning focus on architectures that classify single DNA reads independently from each other. this work, attempt to solve the task of directly predicting the distribution over the taxa of whole metagenomic read sets. We formulate this task as a Multiple Instance Learning (MIL) problem. We extend architectures used in single-read taxonomic classification with two different types of permutation-invarian MIL pooling layers: a) deepsets and b) attention- based pooling. We illustrate that our architecture can exploit the co-occurrence of species in metagenomic read sets and outperforms the single-read architectures predicting the distribution over the taxa at higher taxonomic ranks.
INTRODUCTION
Over the last decades, advancements in sequencing technology have led a rapid decrease of the cost of genome sequencing (Wetterstrand] (0033) while the amount of sequencing data being generated has vastly increased. This attributable the fact that genome sequeneing is tool of utmost importance for a variety of fields, such as biology and medicine, where it is used to identify changes in genes or aid in the discovery of potential drugs (Methé et al., (2122] subfield of biology, which is concerned with the study of genetic material found in samples taken directly from the environment (Consortium et al., 2016 Howe et al. 2014). DNA fragments found in those samples can be sequenced using various sequencing technologies, such Illumina, PacBio, and Oxford Nanopore (Quail et al., 2012). This process results in substrings sampled from random positions in the genomes of the organisms, called DNA reads. The reads obtained from sequencing are noisy, meaning that some of the letters (called base pairs) are flipped to a different letter or, in some cases, the deletion or insertion of additional base pairs can occur. The error rate and the distribution of the noise is dependent on the technology used to sequence the DNA fragments (Quail al. 2012). Newer long-read technologies can sequence complete genomes of viruses and small bacteria, but with higher error rate (Jain et al. 2016).
2012:Qin Qin et al., (010). Metagenomics is
As an application of metagenomic sequencing, samples can be taken from the human intestine in order to characterize the microbial flora of the human gut (Methé et al.) [012 Qin et al. 2010). Significant efforts have been carried out by projects such as the Human Microbiome Project (HMP) (Methé et al. 2012) and the Metagenomics of the Human Intestinal Tract (MetaHIT) project (Qin et al. 2010 in order to understand how the human microbiome can have an effect on human health. An important step in this process is to classify DNA fragments into various groups at different taxonomic ranks. The NCBI Taxonomy maintains a tree ontology of taxonomic labels [Wheeler et al., 2006). Organisms assigned taxonomic labels and thus placed on the tree. Each level of the tree represents a different taxonomic rank, with finer ranks such as species and genus being close to the leaf nodes and coarser ranks such phylum and class closer the root.

---

Under review as a conference paper at ICLR 2020
One approach that has shown great promise for biological classification tasks is deep learning. In recent years, we have seen various attempts of using deep learning to solve tasks such as variant calling (Poplin et al. 2018) the discovery of DNA-binding motifs (Zou et al. 2018). These methods even outperform more classical approaches, despite the relative lack of biological prior knowledge incorporated into those models.
We consider the problem of metagenomic classification, where each individual read is assigned to label or multiple labels corresponding to its taxon at each taxonomic rank. One could simply identify the taxon at the finest level of the taxonomy and then extract the taxa at all levels of the tree above it by following the path to the root. The problem with this approach is that for certain reads, we might not be able to accurately identify the species of the host organism, but nevertheless be interested coarser taxonomic ranks. This can apply cases where little relevant reference data is available for a sequencing dataset (such as deep sea metagenomies data (Tully et al.) (018)) or New York City metagenomics where only 48% of samples matched known species (Afshinnekoo) et al. ). so more accurate prediction at higher taxonomic ranks may be more informative for downstream analysis (Rojas-Carulaa et al. 2019). Furthermore, in many cases we are only interested in the distribution of organisms in environmental sample, also known as the microbiota, rather than in the classification of individual fragments.
We formulate this task an instance of Multiple Instance Learning (MIL). MIL is specific framework of supervised learning approaches. In contrast to the traditional supervised learning task, where the goal is to predict value or class for each sample, in MII,, given a set of samples, the goal is to assign value to the whole set. A set of items is called a bag, whereas each individual item in the bag is called an instance. In other words, a bag of instances considered to be one data point (Foulds & Frank, 2010). More formally, a bag is function B : X' + N where X' is the space of instances. Given an instance x E X. (zx) counts the number of occurrences of in the bag B. Let B be the class of such bag functions. Then the goal of a MIL model is to learn a bag-level concept e: y where y is the space of our target variable.
In the context of metagenomic classification, we consider the instances to be DNA reads. Our goal is to directly predict the distribution over given set of taxonomic ranks in the read set (the bag). So for each taxon, our output is real number in [0, 1] denoting the portion of the reads in the read set that originated from that particular species. The motivation for this that in a realistic set reads, closely related organisms tend to appear together. It might thus be possible to exploit the co-occurrence of organisms to gain better accuracy (Carbonneau et al. 2018).
Our main contributions are:
A new method to generate synthetic read sets with realistic co-occurrence patterns from collections of reference genomes.
A novel machine learning model for predicting the distribution over taxa in read set, combining state-of-the-art deep DNA classification models with read-set-level aggregation in multiple instance learning setting.
A thorough empirical assessment of our proposed model, showing superior performance in predicting the distributions of higher level taxa from read sets.
In the rest of this paper, we give an overview of previous related work in Section 2 describe our data generation method and machine learning models in Section 3 and analyse the results of our experiments in Section An overview of our proposed architectures is depicted in Figure
2 RELATED WORK
To solve the problem of metagenomic classification, more traditional methods rely on read alignment to classify each DNA fragment. Given a DNA read, one first needs to match k-mers to a large database of reference genomes. This is done to detect candidate segments of the genomes and can be executed quickly by first creating an index of the reference genomes during preprocessing phase (Altschul et al. 1990, Bowe al. 2012 Muggli et al.) (2017). Following this step, needs to approximate string matching techniques to match the string to the candidate segments determined by the k-mer matching step before. A well-known and widely used tool that uses alignment is BLAST,
2

---

Under review as a conference paper at ICLR 2020
Embedding Layer
ResNet
MIL Pooling
MLP
Output Distribution
Embedding
Layer
Max pooling, Pooling:
Mean and
MIL
MILP
Oulput Distribution
GGGT CGGTCGGCAAAATGCTIO WAA.TTTTTTTCAGGAGGCCCMMMMM
NEGTAGGGTCGGAC CAAGGCTTGGGGNNGTTTTTTAACCCCCCCTE
ACGT GGGAAGGGTE CGGACAGA ACGTAC ACGTACACAGITA ACAGT ACAGTAGACAA AGACAR
@G6TCGGCAAAATGCT TAATTGGCTCAAGGC TAGTAAAATTTT TAGTTAAAAAAAATTTAA
(a) GeNet MII. pooling
(b) EmbedPool + MII. pooling
Figure 1: The two proposed architectures for solving the MIL task. The models can process multiple reads (only two reads shown for compactness) independently from each other. During the MII. pooling phase, the outputs for each read are combined to create a representation for the whole read set. Subsequently, the model can use this directly predict the distribution over the taxa.
which is general heuristic tool for aligning genomic sequences. Other aligmment and mapping based tools specifically designed for metagenomics include Centrifuge (Kim et al.) 2016), Kraken (Wood & Salzberg 2014), MetaPhlAn (Segata et al., 2012), and MEGAN (Huson al. 2007). These methods make trade-offs of sensitivity for scalability. For example, BLAST is highly sensitive, but not scalable to databases of unassembled sequencing data, while more approximate methods like Kraken are well suited for such large databases. Moreover, recent deep learning approaches have outperformed these methods by significant margins, especially in high error-rate settings Liang et al. 2019).
Carulla et al. 2019
Most of the previous attempts using machine learning focused on 16S rRNA sequences due to their high sequence conservation across wide range of species. An example is the RDP (Ribosomal Database Project) classifier which uses a Naive Bayes classifier to classify 16S rRNA sequences (Wang al. 2007). The disadvantage of this method the loss of positional information due the encoding of the sequence as a 'bag" of 8-letter words. However, the generalizability of this model sequencing data drawn from other genomic regions unclear. Similarly, La Rosa al. (2015)
use probabilistic topic modeling was used in order to classify 16S rRNA sequences in the taxonomic ranks from phylum to family. Another interesting approach is taken by [Brady Salzherg (2009) which uses Markov models to classify DNA reads and can even be combined with alignment methods to increase performance. In addition, Busia et al. (2019) use CNN architecture to classify 16S sequences, while other approaches also proposed to recurrent neural networks sequences (Ganscha et al. 2018).
More recent attempts for solving the general metagenomic classification problem focus using deep learning to tackle as a supervised classification task. Two examples of such attempts are GeNer Kojas-Carulla al. 2019). which attempts to leverage the hierarchical nature of taxonomic classification, and DeepMicrobes (Liang et al. 2019), which first learns embeddings of k-mers and subsequently uses those to classify each read. We use GeNer and simplified version of DeepMicrobes as baselines and explain them in more detail in Section
MODELS AND METHODS
We implemented two deep neural networks for predicting the taxa individual reads which we use baselines: GeNer PRojas-Carulaa al. 2019) and simplified version of DeepMicrobes (Liang
3

---

Under review as a conference paper at ICLR 2020
Rank abundance curve
plylum
order
genus
115
atoee abartence
3.0
102
30P
50.1
3074
10
k00
10"7
10.51
17.
Fank
Rlani
Dant
neittbe chunderce
11P
50
20
to-1
302
2013
3000
1500
Figure 2: Rank-abundance curve for each laxonomic rank. All taxa are sorted using their abundance. Their abundance level is shown on the y-axis.
al. 2 described in sections 3.2.1 and 3.2.2 respectively. We refer those models collectively as single-read models and we extend those in order to solve the MIL problem described above.
The full source code is provided online at http:://ggthabb.ccm//tttgaggogg MetaSetMI1
3.1 DATASET GENERATION
For training, validation and evaluation, we use synthetic reads generated from bacterial genomes from the NCBI RefSeq database (Wheeler et al. 2006) from which comprising 1 862 species similar to the dataset used in KoojssCarulla et al. (2019). We use NCBI's Entrez tool (Schuler et al. 1996), to download the genomes and the taxonomic data. The number of taxa in each taxonomic rank is summarized in Table ST in Appendix
subset of 3 332 genomes
For training the single-read models, create mini-batchar in which the reads are sampled by selecting genomes uniformly at random. Training of the MIL models is different where a batch consists of small number of bags of reads, with each bag containing reads sampled using more realistic distribution over the genomes. The procedure used is similar to the one used by the CAMISIM simulator (Fritz et al. (019) and described in more detail in Section S..1ion An example rank-abundance curve for each laxonomic rank generated by this procedure is shown in Figure 2
From the selected genomes, we sample reads to create mini-batches in iterative procedure similar to the procedure described in Rojas- Carulla et al. (2019). For the generation of reads, we use the software InSilicoSeq (Gourlé et al., 2018). We create datasets of two types order to carry out our experiments: 151 bp reads (default length of InSilicoSey) with no emoor and with Illumina NovaSeq type noise. We refer to those two types of datasets as error-free and novaseg, respectively. In our experiments, we train all models on both dataset types. For validation and evaluation only use datasets of novaseq-yy typpe reads in order to determine whether the models are effective at removing noise from the reads and whether it is beneficial to train with noisy reads.
Every bag is supposed to simulate a different microbial community and hence the generation procedure repeated for each bag. The more realistic bags allow the MIL models to capture the interactions between the reads coming from related species and capture potential overlap in the reads originating from the same taxa. The validation and evaluation datasets for both single-read models and the MII.
4

---

Under review as a conference paper at ICLR 2020
models use this more realistic approach. Hyperparameter search was also performed for all models (details on the exact parameters can also be found Appendix B
3.1.1 SAMPLING REALISTIC SET READS
In order to sample bags with a more realistic community of bacteria, we use a method similar to Fritz et al. (2019). Given a set of all the taxa T at higher level (e.g., genus or family), we sample ' numbers from a lognormal distribution with μ = and o = 2:
(In 2 - 12)
T, Lognormal xo 2T
exp
20²
Then, for a taxon & with genomes associated with it, we choose to include in our microbial community only 1 random genomes where I; is sampled from a geometric distribution with y =
P(X - = 1 (1
(2)
To calculate the abundance of genome 9j belonging to taxon 4, 4 random numbers V1 ... Yi, are sampled from lognormal distribution as in equation ① calculated as:
The abundance for the genome is then
A
Ek=1 Yn T.
All abundances are finally normalized to produce a probability vector over all the genomes in the dataset. When sampling a read set, genome is selected by sampling from the distribution produced. Reads are then simulated from the genome sample using the software package InSilicoSeq.
3.2
BASELINE MACHINE LEARNING MODELS
3.2.1 GENET
GeNer leverages the hierarchical nature of the taxonomy of species to simultancously classify DNA reads all taxonomic ranks (Rojas-Carul embedding as described in Gehring et al. (2017). Given an imput
al. 2019). The procedure similar to positional
In). an embedding w (W....., w,,) is computed, where w, E R The vocabulary of size 5 corresponds to the symbols for the four possible nucleotides A, C, TT, G, and N (for unknown base pairs in the read). Embeddings of the absolute positions for each letter are also computed create p = (Pi..... ........... Pn). where P. E 5 The one-hot representation of the sequence, o, is added to the other two embeddings to create the matrix w + p + o. Subsequently, the resulting matrix is passed to a ResNet-like neural network which produces final low-dimensional representation of the read. The main novelty of the architecture is the final layer used for classification which comprises multiple softmax layers, one for each taxonomic rank. These layers are connected to each other so that information from higher ranks can be propagated towards the lower ranks. More formally, the output of softmax layer i can be written follows:
ReLU (R,LU(Wh)) + ReLU (U.):-))
(4)
where W, and U, are trainable parameters, h; is the output of the ResNet network and y:-1 is the previous softmax output. (-) is the rectified linear unit function. To train the model, an averaged cross-entropy loss for each softmax layer used.
3.2.2 EMBEDPOOL
Liang et al. (2019) introduce multiple architectures for performing single-read classification among which the best is DeepMicrobes. It involves embedding k-mers (k = 12) into latent representation. followed by a bidirectional LSTM, a self-attention layer, and a multi-layer perceptron (MLP). Unlike GeNer, this model only be trained classify single taxonomic rank. Due to limited computational resources (the model requires a significant amount of GPU memory because of the very large embedding matrix), implemented EmbedPool, simpler version of DeepMicrobes (also
5

---

Under review as a conference paper at ICLR 2020
described in the original paper) to use as baseline. In order to classify multiple taxonomic ranks, one could run multiple instances of the model, each rumning on a different GPU. However, each model would be independent of the others and they would not take advantage of the hierarchical structure of the taxonomic tree. EmbedPool is a model that consists of an embedding layer for k-mers, where we set k 11 in order to fit it into GPU memory. Both max- and mean-pooling on the resulting matrix and concatenated together to yield a low-dimensional representation of the read. Since the embedding dimension is set 100, after concatenation, this results in a vector of size 200. An MLP with one hidden layer of 3 000 units subsequently classifies the read. ReLU is used as the activation function. As the authors explained. most of the performance is owed to the use of the k-mer embedding and therefore the reduction in performance relative to DeepMicrobes is not expected to significant. The model is trained end-to-end using cross-entropy loss.
performed
3.3 PROPOSED MULTIPLE INSTANCE LEARNING MODELS
3.3.1 GENET + MIL POOLING
A mini-batch of bags of reads is used input. The first part of GeNet, consisting of the embedding and the ResNet-like neural network, is used to process each read individually. A pooling layer is then used group all reads in each bag to create bag-level embeddings. This is also referred to MIL pooling (Foulds & Frank 2010 Carbonneau et al. 2018). The output is passed to the final layers of GeNer in order to output a probability distribution over the taxa each taxonomic rank. As a loss function we use the -hhannn- (D (s)) divergence (Lin 199T) between the predicted distribution and the actual distribution of the bag-
Given that bag is a set, we require that a MIL pooling layer is permutation invariant, that is, permuting the reads of the bag should still produce the same result. To this end, we utilize DeepSets (Zaheer et al. 2017). DeepSets can be formally described as follows:
f(x) = (Σ
Σ ((z) L
(5)
In other words, each element of set X first processed by function Φ(.). The outputs are all summed together and the result is subsequently transformed by function p(-). Zaheer et al. (2017) proved that all valid functions operating on subsets of countable sets or on fixed-sized subsets of uncountable sets can be written in this form. In our case, the inputs are embeddings in R55x f. where L is the length of a read. In addition, we only input bags of fixed size and hence the assumptions of Theorem 2 in Zaheer et al. (2017) are satisfied. p(-) is modelled with small MLP with one hidden layer while the ResNet part of the network models the function (()).
Alternatively to DeepSets, we also consider attention-based based pooling layer as seen in llse al. (2018) motivated by the fact that would allow the model to attend to specific reads originating from each species. In attention-based pooling, the elements of the set are combined in different ways to create a set z = 21.... Zg. such that the set remains invariant when we permute the elements of the input set. This can be written as follows:
3j
(6)
exp(ur tanh(Va:))
Cy.. Σ I=1 exp(w" (anh(Va/))
where 3e is an element of the input set, and V and wy are trainable parameters. The weights a,,s are therefore calculated with an MLP with hidden layer with tanh non- linearitty and softmax activation at the end. [sse et al. (2018) also attempt to increase the flexibility of the MIL pooling by introducing a gating mechanism as shown below:
exp(w) (tanh(Væ:)) o oo((x)))
aj,k EK exp(wi (lanh(V ( (Vaf)) o o (Ux()))
(8)
where U is an additional learnable matrix, a is the sigmoid activation function and @ is the element- wise product. As shown in Appendix B. for our models, using the gating mechanism is additional

---

Under review as a conference paper at ICLR 2020
Table Performance sss/ In nz) of all models trained on each dataset (higher is better). Our MIL models achieve superior performance at higher taxonomic ranks up to Family. EmbedPool was only trained at the Species level since training time exceeded our cluster limits. See subsection 4.2 for more details.
novaseg
error free
Phylum
Family
Species
Phylum 0.896 + 0.016
Family
Species
GeNet Rojas-Cann Cargs-Carllleeee al. 2019
0.571 0.017 N/A
0.892 - 0:011
0.879
0.008
DJ895 + 0.011 N/A
0.878 0.008
EmbedPoot Luung
NIA
0.903
0.009
NIA
0.902 0.009
GeNet Deepset (ours)
0.985
0.008
0.929
0.022
0.852
0.034
0.985 0.984
0.008
O. 0.929 0. 0.022
0.852 0.034
GeNer Artention (ours)
0.883
0.010
0.921
0.024 N/A
0.840
0.034
0.008
0.924 + 0.025
0.849 0.033
Embedpool Deepeet (ours)
Emhedppool Attention (ours)
N/A
N/A
0.863 + 0.033
N/A
N/A
0.803 0.033
hyperparameter. Following the attention mechanism, the output z is flattened to create single vector for each bag which is subsequently processed by GeNer's final layers output the predicted distributions. The overall architecture can seen Figure
3.3.2 EMBEDPOOL. MIL POOLING
Similarly to subsection 3.3.1 we use EmbedPool to process the reads individually. MIL. pooling
layer is added after the mean- and max- pooling layers, the output of which is fed to the rest of the model to predict the distribution. JS-divergence used as loss function. For MIL pooling, we use DeepSets and attention-based pooling as before. An overview of the model can again be seen in Figure
4 RESULTS AND DISCUSSION
In this section, we analyze the results of the two baselines on solving the single-read prediction task. Then evaluate their performance the proposed MIL task and compare them MIL models. Table iillustrts the performance of the models trained on novaseq and error-free reads. However, in both cases the models are evaluated on novaseq reads in order test their robustness to noise.
SINGLE-READ PREDICTIONS
In Rojas-CCruul et al. 2019), GeNer trained on PacBio reads of length 10 000 bp and Illumina reads of length 000 bp. Since in most cases genome sequencing technologies like Illumina produce shorter reads in the range of 100 bp 300 bp (Quail et al., 2012), chose train all our models on reads of length 151 bp. In the single-read prediction task GeNer does not perform very well on our evaluation dataset neither at Phylum nor Species levels. This is attributed it might be unable to extract useful features shared across the whole genome from shorter reads, especially because one-hot encoding used rather than k-mer encoding. On the other hand, even though EmbedPool seemed to be performing well during training, achieving training accuracy of 0.789, when the distribution of the reads in the mini-batch is changed (as is the case with our more realistic evaluation dataset), the accuracy drops to 0. 0.223. This signifies that EmbedPool is not able to accurately classify all species equally well. GeNer however seems to be more robust to the change of the mini-batch distribution since the accuracy does not drop when moving from the training dataset to the more realistic evaluation dataset. In addition, training with noisy reads seems to not improve results for EmbedPool when evaluating the classifier error-free reads seems to achieve better results for GeNer even when evaluating on novaseq reads. A table with the accuracy achieved by both baselines in the single-read prediction task can be found in Appendix
the fact that
noisy reads. However, training with
4.2 READ-SET-BASED PREDICTIONS
In each taxonomic rank t, the upper bound for the JS-divergence differs because of different numbers of taxa n, belonging that rank. Therefore, normalize results and use 1 DJs/ In n, the metric for comparison, where a value of 1.0 means the model achieved perfect performance. Table shows comparison of our MII. models and the achieved scores. A table of the raw DJs values can
7

---

Under review as a conference paper at ICLR 2020
be found in Appendix A] For the standard GeNet and EmbedPool, the microbiota distribution was calculated by classifying each read independently while for the rest, the distribution was predicted directly by the models. An example of the output of the MIL models is shown in Figure [3] All models were evaluated on a total of 100 bags of 2048 novaseq-type reads each. Both GeNet + Deepset and GeNer Attention perform better than standard GeNet higher taxonomic ranks. As explained in Section | we believe that the improvement in accuracy is owed to the fact that the models can exploit the co-occurrence of species in realistic settings or detect overlaps of reads in a bag. A drawback of our MIL models is that, since the performance is owed to the special structure of the bags, it is unlikely that they would perform well when presented with bags with an unrealistic distribution of species (e.g., bag with uniformly random distribution over all species). Therefore, it is clear that the models achieve a trade-off between flexibility and performance. Moreover, our proposed MIL models perform poorly on the finer taxonomic ranks, possibly because in the MII. setting, the models only observe a summary of the bag rather than a label for each instance and is therefore harder for them to learn adequate features. However, the greater performance beneficial for some real-world metagenomic datasets where sufficient reference data is not available to train deep learning models accurately (Afshinnekoo al., 2015 Tully et al.) 2018 A comparison of GeNet + Deepset, our best performing model and standard GeNel can be seen in Figure ST in Appendix A
higher levels can prove
0.200
Genet
Deepset
0.175
Target
Probablity
0.100
0.075
0.050
0.025
0.000
10
20
30
40
50
60
70
Taxon
Figure 3: Distribution of taxa at the class rank. The target distribution is denoted in orange and the output of the model is denoted in blue.
5 CONCLUSIONS
In this work. we tackle the problem of directly predicting the distribution of the microbiota in metagenomic samples. In contrast to previous methods that are based on classifying single reads, we formulate the problem as a Multiple Instance Learning task and use permutation invariant pooling layers order to learn oww-dimensional embeddings for whole sets of reads. We show that our proposed method can perform better than the baseline models at the higher laxonomic ranks. The MIL models presented could be used as an initial step to filter or preselect the potential genomes that more traditional alignment methods would need to take input in order to increase their performance.
Further work could include exploring alternative base architectures or more sophisticated pooling methods that can better capture the interactions between reads. For example, one could use Janossy pooling (Murphy et af.] 2018). another permutation invariant method that can capture k-order interactions between the elements of a probabilistic component, such as a Gaussian process over DNA sequences (Fortuin et al. 2018 18).. allow for uncertainty estimates on the predictions. Finally, explained, a possible issue that observing only the summary of the read set can make it more difficult for the model to learn adequate features for the individual reads. solution to this could be first learn better instance-level embeddings to use as input, in order to aid the model in learning suitable bag-level embeddings.
Also, the models could potentially be combined with
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Ebrahim Afshinnekoo, Cem Meydan, Shanin Chowdhury, Dyala Jaroudi, Collin Boyer, Nick Bern- stein, Julia M Maritz, Darryl Reeves, Jorge Gandara, Sagar Chhangawala, al. Geospatial resolution of human and bacterial diversity with city-scale metagenomics. Cell systems, 1(1): 72-87, 2015.
Stephen Altschul, Warren Gish, Webb Miller, Eugene W Myers, and David Lipman. Basic local alignment search tool. Journal of molecular biology, 215(3):403-416 1990.
Alexander Bowe, Taku Onodera, Kunihiko Sadakane, and Tetsuo Shibuya. Succinct de bruijn graphs. In International Workshop on Algorithms in Bioinformatics, pp. 225-235. Springer, 2012.
Arthur Brady and Steven L Salzberg. Phymm and phymmbl: metagenomic phylogenetic classification with interpolated markov models. Nature methods, 6(9):673, 2009.
Akosua Busia, George E Dahl, Clara Fannjiang, David H Alexander, Elizabeth Dorfman, Ryan Poplin, Cory McLean, Pi-Chuan Chang, and Mark DePristo. deep learning approach to pattern recognition for short dna sequences. bioRxiv, pp. 353474, 2019.
Marc-André Carbonneau, Veronika Cheplygina, Eric Granger, and Ghyslain Gagnon. Multiple instance learning: A survey of problem characteristics and applications. Pattern Recognition, 77: 329-353, 2018.
Charles Y Chiu and Steven A Miller. Clinical metagenomics. Nat Rev Genet, 20:341-355, 2019.
MetaSUB International Consortium al. The metagenomics and metadesign of the subways and urban biomes (metasub) international consortium inaugural meeting report, 2016.
Vincent Fortuin, Gideon Dresdner, Heiko Strathmann, and Gunnar Rätsch. Scalable gaussian processes on discrete domains. arXiv preprint arXiv: 10368, 2018.
James Foulds and Eibe Frank. review of multi-instance learning assumptions. The Knowledge Engineering Review, 25(1):1-25 2010.
Adrian Fritz, Peter Hofmann, Stephan Majda, Eik Dahms, Johannes Dröge, Jessika Fiedler, Till R Lesker, Peter Belmann, Matthew DeMaere, Aaron E Darling, et al. CAMISIM: simulating metagenomes and microbial communities. Microbiome, 7(1):17, 2019..
Stefan Ganscha, Vincent Fortuin, Max Horn, Eirini Arvaniti, and Manfred Claassen. Supervised learning on synthetic data for reverse engineering gene regulatory networks from experimental time-series. bioRxiv, pp. 356477, 2018.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1243-1252. JMLR. org, 2017.
Hadrien Gourlé, Oskar Karlsson- Lindsjo Julictte Hayer, and Erik Bongeam- Rudlofff. Simulating illumina metagenomic data with insilicoseq. Bioinformatics, 35(3):521-522, 2018.
Adina Chuang Howe, Janet K Jansson, Stephanie A Malfatti, Susannah G Tringe, James M Tiedje, and Titus Brown. Tackling soil diversity with the assembly of large, complex metagenomes. Proceedings of the National Academy of Sciences, (11(14):40444999 2014.
Daniel H Huson, Alexander F Auch, Ji Qi, and Stephan C Schuster. Megan analysis of metagenomic data. Genome research, (73))3377-366, 2007.
Maximilian Ilse, Jakub M Tomezak, and Max Welling, Attention-based deep multiple instance learning, arXiv preprint 100.v:180..04712, 2018.
Miten Jain, Hugh E Olsen, Benedict Paten, and Mark Akeson. The oxford nanopore minion: delivery of nanopore sequencing to the genomics community. Genome biology. 17(1):239, 2016.
Dachwan Kim, Li Song, Florian Breitwieser, and Steven L Salzberg. Centrifuge: rapid and sensitive classification of metagenomic sequences. Genome research, 26( 12::111-11 2016.

---

Under review as a conference paper at ICLR 2020
Massimo La Rosa, Antonino Fiannaca, Riccardo Rizzo, and Alfonso Urso. Probabilistic topic modeling for the analysis and classification of genomic sequences. BMC bioinformatics, 16(6):S2, 2015.
Qiaoxing Liang, Paul W Bible. Yu Liu, Bin Zou, and Lai Wei. Deepmicrobes: taxonomic classification for metagenomics with deep learning. bioRxiv, pp. 694851, 2019.
Jianhua Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory, 37(1):145-151, 1991.
Barbara A Methé, Karen E. Nelson, Mihai Pop, Heather H Creasy, Michelle G Giglio, Curtis Huttenhower, Dirk Gevers, Joseph F Petrosino, Sahar Abubucker, Jonathan H Badger, et al. A framework for human microbiome research. nature, 486(7402):215, 2012.
Martin D Muggli, Alexander Bowe, Noelle R Noyes, Paul S Morley, Keith E Belk, Robert Ray- mond, Travis Gagie, Simon J Puglisi, and Christina Boucher. Succinct colored de bruijn graphs. Bioinformatics, 33(20):3181-3188 2017.
Ryan L Murphy, Balasubramaniam Srinivasan, Vinayak Rao, and Bruno Ribeiro. Janossy pool- ing: Learning deep permutation-invariant functions for variable-size inputs. arXiv preprint arXiv: 18X1v: 1.01900, 2018.
Stephen Nayfach, Zhou Jason Shi, Rekha Seshadri, Katherine S Pollard, and Nikos Kyrpides. New insights from uncultivated genomes of the global human gut microbiome. Nature, 568(7753):505. 2019.
Ryan Poplin, Pi-Chuan Chang, David Alexander, Scott Schwartz, Thomas Colthurst, Alexander Ku, Dan Newburger, Jojo Dijamco, Nam Nguyen, Pegah Afshar, et al. A universal SNP and small-indel variant caller using deep neural networks. Nature biotechnology, 36(10):983, 2018.
Junjie Qin, Ruiqiang Li, Jeroen Raes, Manimozhiyan Arumugam, Kristoffer Solvsten Burgdorf, Chaysavanh Manichanh, Trine Nielsen, Nicolas Pons, Florence Levenez, Takuji Yamada, et human gut microbial gene catalogue established by metagenomic sequencing, nature, 464(7285); 59, 2010.
Michael A Quail, Miriam Smith, Paul Coupland, Thomas D Otto, Simon R Harris, Thomas R Connor, Anna Bertoni, Harold Swerdlow, and Yong Gu. A tale of three next generation sequencing platforms: comparison of ion torrent, pacific biosciences and illumina miseq sequencers. BMC genomics, 13(1):341, 2012.
Mateo Rojas-Carulla, Ilya Tolstikhin, Guillermo Luque, Nicholas Youngblut, Ruth Ley, and Bernhard Schölkopf. Genet: Deep representations for metagenomics. arXiv preprint 19Xiv:1901.11000 2019.
Gregory D Schuler, Jonathan A Epstein, Hitomi Ohkawa, and Jonathan Kans. [10] entrez: Molecular biology database and retrieval system. In Methods in enzymology, volume 266, pp. 141-162. Elsevier, 1996.
Nicola Segata, Levi Waldron, Annalisa Ballarini, Vagheesh Narasimhan, Olivier Jousson, and Curtis Huttenhower. Metagenomic microbial community profiling using unique clade-specific marker genes. Nature methods, 988:8111, 2012
Benjamin J Tully, Elaina D Graham, and John F Heidelberg. The reconstruction of 2,631 draft metagenome-assemblee genomes from the global oceans. Scientific data, 5:1 5:100, 2018.
Qiong Wang, George M Garrity, James M Tiedje, and James R Cole. Naive bayesian classifier for rapid assignment of rma sequences into the new bacterial taxonomy. Appl. Environ. Microbiol, 73 (16):5261 2007.
Kris A Wetterstrand. DNA sequencing costs: data from the NHGRI genome sequencing program (GSP), 2013.
10

---

Under review as a conference paper at ICLR 2020
David L Wheeler, Tanya Barrett, Dennis A Benson, Stephen H Bryant, Kathi Canese, Vyacheslav Chetvernin, Deanna M Church, Michael DiCuccio, Ron Edgar, Scott Federhen, et al. Database resources of the national center for biotechnology information. Nucleic acids research, 35(uupp 1): D5-D12, 2006.
Derrick E Wood and Steven L Salzberg. Kraken: ultrafast metagenomic sequence classification using exact alignments. Genome biology. 15(3):R46, 2014.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan R Salakhutdinov, and Alexander Smola. Deep sets. In Advances in neural information processing systems, pp. 3391-3401, 2017.
James Zou, Mikael Huss, Abubakar Abid, Pejman Mohammadi, Ali Torkamani, and Amalio Telenti. A primer deep learning in genomics. Nature genetics, pp. 2018.

---

Under review as a conference paper at ICLR 2020
A SUPPLEMENTARY MATERIAL
To train and test our models, we have downloaded 3332 genomes from the NCBI RefSeq database (Wheeler 2006). The full list of accession numbers for the genomes used in dataset can be found in our GitHub repository
Table S1: Number of taxa per rank in our dataset. The selected accession numbers are a subset of the dataset used by Rojas EogjasCCaullla al. (2019), See subsection 3.1
Rank
# of taxa
Phylum
37
Class
77
Order
167
Family
349
Genus
824
Species
1862
The accuracy of the two baseline models at solving the single-read prediction task was evaluated and the results shown Table S2
Table S2: Accuracy of the two base models trained on each dataset (higher is better).
novaseq
error-free
Phylum
Species
Phylum
Species
GeNet EmbedPool
0.258 + 0.032
0.100 0.015
0.290 + 0.034
0.097 + 0.017
N/A
0.223 + 0.028
N/A
0.224 + 0.030
Subsequently, all models were evaluated solving the MIL task. The IS-divergence achieved by all models is shown in Table [33 while comparison our best performing model, GeNet Deepset, and GeNer is depicted in Figure ST
Table S3: JS-divergence for all models trained each dataset. Our MIL models achieve superior
performance at higher taxonomic ranks up to Family. See subsection 4.2 for more details.
movaseq
error free
Phylum 0.409
Family
Species
Phylum NIA
Family 0.003
Species
Rojas-Canil et
0.064
0.067
EmbedPoat Lling el al. 2019
N/A
N/A
0.733 0.0664
NIA
0.741 0.067
GoNer Deepset
0.131
1.115:+0.255
0.116
0.225
GeNet Attention (ours)
0.062 0.035
0.462 = 0.139
1.135 0.253
0.008 0.030
0-446 + 0.145
1.140 0.251
Embedpool Deepect (ours)
NAA
NIA
1.101 0.2228
N/A
N/A
1.101 0.227
Embedpool Attention (ours)
N/A
N/A
1.107 0.247
N/A
N/A
1.105 0.245
12

---

Under review as a conference paper at ICLR 2020
Performance comparison of Genet Genet Deepent
Genet
Genet
Gamet Coepat
0.98
Ganat
0.96
Jensen Shannon divergence
0.94
D m/
0.6
0.90
0.1
0.56
Plylen
Grma
Sprrio
Plyton
Order
Taxonomic rank
Taxonomic rank
Figure SI: Performance comparison of GeNet vs GeNet + Deepset. GeNet + Deepset achieves superior performance taxonomic ranks upto Family.
13

---

Under review as a conference paper at ICLR 2020
B HYPERPARAMETER GRID FOR THE TRAINED MODELS.
To train our models, we performed random search over the following hyperparameter grid:
Table S4: Hyperparameter grid
General parameters for single read models
Batch Size
64, 128, 256, 512, 1024, 2048
General parameters for MIL models
Bag Size
64, 128, 512, 1024, 2048 1, 8
Batch Size
GeNet
Output size of ResNet
128, 256, 512, 1024 True, False True, Falsc
Use GeNet initialization scheme
BatchNorm rumning statisties
Optimizer
Adam, SGD
Learning rate
0.001, 0.0005, 1.0 (for SGD) 0.0, 0.9, 0.99
Nesterov momentum (SGD only)
EmbedPool
Size of MLP hidden layer Optimizer
1000, 3000
Adam, RMSprop, SGD 0.0, 0.5, 0.9, 0.99 0.001, 0.0005
Nesterov momentum (SGD only)
Learning rate
Deepset pooling layer
Deepset hidden layer size
128, 256, 1024 128, 1024
Deepset output size Dropout before p network Deepset activation
0.0. 0.2, 0.5, 0.8 RELU, Tanh, ELU
Attention pooling layer
Hidden layer size
128, 256, 512, 1024 False, True 1, 10, 30, 60
Gated attention
Attention rows
14


Reviews:
[{"review_id": "SygY7oEctB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors tackle the task of taxonomic classification of meta genomic read sets. The combine 2 existing approaches for taxonomic classification, with established methods for Multiple Instance Learning, namely DeepSets and an attention-based pooling layer. \n\nWhile the domain of taxonomic classification is interesting, I find there is a lack of novelty on the machine learning part. The authors combine well established methods in a straight-forward manner and while the resulting increase in performance for some datasets may be relevant in the domain, the conceptual advances are too incremental for a machine learning audience. \n\nUpdate: I have read the response and am still think there is a lack of novelty here. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Skgk1ABTKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "~The authors propose the addition of multiple instance learning mechanism to existing deep learning models to predict taxonomic labels from metagenomic sequences.~\n\nI appreciate the focus area and importance of the problem the authors have outlined. However, I do not think the authors have achieved the conclusions they mention on page 2, as well as other issues throughout the work. I also think the inclusion of the multiple instance learning framework is incremental and does not provide sufficient benefit.\n\n“A new method to generate synthetic read sets with realistic co-occurence patterns from collections of reference genomes”. I do not think there was any systematic analysis of the parameterization of their generative framework. I would appreciate empirical comparison of previous synthetic read generation techniques to the current proposed framework. Also, there is no comparison of the generative framework to real data. How are the parameters chosen in section 3.1.1? Finally, how the authors propose to alleviate bias of composition of databases? Rare species that may be present in abundance in metagenomic data may be swamped out by more common species sequenced again and again in databases.\n\n“A thorough empirical assessment of our proposed model, showing superior performance in prediction the distributions of higher level taxa from read sets.” A comparison to existing alignment-based methods is absolutely required for this work. The authors of GeNet compare to state-of-art Kraken and Centrifuge, and when reading Rojas-Carulla et al., these models have still performed worse than Kraken and Centrifuge, and that should be reported in your assessment.\n\nA few minor points:\n\n-More description of your neural network architecture is needed. I’m not sure what a “ResNet-like neural network” actually means, and how something built for images deals with sequences. There are also different ResNets with different numbers of parameters.\n\n-Why isn’t a 1D convolutional neural network used to process the input sequences? This would make the sequences translation invariant, and would have a similar effect to working with kmers, where k = convolution width.\n\n-It may be useful to interpret the attention mechanism to understand which reads are likely influencing the decision of taxonomic assignment.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkgk_SKkqS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I enjoyed reading this paper and found much of their exposition clear. Also found their extension of previous single read metagenomic classification models with DeepSets and attentional pooling layers to be well explained. However, there are two significant flaws that unfortunately make this paper incomplete as written:\n\n1) The abstract states that this paper will \"attempt to solve the task of directly predicting the distribution over the taxa of whole metagenomic read sets\". However, \"whole metagenomic read sets\" typically contain many millions of reads, whereas the maximum bag size explored in this paper is 2048.  The authors never explain how their MIL metagenomic model should be applied to a full metagenomic sequencing dataset. Should the MIL model be applied to random 2048 read subsets of the many-million read real world datasets? Should these bags of reads be sampled with or without replacement? And, when applied to a whole metagenomic read set, are the improvements in classification accuracy observed for 2048 read bags recapitulated?\n\n2) There is no comparison to standard metagenomic classification tools such as Kraken and Centrifuge. While previous work such as GeNet have compared to these tools the read generation and testing assumptions in this paper are not identical. Also, GeNet was found to be inferior to Kraken and Centrifuge in many scenarios, it would be good to know where in the spectrum of accuracy these new models fall. \n\nUpdate: Having read the rebuttal, my review stands. This paper will be much better once the authors add in estimation from full read sets and evaluate against standard tools. Presumably that will have to be for a future conference submission. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The work proposes a modification to existing architectures applied to predict taxonomic labels from metagenomic sequences. Reviewers agreed that the problem was well motivated, but that current experiments lack comparisons with existing standard baselines in the area. I recommend the authors update their work to included the additional experiments suggested by the reviewers.
Decision: Reject