Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: I enjoyed reading this paper and found much of their exposition clear. Also found their extension of previous single read metagenomic classification models with DeepSets and attentional pooling layers to be well explained. However, there are two significant flaws that unfortunately make this paper incomplete as written:

1) The abstract states that this paper will "attempt to solve the task of directly predicting the distribution over the taxa of whole metagenomic read sets". However, "whole metagenomic read sets" typically contain many millions of reads, whereas the maximum bag size explored in this paper is 2048.  The authors never explain how their MIL metagenomic model should be applied to a full metagenomic sequencing dataset. Should the MIL model be applied to random 2048 read subsets of the many-million read real world datasets? Should these bags of reads be sampled with or without replacement? And, when applied to a whole metagenomic read set, are the improvements in classification accuracy observed for 2048 read bags recapitulated?

2) There is no comparison to standard metagenomic classification tools such as Kraken and Centrifuge. While previous work such as GeNet have compared to these tools the read generation and testing assumptions in this paper are not identical. Also, GeNet was found to be inferior to Kraken and Centrifuge in many scenarios, it would be good to know where in the spectrum of accuracy these new models fall. 

Update: Having read the rebuttal, my review stands. This paper will be much better once the authors add in estimation from full read sets and evaluate against standard tools. Presumably that will have to be for a future conference submission. 
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: ~The authors propose the addition of multiple instance learning mechanism to existing deep learning models to predict taxonomic labels from metagenomic sequences.~

I appreciate the focus area and importance of the problem the authors have outlined. However, I do not think the authors have achieved the conclusions they mention on page 2, as well as other issues throughout the work. I also think the inclusion of the multiple instance learning framework is incremental and does not provide sufficient benefit.

“A new method to generate synthetic read sets with realistic co-occurence patterns from collections of reference genomes”. I do not think there was any systematic analysis of the parameterization of their generative framework. I would appreciate empirical comparison of previous synthetic read generation techniques to the current proposed framework. Also, there is no comparison of the generative framework to real data. How are the parameters chosen in section 3.1.1? Finally, how the authors propose to alleviate bias of composition of databases? Rare species that may be present in abundance in metagenomic data may be swamped out by more common species sequenced again and again in databases.

“A thorough empirical assessment of our proposed model, showing superior performance in prediction the distributions of higher level taxa from read sets.” A comparison to existing alignment-based methods is absolutely required for this work. The authors of GeNet compare to state-of-art Kraken and Centrifuge, and when reading Rojas-Carulla et al., these models have still performed worse than Kraken and Centrifuge, and that should be reported in your assessment.

A few minor points:

-More description of your neural network architecture is needed. I’m not sure what a “ResNet-like neural network” actually means, and how something built for images deals with sequences. There are also different ResNets with different numbers of parameters.

-Why isn’t a 1D convolutional neural network used to process the input sequences? This would make the sequences translation invariant, and would have a similar effect to working with kmers, where k = convolution width.

-It may be useful to interpret the attention mechanism to understand which reads are likely influencing the decision of taxonomic assignment.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The authors tackle the task of taxonomic classification of meta genomic read sets. The combine 2 existing approaches for taxonomic classification, with established methods for Multiple Instance Learning, namely DeepSets and an attention-based pooling layer. 

While the domain of taxonomic classification is interesting, I find there is a lack of novelty on the machine learning part. The authors combine well established methods in a straight-forward manner and while the resulting increase in performance for some datasets may be relevant in the domain, the conceptual advances are too incremental for a machine learning audience. 

Update: I have read the response and am still think there is a lack of novelty here. 
review_title: Official Blind Review #1

Meta-review: The work proposes a modification to existing architectures applied to predict taxonomic labels from metagenomic sequences. Reviewers agreed that the problem was well motivated, but that current experiments lack comparisons with existing standard baselines in the area. I recommend the authors update their work to included the additional experiments suggested by the reviewers.
Decision: Reject