Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the problem of de novo assembly, which is one of the most complicated tasks in bioinformatics. De novo assembly is a task to reconstruct the whole genomic sequence given numerous broken pieces of the original sequence. Existing de novo assembly approaches, based on conventional graph algorithms (e.g., de bruijn graph and string graph) require extreme computational cost, restricting further applications.

In this paper, the authors proposed a novel method that reconstructs the original sequence from the assembly graph by using a graph neural network (GNN). When the assembly graph is fed to the GNN, the model outputs the paths to reconstruct the original sequence. The authors demonstrated the effectiveness of the proposed method by comparing reconstruction time and accuracy (i.e., length of the reconstructed genome) with one of the de novo assembly software (Raven) on human genome reference data.

main_review: Existing de novo assembly algorithms or software require extreme computational resources in both time and space. Neural network-based de novo assembly can reduce these extreme costs; therefore, this paper has nontrivial contributions on genome assembly area, and I think this work is promising.

On the other hand, it would be great if the authors address issues as follows:

●	Related work

I think this paper is one of the applications or the extensions of existing work [1], [2], [3], [4]. Could the authors briefly introduce those topics or research areas in the revised manuscript?

●	Experiment (dataset)

In this work, the authors used the human reference genome (CHM13) to demonstrate that the model has a capacity to reconstruct the reference sequence. However, as reference genomes such as CHM13 and GRCh38 already exist, biological researchers and practitioners do not need to assemble the human genome at usual times. Thus, conducting the experiment to explore the generalizability or transferability on unseen (i.e., non-human) sequences is essential to further demonstrate the utility of the proposed method. In other words, the proposed method should work well on assembly graphs of other species, which do not have reference genomes. 

●	Experiment (evaluation)

To measure the accuracy of the assembly, the authors compared the lengths of the reconstructed genome. Obviously, this cannot exactly represent the accuracy of the assembly. Thus, I think the authors should supplement other accuracy metrics in the revised version.

[1] Veličković et al., “Neural Execution of Graph Algorithms”, ICLR 2020.

[2] Vrček et al., “A step towards neural genome assembly”, NeurIPS 2020 workshop.

[3] Joshi et al., “Learning TSP Requires Rethinking Generalization”, CP 2021.

[4] Cappart et al., “Combinatorial Optimization and Reasoning with Graph Neural Networks”, IJCAI 2021.

summary_of_the_review: This paper proposed a novel neural network-based de novo assembly method. Since the proposed method can reduce extreme computational costs during the genome reconstruction process, it could be a promising direction for de novo assembly algorithms.

However, the utility of the proposed method is not sufficiently demonstrated from the perspective of biology researchers and practitioners. Also, the technical novelty of the proposed method seems to be limited in terms of machine learning techniques. 

Therefore, I am leaning towards rejecting this paper at this moment.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a graph convolutional network to assemble a genome from long, perfectly accurate sequencing reads. They show that their model performs better when considering overlap lengths and graph topology versus just the overlap lengths. The proposed method is compared to other methods, such as Raven, and shown to reconstruct the genome much faster. The work was demonstrated on synthetic data without any read errors and thus serves as a prototype for real de novo genome assembly problems, which are much longer and contain noise.
main_review: Strengths
- uses synthetic data to allow for quantifying accuracy of reconstructed genomes
- benchmark against a recent previous method, Raven, was performed for both accuracy and execution time

Limitations: 
- The synthetic sequencing reads do not contain any errors. This may  be a reasonable approximation only for HiFi reads. The authors should include simulations with varying levels of noise in order to test the efficacy of the proposed method versus Raven. If this is not done, then the scope of the paper should be about genome assembly for HiFi reads, instead of the more generic language used throughout the text. 
- How does the method do with shorter reads? Perhaps a sweep of this could be informative to set lower bounds. This is important for practitioners who may erroneously use this thinking it is a general assembler.  
- The range of synthetic genomes are very small, up to 10 Mbp. As the authors stated, a human chromosome is 250 Mbp and there are of course 23 pairs of chromosomes.  Thus, it is unclear how well the GCN can assemble realistic data (not HiFi) for which there is plenty of data. A good use case could be metagenomics, where de novo assembly of contigs is critical. There needs to be some demonstration on real data. A proof of principle using real data would make this paper more compelling.
- With Raven, there is no training. It assembles the genome with an algorithm. On the other hand, the GNN has parameters that require training. So, the reported execution time for GNN includes just the inference time and not the training, right? Is it fair to compare just the inference time of the GNN on a very powerful A100 vs the whole end-to-end task for the other methods? Perhaps an end-to-end execution time is more appropriate. Also, what is the memory requirement of the GCN? Do people need 48GB of GPU ram? How would the model's execution time be on a GPU that is more common, like a P100 on google colab?
- Can this method deal with allelic differences across chromosomes? 
- The execution time was based on a single Intel Xeon E5-2698 v4 CPU, which has 20 cores. The Raven paper states that they were run on two AMD EPYCTM 7702 64-core processors. How much would this speed up the execution time, using more cores for Raven. 
summary_of_the_review: This paper provides a promising approach to use GCNs to assemble genomes. To the best of my knowledge, this is novel. The demonstration is focused on synthetic data with no noise. While this can be reasonable approximation with HiFi reads, it remains unclear how it extends to other kinds of data, eg. short- and long-read RNA seq for which there exists many data. To be impactful, a comparison on more realistic synthetic data is important, such as noise added and longer genomes. While there are many good things about this paper to be excited by, critically, a demonstration on real data is needed.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents an interesting idea of using graph-based neural networks (GNNs) to find an optimal layout path during de novo genome assembly using the Overal-Layout-Consensus (OLS) method. Existing methods perform this step by using heuristics to simplify the graph (removing edges/nodes)  instead of finding a path to assemble the genome from the assembly graph. Therefore, they fail to resolve complex graphs with repetitive regions and instead remove or trim those altogether. The paper proposes to train a gated GNN to predict probabilities for the edges of an assembly graph. These probabilities can help in defining a path on the assembly graph that can lead to faster de novo assembly during the Layout phase. To obtain ground truth positional information, the paper uses simulated reads generated from the human genome. The GNN model is trained on mini-references (2 Mbp regions) taken from chromosome 11 and the alignment and execution time results are presented for 2Mbp, 5Mbp, and 10Mbp mini-references from chromosomes 11,10, and 12. The results show that the proposed GNN strategy improves over the greedy approach in alignment performance and an existing assembler (Raven) in execution time. 

main_review: Strengths:
+ The paper presents an interesting idea of using Graph-based deep learning to learn the path for genome assembly thus extending the previous neural network-based assembly idea to graphs. 
+ The paper is clearly written. The problem is well defined and the choice of GNN architecture is well motivated.
+ The result regarding the model handling the repetitive regions more effectively than Raven is particularly important for the use of the proposed method for de novo genome assembly.

Weaknesses: 
- The results in the paper are not comprehensive enough to support and highlight the claims in the introduction. They are restricted to a very small subset of the simulated datasets generated from the human genome (focusing on just 3 chromosomes). 
- The choice of the baselines is not entirely clear. While Raven seems like a reasonable assembly method, the results in the paper (Vaser et al, 2021) do not suggest that it is the best assembly tool, thus requiring at least a mention/discussion of other genome assembly methods in the literature. 
- Furthermore, the baseline Raven paper explores different genomes (plants, prokaryotes, eukaryotes) as well as different sequencing technologies. This paper would benefit from a similarly comprehensive study of the performance of the GNN based method across different datasets
- The GNN method requires the ground-truth position information that is obtained from an assembled genome and reads generated via simulation. Is the inference of a model trained from simulations then generalizable to another chromosome or another genome with actual reads? The results seem to suggest the former but it is unclear if the inferences were performed on actual reads of other chromosomes. How will such a framework work for de novo assembly? 
- Does the execution time include the training time of the GNN model? If not, then are the execution time comparisons fair? 
- The scalability of the method should be further explored in comparison to other baselines as graphs with the execution time as a function of the size of the mini references scaling to sizes>10 Mbp. 
- It would be useful to explore the potential cause of the differences in alignment performance of the GNN method (and others) across 2Mbp, 5Mbp, and 10Mbp mini-references. 

summary_of_the_review: The proposed idea in the paper is interesting, however, the limited set of results are insufficient to support its claims and highlight its usefulness for de novo genome assembly.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a graph neural network approach for genome assembly. In the framework of overlap-layout-consensus, a graph structure is formed in which reads are nodes and overlaps between them are edges. In theory, one desires a Hamiltonian path that visits every node and thus forms a reconstruction of the full genome.

The authors suggest that a gated graph convolutional neural network could compute on this graph to identify the true edges which should be followed in the Hamiltonian path. This is a creative and intriguing direction of research.

They specify a procedure to simulate data from a known genome and train the algorithm. They benchmark their method on simulated data versus another recently published genome assembler called Raven.
main_review: The benchmarks aren’t sufficient to demonstrate the potential for this approach. The authors do not use standard metrics for this field or compare to widely used genome assemblers like wtdbg2 and mdBG. These manuscripts are also great references for additional metrics to study.

Ekim, B., Berger, B. & Chikhi, R. Minimizer-space de Bruijn graphs: Whole-genome assembly of long reads in minutes on a personal computer. Cell Syst 12, 958-968.e6 (2021). 
Ruan, J. & Li, H. Fast and accurate long-read assembly with wtdbg2. Nat Methods 17, 155–158 (2020). 

The use of only simulated data is another problem. There is often a very large difference between solving this problem on simulated versus real data. For this reason, the field has needed to hold competitions to properly benchmark methods, such as the Assemblathon.

The authors’ simulated data also contains no sequencing errors. I completely disagree with the authors that this assumption is not “far-fetched”. The authors must benchmark their method on data containing sequencing errors, which are an inevitable presence in real data. If they believe error correction methods are a useful stage in a full pipeline, they are welcome to include such a method.

On p. 7, the authors write, “We start the traversal from the starting node of the ground-truth path and take w steps, where w is a hyperparameter and we refer to it as the walk length." How do you make use of data from the later portion of the genome if every SGD step begins from the start and goes for 10 graph steps?
summary_of_the_review: Processing assembly graphs with graph neural networks is an intriguing direction, but this work hasn't sufficiently demonstrated its value.
review_title: null

Meta-review: The paper demonstrates that one phase of de novo assembly, specifically the layout phase, can be replaced with graph-neural-network based methods. The paper clarifies in the rebuttal that it focuses on building a method for assembling high-quality long reads. 

All four reviewers rated the paper as below the acceptance threshold. The reviewers largely agree that the idea of using GNNs to assemble a genome from reads is novel, interesting, and has the potential to be very useful. 
The reviewers raise the following concerns: The paper only considers synthetic data, and the synthetic reads used in the simulations are error-free. In practice, reads are not error-free, and thus simulations on real data or at the very least on reads with errors are needed. The authors acknowledge that, and state that they'll provide such experiments in future work. In summary, the reviewers found the experiments to be insufficient to support the claims, even though it is understood by the reviewers and me that the paper only presents a proof-of-concept idea. I agree with the reviewers that simulations on erroneous reads, ideally real data, would be needed for acceptance.

I recommend to reject the paper, since the paper provides insufficient experiments to understand the merits of the proposed approach.
Decision: Reject