Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The authors tackle the problem of annotating genes in newly-sequenced genomes. They develop a model called GeneDecoder which uses a combination of a CRF, LSTM and dilated convolutional layers. The authors show that the model relearns several properties of genes, including directionality and length distribution. The model achieves similar predictive performance to existing method Augustus. 
strength_and_weaknesses: Overall, the problem of annotating genes is important, the method is reasonable and the manuscript is understandable. The methodological novelty is low, as the method is a combination of standard CRF, LSTM and dilated CNN models. The experimental setup is flawed and the results are poor. 

I couldn't tell what the model used by GeneDecoder is. It involves a neural network, whose architecture is described in A.4. However it also uses a Latent CRF. I think the NN outputs a representation which forms the input to the CRF? This isn't actually stated.

Benchmarking gene prediction tools is challenging because many of the annotations within gene databases (e.g. GENCODE) are derived from computational predictors. Thus it can be hard to tell whether a predictor is good at discovering real biology or simply recapitulating the errors made by previous predictors. There exists a benchmark for this task, G3PO (Scalzitti et al 2020), cited by the authors.  Unfortunately, the authors did not use this benchmark and instead used an ad-hoc strategy with many issues (see below).

The authors compared to pretrained versions of existing models, so differences in performance could result from differing training sets. When the authors compared against Augustus using a training set similar to Augustus's, results were similar or worse.

The authors trained and tested on genes of the same species. This not a proper simulation of the target application, in which a new species is sequenced and its genome must be annotated from scratch. 

In splitting genes into train and test sets, the authors ensure that all isoforms of the same gene are placed in the same set. However, since many genes overlap, the same sequences likely appear in both train and test sets. This pitfall is described in detail in the following paper. A better strategy would be to split train and test by chromosome (or by species; see previous note). 
https://pubmed.ncbi.nlm.nih.gov/34837041/
clarity,_quality,_novelty_and_reproducibility: See above.
summary_of_the_review: See above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors have presented a novel method of gene finding - where the task is to label spans of text the functional labels.  The presented method uses a neural algorithm that leverages conditional random fields (CRF) over the state of the art HMM methods. They provide a way to fuse pre-trained representations of genomic sequences with their architecture for this task. The novelty is in terms of architecture, incorporating domain information in the form of allowed direction of state changes and experimentations across organisms.
strength_and_weaknesses: Strengths

State of the art gene finding

Pioneer attempt to combine pre-trained embeddings with gene finding

Incorporation of domain knowledge to improve performance

Exhaustive experimentation

Cross organism experiment verification


Weaknesses

No clarity or intuition on why CNN + LSTM does better than pre-trained models

While an empirical ablation has been done, it seems non-intuitive that the pre-trained embeddings do not help much, considering they are tested on similar downstream non-coding tasks. 

Novelty needs to be explained for architecture 

For example, how does the combination of embeddings work with neural HMM based formulation work as compared to the state of the art HMM methods?

The efficiency study for these methods is missing. What do we gain by using this architecture (which takes time to compute) against others? This is relevant especially because the values of Augustus’ baseline performance is quite close.

No statistical significance values reported
clarity,_quality,_novelty_and_reproducibility: Clarity - Paper is very easy to read and follow

Quality - Moderate quality work - requires more analysis to justify the findings

Novelty - Moderately novel - CRFs are the mainstay of POS tagging tasks and this is the first application in the gene embedding space. Though novel, more analysis is required to ascertain the need with respect to state of the art methods.


Reproducible - The paper is easily reproducible and gives clarity.
summary_of_the_review: The authors have proposed a new neural architecture, with elegant ways of including domain knowledge, for the task of gene finding. This could potentially be a new fine tuning task for gene embedding research and help practitioners in unearthing genes among unlabelled data. The experiments have been conducted well and the problem is relevant and exciting.


At this point, I would recommend a weak reject because I am not convinced this neural architecture, though it incorporates domain knowledge, is significantly better than the state of the art methods in any setting, (for example, Augustus also seems to perform well in cross organism settings). 

===============================
Several of the comments have been addressed. So increasing the score.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper develops a new algorithm for gene finding which is compatible with embeddings obtained from deep and transfer learning. 
strength_and_weaknesses: The paper argues the prior models for gene prediction “lack the flexibility to incorporate deep learning representation learning techniques” as a potential drawback. This sets the motivation of the paper to develop a deep learning compatible gene prediction model. However I do not see why deep learning compatibility, per se, is a drawback of the prior works. Thus, in my opinion, the paper is poorly motivated from both the biological and methodological perspectives. The issue becomes even more problematic, as the paper demonstrates that the proposed deep learning-compatible method does not significantly improve the performance in any tangible aspect over the baseline considered. Unfortunately, all these are on top of the fact that the paper misses a key family of baseline algorithms for gene finding, i.e., GeneMark, GeneMark-S, etc. which is discussed in the paper but never compared with. 
clarity,_quality,_novelty_and_reproducibility: The quality is not above acceptance threshold for ICLR.
summary_of_the_review: The paper can be substantially improved by 1) better motivating the need for a new gene finding algorithm, 2) discussing the proposed method more clearly without over-referring to the Appendix, which I found highly distracting, 3) including key remaining baselines, and 4) editing the grammatical errors which occasionally harms the clarity of the message. I do not thing in the current state the paper is ready for publication.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This manuscript describes a straightforward application of standard deep learning techniques to the important problem of gene finding in eukaryotic genomes.  It is actually quite surprising that something like this has not been done before, since this is a central problem in genomics that is still not well solved despite decades of work in the area.  State-of-the-art methods have used HMMs for more than two decades. This paper provides convincing evidence that a deep neural network approach to this problem has promise.
strength_and_weaknesses: The paper does a very good job of providing a concise and accessible description of the gene finding problem for an ICLR audience.

Not much novelty from a machine learning perspective.

The main missing piece in this paper, in my opinion, is an investigation into the types of errors that the system makes in comparison to other methods.  I want to know whether the system is missing some exons entirely, just getting the splice sites wrong, hallucinating new genes that don't exist, etc.

The other missing piece is an evaluation of how well the system works in practice on an actual genome, rather than on snippets of a genome selected around a gene.  As is, the paper is more like a proof of principle than a full solution.

clarity,_quality,_novelty_and_reproducibility: I think the paragraph beginning "The current state of the art" should include citations.  E.g., the first sentence of this paragraph should include citations.  And the first mention of software packages such as Glimmer should be accompanied by a citation.

In Section 4.1, there are some confusing statements about how the model is set up, where we are told, for example, that random flanks were added, but not precisely how these flank lengths were selected.  The text implies that this was handled differently in different experiments.  These choices need to be clearly described, for reproducibility. Similarly, why is the gene length capped at two different values, and which experiments are these caps used for?

I was confused by the organization of the results.  I thought Section 4.2 was going to provide an initial comparison of the proposed method to the current state of the art.  But that section simply lists the what those methods are, and then we skip immediately to the ablation experiments.  I think the ablation experiments should come after the results that show that the method works well across many species. This seems to be what appears in Section 4.6.

Minor:

"dominated the field for more than a decade" -> "... more than two decades"

"HMM model" is redundant.

The Stanke and Wacke citation used \cite{} rather than \citep{}.  This happens several other places as well.

"and highlighting" -> "and highlights"

"recommendation are" -> "recommendations are"

"identification ... are" -> "identification ... is"

Section 3.1 should clarify what the output labels are.

Throughout, there is no need to capitalize the name of a method, even if the associated abbreviation necessarily uses capitals.  E.g. "Linear chain Conditional Random Field (CRF)" -> "linear chain conditional random field (CRF)"

"choice ... depend" -> "choice ... depends"

"takes a onehot encoded sequences" -> "takes a onehot encoded sequence"

"of the CRF:" -> "of the CRF is used:"

"state:" -> "state"

"Figure 3.1" -> "Figure 1"

"to label to unnormalised label" -> ??

"on the either across" -> ??

"one the hidden" -> "on the hidden"

"asses" -> "assess"

"inference set" -> "inference"

The description of annotation sources in Section 3.1 is not clear.  Provide URLs and make clear what those citations refer to.

"A. Thaliana" -> "A. thaliana"

Section 4.3 should do a better job of explicitly using the labeling scheme from Table 1, so we know which line is being discussed at each step.

Section 4.4 mentions the name "GeneDecoder" without explaining that this is the name of the proposed method.  The name is formally introduced later.  I think you should state the name in the introduction.

"both strand" -> "both strands"

"genes that lay" -> "genes that lie"

"of sequenced" -> "of the sequenced"

"models" -> "model's"

"evolutionary" -> "evolutionarily"

"The benchmark limited and specific" -> ??

"model ," -> "model,"

"but it is still has" -> "but it still has"

"previous findings" -> What previous findings? This either requires a citation or a reference to some previous result in the current paper.

"Human" -> no italics, no capitalization

"models closely" -> "model closely"

"mebeddings" -> "embeddings"


summary_of_the_review: From a machine learning perspective, very little here is new or innovative, but that's OK when the task is simply to apply standard methods to solve an important problem. The writing and organization are clear, though the paper is poorly proofread, with many typos and grammatical errors.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper propose a deep learning method for predicting functional coding sequences (genes) within genomes. The work is overall well presented and clear to read, and shows state of the art performance comparable to benchmarks. However, it is limited in terms of technique novelty and there's need for more extensive comparison and testing, which would better help to demonstrate the superiority of this method.

justification_for_why_not_higher_score: While the reviewers have more polarized scores, overall all reviewers noted there's limited novelty from technique and machine learning perspective and it's also lacking the level of extensive evaluation, testing, that several reviewers mentioned. The authors should be commended for revising and replying to the comments and it's notable they also will be running more of these validation so it would be great to integrate these suggestions for a future submission.
justification_for_why_not_lower_score: NA
Decision: Reject