Gene finding revisited: improved robustness through structured decoding from learning embeddings
Authors: Frederikke Isa Marin
Dennis Pultz
Wouter Boomsma
Venue: Submitted to ICLR 2023 (2023)
Keywords: gene finding
graphical model
gene prediction
gene splicing
conditional random fields
structured decoding
DNA
learned embeddings

Abstract: Gene finding is the task of identifying the locations of coding sequences within the vast amount of genetic code contained in the genome. With an ever increasing quantity of raw genome sequences, gene finding is an important avenue towards understanding the genetic information of (novel) organisms, as well as learning shared patterns across evolutionarily diverse species. The current state of the art are graphical models usually trained per organism and requiring manually curated data sets. However, these models lack the flexibility to incorporate deep learning representation learning techniques that have in recent years been transformative in the analysis of protein sequences, and which could potentially help gene finders exploit the growing number of sequenced genomes to expand performance across multiple organisms. Here, we propose a novel approach, combining learned embeddings of raw genetic sequences with exact
decoding using a latent conditional random field. We show that the model achieves performance matching the current state of the art, while increasing training robustness, and removing the need for manually fitted length distributions. As language models for DNA improve, this paves the way for more performant cross-organism gene-finders.  

Under review as a conference paper at ICLR 2023
GENE FINDING REVISITED: IMPROVED ROBUSTNESS THROUGH STRUCTURED DECODING FROM LEARNING EMBEDDINGS
Anonymous authors Paper under double-blind review
ABSTRACT
Gene finding is the task of identifying the locations of coding sequences within the vast amount of genetic code contained in the genome. With an ever increasing quantity of raw genome sequences, gene finding is an important avenue towards understanding the genetic information of (novel) organisms, as well as learning shared patterns across evolutionarily diverse species. The current state of the art are graphical models usually trained per organism and requiring manually curated data sets. However, these models lack the flexibility to incorporate deep learn- ing representation learning techniques that have in recent years been transforma- live in the analysis of protein sequences, and which could potentially help gene finders exploit the growing number of the sequenced genomes to expand perfor- mance across multiple organisms. Here, we propose novel approach, combining learned embeddings of raw genetic sequences with exact decoding using a latent conditional random field. We show that the model achieves performance match- ing the current state of the art, while increasing training robustness, and removing the need for manually fitted length distributions. As language models for DNA improve, this paves the way for more performant cross-organism gene- gene-finders.
INTRODUCTION
Genes are patches of deoxyribonucleic acid (DNA) in our genome that encode functional and struc- tural products of the cell. The central dogma of biology states that these segments are transcribed into ribonucleic acid (RNA) and in many cases translated into the amino acid sequences of proteins. In recent years, the machine learning community has dedicated considerable attention specifically to studying proteins, and solving various protein-relate tasks, with the aid of deep learning. This fo- cus has resulted in impressive advances within the field (Detlefsen et al. 2022 Jumper al. 2021 Rao al.) 2020; Shin et al. 202Tp. Less attention has been paid to the DNA sequences themsel ves, despite the fact that finding genes in genome remains important open problem. Due to techno- logical advances, the rate by which genomes are sequenced is rising much more rapidly than we can reliably annotate genes experimentally, and without proper gene annotations, we lack information about the proteins encoded in these sequences. In particular, for taxonomies that are sparsely char- acterised or highly diverse, such as fungi, this hinders from extracting essential information from newly sequenced genomes.
The wealth of available genomic data suggests that this is area ripe for high-capacity deep learning approaches that automatically detect the most salient features in the data. This potential has in recent years been clearly demonstrated in the realm proteins where deep learning has proven
extremely effective in both the supervised setting (Alley et al. 2019; Hsu et al. 2022 Jumper et al.) 202T) and the unsupervised setting (Rao et al.J 2021; 2020, Vig et al. 2021). In particular, embeddings obtained in transformer based protein language models have pushed the boundaries for performance in many downstream sequence- based prediction tasks. The advantages of such models two-fold: 1) they enable pre-training in cases where unlabelled data far outweighs labelled data and 2) they have demonstrated the ability to learn across diverse proteins. We are currently witnessing an emerging interest in language models for DNA well, but progress in this area has proven more difficult than for its protein counterpart. In a completely unsupervised setting the amount of DNA data is orders of magnitude larger than that of proteins, and the signals

---

Under review as a conference paper at ICLR 2023
correspondingly sparse. For instance, Eukaryotic genomes consist of millions to billions of DNA base pairs but only a small percentage are genes and an even smaller percentage codes for protein (approx. 1 % in the human genome). Genes also have intricate structure which places demands on a high degrees of consistency between output labels predicted at different positions in the sequence. In particular, genes contain both coding segments (called CDS exon) and intron segments. Only the coding segments are retained in the final protein, while the introns are removed. The process in which introns are removed is called splicing, which occurs after the gene is transcribed from DNA to RNA. After introns are spliced out, the RNA is translated to amino acid sequences (the protein product). Each amino acid is encoded by a codon (a triplet of DNA nucleotides) in the RNA sequence. Due to this codon structure, the annotation of the coding sequences in the gene must be extremely accurate, as shifting the frame of the codon with just one will result in a nonsensical protein. Gene prediction thus consists of correctly annotating the boundaries of a gene as well as the intron splice sites (donor/acceptor sites), a task challenged both by the imbalanced data but also by the extreme accuracy needed.
The current state-of-the-art in gene-finding relies on Hidden Markov Models (HMMs) and exact decoding (e.g. Viterbi) to ensure the required consistency among predictions at different output positions. To make these methods work well in practice, considerable effort has been put into hand-coded length distributions inside the HMM transition matrix, and a careful curation of the training data to ensure that the length statisties are representative for the genome in question. The resulting HMMs have dominated the field for more than two decades. However, their performance still leaves lot to be desired, they are generally difficult to train, and have no mechanism for incorporating learned embeddings and context dependent learned length distributions. These models can be improved by incorporating them with external hints and constructing pipelines (Hoff et al.) 2016) but they are not compatible with deep learning advents that have revolutionised adjacent fields. The goal with this paper is to develop new approach that is compatible with contemporary deep learning practices, can be trained without manual feature engineering and careful data curation, while maintaining the capability for exact decoding.
Here we present an approach, which we term GeneDecoder, to gene prediction that is able to both incorporate prior knowledge of gene structure in the form of a latent graphs in Conditional Random Fields as well as embeddings learned directly from the DNA sequence. This approach proves easy to train naively while still achieving high performance across range of diverse genomes. We highlight that the resulting model is very flexible and open to improvement either by including external hints or by improvement of the current DNA sequence embeddings. We benchmark against three other supervised algorithms (Augustus, Snap, GlimmerH\MM) and find that the performance of our model competes with that of the otate-of-the-art (Scalzitti et al.] 2020) without strong effort put into model-selection. However, as pre-trained DNA models start to emerge and improve we expect that the full potential of this approach will be realised.
2 RELATED WORK
Current gene prediction algorithms are Hidden Markov Models (HMM) or Generalized HMMs.
These include Augustus (Stanke & Waack 2003). Snap (Korf) 2004).. GlimmerHMM (Majoros) et al. 2004) and Genemark.hmm (Borodovsky & Lomsadze 2011). All these models are trained fully supervised and on per-organism basis. Genemark also exists in version that is similarly trained on one organism but in an iterative self-supervised manner (Ter-Hovhannisan et al. 2008 In practice, gene prediction is often done through meta-prediction pipelines such as Braker (Hoff et al.) 2016), Maker2 (Holt& & Yandell 201T) and Snowyowl (Reid et al. 2014), which typically combine preexisting HMMs with external hints (e.g. protein or RNA alignments) and/or iterated training. Focusing on the individual gene predictors, Augustus is currently the tool of choice in the supervised setting, according to recent benchmark study (Scalzitti et al. 2020). It is an HMM with explicit length distributions for introns and CDS states. The model also includes multiple types of intron and exon states emitting either fixed-length sequences or sequences from a length distribution given by a specific choice of self-transition between states. This intron model has been shown to be key to its performance; without it Augustus was found to be incapable of modelling length distri- butions in introns correctly (Stanke & Waack, 2003). Models like Snap and GlimmerHMM follow similar idcas, but differ in their transition structure. In particular, GlimmerHMM includes a splice site model from Genesplicer (Pertea al. 2001). These HMM-gene predictors are known to be
2

---

Under review as a conference paper at ICLR 2023
Table I:
Label sets used in the CRF model. The Direction labels used support processing
genes in both the forward (F) and reverse (R) directions. The Codon numbering is required to keep track of the reading frame. To avoid the need for users specify these labels, we use latent CRF which internally processes the full label set, but emits only the base labels.
Description
Base label
Direction
Direction + Codon
Exon/CDS
E
Ep, En 18,1F DF, DR AF,Ap
FI, F2,r, E3,F, FI,R, E2,n, Es,n 11,,, 12, 13,F, L1 12, 13.R DI,F, D2,F, D2,F, D3,F, DI,R, D2,R, D3.R AI,F, A2,8, As,F, AI, A2, As,R
Intron
Donor splice site
D
Acceptor splice site
A
Non-coding/ntergenicc
NC
sensitive the quality of the input data set. For instance, the Augustus training guidelines specifies requirements for non-redundancy, non-overlapping genes and restrictions to single transcripts per genes. These considerations are not merely theoretical. for this paper, have attempted to retrain several of these models on own data set splits, but were unable to obtain competitive results. The Augustus guidelines also report diminishing returns for data sets with more than 1000 genes, and highlights the importance of quality the training set in this scenario. These recommendations are sensible a low-data regime, but we might wish to relax them we obtain more genomic data. It would be convenient if we could train gene predictor models on lower quality data sets, including multiple, potentially conflicting tran- scripts arising for instance from alternative splicing. The goal of this paper to explore modelling strategies towards this goal.
the preparation of the baseline results
quantity in
Eventually, we would also like train cross-genome gene predictors, for instance using featuriza- tions obtained from pre-trained language models. Following the success of pre-trained models on protein sequences, DNA language modelling starting emerge field. Currently, two such models are available: DNABert (Ji et al. (222T) and GPN (Benegas et al. (022). Both are trained on single organism, each across a whole genome with sequence window of 512 nucleotides. While Ber is standard transformer based model for which long sequences can be computationally prohibitive, GPN (Benegas et al., (002)) is based on dilated convolutions where it could possible to extend the sequence length. In this work we perform preliminary explorations of using GPN embeddings for gene prediction.
METHODS
3.1 MODEL
Latent conditional random fields Genomic datasets are highly unbalanced, containing only very small percentage of the donor and acceptor labels. Since identification of these labels is paramount for correct annotation, the model must be designed to deal robustly with this. Furthermore, wish to encode prior knowledge of gene structure in order to produce consistent predictions. Finally, the model must be flexible in the type of input used, so that context dependent embeddings can be learned directly from the sequence. To fulfil these requirements, we choose Linear chain Condi- tional Random Field (CRF) (Lafferty et al. 200T) model. The CRF can, like the HMM, encode gene structure in its graph, but because it models the conditional probability of the output labels, rather than the joint distribution of input and output, it is well suited for integration with a embedding model. In doing so, we can hope to learn richer representation of the input, while still training the model end-to-end.
The input to the embedding model are one-hot encoded values X = unknown nucleotides. As output labels, we have Exon (E), Intron (!), Donor (D), Acceptor (A) and Non-coding (NC). The CRF learns transition potentials (i.e. unnormalized transition probabilities) between these states, but we can manually set particular of these potentials to inf to allow only biologically meaningful transitions, thus following similar strategy that employed in classic HMM gene predictors. Two important restrictions are 1) directionality, ensuring that forward and reverse coded genes are not intermixed, and 2) codon-awareness, which ensures that the coding
where N is
3

---

Under review as a conference paper at ICLR 2023
Dilated CNN
LSTM
L-CRF
A
E
h
2/1
G
C
E
G
T
G
G
hi
Featurization emmbeddigg
Decoding
Figure 1: Overview of the model. The current best version GeneDecoder has feature model that processes the input first with dilated convolutional layers with residual connections and thereafter a bidirectional LSTM. The feature model outputs an-normalized label probabilities per positions which are then processed by the latent CRF to output label sequences in manner consistent with the allowed transitions and emissions.
regions genes are always multiple of three. To encode this in the transition potentials, we expand the label set (Table T.. Note that in addition to the states, it is necessary to expand also the intron, acceptor and donor states such that codon position can be tracked through the intron regions.
The likelihood for CRF takes the form:
N
P(y|x, 0) = Z
exp
Σ n=1
(T (To-1.00 + (xx))
(1)
T' is the learned transition weights of the observed labels and 0 the learned input embedding. The partition function Z is the summed score over all possible label sequences:
2 = Σ exp Σ N WEY
(Te, 1.0% + +((r))
(2)
In the standard CRF, the states appearing the transition table will be observed during training. In our case, we do not wish to specify the direction and codon-index apriori. HMM-based genefinders solve this problem by formulating the Markov process in a latent discrete space, and coupling to the output states through learned emission probability table. A similar solution is not easily tractable for CRFs. However, as shown by Morency al, it becomes feasible if we introduce many-to-one deterministic mapping between latent and observed states (Morency et al. (2007). This a natural assumption to make our case, as illustrated by the individual rows in Table [ The corresponding Latent CRF (L-CRF), includes summation over the set of latent states emitting given observed state.
N
P(y|x, 0)
7 exp
Σ Σ KEE.
(Twin + En.o..) + (xn)
(3)
n=l
W MEH
where Hy denotes the set of hidden states emitting the value taken Vn- The learned transition matrix T is now over the hidden states and the emission matrix denotes the emission potentials from hidden to observed states. Similarly, the partition function also sums over the set of hidden
4

---

Under review as a conference paper at ICLR 2023
states that emits y:
== E exp
Σ
Σ hellyn
(Tw + Emgnn + ((xn)
(4)
S'EY n=1 WEEl
In our case, the Hy,, matrix is given by the rows in Table [ matrix T' are learned (see examples of an udjacency matrices in Figure A.2). We assume that gene structure is similar on the forward and reverse strands and therefore share the learned transition potentials between the two directions. Figure I shows full overview of our model. In the remainder of the paper, we will refer to this model decoding architecture as CRF and L-CRF interchangably.
The non-blocked entries the transition
Feature Model Embeddings Although the embedding function 0 in CRF was traditionally based on manually engineered features, modern autograd frameworks make possible to learn such func- tions using e.g. neural networks, and train both the featurization and decoder models end-to-end. This allows us to use a complex, context dependent mapping from the one-hot encoded input se- quence the conditioning input tion of a (dilated) CNN and an LSTM. The former ensures that we see beyond single nucleotides, avoiding the need to manually encode at the codon level, and the latter models the remaining quential signal. We disregarded more expressive architectures such as transformers because we in the purely-supervised setting are in a limited data regime. However, we anticipate that pre-trained language models for DNA will eventually replace the need for learning new feature embeddings per organism (actually, one could meaningfully define models). We briefly explore this potential in the result section below.
CRE. As proof of concept,
here
simple combina-
success criterion for DNA language
Training The model were trained using AdamW on the on the Negative Log Likelihood, using batchsize of 64. We used early stopping based on the validation set likelihood to avoid overfitting.
Inference Viterbi decoding is used find the best label sequence under the model. Alternatively sampling can be performed according to the label probabilities, either across the entire sequence or for segments. Posterior marginal probabilities per position are calculated with the forward-backward algorithm and can be used to assess model certainty for specific positions.
Choice of performance metrics Accurately predicting the direction, as well as the donor and ac- ceptor sites (i.e. start and end of introns), is highly important for predicting the correct protein. For this reason Matthews Correlation Coefficient as well as Macro weighted Fl score reported a the label set containing exon/CDS, intron, donor and acceptor labels.
4 EXPERIMENTS
4.1 DATA
Sequence and general feature format (gff3) annotation files are obtained for each genome as fol- lows: Human Geneode, Aspergillus Nidulans and Saccharomyces Cerevisiae EnsembleFungi. Arabidopsis Thaliana- EnsemblPlants, C. Elegans Wormbase (Frankish et al. 2021 Cunningham Howe et 2017).
al. 2022
No augmentation or inspection of the data was performed, with the exception of ensuring the total length of CDS segments in each gene was divisible by 3, ensuring that valid protein product was possible. Gene annotations were extracted with a random flanks of a length between 1-2000 nu- cleotides. These relatively short flank lengths were chosen decrease training time. The maximum allowed gene length was capped according to the gene length distribution in the species to ensure Representative data set but simultaneously exclude extreme outliers in gene length that would heav- ily increase compute time. The data set was homology partitioned at threshold of 80% protein sequence similarity into train, validation and test sets. The test sets were used for benchmarking. The length of the flank can influence the performance depending on the inference task. For some ablation experiments simpler feature models have been chosen in order to reduce compute, these results may therefore reflect only relative performance rather than the full potential.
5

---

Under review as a conference paper at ICLR 2023
4.2 BENCHMARK ALGORIHHM
We benchmark method against three other widely used gene prediction software (Augustus, GeneMark.hmm, Snap and GlimmerHMMM, Given the sensitivity of these methods to the training sets, pre-trained parameter set for each species rule out that sequences in our test set are part of the training set of the baseline models. Our reported results are thus conservative estimate of our relative performance. There exists several versions of the Gene algorithm depending on the desired user case (Borodovsky & Lomsadze [001] Bese- mer al. 200T Lukashin & Borodovsky 1998). We chose GeneMark.hmin this a supervised pre-trained per species algorithm, which matches the scope here.
used where available. This means we cannot
4.3 BENCHMARKING ACROSS EVOLUTIONARII DIVERSE SPECIES
To demonstrate the performance of our approach we benchmark against four widely used programs on five phylogenetically diverse well-studied species typically used in benchmarking (Table Our model, GeneDecoder, was trained separately genes from each organism, which is the standard for such gene-finders (Stanke et al. 2006 BBorodovsky & Lomsadze 2011). For the other models, pre- trained versions of the programs for the given species, were used in order to not reduce performance by re-training on a data set that was insufficiently curated for their model.
We find that only Augustus achieves similar performance markedly lower across all species we it tested on. GlimmerHMM and ManeMark.hmm shows better performance but it still has significantly lower predictive power compared to Augustus. This further cements previous findings that Augustus achieves state of the art performance (Scalzitti et al.. 2020).
model, with Snap performing
Our model, GeneDecoder competes with outperforms Augustus, which is state-of-the art gene prediction software, across the benchmark, with the exception of the lower Fl score for S. Cere- visiae. The low F1 score for S. Cerevisiae across all algorithms is due to the low number of intron- containing genes. Less than 1 % genes contain introns in the data set for this organism. Since the Fl score is highly influenced by performance on underrepresented label categories this reduced per- formance comes from poor prediction of introns and donor/acceptor sites. It was not possible to obtain Augustus' training data set for this species, but according to the official training instructions of the software, data sets must be curated to contain high proportion of intron-containing genes. This would explain discrepancy in the Fl score for Augustus and GeneDecoder, however it also highlights that our model is not highly influenced by the quality of the data set. We expect that the perfomance of GeneDecoder would improve with a curated dataset or oversampling of intron containing genes.
We further compare Augustus with our model by training and testing on the original Augustus data sets for Human (Stanke 2004). The training data set is in low data regime, consisting of 1284 training genes, which is not ideal for deep learning settings. Furthermore, the Augustus data set strictly homology reduced, allowing more than 80% sequence similarity the protein level. Nevertheless, our model matches the performance of Augustus on its own data set (see Figure Figure(A.5). The difference in predictive performance on non coding labels are likely due to a manually set self- transition probability of 0.99 for non coding labels in the Augustus model. This improves Augustus' performance on non coding labels but might come at the expense of predicting other labels. While improving Augustus performance requires the use of external hints chosen by the user, our model may be improved by learning better embeddings.
4.4 LEARNED EMBEDDINGS IMPROVE PREDICTIONS
To explore the performance contribution of different aspects of our model we performed ablations, where parts of the model were excluded simplified part of it (Table 3). Note that these models were trained on a smaller set the A. thaliana data set.
First, to test the capabilities of the CRF alone, single linear layer is implemented to map each position from the one-hot encoded DNA sequence to the hidden states. Two graphs of different complexity were used for this experiment (see Table Figure Linear- CRFs exhibit very poor predictive power, the more complex codon graph, which encodes knowledge of codon structure, clearly performs better that the simple graph, illustrating the need
A.2 for graphs). While both

---

Under review as a conference paper at ICLR 2023
Table 2: Comparison of algorithms
Model
Human MCC
Nidulans F1
A. Thalaiana
S. Cerevisiae
C. Elegans
F1
MCC
MCC
F1
MCC
F1
MCC
F1
GeneDecoder
0.81
0.83
0.90
0.91
0.90
0.82
0.88
0.46
0.92
0.94
Augustus
0.78
0.81
0.87
0.81
0.87
0.70
0.88
0.53
0.76
0.75
Snap
0.36
0.41
N/A
N/A
0.54
0.54
0.85
0.45
0.55
0.58
GlimmerHMM
0.74
0.76
N/A
N/A
0.77
0.78
N/A
N/A
0.78
0.80
GeneMark.hmm
0.62
0.71
N/A
N/A
0.64
0.56
N/A
N/A
0.59
0.70
Table 3: Ablations of feature model on A. Thaliana
FEATURE MODEL
MCC
Fl
Linear +
CRF (simple graph)
0.15
0.11
Linear CRF (codon graph)
0.34
0.19
LSTM
0.79
0.75
LSTM + CRF
0.86
0.84
DilCNN + LSTM + CRF (GeneDecodery
0.89
0.85
GPN + CRF
0.74
0.75
for a higher graph complexity to improve modelling performance. This consistent with similar observations made for HMM-based gene predictors SeeacctorsSStanke (Stanke & Waack (2003).
By introducing a feature model such as an LSTM, performance greatly enhanced. While LSTM trained alone also displays decent performance it is evident from Figure A.5 that without the CRF, splice sites are often not labelled correctly which is vital for producing consistent predictions.
Interestingly training linear layer fixed embeddings from the GPN language model (Benegas et al. 2022) achieves high performance (Figure A.5). This indicates that the masked pre-traiming of the GPN model captures DNA sequence patterns relevant for gene prediction, despite only having trained on a single organism and primarily on non non-coding DNA. The emergent field of DNA lan- guage modelling could have significant effect downstream DNA modelling tasks, as they have had in the protein field. These ablation studies reveal that with a good feature model, there less need for complex graph structure. The feature model can instead learn complex dependencies directly from the sequence.
The best performing model, which refer as GeneDecoder, bination of dilated convolutions well as bidirectional LSTM, which we al. We hypothesise that the inereased performance over a LSTM alone is due better long range information provided by the dilated convolutions. The details of the architecture are available in Appendix A.4).
feature model with a com-
4.5 LEARNED EMBEDDINGS ACCURATELY MODEL LENGTH DISTRIBUTIONS
Accurately capturing length distributions of introns and exons is essential for well performing gene predictor. Along with explicitly modelling exon length distributions, [Stanke Waack (2003) had to introduce an intron sub-model with states modelling both explicit length distributions as well as fixed length emissions, to meaningfully capture intron lengths.
The latent CRF without learned embedding is not sufficient to reproduce the true lengths resulting in a flattened dis- tribution similar to Stanke & Waack (2003). However, with the learned embeddings, Genede captures the length distributions faithfully. especially for exons. trained embeddings GPN model is also able to capture the length distributions accurately. The GPN embeddings were not fine-tuned in this case. This further cements that incorporating deep learning, and particularly unsupervised pre- me-trieeed embeddings, promising direction for Recoder and
We find that our model readily learns these signals directly from data (Figure 4.5)
very promising that the pre-

---

Under review as a conference paper at ICLR 2023
genefinders in general. Especially when expanding the model to perform cross-species prediction. However, DNA language modelling still a nascent field and there are currently many unanswered questions on how to transfer the masked modelling concept to DNA sequences as compared to pro- teins. These concern both the sparsity of information, the longer sequences and the much longer range signal that likely needs be captured for high performance in many downstream tasks. The GPN model is trained on DNA segments of 512 nucleotides which might exclude capturing many longer range signals and affect performance.
Intron Length Distributions
True
Linear
GeneDecoder
GPN
200
400
600
800
1000
Exon Length Distributions
500
750
1000
1250
1500
1750
2000
Length (Nucleotides)
Figure 2: Learned embeddings capture length distributions of more accurately
4.6 LATENT STRUCTURE CAN INFER UNOBSERVED LABELS
To test the capacity of the latent graph of our model we explore whether can learn unobserved states directly from the sequence. One such example inferring the directionality of genes. Genes can be encoded on both strands of the double-helical DNA but usually only one strand is sequenced. Genes that lie in the "reading" direction of sequenced strand are designated as forward and genes that would be read on the opposite strand reverse. We train our model on directionless set of observed labels (see simple-labels-DD provide any information about directionality, the model must learn this directly from the sequence to be able to distinguish between directions.
Appendix [..]). Since the training data does not
Figure 4.5 4 shows the comparison between model trained the directionless label set well as a set including directional labels (see simple-direction -DA in Appendix [A.I). I). Although there is reduction in performance between the training regiments, the model is still able to learn and infer direction. Surprisingly the reduced performance does not originate from confusion between forward and reverse labels but rather due the model's propensity predict non-coding labels (seele A.5)
5 CONCLUSION
Here we present a novel approach, GeneDecoder, to gene prediction combining the advantages of graphical models with the flexibility and potential of deep learned embeddings.
8

---

Under review as a conference paper at ICLR 2023
1.0
0.8
True Postitive
0.6
direction
0.4
wio direct on
0.2
0.0
Ef
DF
1F
Ap
ER Label
DR
1x
AR
NC
Figure 3: Inferring unobserved labels directional labels
We find that the Latent CRF gives us not only the ability to preserve the graph-encoding of prior knowledge, as in the state-of-the-art HMMs gene predictors, but also offers an advantage by al- lowing us learn embeddings directly from the sequence. This advantage is not only visible in the performance, but also in the low effort required to construct data sets and train the model. The greatest advantage, however, might well be the great flexibility of the learned embeddings and the potential to improve them as the DNA modelling field advances.
Even now this seems evident from the preliminary studies performed here using embeddings from a DNA language model trained only on a single species, with very short input sequences. We expect that as the field progresses language models more specifically tailored DNA will emerge, modelling longer range interactions and training across multiple species. Such results have already been demonstrated by the success of the Enformer model (Avsec et al. 202I), which demonstrated the importance of long range information as well as the ability to learn downstream tasks that were not trained on.
One potential limitation of model is that the amount of non-coding DNA (in cases the flanking regions) in the training data set can affect the performance the model on data with significantly more non-coding DNA. This issue be resolved in the training process by scaling the training to larger data sets containing longer non-coding regions. As this presents a technical challenge, along with time and resource challenge, rather than scientific one we leave it for future work. Our model provides the capacity to include external hints in the inference process. Either via sampling high probability pathways around fixed labels but also via modification of the hidden state probabilities. The latter method can even be used flexibility of out model. Lastly, model per position certainty is directly provided model in the form of posterior marginal probabilities. We leave it for future exploration to rigorously benchmark this.
strategy during training, highlighting the
We also expect that language modelling will alleviate issues like this by modelling better and more informative embeddings of the raw sequence. But we leave this for future work. In this regard we view the model presented here as an initial iteration that can only improve as DNA mod- elling does. We anticipate that future iterations of Gene will incorporate some form of self-supervised pre-training across various species which could vastly improve not only prediction tasks in species with rich gene annotation data but also in novel understudied organisms.
CODE AVAILABILITY
Code will be available on github upon publication.
REFERENCES
Ethan C. Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi, and George M. Church. Unified rational protein engineering with sequence-hased deep representation learn-

---

Under review as a conference paper at ICLR 2023
ing. Nature methods, 16(12):1315-122) December 2019. ISSN 1548-7091. doi: 10.
1088/44159201---5988... URL attpp::/www. PMC706768271
nlm. nih. gww wttp://////atccle//ee//
Ziga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R. Ledsam, Agmieszka Trabska-Barwinska. Kyle R. Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R. Kelley. Effective
gene expression prediction from sequence by integrating long-range interactions. Nature Meth-
ods, 18(10):116-1223, October 2021. ISSN 1548-7105. doi: 10. 1001008//1152--01-001222
URL https: / www. nature. com/ httaaaatccessssssa//// 000-9922222-11111 01252-2
Number: 10
Publisher: Nature Publishing Group.
Gonzalo Benegas, Sanjit Singh Batra, and Yun S. Song. DNA language models are powerful zero-
shot predictors of non-coding variant effects. preprint, Bioinformatics, August 2022. URLhttp
h//iiorxiw.orr//Looko/////o 1tt.:/101111122222110 08 22. 504706
John Besemer, Alexandre Lomsadze, and Mark Borodovsky. GeneMarkS: self-training method for prediction of gene starts microbial genomes. Implications for finding sequence motifs in regulatory regions. Nucleic Acids Research, 29(12):2607-2618, June 2001. ISSN 0305-1048.
URL htttps://ww. www..nncii.ooo nlm. nih. http//mc/aaatiiicee///SSSSS
Mark Borodovsky and Alex Lomsadze. Eukaryotic Gene Prediction Using GeneMark. hmmm- and GeneMark-ES. Current protocols in bioinformatics / editoral board, Andreas D. Baxe- vanis ... [er al.), CHAPTER:Unit 4.610, September 2011. ISSN 1934-3396. doi: 10.1002/ 2511250953.b04006s.5 PMC32043787
URL https: //wp:::/www.. nebi nlm. nih, wtwphh/ooww//aa....o./
Fiona Cunningham, James E Allen, Jamie Allen, Jorge Alvarez-Jarreta, M Ridwan Amode, Irina M Orimoooye, Andrey G Azov, If Barnes, Ruth Bennett, Andrew Berry, Jyothish Bhai, Alexandra Bignell, Konstantinos Billis, Sanjay Boddu, Lucy Brooks, Mehrnaz Charkhchi, Carla Cummins, Luca Da Rin Fioretto, Claire Davidson, Kamalkumar Dodiya, Sarah Donaldson, Bilal El Houdaigui, Tamara El Naboulsi, Reham Fatima, Carlos Garcia Giron, Thiago Genez, Jose Gonzalez Martinez, Cristina Huujaaro-Clarke Arthur Gymer. Matthew Hardy, Zoe Hollis, Thibaut Hourlier, Toby Hunt, Thomas Juettemann, Vinay Kaikala, Mike Kay, Ilias Lavidas, Tuan Le, Diana Lemos, José Carlos Marugán, Shamika Mohanan, Aleena Mushtaq, Mare Naven, Denye N Ogeh, Anne Parker, Andrew Parton, Malcolm Perry, Ivana Pilizota, Irina Prosovetskaia, Manoj Pandian Sakthivel, Ahamed Imran Abdul Salam, Bianca M Schmitt, Helen Schuilenburg. Dan Sheppard, José G Pérez-Silva, William Stark, Emily Steed, Kyösti Sutinen, Ranjit Sukumaran, Dulika Sumathipala, Marie-Marth Suner, Michal Szpak, Anja Thormann, Francesca Floriana Tricomi, David Lrbina-Gómez, Andres Veidenberg, Thomas A Walsh, Bran- don Walts, Natalie Willhoft, Andrea Winterbottom, Elizabeth Wass, Marc Chakiachvili, Bethany Flint, Adam Frankish, Stefano Giorgetti, Leanne Haggerty, Sarah E Hunt, Garth R Issley, Jane Loveland, Fergal J Martin, Benjamin Moore, Jonathan M Mudge, Matthieu Muffato, Emily Perry, Magali Ruffier, John Tate, David Thybert, Stephen J Trevanion, Sarah Dyer, Peter W Harrison, Kevin L Howe, Andrew D Yates, Daniel R Zerbino, and Paul Flicek. Ensembl 2022. Nucleic Acids Research, 50(D1):D988 D995, January 2022. ISSN 0305- 1093 par
Armean, Olanrewaju
doi: 10. 10.109//nar/gkab 1049 1049.
URL httptttts///ooooooot http://..00000/////a//aa00001 qkab1049
Nicki Skafte Detlefsen, Søren Hauberg, and Wouter Boomsma. Leaninng meaningful represen- tations of protein sequences. Nature Communications, 13(1):1914, April 2022. ISSN 2041-
1723. doi: 10. 11701//////41-00222222
htLhttp::: / www..w.. nature. com/ arrt cles/
841467-022-22 2222222224444-2
Number: Publisher: Nature Publishing Group.
Adam Frankish, Mark Diekhans, Irwin Jungreis, Julien Lagarde, Jane Loveland, Jonathan M Mudge, Cristina Sisu, James C. Wright, Joel Armstrong, If Barnes, Andrew Berry, Alexandra Bignell, Carles Boix, Silvia Carbonell Sala, Fiona Cunningham, Tomás Di Domenico, Sarah Donaldson, Ian T Fiddes, Carlos García Girón, Jose Manuel Gonzalez, Tiago Grego, Matthew Hardy, Thibaut Hourlier, Kevin L Howe, Toby Hunt, Osagie G Izuogu, Rory Johnson, Fergal Martin, Laura Martínez, Shamika Mohanan, Paul Muir, Fabio C P Navarro, Anne Parker, Baikang Pei, Fermando Pozo, Ferriol Calvet Riera, Magali Ruffier. Bianca M Schmitt, Eloise Stapleton, Marie-Marthe Suner, Irina Sycheva, Barbara Us7czynska-Ratajena Maxim Y Wolf, Jinuri Xu,
10

---

Under review as a conference paper at ICLR 2023
Yucheng T Yang, Andrew Yates, Daniel Zerbino, Yan Zhang. Jyoti S Choudhary, Mark Ger- stein, Roderic Guigó, Tim J Hubbard, Manolis Kellis, Benedict Paten, Michael L Tress, and Paul Flicek. GENCODE 2021. Nucleic Acids Research, (9(11):9916-D923, January 2021. ISSN 0305-1048.
doi: 10. 20.103//ar/gkaaa0005 URL https://doi.ooo orgp:////.oorr//////// 10t/3//00.00////// 0933/nar/
gkaa1087
Katharina J. Hoff, Simone Lange, Alexandre Lomsadze, Mark Borodovsky, and Mario Stanke. BRAKERI: Unsupervised RNA-Seq- Based Genome Annotation with GeneMark-ET and AU-
GUSTUS. Bioinformaticx. 32(5):767-769, March 2016. ISSN 1367-4803. doi: 10.1093
bioinformatics//tv66... URL https:: http///wwwc/ccccce nlm. nih. PKC60781677
Carson Holt and Mark Yandell. MAKER2: an annotation pipeline and genome-database manage- ment tool for second generraton genome projects. BMC Bioinformatics, 12(1):491, December 2011. ISSN 1471-2105. doi: 00.1186/471-2210--12-44 URL https: //ooi. org/1C.
Kevin L. Howe, Bruce J. Bolt, Myriam Shafie, Paul Kersey, and Matthew Berriman. Worm- Base ParaSite a comprehensive resource for helminth genomics. Molecular and Bio- chemical Parasitology, 215:2-10, July 2017. ISSN 0166-6851. doi: 10.1016/j.molbiopara. 2016.11.005. URL https:/ /tt::://wwww.o. wttp///wwiiincnirreectc.c.c com ccm./science/articeeee..
S01 6016668511630168
Chloe Hsu, Hunter Nisonoff, Clara Fannjiang, and Jennifer Listgarten. Learning protein fitness models from evolutionary and assay-labeled data. Nature Biotechnology, 40(7): July 2022. ISSN 1546-1696. doi: 100003//41117772--01111111
URL https: www nature. Number: 7 Publisher: Nature Publishing Group.
Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana V Davuluri. DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA- DNA-language in genome. Bioinformat- ics, 12:2)12211222222 August 2021, ISSN 1367-4803, 1460-2059, doi: 10. 10.1009//hoonnffrmmaiick btab083. URL https: http///caaemicccm c oup. com 2112/6128680
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J. Ballard, Andrew Cowie, Bernardino Romera- Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Se- bastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Push- meet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with AlphaFold. Nature, 596(7873):583 589, August 2021. ISSN 1476-4687 doi: 10. 1038//418816666 021660022-38899 URL/attps: www http:///www..ttt..eeeat nwwura............. www.:re.cco...ooooo art httpe//aeaaaaaacceeeeaaa-------. Publisher: Nature Publishing Group.
Number: 7873
Ian Korf. Gene finding in novel genomes.
BMC Bioinformatics, 5(1):59, May 2004. URL http::://oll.rr//100.
ISSN 1471-2105.
doi: 10.1186/1471-2155-5
1471 1471-2105-5-59
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. In Proceedings of the Eigh- teenth International Conference Machine Learning, ICML '01, pp. 282-289, San Francisco, CA, USA, June 2001. Morgan Kaufmann Publishers Inc. ISBN 978-1-55880-778-1
Alexander V. Lukashin and Mark Borodovsky. GennMMrk.hmmm New solutions for gene finding. Nucleic Acids Research, 107-11111 February 1998, ISSN 0305-1048. doi: 10.1093/aar 26.4. 100. URL httL:tts:::/// doi org 1ww.:1.......//111 1107
W. H. Majoros, M. Pertea, and S. L. Salzberg. TigrScan and GlimmerHMM:: two open source ab initio eukaryotic gene-finders. Bioinformatics, 20(16))2878-293 2879, Novem- ber 2004. ISSN 1367-4803, 1460-2059.
doi: 10.009//binnoomattt//hhhhhh.1. URL
oup htt.a///ee.....ooooo//oo/////aaa httpp///.eattc....ooooooooooaaaa
T093/bloinformaticooo

---

Under review as a conference paper at ICLR 2023
Louis- Louis-Philiipe Morency, Ariadna Quattoni, and Trevor Darrell. Latent-Dynamic Discriminative Models for Continuous Gesture Recognition. In 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8. Minneapolis, MN. USA. June 2007. IEEE. ISBN 978-1-4244-
1179-5 978-1-4244-1180-41 188--1 doi: 10.1 10.1199CCVPR22207.3329999 URL http:://eeexpp:
feee..rrg/dccummnt//222000
Mihaela Pertea, Xiaoying Lin, and Steven L. Salzberg. GeneSplicer: a new computational method for splice site prediction. Nucleic Acids Research, 2965):1185-1190 March 2001. ISSN 0305- 1048. http://wwwwaaa.......oaaaa.oo/ta
Roshan Rao, Joshua Meier, Tom Sercu, Sergey Ovchinnikov, and Alexander Rives. Transformer pro- tein language models are unsupervised structure learners, preprint, Synthetic Biology, December 2020. httt::////oooxii org Lookup/ /oi/ 10.00000000 1101 2020 12. 15. 1...115.42256.. 422761
Roshan M. Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. MSA Transformer. In Proceedings of the 38th International Conference on Machine Learning, pp. 8844-8856. PMLR, July 2021. URL httplttts//ppocceee/teeee ISSN: 2640-3498.
mlr.
gress/v139/Es raollaaaaalaa.ommm html
lan Reid, Nicholas O'Toole, Omar Zabaneh, Reza Nourzadeh, Mahmoud Dahdouli, Mostafa Ab- dellateef, Paul MK Gordon, Jung Soh, Gregory Butler, Christoph W. Sensen, and Adrian Tsang.
SnowyOwl: accurate prediction fungal genes by using RNA-Seq and homology information to select among ab initio models. BMC Bioinformatics, (5(1):229, July 2014. ISSN 1471-2105, doi: 10.1186/1471-2105 229.
Nicolas Scalzitti, Anne Geannin-Girardee Pierre Collet, Olivier Poch, and Julie D. Thompson. A benchmark study of initio gene prediction methods in diverse eukaryotic organisms. BMC Genomics, 21(1); April 2020. ISSN 1471-2164. doi: 10.
URL
lattps://doi. org.org////// 111...11111//111411 000-084440200077----000
Jung-Eun Shin, Adam J. Riesselman, Aaron W. Kollasch, Conor McMahon, Elana Simon, Chris Sander, Aashish Manglik, Andrew C. Kruse, and Debora S. Marks. Protein design and variant prediction using autoregressive generative models. Nature Communications, 12(1):2403, April 2021. ISSN 2041-1723, doi: 10110036644467--11222773333
httht:::: www. nature. Number: Publisher: Nature Publishing Group.
com/ http:/artceesss4110 h4tp://44445-//
Mario Stanke. Augustus: datasets, January 2004.
www.biiiiiiii.iiiiiiiiieeeeeeeee fww. de lee/auuusuus datasets/ Publisher: Institute for Mathematics and Computer Science, University of Greifswald.
Mario Stanke and Stephan Waack. Gene prediction with hidden Markov model and a new intron submodel. Bioinformatics, (9(uppl 2) September 2003. ISSN 1367-4803. doi: 10. wtwpooo...rr..... 1093/
(093//ioinformatiis//ggggo URL
btç1080
Mario Stanke, Oliver Keller, Irfan Gunduz, Alec Hayes. Stephan Waack, and Burkhard Morgenstern. AUGUSTUS: ab initio prediction of alternative transcripts. Nucleic Acids Research, 34(suppl 2): July 2006. ISSN 0305- doi: 10. 10.10933nar/gk12000 URL httpp:://doi.
Vardges Ter-Hovhannisyan, Alexandre Lomsadze, Yury O. Chernoff, and Mark Borodovsky. Gene prediction in novel fungal genomes using an ab initio algorithm with unsupervised training.
Genome Research, 1811):19999111 1990, December 2008. ISSN 1088 1088-9051, 1549-5469. doi: 10.1101/gr.011612.106
URL https: /tt:://genno eshlp. org/ wwtttot/tttttcccecco /tt/eenn///////119/ 18/12/1979 Cold Spring Harbor Laboratory Press Distributor: Cold Spring Harbor Laboratory Press Institution: Cold Spring Harbor Laboratory Press Label: Cold Spring Harbor Laboratory Press Publisher: Cold Spring Harbor Lab.
Company:
Jesse Vig, Ali Madani, Lav R. Varshney, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. BERTology Meets Biology: Interpreting Attention in Protein Language Models, March 2021. wtt.ttpp: wwwperwwwv.oroggreeeeeeeee wwt.://g/////oaooooooooooo 15222 arXiv:2006. 11000006.12222 15222 [cs, q-bio].
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 LABEL SETS
Generally five types labels are used:
E: Exon/CDS (i.e. the coding part of a sequence)
I Intron
D: Donor site (one the two splice sites)
A Acceptor site (one of the two splice sites)
NC: : Non-coding or intergenic regions of the genome.
Subscript F and R (e.g- Er and Er) used to denote the forward and reverse direction of the gene. Numbered subscripts are used to denote codon structure (i.e. Fi,r the first exon nucleotide in codon in the forward direction).
Simple-DA:
The simplest set of states used.
Y E.D.I,A,CC
Simple-direction-DA:
Expands the set of states to include information about the direction/strand of the gene.
EF, Ap, ER, ER, IR. IR AR.
Codon-direction-DA:
Further expands the set states to include not only direction but also codon structure.
= [ EL,, E2 Ea DI,, D2,r, Da,F, I II Ja ALE, A2, As,s
Ei Eg Es DI.R. D2, Da, I, 12 la A A2R AaR.
A.2 GRAPHS
Graphs are represented as adjacency matrices and are named according the label sets described in Appendix A.T A grey cell signifies a forbidden transition while the weights the green cells are learned by model.
13

---

Under review as a conference paper at ICLR 2023
DF
4
Ar
ER
DR
ln
AR
NC
Er
D=
IF
A
Ex
Da
Is
AR
NC
Figure 4: Graph of allowed transitions for simple-direction DA labels
ELr
2,
3
D..F A.
D
D,.F
2
3
LF
2,
F
E
D R
2 R
D
R
2.
3.R
A
2.R
A
NC
E1.F
E
E3,F
D1.F
D2.F
Da.F
h.
2.
l.,
A1.F
A2.r
As.F
E1.R
E
E3,R
D1.8
D2.8
D3.8
11.8
I3
AI,
Az.
A3,8
NC
Figure 5: Graph of allowed transitions for codon-direction DA labels
14

---

Under review as a conference paper at ICLR 2023
A.3 EMISSIONS
A grey cell signifies forbidden emission while the weights of the green cells are set to 1 (i.e. an allowed emissions).
A
NC
Ei.
E2.r
Ea.F
Di.
D2.
Da
l2.
A1.F
A2,
A3.F
E1,R
E2,R
E3,R
D2.R
Dz.R
Da.R R
11,
I,,
13,8
AIa
Az,
A3,a
NC
Figure 6: Allowed emissions from codon-direction labels to simple- DA labels
Er
IF DF
AF
ER
DR
lR
AR
NC
E1.F
E
E3,F
Di.F
D2.F
D3.F
l
1,,F
l3,
A1,F
A2.F
A3,F-
E1,8
E
E3,8*
D1.8
D
D 3.
1
13.8
A1,
Az, R
A3
NC
Figure 7: Allowed emissions from codon-dirrection-DA labels to simple-direction- DA labels
15

---

Under review as a conference paper at ICLR 2023
A.4 GENEDECODER FEATURE MODEL ARCHITECHTURE
Feature Model
Where X is the number of channels in the input, and [HH] is the size of the set of hidden states. In most cases X GA,C,G,T,NF
DilatedCNN(
Layer Convld(ín channels = |X| out channels = 50, kernel size (9,), stride
I, dilation ReLU
Layer Conv Id(ín channels 50, out channels = 50, kernel size (9,), stride
1, dilation 2), ReLU
Layer Conv Idin channels 50, out channels = 50, kernel size 9,), stride
1, dilation 4), ReLU
Layer Conv Id(in channels 50, channels 50, kernel size 9,), stride 1, dilation = 8), ReLU
Layer 5 Convid(in channels 50, out channels 50, kernel size 9.), stride 1, dilation -1)), ReLU
Layer 6: Conv channels 50, out channels = 500, kernel size 9,), stride dilation 32), ReLU
LSTM(
LSTM layer: lstm(50, hidden layers 100, bidirectional True)
Output layer: linear(input size 200, output size [[il], bias True,
dropout(O.2)). ReLU
A.5
SUPPLEMENTARY FIGURES
1.0
0.8
True Postitive
0.6
w/ CRF
w/o CRF
0.4
0.2
0.0
Ep
D
18
Ap
Ex Label
DR
1R
Aa
NC
Figure 8: Comparison of true positive rate per label for LSTM-CRE, LSTM and Linear-GPN-CE model respectively.
16

---

Under review as a conference paper at ICLR 2023
6.005
0.67
6,052
0,0013
0.15
0.062
0.07
0.5
0.22
0.0031
0.0227
0,003
3,004
6.94
6.079
0.12
0.20.5
0,003
7.30
0.73
0.00012
0.002
0.0033
9.044
6.09%
0.71
0.14
Predicted Label
Figure 9: Confusion matrix for inference of direction on test set performance of model trained on simple-DA label set
1.0
0.8
True Postitive
0.6
Augustus GeneDecader
0.4
0.2
0.0
DF
1F
A
ER
D
fa
Aa
NC
Label
Figure 10:
True positive rate per label for training and testing on the original Augustus Human
datasets
17


Reviews:
[{"review_id": "eSust7vS9Hi", "reviewer": "Reviewer_rvN5", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors tackle the problem of annotating genes in newly-sequenced genomes. They develop a model called GeneDecoder which uses a combination of a CRF, LSTM and dilated convolutional layers. The authors show that the model relearns several properties of genes, including directionality and length distribution. The model achieves similar predictive performance to existing method Augustus. ", "summary_of_the_review": "See above.", "strength_and_weaknesses": "Overall, the problem of annotating genes is important, the method is reasonable and the manuscript is understandable. The methodological novelty is low, as the method is a combination of standard CRF, LSTM and dilated CNN models. The experimental setup is flawed and the results are poor. \n\nI couldn't tell what the model used by GeneDecoder is. It involves a neural network, whose architecture is described in A.4. However it also uses a Latent CRF. I think the NN outputs a representation which forms the input to the CRF? This isn't actually stated.\n\nBenchmarking gene prediction tools is challenging because many of the annotations within gene databases (e.g. GENCODE) are derived from computational predictors. Thus it can be hard to tell whether a predictor is good at discovering real biology or simply recapitulating the errors made by previous predictors. There exists a benchmark for this task, G3PO (Scalzitti et al 2020), cited by the authors.  Unfortunately, the authors did not use this benchmark and instead used an ad-hoc strategy with many issues (see below).\n\nThe authors compared to pretrained versions of existing models, so differences in performance could result from differing training sets. When the authors compared against Augustus using a training set similar to Augustus's, results were similar or worse.\n\nThe authors trained and tested on genes of the same species. This not a proper simulation of the target application, in which a new species is sequenced and its genome must be annotated from scratch. \n\nIn splitting genes into train and test sets, the authors ensure that all isoforms of the same gene are placed in the same set. However, since many genes overlap, the same sequences likely appear in both train and test sets. This pitfall is described in detail in the following paper. A better strategy would be to split train and test by chromosome (or by species; see previous note). \nhttps://pubmed.ncbi.nlm.nih.gov/34837041/", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "eDuE4oJ1pm", "reviewer": "Reviewer_ggXe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors have presented a novel method of gene finding - where the task is to label spans of text the functional labels.  The presented method uses a neural algorithm that leverages conditional random fields (CRF) over the state of the art HMM methods. They provide a way to fuse pre-trained representations of genomic sequences with their architecture for this task. The novelty is in terms of architecture, incorporating domain information in the form of allowed direction of state changes and experimentations across organisms.", "summary_of_the_review": "The authors have proposed a new neural architecture, with elegant ways of including domain knowledge, for the task of gene finding. This could potentially be a new fine tuning task for gene embedding research and help practitioners in unearthing genes among unlabelled data. The experiments have been conducted well and the problem is relevant and exciting.\n\n\nAt this point, I would recommend a weak reject because I am not convinced this neural architecture, though it incorporates domain knowledge, is significantly better than the state of the art methods in any setting, (for example, Augustus also seems to perform well in cross organism settings). \n\n===============================\nSeveral of the comments have been addressed. So increasing the score.", "strength_and_weaknesses": "Strengths\n\nState of the art gene finding\n\nPioneer attempt to combine pre-trained embeddings with gene finding\n\nIncorporation of domain knowledge to improve performance\n\nExhaustive experimentation\n\nCross organism experiment verification\n\n\nWeaknesses\n\nNo clarity or intuition on why CNN + LSTM does better than pre-trained models\n\nWhile an empirical ablation has been done, it seems non-intuitive that the pre-trained embeddings do not help much, considering they are tested on similar downstream non-coding tasks. \n\nNovelty needs to be explained for architecture \n\nFor example, how does the combination of embeddings work with neural HMM based formulation work as compared to the state of the art HMM methods?\n\nThe efficiency study for these methods is missing. What do we gain by using this architecture (which takes time to compute) against others? This is relevant especially because the values of Augustus’ baseline performance is quite close.\n\nNo statistical significance values reported", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XK8yl109O-5", "reviewer": "Reviewer_6LHj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This manuscript describes a straightforward application of standard deep learning techniques to the important problem of gene finding in eukaryotic genomes.  It is actually quite surprising that something like this has not been done before, since this is a central problem in genomics that is still not well solved despite decades of work in the area.  State-of-the-art methods have used HMMs for more than two decades. This paper provides convincing evidence that a deep neural network approach to this problem has promise.", "summary_of_the_review": "From a machine learning perspective, very little here is new or innovative, but that's OK when the task is simply to apply standard methods to solve an important problem. The writing and organization are clear, though the paper is poorly proofread, with many typos and grammatical errors.\n\n\n", "strength_and_weaknesses": "The paper does a very good job of providing a concise and accessible description of the gene finding problem for an ICLR audience.\n\nNot much novelty from a machine learning perspective.\n\nThe main missing piece in this paper, in my opinion, is an investigation into the types of errors that the system makes in comparison to other methods.  I want to know whether the system is missing some exons entirely, just getting the splice sites wrong, hallucinating new genes that don't exist, etc.\n\nThe other missing piece is an evaluation of how well the system works in practice on an actual genome, rather than on snippets of a genome selected around a gene.  As is, the paper is more like a proof of principle than a full solution.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Qi1IJYq0VWd", "reviewer": "Reviewer_DUdq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper develops a new algorithm for gene finding which is compatible with embeddings obtained from deep and transfer learning. ", "summary_of_the_review": "The paper can be substantially improved by 1) better motivating the need for a new gene finding algorithm, 2) discussing the proposed method more clearly without over-referring to the Appendix, which I found highly distracting, 3) including key remaining baselines, and 4) editing the grammatical errors which occasionally harms the clarity of the message. I do not thing in the current state the paper is ready for publication.", "strength_and_weaknesses": "The paper argues the prior models for gene prediction “lack the flexibility to incorporate deep learning representation learning techniques” as a potential drawback. This sets the motivation of the paper to develop a deep learning compatible gene prediction model. However I do not see why deep learning compatibility, per se, is a drawback of the prior works. Thus, in my opinion, the paper is poorly motivated from both the biological and methodological perspectives. The issue becomes even more problematic, as the paper demonstrates that the proposed deep learning-compatible method does not significantly improve the performance in any tangible aspect over the baseline considered. Unfortunately, all these are on top of the fact that the paper misses a key family of baseline algorithms for gene finding, i.e., GeneMark, GeneMark-S, etc. which is discussed in the paper but never compared with. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject