RNAformer: Axial-Attention For Homology-Aware RNA Secondary Structure Prediction

Authors: Jörg K.H. Franke, Frederic Runge, Ryan Koksal, Dominika Matus, Rolf Backofen, Frank Hutter
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: Structural Biology, Deep Learning, RNA Secondary Structure Prediction, Axial Attention

Abstract:
Predicting RNA secondary structure is essential for understanding RNA function and developing RNA-based therapeutics. Despite recent advances in deep learning for structural biology, its application to RNA secondary structure prediction remains contentious. A primary concern is the control of homology between training and test data. Moreover, deep learning approaches often incorporate complex multi-model systems, ensemble strategies, or require external data. Here, we present the RNAformer, a scalable axial-attention-based deep learning model designed to predict secondary structure directly from a single RNA sequence without additional requirements. We demonstrate the benefits of this lean architecture by learning an accurate biophysical RNA folding model using synthetic data. Trained on experimental data, our model overcomes previously reported caveats in deep learning approaches with a novel homology-aware data pipeline. The RNAformer achieves state-of-the-art performance on RNA secondary structure prediction, outperforming both traditional non-learning-based methods and existing deep learning approaches, while carefully considering sequence and structure similarities.

Decision: reject
Reviewer scores: 3, 6, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730540668333

thread_type:
official_review

<thread_object>
creator: reviewer_ued4

<content>
summary:
The paper introduces RNAformer, a deep learning model that predicts RNA secondary structures from sequences using axial-attention mechanisms. It achieves state-of-the-art performance and overcomes previous limitations by employing a homology-aware data pipeline, making it superior to traditional and existing deep learning methods.

strengths:
RNAformer stands out for its scalability, efficiency with varying sequence lengths, and a robust homology-aware approach to data curation. It simplifies the prediction process without additional data requirements and offers linear scaling capabilities, outperforming other models in accuracy and generalization.

weaknesses:
1. In the introduction, why is it stated that AlphaFold 3 struggles to capture RNA topologies and secondary structure features? The article should provide a more detailed explanation. Since structural information is more conserved, why can't we solely use structure information for training and test data splitting?

2. What is meant by "row- and column-wise embedding" on page 4? According to the mathematical expression, $L^{(0)}$ is a 3D matrix; why is it described as being in a 2D latent space? The mathematical representation and textual explanation in this section are quite confusing, making it difficult for readers to understand.

3. From lines 156, 174, 179, and line 188, it appears that the strategies used in this article are derived from existing literature. Therefore, this work seems to be incremental in the context of RNA structure prediction. Where does the unique innovation of this article lie?

4. From a methodological standpoint, how does the author emphasize the issue of homology in data splitting?

5. The code repository provided in the article is inaccessible. Could the authors please provide an updated link for the code and data, so that readers can review and reproduce the results?

6. The deduplication tools should be CD-HIT-EST and BLAST-N.

7. Add the evaluation results for the INF (Interaction Network Fidelity) metric. Reference for INF: Parisien et al. RNA (2009), 15:1875–1885.

8. Why was the RNAFold-generated structure chosen as the true label for the synthetic dataset? According to later test results, there are other energy minimization-based prediction methods (e.g., MXfold2) that perform better.

questions:
See "Weaknesses".
</content>
</thread_object>

</thread>
<thread>
creation date:
1730567390280

thread_type:
official_review

<thread_object>
creator: reviewer_akqc

<content>
summary:
This paper introduces RNAformer, a transformer-based model that utilizes axial attention for RNA secondary structure prediction, relying solely on a single RNA sequence as input without requiring multiple sequence alignment (MSA) data. It also introduces a data preprocessing pipeline for constructing homology-aware RNA datasets.

strengths:
1. The proposed methods work reasonably well.
2. The paper provides experimental validation of the model’s performance and scalability.
3. The code is available.

weaknesses:
1. I think the authors should clarify why they chose to input the single RNA sequence into the model while using axial attention. Typically, axial attention is designed to handle both row and column interactions—where row attention captures sequence-specific information and column attention enables the model to consider homologous relationships across multiple sequences, such as in multiple sequence alignments (MSAs).  In RNAformer, however, axial attention is applied solely to the single sequence, instead of the MSA input.  Understanding the motivation for this choice is essential, as it diverges from the usual application of axial attention in models that integrate homologous data.  The authors could explain how this approach benefits RNA secondary structure prediction and whether it offers advantages in terms of computational efficiency, interpretability, or applicability to cases where MSA data is unavailable.

2. The paper lacks unique contributions to RNA secondary structure prediction, particularly regarding its axial attention module. What is the specific difference between RNAformer’s axial attention module and those in Evoformer in [1] or [2]?

3. The methodology for preparing homology-aware RNA datasets needs further clarification. In lines 269–273, how were the "50 random PDB samples" and the "TS-Hard" set selected? How were the data split across the four test sets and three validation sets? What is meant by “we gather TS1, TS2, and TS3 into the test set, TS-PDB, containing 125 samples”? Does “Rfam TS” in Table 1 refer to a combination of TS1, TS2, and TS3? Also, do the "FT-Homolog" and "FT-Non-Homolog" sets overlap with the Rfam training, validation, and test sets? In line 291, what does “2) only for TS-PDB” mean?

[1] Abramson, Josh, et al. "Accurate structure prediction of biomolecular interactions with AlphaFold 3." Nature (2024): 1-3.

[2] Rao, Roshan M., et al. "MSA transformer." International Conference on Machine Learning. PMLR, 2021.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730659949063

thread_type:
official_review

<thread_object>
creator: reviewer_bzzh

<content>
summary:
I will cut directly into strength/weakness/questions.

strengths:
- The proposed data preparation pipeline, which emphasizes a homology-aware train/validation/test split, addresses a critical need in this research community. Given the limited availability of RNA structural data, large deep learning models trained on random splits are at heightened risk of overfitting. 
- I like this idea about validating deep learning models through alignment to biophysical techniques such as RNAfold. 
- These experimental results are useful. It is good to see in table 3 that careful data preparation has brought deep learning models to a more realistic and often comparable level of performance to thermodynamic models like RNAfold and Linearfold.

weaknesses:
- The machine learning component of this paper is a bit limited in terms of novelty. The axial attention (mainly comprising of row-wise and column-wise self attention) is indeed quite similar to those that have been used in alphafold2 and MSA transformer. A lot of techniques have also been adopted from other works, for example the pretraining-finetuning training process which was initially used by spotrna.
- Between certain non-coding RNA families, there can be substantial sequence/secondary structure similarity, so the the synthetic dataset generated in section 3.1 may be flawed in theory. I think this is something worth checking out.

questions:
- The covariance models e-value cutoff at 0.1, how do you come to this value (empirically or theoretically) and is it effective?
- Data splitting subsection on page 6 is a bit confusing. So essentially, you created two types of dataset, one effectively removing homology information (TS-Hard and FT-Non-Homolog, for test and finetuning), ther other one only accounting for sequence similarity (TS-PDB and FT-homology) so that the comparison to prior baselines is fair. 
- Why is e2efold missing from Table 3? It could probably serve as a valuable cautionary example.
- In section 4.2.1, although the sectio is called "Impact of homology on prediction quality", I don't really see how the content of that section alone is relevant for this topic, since it was mainly about comparing to alphafold 3.
- The linear scaling behaviour shown in section 4.2 may benefit from some comparisons to some other deep learning baselines.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730673557052

thread_type:
official_review

<thread_object>
creator: reviewer_ulq7

<content>
summary:
This paper makes a valuable contribution to RNA secondary structure prediction by introducing RNAformer, an innovative model that leverages axial attention alongside a rigorously curated dataset and evaluation pipeline. The authors carefully address homology concerns, a major problem in previous machine learning models, which significantly enhances the reliability of their findings. The experiments are thorough and demonstrate substantial improvements over existing methods, particularly in generalizing to unseen RNA families. Overall, the combination of a well-designed model and rigorous evaluation positions this work as a strong candidate for acceptance.

strengths:
* **Rigorous Dataset Construction**: A major strength of this paper is its rigorous approach to dataset construction, addressing an important issue in computational biology that previous models have often overlooked—the need to prevent homologous (highly similar) sequences across training, validation, and test sets to ensure robust evaluations. After assembling the initial dataset, the authors implement a comprehensive three-step data filtering process: (1) they use CD-Hit to remove sequences with over 80% similarity, reducing redundancy; (2) a BLAST search is conducted with a high e-value threshold to further exclude overlaps between training/validation and test sequences; and (3) they build covariance models using tools like LocaRNA-P and Infernal to filter any remaining sequences with structural or sequence similarities, applying a stringent e-value cutoff. This multi-layered filtering pipeline effectively minimizes overlap and ensures that the train/validation and test sets have minimal similarity. Additionally, by defining distinct FT-Homolog and FT-Non-Homolog test sets, the paper provides a structured approach to evaluate the model’s performance across varying levels of similarity, showcasing RNAformer’s generalization capabilities. 

* **Well-Designed Experiments with Superior Performance over Baselines**: The experimental design of this paper is well-conceived, with RNAformer consistently outperforming baseline methods in a rigorous experimental setting. The authors conduct two complementary experiments to highlight the model's capabilities. First, they evaluate on synthetic data, comparing RNAformer to the current state-of-the-art RNA 3D structure prediction model, AlphaFold 3 (AF3), demonstrating RNAformer’s superior performance. Second, they evaluate on experimental data with strict homology control, providing a thorough assessment of generalization. Even though baseline methods often use less stringent data splits or rely on additional information, RNAformer achieves better results. Together, these two experiments effectively underscore RNAformer’s superiority. 

* **Model Simplicity**: Another commendable aspect of this paper is the simplicity of the RNAformer model. By avoiding reliance on multiple sequence alignments (MSAs) and additional complex features, RNAformer achieves high performance with a streamlined architecture. This simplicity reduces computational overhead and enhances the model’s generalizability, making RNAformer a practical and effective alternative to more complex methods in RNA secondary structure prediction.

weaknesses:
* **Lack of Explanation for Column and Row Axial Attention**: One limitation of the paper is the insufficient discussion or justification for using column and row axial attention in RNAformer. This attention mechanism is applied in each RNAformer block to process the 2D latent representation of the RNA sequence. In certain models, such as MSA Transformer (https://www.biorxiv.org/content/10.1101/2021.02.12.430858v3), row and column attention is intuitive due to the distinct structural roles of rows and columns in the input data. However, in RNAformer, there is no inherent difference between rows and columns, as the input is a single RNA sequence without a natural row-column structure. Additionally, the output contact map is a symmetric $\( L \times L \)$ matrix, where rows and columns are effectively equivalent. The authors may want to clarify why this architecture was chosen and how it specifically benefits the model in this context.

* **Insufficient Discussion of Sparse Adjacency Loss**: While the paper introduces a specially designed sparse adjacency loss, the discussion surrounding its design and impact on model performance is limited. Sparse adjacency loss is an important component, especially given the sparsity of RNA contact maps. Understanding how this loss function influences the model's predictions could provide valuable insights. The authors could expand on why this specific loss was chosen, its advantages over standard loss functions, and its contribution to handling sparse data effectively.

* **Need for Additional Baselines on Synthetic Data**: When comparing results on synthetic data, the paper only uses one baseline method, AlphaFold 3 (AF3), a 3D structure prediction model. In this comparison, the authors first predict the tertiary structure with AF3 and then extract the secondary structure, which may not provide an entirely adequate baseline. Although AF3 is considered state-of-the-art for 3D structure prediction, this does not necessarily imply it is optimal for secondary structure prediction, as the paper only claims AF3’s superiority in 3D tasks. Including additional commonly used secondary structure prediction models as baselines would strengthen the evaluation and provide a more comprehensive comparison.

questions:
I have no question.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731205044419

thread_type:
official_review

<thread_object>
creator: reviewer_qpbb

<content>
summary:
The paper introduces RNAformer, a deep learning model designed for predicting RNA secondary structures from a single sequence without additional requirements like multiple sequence alignments or thermodynamic parameters. The model uses an axial-attention mechanism that is independent of input size, making it suitable for RNA sequences of varying lengths. RNAformer is trained on a homology-aware data pipeline that addresses sequence and structure similarities to ensure a clean split between training and test data.

strengths:
RNAformer is a single deep learning model that relies solely on RNA sequence input and does not require additional information like multiple sequence alignments (MSAs), embeddings, or ensemble techniques. Also, the Axial-Attention makes the model to be efficient to the length of RNA sequence.

weaknesses:
The novelty is limited, where the key technique Axial-Attention has already been proposed in 2019. The authors do not appear to have designed the algorithm specifically for RNA data. Rigorous data splitting is too common sense to be a significant contribution to be published at an AI conference. Compared to baselines, it seems to be have weak constraints on RNA secondary structure, such as ES and TP post processing.

questions:
Q1: How can your algorithm ensure the RNA secondary structure constraints, such as symmetry, no sharp loop within three bases, and so on. Refer to [1].

Q2: Could you compare your algorithm to recent studies[1,2,3]?

Q3: Regarding the Axial-Attention, what you have done to make it spcified to your problem? What is the difference when applying it to image data and RNA data?

Q4: What is the distribution of sequence length? What are the average and maximum lengths?

[1] Tan, Cheng, et al. "Deciphering RNA Secondary Structure Prediction: A Probabilistic K-Rook Matching Perspective." Forty-first International Conference on Machine Learning.

[2] Chen, Xinshi, et al. "RNA secondary structure prediction by learning unrolled algorithms." arXiv preprint arXiv:2002.05810 (2020).

[3] Jung, Andrew J., et al. "RTfold: RNA secondary structure prediction using deep learning with domain inductive bias." The 2022 ICML Workshop on Computational Biology. Baltimore, Maryland, USA. 2022.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733310479982

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
References for final general response

comment:
[1] Multiple sequence alignment-based RNA language model and its application to structural inference. Nucleic Acids Research, 2024


[2] Busaranuvong, Palawat, et al. "Graph Convolutional Network for predicting secondary structure of RNA." Research Square (2024)


[3] Chen, Chun-Chi, and Yi-Ming Chan. "REDfold: accurate RNA secondary structure prediction using residual encoder-decoder network." BMC bioinformatics 24.1 (2023)


[4] Wei, Sen, and Shengjin Wang. "Structural stability-aware deep learning: advancing RNA secondary structure prediction." Fourth International Conference on Biomedicine and Bioinformatics Engineering (ICBBE 2024). Vol. 13252. SPIE, 2024.


[5] Yang, Enbin, et al. "GCNfold: A novel lightweight model with valid extractors for RNA secondary structure prediction." Computers in Biology and Medicine 164 (2023)


[6] Zhao, Qi, et al. "RNA independent fragment partition method based on deep learning for RNA secondary structure prediction." Scientific Reports 13.1 (2023)


[7] Khrisna, et. al. "The Use of Convolutional Neural Networks for RNA Protein Prediction." 2023 3rd International Conference on Intelligent Cybernetics Technology & Applications (ICICyTA). IEEE, 2023.

[8] Flamm, Christoph, et al. "Caveats to deep learning approaches to RNA secondary structure prediction." Frontiers in Bioinformatics 2 (2022): 835422.

[9] Szikszai, Marcell, et al. "Deep learning models for RNA secondary structure prediction (probably) do not generalize across families." Bioinformatics 38.16 (2022): 3892-3899.

[10] Qiu, Xiangyun. "Sequence similarity governs generalizability of de novo deep learning models for RNA secondary structure prediction." PLOS Computational Biology 19.4 (2023): e1011047.

[11] Rivas, Elena, Raymond Lang, and Sean R. Eddy. "A range of complex probabilistic models for RNA secondary structure prediction that includes the nearest-neighbor model and more." RNA 18.2 (2012): 193-212.

[12] Tan, C., Gao, Z., & Li, S. Z. (2022). RFold: RNA Secondary Structure Prediction with Decoupled Optimization. arXiv preprint arXiv:2212.14041.


[13] Chen, X., Li, Y., Umarov, R., Gao, X., & Song, L. RNA Secondary Structure Prediction By Learning Unrolled Algorithms. In International Conference on Learning Representations.


[14] Jung, A. J., Lee, L. J., Gao, A. J., & Frey, B. J. (2022). RTfold: RNA secondary structure prediction using deep learning with domain inductive bias. In The 2022 ICML Workshop on Computational Biology. Baltimore, Maryland, USA.


[15] Franke, J., Runge, F., & Hutter, F. (2022). Probabilistic transformer: Modelling ambiguities and distributions for rna folding and molecule design. Advances in Neural Information Processing Systems, 35, 26856-26873.


[16] Singh, J., Hanson, J., Paliwal, K., & Zhou, Y. (2019). RNA secondary structure prediction using an ensemble of two-dimensional deep neural networks and transfer learning. Nature communications, 10(1), 5407.


[17] Fu, L., Cao, Y., Wu, J., Peng, Q., Nie, Q., & Xie, X. (2022). UFold: fast and accurate RNA secondary structure prediction with deep learning. Nucleic acids research, 50(3), e14-e14.


[18] Chen, J., Hu, Z., Sun, S., Tan, Q., Wang, Y., Yu, Q., ... & Li, Y. (2022). Interpretable RNA foundation model from unannotated data for highly accurate RNA structure and function predictions. arXiv preprint arXiv:2204.00300.


[19] Abramson, J., Adler, J., Dunger, J., Evans, R., Green, T., Pritzel, A., ... & Jumper, J. M. (2024). Accurate structure prediction of biomolecular interactions with AlphaFold 3. Nature, 1-3.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733310519368

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Our valuable contribution and good reasons to accept

comment:
Dear Reviewers, Area Chairs, Senior Area Chairs, and Program Committee,

We respectfully disagree with some aspects of the current rating and would like to highlight key points why **we see our work as a valuable contribution and good reasons to accept** our RNA secondary structure prediction approach:

- We introduce the RNAformer based on axial-attention and argue that **attention is better suited than the commonly used CNNs [1-7] for processing sequences with varying length** such as RNA and show that such a **lean architecture built from existing components achieves SOTA results** without the need for ensamples, pre/post processing or additional MSA.
- We **address the caveats raised by the RNA community** [8-11] by introducing a novel data pre-processing pipeline that allows us to split training and test data while accounting for homologies. This homology-aware data split and evaluation is not common and many approaches still evaluate on homologous data [3, 5, 6, 12-19].
- We show the importance of removing homologies between training and test data in an experiment where we **outperform AlphaFold 3 in terms of secondary structure accuracy** (more precisely, we show that AlphaFold 3 performs worse on samples where there is no MSA available while RNAformer’s performance is constant, consistently outperforming AlphaFold 3).
- Even so, we consider homologies, RNAformer achieves SOTA results and **outperforms models that use homologous data or additional information such as MSA on experimentally derived PDB** data.

We thank the reviewers for their feedback and discussion. We responded to all reviews, answered all questions, and updated the manuscript according to the feedback from the reviewers. We summarize the reviewer interaction as follows: 

- Reviewer qpbb (rating 3, confidence 3) didn’t acknowledge the SOTA results, emphasized the **lean, efficient, and single sequence-based design** but criticized the lack of a novel architecture specific to RNA secondary structure, and claimed the RNA data split was common sense. We explained that our point is that current deep learning architectures are already sufficiently well suited to achieve SOTA results and explained the homology issues in recent related works. We answered all questions but we got **zero responses**. 
- Reviewer ULQ7 (rating 6, confidence 4) emphasizes the **“model simplicity”, “well-designed experiments with superior performance”, and the “rigorous dataset construction”**. The reviewer named weaknesses that we addressed satisfactorily according to the reviewer's response.
- Reviewer BzZh (rating 5, confidence 4) didn’t write a summary but **named as the strength that the data pipeline addresses critical aspects of the research community, the biophysical experiment and that the “experimental results are useful”**. The reviewer criticized the lack of a novel architecture specific to RNA secondary structure, and the similarity in the synthetic dataset. Again, we explained that our point is that current deep learning architectures are already good enough to achieve SOTA results, addressed the data concern, and answered the questions but we got **zero responses**. 
- Reviewer aKqc  (rating 3→6, confidence 3) was most active in the discussion and gave good feedback, thanks again. The reviewer emphasizes **“model’s performance and scalability”** and named as weakness the motivation for the axial attention, lack of specific architecture for RNA secondary structure, and that the methodology of the datasets needs clarification. We addressed all these points and the reviewer increased the rating from recject to minor accept. 
- Reviewer UEd4  (rating 5, confidence 3) emphasized the **scalability, efficiency, and homology-aware approach and that our model “outperforming other models in accuracy and generalization”**. The reviewer criticized mainly the manuscript by asking for better explanation and naming issues. The reviewer sees a lack of novelty on the algorithmic level, but the reviewer also admits that they are “not particularly knowledgeable about the application domain” and that they will follow other reviewers’ decisions.  

In total, the reviewers highlighted RNAformer's scalability, efficient design, the homology-aware data processing, and ability to achieve strong performance using sequence data alone as its major strengths. **We addressed all concerns raised by the reviewers**, answered all questions, and updated the manuscript and title according to the feedback from the reviewers. 

@AC: Due to the lack of the two remaining critical reviewers' responses, we respectfully ask you to **make your own assessment** of our work for the meta-review, **rather than relying on the current scores**. Again, we believe **our work makes a valuable contribution** to the field.

Thank you for your consideration and your service to ICLR.

Best regards, The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes RNAformer, a pretty lean approach for RNA secondary structure prediction, relying primarily on a single sequence and axial-attention. The authors claim that using a homology-aware data pipeline and avoiding MSAs or complex ensembles, their model still achieves what they call SOTA performance on experimentally validated PDB structures.

The authors were active in responding to the reviews, going through all comments in detail. I personally highly appreciate this!
They clarified the reasoning behind their data splits, pointed out that previous methods often evaluated on homologous datasets, and explained how axial-attention can handle sequence lengths and capture pairwise structure information.

After the rebuttal phase, not all reviewers changed their view. One reviewer (aKqc) initially skeptical about the motivation for axial-attention, ended up slightly more positive after the authors provided extra experiments and data. Another reviewer (ULQ7) remained supportive, acknowledging the model’s simplicity and the careful homology-aware evaluation. On the other hand, a couple of reviewers (qpbb and UEd4) felt that the algorithmic novelty was too limited, even after multiple attempts by authors to highlight the training data pipeline and sparse adjacency loss. They never re-engaged positively. Personally I am also on the side of thinking the novelty can be better justified or improved. 

I think the paper’s main contributions—showing strong results without MSAs and carefully ensuring no test-train homology—is relevant and addresses key criticisms in RNA structure prediction research.