Beyond Sequence: Impact of Geometric Context for RNA Property Prediction
Authors: Junjie Xu
Artem Moskalev
Tommaso Mansi
Mangal Prakash
Rui Liao
Venue: ICLR 2025 Poster (2025)
Keywords: Geometric deep learning
Graph Neural Networks
GNNs
RNA property prediction
Datasets and Benchmarks
Primary area: datasets and benchmarks

Abstract: Accurate prediction of RNA properties, such as stability and interactions, is crucial for advancing our understanding of biological processes and developing RNA-based therapeutics. RNA structures can be represented as 1D sequences, 2D topological graphs, or 3D all-atom models, each offering different insights into its function. Existing works predominantly focus on 1D sequence-based models, which overlook the geometric context provided by 2D and 3D geometries. This study presents the first systematic evaluation of incorporating explicit 2D and 3D geometric information into RNA property prediction, considering not only performance but also real-world challenges such as limited data availability, partial labeling, sequencing noise, and computational efficiency. To this end, we introduce a newly curated set of RNA datasets with enhanced 2D and 3D structural annotations, providing a resource for model evaluation on RNA data. Our findings reveal that models with explicit geometry encoding generally outperform sequence-based models, with an average prediction RMSE reduction of around 12% across all various RNA tasks and excelling in low-data and partial labeling regimes, underscoring the value of explicitly incorporating geometric context. On the other hand, geometry-unaware sequence-based models are more robust under sequencing noise but often require around 2-5x training data to match the performance of geometry-aware models. Our study offers further insights into the trade-offs between different RNA representations in practical applications and addresses a significant gap in evaluating deep learning models for RNA tasks.

Published as a conference paper at ICLR 2025
BEYOND SEQUENCE: IMPACT OF GEOMETRIC CONTEXT FOR RNA PROPERTY PREDICTION
Junjie Xu +1, Artem Moskalev' i, Tommaso Mansi', Mangal Prakash' 3, Rui Liao
'Johnson & Johnson Innovative Medicine, 'The Pennsylvania State University
junjiexu&psu.
famoskal2, tmansi, mpraka12, rliao2]eits.jmm jnj.com
ABSTRACT
Accurate prediction of RNA properties, such stability and interactions, is cru- cial for advancing our understanding of biological processes and developing RNA- based therapeutics. RNA structures can be represented topological graphs, or 3D all-atom models, each offering different insights into its function. Existing works predominantly focus ID sequence- based models, which overlook the geometric context provided by 2D and 3D geometries. This study presents the first systematic evaluation of incorporating explicit 2D and 3D geometric information into RNA property prediction, considering not only per- formance but also real-world challenges such as limited data availability, partial labeling, sequencing noise, and computational efficiency. To this end, we intro- duce a newly curated set of RNA datasets with enhanced 2D and 3D structural annotations, providing a resource for model evaluation on RNA data. Our find- ings reveal that models with explicit geometry encoding generally outperform sequence-based models, with an average prediction RMSE reduction of around 12% across various RNA tasks and excelling in low-data and partial labeling regimes, underscoring the value of explicitly incorporating geometric context. On the other hand, gcometry-unawan sequence-bra based models are more robust under sequencing noise but often require around 2 5x training data match the per- formance of geometry-aware models. Our study offers further insights into the trade-off's between different RNA representations in practical applications and ad- dresses a significant gap in evaluating deep learning models for RNA tasks. All datasets and code will be released upon acceptance.
ID sequences, 2D
INTRODUCTION
RNA plays a central role in the machinery of life, serving as a crucial intermediary between nu- cleotide and amino acid worlds (Sahin et al., 2014). Beyond messenger role, RNA is involved in diverse biological processes, including gene regulation, catalytic activity, and structural support within ribosomes (Sharp) 2009. [Stobbll et al., 2016). This versatility makes RNA a key target for fundamental biological research and therapeutic interventions. As understanding of RNA com- plexity grows, so does the need for advanced computational tools for its analysis.
Modeling RNA is challenging due its intricate secondary and tertiary structures, dynamic confor- mational changes, and interactions with cellular components (Han et al. 2013). Furthermore, RNA analysis is hindered by the practical challenges of RNA data acquisition which include sequencing errors (Ozsolak & Milos 201 1), batch effects (Tran et al., (020). incomplete sequencing (Alfonzo) al. 2021), partial labeling (Wayment-Steele et al. 2022b), and high costs of obtaining large la- beled datasets (Byron et al. 2016). Moreover, RNA molecule can be represented in different ways: as ID nucleotide sequence. 2D graph of base pairings, or a 3D atomic structure, Each represen- tation highlights different aspects of RNA, presenting both opportunities and challenges for model design and selection (Fig. I).
* "This work done while the author intern Johnson & Johnson.
I Equal contribution as first authors
I Equal contribution as last authors

---

Published as a conference paper at ICLR 2025
RNA
Models
Performance: RMSE
20
SD
2D Structure
Tonsfommn
Plarar Graph
Sequence
..GGAAUCCA..
1D
2D
3D
THIS STUDY
3D Structure
Robustness: & RMSE
da Geometric GNNN Graph
Analysis
1D
2D
3D
Robustness
Performance:
Data Requirement: samples
RMSE
Generalization to noise
partial data
2D
3D
Figure 1: Overview of the study. (a) Left panel: RNA sequences represented in ID, 2D, and 3D structures, processed by ID sequence, 2D GNN, and 3D GNN models. Our analysis includes prediction error, robustness and generalization to sequencing noise, and performance under lim- ited training data and partial labelings. (b) Right panel: Comparative performance of ID, 2D, and 3D methods across experimental conditions. Histograms show RMSE performance, relative RMSE changes with increasing noise, and data requirements for optimal performance. Lower values indi- cate better performance in all metrics.
In this work, we systematically study the performance of various machine learning models for RNA property prediction, extending beyond traditional sequence- based approaches (He et 2021 Soylu & Sefer 2023) [Prakash et al., [2024] Yazdani- Mizzani-Jahromi et al.) (025) to include methods that process RNA with its 2D or 3D geometry. While 2D and 3D RNA representations offer potentially richer information, they also present unique challenges. In the absence of high-quality experimental data, accessing or 3D RNA structures requires running structure prediction algorithms prone noise and mistakes, especially in the presence of sequencing errors (Schneider al. 2023 Wang al. 2023). A few mutations in the nucleotide sequence owing to sequencing mistakes can hugely alter the 2D and 3D structure (Fig. 4), potentially undermining the benefit of using additional geometric context. Furthermore, real-world RNA datasets often suffer from partial labeling (Wayment-Stedd al. (202bb) and scarcity of training samples (Wint al. (022), which may affect geometry- aware methods differently than sequence-based approaches. These challenges raise fundamental question: what extent does explicit 2D and 3D geometry contribute to RNA property prediction, and under what cireumstances might it offer advantages over geometry-free sequence models?
In this study, we seek to answer this question by making the following contributions:
We introduce a diverse collection of RNA datasets, including newly annotated 2D and 3D struc- tures, covering various prediction tasks nucleotide and sequence levels across multiple species.
We provide a unified testing environment to evaluate different types of machine learning models for RNA property prediction, including sequence models for ID, graph neural networks for 2D, and equivariant geometric networks for 3D RNA representations.
We conduct a comprehensive analysis of how different models perform under various conditions, such as limited data and labels, different types of sequencing errors, and out-of- -itof--distribttion scenar- ios. We highlight the trade-off's and contexts in which each modeling approach most effective, guiding researchers in selecting suitable models for specific RNA analysis challenges.
We also introduce novel modifications to existing 3D geometric models based on biological prior, specifically optimizing them for handling large-scale point cloud RNA data, thus improving the efficiency and performance 3D models significantly.
Our study reveals several key insights: (i) 2D models generally outperform ID models, with spec- tral GNNs reducing prediction error by about 12% on average across all datasets, highlighting the importance of explicitly considering RNA structural information; (ii) 3D equivariant GNNs outper-
2

---

Published as a conference paper at ICLR 2025
form ID and some 2D methods in noise-free scenario but sensitive noise, exhibiting up to 56% decrease in prediction quality under high sequencing error rates; (iii) geometry- sequence models remain the most robust sequencing noise, showing only 14-27% increase in prediction error compared to noise-free conditions, however they require around 2 5x more training data to match the performance of geometry-aware models.
DATASETS AND MODELS
Here, discuss datasets selected for our study with RNA-level prediction labels. These datasets are selected to vary from small to large-scale and to encompass both nucleotide-level tasks and sequence-level tasks. We perform extensive evaluation across these datasets, leveraging three different model families (1D, 2D, 3D) spanning representative models in total.
2.1 DATASETS
The datasets vary in size based the number of sequences and sequence lengths: the small dataset Tc-Riboswitches (Groher et al. 2018), the medium datasets Open Vaccine COVID-19 19 (Wayment- Steele al. 2022b) and Ribonanza-2k (He et al.) 2024), and the large dataset Fungal (Wint al., 2022). All datasets provide regression labels. Detailed statistics for these datasets are provided in Appendix E.1
Te-Riboswitches: 355 mRNA sequences (67-73 nucleotides) with sequence-level labels for riboswitch switching behavior, important for optimizing gene regulation in synthetic biology and gene therapy.
2. Open Vaccine COVID-19: 4,082 RNA sequences (each of 107 nucleotides) with nucleotide- level degradation rate labels, crucial for predicting RNA stability in mRNA vaccine development.
Ribonanza-2k: 2,260 RNA sequences (each of 120 nucleotides) with nucleotide-level experi- mental reactivity labels, supporting RNA structure modeling and RNA-based drug design.
4. Fungal: 7,056 coding and tRNA sequences (150-3,000 nucleotides) from 450 fungal species, used for sequence-level protein expression prediction.
2.2 DATA PREPROCESSING AND CURATION
For the Open Vaccine COVID-19 19 dataset, we filter out sequences with signal-to-noise ratio (SNR) below 1, as recommended by the dataset authors (Wayment-Stecll
al.) 2022b), to ensure that only sequences with significant signal relative to background noise are included, thereby enhancing the reliability of modeling. For the other datasets, we use the original sequences since no SNR annotations are available.
Since all the RNA datasets come with sequences only, we employ EternaFold (Wayment Steele et al. 2022a) and RhoFold (Shen et al. 2022) infer 2D and 3D molecular structures respectively. We selected EternaFold and RhoFold due to their state-of-the-at performances acknowledged in recent works (Wayment-Steele et al. 2020 2022b He al. 2024) Additionally, RhoFold typically runs in seconds to a minute per sequence, unlike other 3D structure prediction tools which usually take hours, and hence not suitable for large datasets.
For ID modeling, we use the original RNA sequences without structural augmentation which equates to processing a plain string of nucleotides. The 2D datasets represent each RNA sequence as graph with nodes for nucleotides and edges for bonds between nucleotides. The node features are dimensniiinnl, incorporating one-hot nucleotide identity ('A', 'C', 'G'. 'U') alongside the sum and mean base-pairing probabilities (BPP), which are available from 2D structure prediction tools. In 3D, each RNA molecule is represented as graph, with nodes corresponding to individual atoms. Node features represent one-hot atom identity.
2.3 MODELSS
We select well-established model architectures recognized for their otate-of-the-art performance for molecular property prediction tasks in various domains. 1D Model: Transformer1 D (Honda al.,
3

---

Published as a conference paper at ICLR 2025
2019
He et al., 20 RNA-FM -M (Chnn et al., 2022). SpliceBERT (Chen et al.
2023)
2D Mod-
els:
GCN (Kipf & Welling 2017 Wieder et al. 2020). GAT Veličković et al.
Ye et al. (He al. [002]} Transformer (Shi et al. 2020 Li et al. 2022), and GraphGGP (Rampášek et al. 2022 Zhu
2018
2022).
ChebNet (Defferrard al. 2016 Knyazev al. 2018 Transformer DDD IDD
Graph
al.
2023); 3D Models: Sch (Schütt et al. 2017 Han al. 2022), EGNN (Satorras et al.
2021 1
FAENet (Duval et al. 2023), DimeNet (Gasteiger et al. (020), GVP (Jing et al. 2020). and
Fast FastEGNN
(Zhang et al.p. Detailed descriptions of all the models can be found in Appendix Sec.
Training and evaluation All models were trained on NVIDIA A100 GPU. To ensure hyperpa- rameter parity for each baseline, hyperparameters were optimized using Optuna (Akiba et al. 2019), restricting the search to models with fewer than 10 million parameters that fit within the GPU mem- ory constraint of 80GB. All model hyperparameters, training, and evaluation details are reported in Appendix H We ran all models for random data splits (alain:val:test split of 70:15:15) and we port average performance with a standard deviation across splits. The mean column-wise root mean squared error (MCRMSE), introduced in Wayment-Steed et al. (2020). is used the evaluation metric. It is defined as MCRMSE(S. D) V the dataset, and 9J and y/ are the predicted and true values for data point i.
E (ir -y) where / represents model, D is
TASK DEFINITIONS
Here, we introduce the downstream tasks for evaluating models for RNA property prediction. Each task is designed to quantify specific behaviors under various real-world experimental conditions.
Task I: Impact of structural information on prediction performance This task aims to evaluate how incorporating RNA structural information affects prediction quality. We compare the perfor- mance of models using ID (sequence-only), 2D, and 3D RNA representations to determine if and to what extent geometric data improves property prediction.
Task 2: Model efficiency in limited training data settings Acquiring high-quality comprehen- sive RNA datasets often challenging and resource-intensive thus limiting the amount of labeled data for training (Teufel & Sobetzko/ [022 Byron et al. 2016). This task aims to investigate how model performance depends on the amount of training data used, evaluating the sample efficiency of each family of models. In other words, given a dataset D = {X, YJ, let D. = (Xa, Y. be a subset where the training set is reduced fraction a. We train models on different sets of D.. datasets with decreasing a.
Task 3: Performance with partial sequence labeling Due to the high cost of measuring prop- erties for every nucleotide in RNA sequence, real-world datasets often contain partial annota- tions (Wayment-Steele et al. 2022b) where labels are only available for the first small part of the sequence. This task is relevant for nucleotide-level datasets and it aims to investigate how well model can generalize to a whole RNA sequence when labels are only available for a portion of it.
Task 4: Robustness to sequencing noise Acquiring RNA data requires sequencing. In practice sequencing procedure may introduce sequencing errors (random mutations of nucleotides) that vary depending on the sequencing technology and platform (Ozsolak & Milos| 2011: Fox et al. 2014). These errors affect the raw sequence data, and propagate to structural noise in 2D and 3D. The goal of this task assess how well models can maintain reliable performance when trained and tested under the same distribution of realistic levels of sequencing noise observed in practice, ensuring robustness across a consistent noise environment. This reflects real-world cases where a specific sequencing method produces noisy data, but the noise characteristics are stable across training and deployment.
Task 5: Generalization to Dut-of-Distribation (OOD) data This task focuses on different practical challenge: models trained on high-quality RNA sequences are often deployed in conditions where the data exhibits different noise characteristics due to batch effect (Tran et al. 2020 or the use of different sequencing platforms (Tom et al. 2017). Here, the objective is to evaluate how well models generalize to OOD datasets with different levels of sequencing noise, assessing the
4

---

Published as a conference paper at ICLR 2025
Table Comparison of 2D, and 3D models across datasets. Bold indicates the best, underline the second-best. 'OOM' means out-of-memory. ChebNet excels by capturing global graph informa- tion. Overall, 2D models outperform ID models, highlighting the value of structural information. Although 3D models face challenges with scalability and noisy predictions, our nucleotide pooling strategy, based on biological prior, enhances their performance on shorter sequences, allowing 3D encoding to occasionally surpass ID models. See Sec. 4. for details nucleotide pooling strategy.
Model
COVID
Ribonanza
Te-Ribo
Fungal
ID model
Transformer
0.361
0.705++0.015
0.705
1.417=0.005
RNA-FM
0.591
0.990 144
0.693
1,420=0.023
SpliceBERT
0.588+10.077
1.022:+0.144
0.708
1.435=00.059
2D model
Transformer ID2D
0.305+00002
0.514+0.004
0.633+0.001
0OM
GCN
0.359+000009
0.701+0.004 0.595+00.066
1.192-0.077
GAT
0.315:10.066
0.53440.006
0.685=0.024
1.112=0.035
ChebNet
0.279--0.007
0.408
0.621+0.022
0.973-
Graph Transformer
0.318+00008
0.51540.001
0.710=0.041
1.317=0.000
GraphGPS
0.332:10.013
0.523+0.003
0.715
1.025=0.081
model (without poolin))
EGNN
0.480110.025
0.808:0.023
0.725
OOM
SchNet
0.499++0.003
0.843+00004
0.696+0.008
OOM
FAENet
0.486110.010
0.83410.003
0.703
OOM
DimeNet
0.49711.001
0.71210.004 0.85510.006
OOM
GVP
0.467++0001
0.797+0.012
0.744+0.004
OOM
FastEGNN
0.477-10.005
0.816=1.0014
0.753
0OM
3D model (with nucleotide pooling)
EGNN (pooling)
0.364++0.03
0.619+00007
0.663
OOM
SchNet (pooling)
0.300:10.006
0.6851100066
0.655
OOM
FastEGNN (pooling)
0.444+-0.033
0.753+0.015
0.710=0.011
OOM
extent of performance degradation as noise levels increase. This task simulates the scenario where model encounters noisier data than was trained on, highlighting its ability adapt unexpected experimental conditions.
For detailed descriptions and motivations of the five task settings, please refer Appendix D
4 EXPERIMENTS AND RESULTS
4.1 IMPACT OF EXPLICIT GEOMETRY LEARNING ON MODEL PERFORMANCE
We begin by addressing Task 1, where we compare the performance of model families when trained and evaluated on the downstream RNA datasets. Additionally, we provide runtime and memory comparison Appendix C
2D models consistently outperform ID model Results in Table reveal that 2D methods consis- tently outperform the ID sequence model across all datasets. Notably, the Transformer model, which simply augments the attention matrix with adjacency features alongside, achieves around 10% lower prediction MCRMSE on average across datasets than its geometry-free counterpart. This sug- gests that explicitly incorporating structural information crucial, as learning from sequence data alone proves to be insufficient. Further experiments, detailed in the Appendix F.4 investigate the learned attention maps both the Transformer D2D and the Transformer!D model and their corre- lation with structural information and reveal that mansformeriD2D attention maps much more closely aligned with the topological structure of nucleotide graph, reinforcing the conclusion that explicit encoding of structural information is essential for improved performance.
5

---

Published as a conference paper at ICLR 2025
For the foundation models, RNA-FM and SpliceBERT, we observe that for out of datasets (Covid and Sibonanza-2kk) RNA foundation models perform worse than the simple supervised transformer baseline whereas for the other two datasets (Tc-Riboswitches and Fungal), transformer and RNA foundation models achieve similar performance. This is consistent with recent works in multiple biology-related domains demonstrating specialized foundation models are yet to surpass simple su- pervised learning baselines (Xu et al. [2024b] Kedzierska et al.] 2023 Yang et al. 2024). Hence, we choose Transformer as the model of choice for subsequent experiments.
Spectral GNN outperforms spatial GNNs in 2D ChebNet, a spectral method, outperforms spatial methods such as GCN, GAT, Graph Transformer, and GraphGPS, achieving prediction MCRMSE 2.5% lower than the next best 2D model across datasets. Spatial GNNs aggregate node features layer by layer, emphasizing local information within a fixed distance. While computation- ally efficient, these methods are limited by the 1-Weisfeiler-Lehman the expressive power of bosed updates (Xu ct al. 2018). In addition, spatial GNNs may suffer from a limited receptive field while spectral methods approximate global graph features, enabling a global receptive field since the first layer Wang & Zhang (2022): Bo et al. (2023); Xu et al. (2024a). This allows ChebNet to effectively process global information which is important for RNA data due to potential long-range interaction between nucleotides.
(WL) test, which constrains
Challenges of modeling geometric context in all-atom resolution Contrary to our expectations, 3D models at all-atom resolution (EGNN w/o pooling, SchNet w/o pooling DimeNet (w/o pooling), FAENet (w/o pooling), GVP (w/o pooling) and FastEGNN (w/o pooling) in Table I show relatively high prediction MCRMSE across datasets, underperforming compared to ID and 2D methods. We hypothesize this is due to two factors.
First, all-atom 3D models rely on a limited local neighborhood of adjacent atoms, limiting their receptive fields and preventing them from capturing long-range dependencies (see Appendix Ta- ble F.T. which can be crucial for determining RNA properties (Shetty et al.) 2013] Alshareedah et al. 2019). Expanding the local neighborhood for these methods becomes challenging due the overwhelming scale of large molecular systems in all-atom resolution. Second, the performance of 3D models is often limited by the inherent inaccuracies in 3D structure prediction tools, which are generally less reliable compared to 2D structure prediction methods (Ponce-Sall Soluce-Salvatiera et al., 2019). While FastEGNN is designed for larger molecules, its reliance on the center of mass virtual node initialization may not align well with tasks like RNA property prediction, which inherently have a sequence prior. This may explain the limited performance of FastEGNN in this context and RNA-specific virtual node initialization and carefully designed pooling mechanisms may be needed to further adopt this architecture for large-molecules with a sequence prior.
To address the receptive field limitation of all-atom methods, employ biological prior by pool- ing atomic features into nucleotide-level representations after few layers of all-atom operations. This strategy is aligned with RNA's natural secondary structure, where atoms group into nucleotides, and nuclcotides form the complete RNA molecule (Deng et al.) (023).. This novel strategy allows us maintain all-atom resolution in the initial layers while increasing the receptive field via pool- ing. By balancing the number of all-atom and nucleotide layers as hyperparameters, we can balance fine- and bearse-grained all-atom and nucleotide resolution. Compared other 3D models, EGNN, FastEGNN, and SchNet perform slightly better, and hence we introduce the nucleotide pooled vari- ants for these models. We call 3D models with pooling EGNN (nuc. pooling), FastEGNN (nuc. pooling), and SchNet (nuc. pooling). This strategy significantly enhances 3D model performance, reducing prediction MCRMSE by ~00% compared to the original EGNN, SchNet, and FastEGNN and outperforming ID models on the Ribonanza-2k and Tc- Rbossibsssscees datasets by 5% on aver- age. On the COVID dataset, EGNN (nuc. pooling) matches the Transformerl model but still trails behind 2D models. All subsequent experiments report results using nucleotide- -pooled versions of the 3D models.
Next, we investigated the second hypothesis regarding higher noise in 3D structures by quantifying variability in predicted structures across different 3D prediction tools in Appendix G We observe substantial variability (between 11-45A RMSD across structures given by different 3D structure prediction tools), suggesting considerable noise in 3D predictions, which likely contributes to the poorer performance of 3D models.

---

Published as a conference paper at ICLR 2025
4.2 MODEL EFFICIENCY UNDER LIMITED DATA AND PARTIAL SEQUENCE LABELING
In this section, we combine the analysis of Tasks and 3, assessing model performance in scenarios with limited training data or partial labels.
To analyze how the amount of training data influences model performance, we run experiments with varying portions of the full datasets (25%, 50%, 75%, and 100%) on the medium- and large-scale datasets: COVID, Ribonanza-2k, and Fungal (Appendix Fig. g a) for illustration). The small size of the Te-Riboswitches dataset is excluded from the analysis, training with lower ratios would have resulted in inadequate sample sizes for meaningful evaluation. Additionally, GPU memory constraints prevent the application of Transformer1 and 3D models on the Fungal dataset due to its large sequence length.
We also evaluate the impact of partial property labels for nucleotide-level tasks, common oceur- rence owing to costly experimental measurements (Wayment-Steele et al. (020) to identify which models are best suited to handle the challenges of incomplete labels RNA property prediction. For this, we use the COVID and Ribonanza datasets these datasets contain nucleotide-level la- bels. We train the models using all training data but with varying proportions labeled nucleotides (20%, 40%, 60%, 80%, and 100%) per sequence, thus simulating incomplete sparse labeling, while testing on fully labeled test sets (Appendix Fig. b) for illustration).
Transformer1D 1D Transformer 1eteeerr12200 1D2D
GCN
ChebNet
GraphGPS
SchNet
GAT
Graph Transformer
EGNN
FasIEGNN
17
16
15
0.40
MCRMSS
1.4
1.3
0.35
1.2
0.5
0.9
0.25
0.50
0.75
1000
0.25
0.50
0.75
1:00
0.25
0.50
0.75
1.00
Fraction training data
Fraction of training data
Fraction of tmining data
(a) COVID
(b) Ribonanza
(c) Fungal
Figure 2: Performance vs. fraction of training data across various datasets. Model performance
improves with increasing data, with lower MCRMSE across all models. 2D models consistently outperform ID models, particularly in low-data regimes, underscoring the value of structural in- formation for generalization. Dotted, solid, and dashed lines denote ID, 2D, and 3D methods, respectively, which applies consistently throughout all figures in this paper.
0.7
1.3
Transtormer1D
Transformer1D20
0.6
1.1
GCN
MCRMSE
MCRMSE
GAT
0.5
0.9
ChebNet
Graph Transformer
0.4
0.7
GraphGPS
EGNN
0.3
0.5
SchNet
0.20
0.40
0.60
0.80
1.00
0.20
0.40
0.60
0.80
1.00
FastEGNN
Fraction of property labels
Fractoo of property labels
(a) COVID
(b) Ribonanza
Performance vs. partial property labels on COVID and Ribonanza datasets. 2D models consistently outperform ID models with sparse labeling, while Transformer and Trans- formerlD2D improve rapidly with denser supervision, emphasizing the need for more labels in transformer-based models.
Figure 3:
More training data improves performance Unsurprisingly, across all models and datasets, clear trend emerges: increasing the amount of available data, whether through higher number of training data points greater proportion of available labels leads to improved performance (Fig. 2 Fig. 3 Appendix Tables in Sec. [.2 However, the degree of performance improvement varies significantly between model types, analyzed next.

---

Published as a conference paper at ICLR 2025
2D models excel in low data and partial label regimes Evaluating model performance differ- ent training and label ratios reveals a notable trend: 2D models consistently outperform ID and 3D models under low data and incomplete labeling regimes. Between 20-50% training data and label levels, 2D models such as ChebNet, Transformerl D221,, D21D, GraphGPS, and GAT significantly outper- form Transformerl D, highlighting the role of additional structural information for model sample efficiency. Interestingly. Transformerl and Transformer 1erormerrDDDD exhibit a faster rate of improvement when more labels available (Fig. 3 suggesting that transformer-basee architectures benefit from denser supervision. Notably, Transformer requires 2 5 more training data/labels to match the performance of the least effective 2D models, which achieve comparable results using only 20% to 50% of the training data needed by Transformer when trained on the full dataset.
3D models outperform ID model in limited data regime despite structural noise For the medium-scale datasets (COVID and Ribonanza), where 3D models can be evaluated, we observe that the 3D models generally outperform are on par with the Transformerl even for lower data and labeling regimes. This suggests that despite the noise introduced by inaccuracies in 3D structure predictions, the explicit geometric encoding in 3D models still provides an advantage over 1D models. EGNN, in particular, is consistently better than or on par with Transformerl) across all training and label ratios. This further emphasizes that models incorporating explicit geometric encoding (whether 2D or 3D) are more data-efficient than those relying solely sequence informa- tion. However, important to note that 3D models not match the efficiency 2D models in these scenarios, likely due to their susceptibility to noise as discussed in Section
Noise Ratio
0.1 ..GGAAAACUCA...
0.2
0.3 GGUAGGGUAA...
ID .GAAAGCUCA....
GGGAAAAUUCA....
2D
3D
Figure 4: Visualization of 1D, 2D, and 3D structures under varying noise ratios (mutation errors during sequencing). Each column represents different noise ratio, showcasing the impact of noise on the structures across different dimensions.
4.3 MODEL. ROBUSTNESS AND GENERALIZZTION UNDER SEQUENCING NOISE
Tasks 4 and both deal with model performance with noise in the data, but focus on different aspects, robustness noise, and ability to generalize across unseen noise distributions.
explained in Sec. 3 sequencing noise is common depending the sequencing method and platform used (Fox al. 2014), thereby introducing errors in sequences that propagate into 2D and 3D structures. Additionally one of the deployment scenarios involves models trained high-quality clean data applied for datasets acquired under noisy conditions owing to different sequencing platforms or experimental batch effects (Tom et al. 2017).
To explore these practical aspects, we design two sets experiments:
Robustness: We introduce sequencing noise into the training, validation, and test sequences to simulate realistic sequencing errors. Nucleotide mutations are applied with probabilities {0.05, 0.1, 0.15, 0.2, 0.25, 0.3), mirroring typical sequencing propagates to the 2D and 3D structures (Fig. 4 via structure prediction tools. Importantly, these mutations random; the likelihood of particular nucleotide mutating into another varies, as
rates (Pfeiffer et al. 2018) which also
8

---

Published as a conference paper at ICLR 2025
is well documented in sequencing studies Pfeiffer et al. (2018). Our noise model reflects these real- world mutation profiles. Crucially, while the input training, validation, and test sequences contain noise, the property labels remain clean. This again reflects practical scenarios where labels experimentally determined independent sequencing and thus unaffected by sequencing errors.
Generalization: Here, models are trained on clean, noise-free data corresponding high-quality sequencing experiments but are tested on datasets with varying levels of noise simulated by se- quencing mutation probabilities in {0.05, 0.1, 0.15, 0.2, 0.25, 0.3). This setup reflects the real- world scenario where models trained on high-quality data will be deployed on OOD data that may come from different sequencers or have been affected by batch effects.
0.9
0.50
0.8
MCRMSE
0.45
MCRMSE
0.7
0.40
0.6
Transformer1D
0.5
Transformer1D2D
0.00
0.05
0.10 0.15 0.20 0.25 0.30
0,00
0.05
0.10 0.15 0.20 0,25 0.30
GCN
Noise Ratio
Noise Ratio
GAT
(a) COVID
(b) Ribonanza
ChebNet
1.6
0.9
Graph Transformer
1.5
GraphGPS
MCRMSE
1.4
MORMSE
0.8
1.3
EGNN
1.2
SchNet
111
FastEGNN
1.0
0.6
0.00 0.05 0.10 0.15 0.20 0.25 0.30
0.00
0.05 0.10 0.15 0.20 0,25 0.30
Noise Ratio
Noise Ratio
(c) Fungal
(d) Tc-Riboswitches
Figure 5: Robustness experiments. Transformer shows the least performance drop under in- creasing noise, maintaining the highest accuracy, with Transformer1D2D following closely. In con- trast, 2D and 3D models, particularly ChebNet and 3D models, are more impacted by noise.
Transformer architectures demonstrate superior robustness and generalization under se- quencing noise Expectedly, across both tasks, increased noise levels generally leads to worse test MCRMSE for all models, indicating decline in prediction performance (Fig. 5 Fig. 6 and
Appendix Tables in Sec. I.3 1.4.
Among all models, Transformer demonstrates the highest robustness and generalization, exhibit- ing the least performance degradation as noise levels increase. Notably, in generalization exper- iments, Transformer1D achieves the best prediction MCRMSE the COVID, Ribonanza, and To-Riboswitches datasets under higher noise levels (Fig. 6 and Appendix (.4). The reliance of I on sequence-only information without considering geometric context, while being a weakness in other scenarios, becomes a strength in case of noisy sequences as minor sequenc- ing errors may severely alter the predicted RNA structures (Fig. 4. While its performance on the Fungal dataset is worse overall, Transformer D maintains remarkably consistent performance as noise increases. Transformer DD2D ID2 ranks just behind Transformer ID outperforming other 2D and 3D models. This can be attributed to Transformer D2D's ability selectively focus on sequence information for noisy data rather than relying on structural data alone as the self-attention only weakly conditioned on the graph topology.
More elaborated 2D and 3D models, which rigidly rely on structural information, are significantly more affected by noise, underperforming compared to plain sequence baseline in both robustness and generalization experiments for high noise levels. For now-to-moderate noise levels (5-10% noise), 2D methods such ChebNet still perform the best. Interestingly, ChebNet shows the worst generalization among 2D models, although it performs on par with other methods robustness experiments. This suggests that while ChebNet struggles with OOD noise, its performance gets

---

Published as a conference paper at ICLR 2025
0.65
1.6
0.60
1.4
MCRMSE
0.55
MCRMSE
1.2
0.50
1.0
0.40
Transformer1D
0.6
0.30
Transformer1D2D
0.00
0.05
0.30
0.00
0.05
0.30
GCN
Noise Ratio
Noise Ratio
(a) COVID
(b) Ribonanza
GAT
ChebNet
1.8
1.4
Graph Transformer
1.6
GraphGPS
MCRMSE
MCRMSE
1.2
EGNN
1.4
1.0
SchNet
1.2
FastEGNN
0.8
1.0
0.00
0.05 0.10 0.15 0.20 0.25 0.30
0.6
0.00 0.05 0.10
0.15 0,20 0.25 0.30
Noise Ratio
Noise Ratio
(c) Fungal
(d) Tc-Riboswitches
Figure 6: Generalization experiments. Transformer outperforms other models on noisy se- quences, achieving the lowest RMSE at higher noise levels. particularly COVID, Ribonanza, and Tc-Riboswitches. Transformer D2D follows closely, showing that transformer-based models gener- alize better under noise than 2D and 3D models, especially in tasks with geometric representations.
better when it is also trained the same noise level used during testing, highlighting the need for retraining for different experimental data batches/noise levels in real-world applications. Across both experiments, 3D models have poor performance for more noisy conditions, particularly with the COVID and Ribonanza datasets, due to their dependence on 3D structures which are also sensitive to the propagation of sequencing errors (Fig 4 bottom). In the smaller Tc-Riboswitches dataset, model performances vary more, likely due to limited data size, but transformer models still consistently demonstrate greater robustness noise.
Across both settings, as the noise ratio rises, the ID model demonstrates the greatest resilience to noise, showing an average test MCRMSE increase of approximately 14% and 27%, respectively, relative to the train and test clean data. In contrast, 2D models exhibit the highest sensitivity, with test MCRMSE increasing from 30% to 82%. Meanwhile, 3D models show intermediate per- formance, with MCRMSE increases of 29% and 56%. Our results reveal higher vulnerability of 2D and 3D models to sequencing noise where geometric context becomes unreliable a faster rate than plain sequence of nucleotides.
CONCLUSION
We present the first comprehensive study the benefits and challenges of the effect of geometric context for RNA property prediction models. With providing a curated set of RNA datasets with annotated 2D and 3D structures, systematically evaluate the performance of 1D, 2D, and 3D models under various real-world conditions, such as limited data, partial labeling, sequencing er- generalization. Our results reveal that 2D models outperform 1D and
rors and 3D models, with spectral graph neural networks excelling even in low-data and partial labeling sce- narios. For 3D models, find that their potential benefits are hindered by the limited receptive field, computational complexity, and structural noise from RNA structure prediction tools. At the same time, ID models demonstrate better robustness compared to 2D and 3D models in noisy and OOD conditions. This study highlights the value and limitations of using geometric context for RNA modeling. Future work could focus on ensembling ID. 2D, and 3D models for complemen- tary strengths, and improving 2D and 3D models to better handle noise from structure prediction tools as elaborated in Appendix E.2 Another promising direction can be to investigate advanced 3D model architectures which incorporate high-degree steerable features as discussed in Appendix E.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
This work aims to advance the computational prediction of RNA properties through the development and evaluation of machine learning models utilizing diverse representations of RNA, including 1D sequences, 2D structures, and 3D geometries. In conducting this research, we are committed to up- holding high ethical standards in all aspects of our study, ensuring that our work promotes scientific integrity, transparency, and responsible use of technology.
Data Integrity and Fair Use: All RNA datasets used in this study were acquired from publicly available and ethically sourced repositories. Where applicable, the original data sources have been duly cited, and all efforts have been made to ensure data privacy and compliance with rel- evant data-sharing agreements. Our curated datasets have been handled responsibly, with proper annotations to minimize errors and misinterpretations.
Minimization of Bias: We recognize that RNA data, particularly when involves limited or incomplete datasets, can introduce biases in model predictions. To mitigate this, we employ diverse set of RNA data and systematically analyze model performance under various conditions, including label scarcity and sequencing errors. By doing so, aim provide balanced insights into the potential advantages and limitations of different modeling approaches, fostering respon- sible model selection and deployment.
The machine learning techniques used in this study are in- tended to assist in scientific discovery and biological understanding, with potential applications in therapeutic interventions. However, we acknowledge the importance of caution in applying computational models in sensitive domains such as healthcare. While our models are designed to improve RNA property prediction, we emphasize that these predictions should not be used in isolation for clinical therapeutic decision-making without further validation and consideration of ethical implications.
Responsible Use of AI in Biology:
REPRODUCIBILITY STATEMENT
We are committed to the transparency and reproducibility of our findings. All methodologies, datasets, and benchmarking environments will be made publicly available to the research commu- nity, allowing others to reproduce, verify, and extend our work. This open approach aims to advance the field of RNA research and promote collaborative progress. To ensure reproducibility, we also provide detailed descriptions of model architecture and training procedures in the main text. The appendix H contains full details on data preprocessing. hyperparameter searching and additional experimental information. All data and code will be provided to upon acceptance.
REFERENCES
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama, Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2623-2631 2019.
Juan D Alfonzo, Jessica A Brown, Peter H Byers, Vivian G Cheung, Richard Maraia, and Robert L Ross. A call for direct sequencing of full-length ras to identify all modifications. Nature genetics, 53(8): 111-111111111,11 2021.
Ibraheem Alshareedah, Taranpreet Kaur, Jason Ngo, Hannah Seppala, Liz-Audrey Djomnang Kounatse, Wei Wang, Mahdi Muhammad Moosa, and Priya R Banerjee. Interplay be- tween short-range attraction and long-range repulsion controls reentrant liquid condensation of ribomucleoprotein-mma complexes. Journal of the American Chemical Society, 141(37):14593- 14602, 2019.
Viplove Arora and Guido Sanguinetti. De novo prediction of ma-protein interactions with graph neural networks. RNA, 28(11):1469-1480 2022.
Minkyung Baek, Ryan McHugh, Ivan Anishchenko, Hanlun Jiang, David Baker, and Frank DiMaio. Accurate prediction protein- nucleic acid complexes using rosettafoldna. Nature methods, 17):11,- 2024.
11

---

Published as a conference paper at ICLR 2025
Ilyes Batatia, David P Kovacs, Gregor Simm, Christoph Ortner, and Gábor Csányi Mace: Higher order equivariant message passing neural networks for fast and accurate force fields. Advances in Neural Information Processing Systems, 35:1 15:11423-1 2022.
Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P Mailoa, Mordechai Ko- rnbluth, Nicola Molinari, Tess E Smidt, and Boris Kozinsky. E (3)=equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature communications, 13(1):
2453, 2022.
Deyu Bo, Xiao Wang, Yang Liu, Yuan Fang. Yawen Li, and Chuan Shi. A survey on spectral graph neural networks. arXiv preprint arXiv:2302.0563// 2023.
Michal Boniecki, Grzegorz Lach, Wayne K Dawson, Konrad Tomala, Pawel Lukasz, Tomasz Soltysinski, Kristian M Rother, and Janusz M Bujnicki. Simrna: a coarse coarse-graines grarssegr method for ma folding simulations and 3d structure prediction. Nucleic acids research, 44(7):e63-e63, 2016.
Sara A Byron, Kendall R Van Keuren-Jensen, David Engelthaler, John Carpten, and David W Craig. Translating ma sequencing into clinical diagnostics: opportunities and challenges. Nature Reviews Genetics, 17(5):5572271 2016.
Jiacheng Cen, Anyi Li, Ning Lin, Yuxiang Ren, Zihe Wang, and Wenbing Huang. Are high- degree representations really unnecessary in equivariant graph neural networks? arXiv preprint arXiv:2410.11443, 2024.
Jiayang Chen, Zhihang Hu, Siqi Sun, Qingxiong Tan, Yixuan Wang, Qinze Yu, Licheng Zong, Liang Hong, Jin Xiao, Tao Shen, et al. Interpretable maa foundation model from unannotated data for highly accurate structure and function predictions. arXiv preprint 2rXiv:2204.00300, 2022.
Ken Chen, Yue Zhou, Maolin Ding, Yu Wang, Zhixiang Ren, and Yuedong Yang. Self-supervised learning on millions of pre-mrna sequences improves sequence- basedd ma splicing prediction. bioRxiv, pp. 2023-01, 2023.
Xinshi Chen, Yu Li, Ramzan Umarov, Xin Gao, and Le Song. Rna secondary structure prediction by learning unrolled algorithms. arXiv preprint arXiv:2002.05810. 2020.
Rhiju Das and David Baker. Automated de novo prediction of native-like ma tertiary structures. Proceedings of the National Academy of Sciences, 104637):1464411666) 2007.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in Neural Information Processing Systems, pp. 3844-3852, 2016.
Jie Deng, Xianyang Fang, Lin Huang, Shanshan Li, Lilei Xu, Keqiong Ye, Jinsong Zhang, Kaiming Zhang, and Qiangfeng Cliff Zhang. Rna structure determination: From 2d to 3d. Fundamental Research, 3(5):727-737, 2023.
Alexandre Agm Duval, Victor Schmidt, Alex Hernández-Gaarcaa Santiago Miret, Fragkiskos D Malliaros, Yoshua Bengio, and David Rolnick. Faenet: Frame averaging equivariant gnn for materials modeling. In International Conference on Machine Learning, pp. 9013-9033. PMLR, 2023.
Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
Edward J Fox, Kate Reid-Band Bayllss Mary J Emond, and Lawrence Loeb. Accuracy of next generation sequencing platforms. Next generation, sequencing & applications, 1, 2014.
J Thorben Frank, Oliver Unke, Klaus-Robert Müller, and Stefan Chmiela. euclidean transformer for fast and stable machine learned force fields. Nature Communications, 15(1):6539, 2024.
Laiyi Fu, Yingxin Cao, Jie Wu, Qinke Peng, Qing Nie, and Xiaohui Xie. Ufold: fast and accurate ma secondary structure prediction with deep learning. Nucleic acids research, 30(3)xe1-el11 2022.
12

---

Published as a conference paper at ICLR 2025
Johannes Gasteiger. Janek Groß, and Stephan Günnemann. Directional message passing for molec- ular graphs. arXiv preprint arXiv:2003.03123. 2020.
Ann-Christe Groher, Sven Jager, Christopher Schneider, Florian Groher, Kay Hamacher, and Beat- rix Suess. Tuning the performance of synthetic riboswitches using machine learning. ACS syn- thetic biology, 8(1):34 44, 2018.
Jiaqi Han, Yu Rong, Tingyang Xu, and Wenbing Huang. Geometrically equivariant graph neural networks: survey, arXiv preprint arXiv:2202.07230. 2022.
Yixing Han, Shouguo Gao, Kathrin Muegge, Wei Zhang, and Bing Zhou. Advanced applications of sequencing and challenges. Bioinformatics and biology insights, 9:BB1-S28991, 2015.
Shujun He, Baizhen Gao, Rushant Sabnis, and Qing Sun. Nucleic transformer: Deep learning on nucleic acids with self-attention and convolutions. bioRxiv, pp. 2021-01, 2021.
Shujun He, Baizhen Gao, Rushant Sabnis, and Qing Sun. Rnadegformer: accurate prediction of mrma degradation at nucleotide resolution with deep learning. Briefings in Bioinformatics, 24(1): bbac581, 2023.
Shujun He, Rui Huang, Jill Townley, Rachael C Kretsch, Thomas Karagianes, David BT Cox, Hamish Blair, Dmitry Penzar, Valeriy Vyaltsev, Elizaveta Aristova, et al. Ribonanza: deep learn- ing of ma structure through dual crowdsoureing. bioRxiv, 2024.
Ivo I. Hofacker, Walter Fontana, Peter F Stadler, I. Sehastian Bonhoeffer, Manfred Tacker, Peter Schuster, et al. Fast folding and comparison of rma secondary structures. Monatshefte fur chemie, 125:167-167, 1994.
Shion Honda, Shoi Shi, and Hiroki R Ueda. Smiles transformer: Pre-trained molecular fingerprint for low data drug discovery. arXiv preprint 1rXiv:1911.04:36 2019.
Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael JL Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons, arXiv preprint 2rXhv:2009.01411 2020.
Chaitanya K. Joshi, Cristian Bodnar, Simon V. Mathis, Taco Cohen, and Pietro Liò. On the ex- pressive power of geometric graph neural networks. In International Conference Machine Learning, 2023.
Wolfgang Kabsch. A solution for the best rotation to relate two sets of vectors. Acta Crystallo- graphica Section A: Crystal Physics, Diffraction, Theoretical and General Crystallography, 32 (5):922-923, 1976.
Kasia Z Kedzierska, Lorin Crawford, Ava P Amini, and Alex X Lu. Assessing the limits of zero-shot foundation models in single-cell biology. bioRxiv, pp. 2023 10, 2023.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
Boris Knyazev, Xiao Lin, Mohamed R Amer, and Graham W Taylor. Spectral multigraph networks for discovering and fusing relationships in molecules. arXiv preprini 181 1.0911. 2018.
Sowmya R Krishnan, Arijit Roy, and M Michael Gromiha. Reliable method for predicting the binding affinity of ma-small molecule interactions using machine learning. Briefings in Bioinfor- matics, 25(2):bbae002, 2024.
Mandar Kulkarni, Jayaraman Thangappan, Indrajit Deb, and Sangwook Wu. Comparative analysis of ma secondary structure accuracy on predicted ma 3d models. Plos one, 18(9))c029000,, 2023.
Han Li, Dan Zhao, and Jianyang Zeng. Kpgt: knowledge- seniedge-guided pre-training of graph transformer for molecular property prediction. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 857-867, 2022.
13

---

Published as a conference paper at ICLR 2025
Artem Moskalev, Mangal Prakash, Rui Liao, and Tommaso Mansi. SE(3)-hyena operator for scalable equivariant learning. In ICML 2024 Workshop on Geometry- Representation Learning and Generative Modeling, 2024. BRLLhtts: http:/ss//oeerreeiw.etttteeeeeee
rJXsKeN6dG
Chandran Nithin, Sebastian Kmiecik, Roman Blaszczyk, Julita Nowicka, and Irina Tuszyáska. Com- parative analysis of ma 3d structure prediction methods: towards enhanced modeling of ma- ligand interactions. Nucleic Acids Research, 52(13):7465-7488 2024.
Fatih Ozsolak and Patrice M Milos. Rna sequencing: advances, challenges and opportunities. Nature reviews genetics, 12(2):87-98, 2011.
Robin Pearce, Gilbert S Omenn, and Yang Zhang. De novo ma tertiary structure prediction at atomic resolution using geometric potentials from deep learning. BioRxiv, pp. 2022-05, 2022.
Rafael Josip Penić, Tin Vlašić, Roland Huber, Yue Wan, and Mile Šikić. Rinalmo: General- purpose ma language models generalize well structure prediction tasks. arXiv preprint arXiv:2403.0004: 2024.
Franziska Pfeiffer, Carsten Gröber, Michael Blank, Kristian Händler, Marc Beyer, Joachim Schultze, and Günter Mayer. Systematic evaluation of error rates and causes in short samples in next-generation sequencing. Scientific reports, 8(1):10950, 2018.
Almudena Salvatera,,,,, Astha, Katarzyna Merdas, Chandran Nithin, Pritha Ghosh, Sunan- dan Mukherjee, and Janusz M Bujnicki. Computational modeling of ma 3d structure based on experimental data. Bioscience reports, 39Q))BSR20188430, 2019.
Mangal Prakash, Artem Moskalev, Peter DiMaggio Jr., Steven Combs, Tommaso Mansi, Justin Scheer, and Rui Liao. Bridging biomolecular modalities for knowledge transfer in bio-language models. In Neurips 2024 Workshop Foundation Models for Science: Progress, Opportunities, and Challenges, 2024. BRRUtttss///ppeeeriiiie
Ladislav Rampášek, Mikhail Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Do- minique Beaini. Recipe for General, Powerful, Scalable Graph Transformer, Advances Neural Information Processing Systems, 35, 2022.
Ugur Sahin, Katalin Karikó, and Özlem Türeci. mana-based therapeutics- developing new class of drugs. Nature reviews Drug discovery, 13(10))759-788 2014.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural net- works. In International conference on machine learning, pp. 9323-9332. PMLR, 2021.
Tamar Schlick and Anna Marie Pyle. Opportunities and challenges in ma structural modeling and design. Biophysical journal, 13(())225-234 2017.
Bohdan Schneider, Blake Alexander Sweeney, Alex Bateman, Jiri Cerny, Tomasz Zok, and Marta Szachniuk. When will ma get its alphafold moment? Nucleic Acids Research, 51(18):9522-9533 2023.
Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus- Robert Müller, Schnet: continuous-tin convolutional neural network for modeling quantum interactions. Advances in neural information processing systems, 30, 2017.
Shantanu Sharma, Feng Ding, and Nikolay V Dokholyan. ifoldma: three-dimensional ma structure prediction and folding. Bioinformatics, 24(17):1951-1952 2008
Phillip A Sharp. The centrality of rna. Cell, 136(4):577-580, 2009.
Tao Shen, Zhihang Hu, Zhangzhi Peng. Jiayang Chen, Peng Xiong, Liang Hong, Liangzhen Zheng, Yixuan Wang, Irwin King, Sheng Wang, et al. E2efold-3d: End-to-end deep learning method for accurate de novo ma 3d structure prediction. arXiv preprint arXiv:2207.01580 2022.
Sumangala Shetty, Snezana Stefanovic, and Mihaela Rita Mihailescu. Hepatitis virus ma: molec- ular switches mediated by long-range ma ma-maa interactions? Nucleic acids research, 41(4):2526- 2540, 2013.
14

---

Published as a conference paper at ICLR 2025
Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjin Wang, and Yu Sun. Masked label prediction: Unified message passing model for eemi-supervised classification. arXiv preprint 20KV::2000.03509 2020.
Jaswinder Singh, Kuldip Paliwal, Tongchuan Zhang, Jaspreet Singh, Thomas Litfin, and Yaoqi Zhou. Improved ma secondary structure and tertiary base-pairing prediction using evolution- ary profile, mutational coupling and two-dimensional transfer learning. Bioinformatics, 37(17): 2589-2600, 2021.
Necla Nisa Soylu and Emre Sefer. Bertoom:: Prediction of 2-0-methylation modifications from ma sequence by transformer architecture based on bert. IEEE/ACM Transactions on Compulational Biology and Bioinformatics, 20(3):2177-2189, 2023.
Eric Strobel, Kyle E Watters, David Loughrey, and Julius Lucks. Rna systems biology: uniting functional discoveries and structural tools to understand global roles of rnas. Current opinion in biotechnology. 19:1 2016.
Marc Teufel and Patrick Sobetzko. Reducing costs for dna and ma sequencing by sample pooling using a metagenomic approach. BMC genomics, 23(1):613, 2022.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang. Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and and translation-equivaraan neural networks for 3d point clouds. arXiv preprint /802. 2018.
Jennifer A Tom, Jens Reeder, William F Forrest, Robert Graham, Julie Hunkapiller, Timothy W Behrens, and Tushar Bhangale. Identifying and mitigating batch effects in whole genome sequencing data. BMC bioinformatics, 18:1-12, 2017.
Hoa Thi Nhu Tran, Kok Siong Ang, Marion Chevrier, Xiaomeng Zhang, Nicole Yee Shin Lee, Michelle Goh, and Jinmiao Chen. A benchmark of batch-effect correction methods for single-cell ma sequencing data. Genome biology, 21:1-32, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, pp. 5998-6008, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30-May 3, 2018, Conference Track Proceedings, 2018.
Hong Wang, Shuyu Wang, Yong Zhang, Shoudong Bi, and Xiaolei Zhu. A brief review of machine learning methods for rna methylation sites prediction. Methods, 203:399-421, 2022.
Wenkai Wang, Chenjie Feng, Renmin Han, Ziyi Wang. Lisha Ye, Zongyang Du, Hong Wei, Fa Zhang, Zhenling Peng, and Jianyi Yang. trrosettana: automated prediction of ma 3d structure with transformer network. Nature Communications, 14(1):7266, 2023.
Xiyuan Wang and Muhan Zhang. How powerful are spectral graph neural networks. In International conference on machine learning, pp. 23341-23362. PMLR, 2022.
Hannah K. Wayment-Steele, Wipapat Kladwang, and Rhiju Das. Rna secondary structure packages ranked and improved by high-throughhuut experiments. bioRxiv, 2020. doi: 1011200/202000522 124511. URL https::///// /tttps::/www.r
05. 29. 124511
Hannah K Styment-Steele, Wipapat Kladwang, Alexandra I Strom, Jeehyung Lee, Adrien Treuille, Alex Becka, Eterna Participants, and Rhiju Das. Rna secondary structure packages evaluated and improved by tightthroughput experiments. Nature methods, 19(10):1234-11 1242, 2022a.
Hannah K Styment-Steele Wipapat Kladwang, Andrew M Watkins, Do Soon Kim, Bojan Tunguz, Walter Reade, Maggie Demkin, Jonathan Romano. Roger Wellington John Nicol, et al. Deep learning models for predicting ma degradation via dual crowdsourcing. Nature Machine Intelligence, 4(12): 174-1 184, 2022b.
15

---

Published as a conference paper at ICLR 2025
Oliver Wieder, Stefan Kohlbacher, Mélaine Kuenemann, Arthur Garon, Pierre Ducrot, Thomas Sei- del, and Thierry Langer. A compact review of molecular property prediction with graph neural networks. Drug Discovery Today: Technologies, 37:1-12, 2020.
Rhondene Wint, Asaf Salamov, and Igor V Grigoriev. Kingdom-wide analysis of fungal protein- coding and trna genes reveals conserved patterns of adaptive evolution. Molecular biology and evolutioon, 39(2)msab337, 2022.
Junjie Xu, Enyan Dai, Dongsheng Luo, Xiang Zhang, and Suhang Wang. Shape-aware graph spec- tral learning, In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pp. 2692-2701, 2024a.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference Learning Representations, 2018.
Xiaojun Xu, Peinan Zhao, and Shi-Jie Chen. Vfold: web server for ma structure and folding thermodynamics prediction. PloS one, 99)ee107044, 2014.
Zongzhe Xu, Ritvik Gupta, Wenduo Cheng, Alexander Shen, Junhong Shen, Ameet Talwalkar, and Mikhail Khodak. Specialized foundation models struggle beat supervised baselines. arXiv preprint arXiv:2411.027990 2024b.
Zichao Yan, William L Hamilton, and Mathieu Blanchette. Graph neural representational learn- ing of ma secondary structures for predicting rna-protein interactions. Bioinformatics, 36 (Supplement.):x276--2244, 2020.
Kevin Yang, Nicolo Fusi, and Alex Lu. Convolutions are competitive with transformers for protein sequence pretraining. Cell Systems, 15(3):286 294,, 2024.
Yuning Yang, Gen Li, Kuan Pang, Wuxinhao Cao, Xiangtao Li, and Zhaolei Zhang. Deciphering 3'utr mediated gene regulation using interpretable deep representation learning. bioRxiv, pp. 2023 09, 2023.
Mehdi Yazdani- Yazdani-Jahromi, Mangal Prakash, Tommaso Mansi, Artem Moskalev, and Rui Liao. HELM: Hierarchical encoding for mRNA language modeling. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL httph:: httpss///ppereeie..o net forum?id-
MMHonUOO10
Xian-bin Ye, Quanlong Guan, Weiqi Luo, Liangda Fang, Zhao-Rong Lai, and Jun Wang. Molecu- lar substructure graph attention network for molecular property identification in drug discovery. Pattern Recognition, 128: 108659, 2022.
Yi Zhang, Jun Wang, and Yi Xiao. 3dma: Building ma 3d structure with improved template library. Computational and structural biotechnology journal, 18:2416-2422 2020.
Yuelin Zhang, Jiacheng Cen, Jiaqi Han, Zhiqiang Zhang, Jun Zhou, and Wenbing Huang, Improving equivariant graph neural networks on large geometric graphs via virtual nodes learning. In Forty- first International Conference on Machine Learning.
Jingjing Zheng, Yurong Qian, Jie He, Zerui Kang, and Lei Deng. Graph neural network with self- supervised learning for noncoding ma- nu-druu resistance association prediction. Journal of Chemi- cal Information and Modeling, 62(15)33666-3644 2022.
Jinhua Zhu, Yingce Xia, Lijun Wu, Shufang Xie, Wengang Zhou, Tao Qin, Houqiang Li, and Tie- Yan Liu. Dual-view molecular pre-training. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 3615-3627, 2023.
16

---

Published as a conference paper at ICLR 2025
APPENDIX OVERVIEW
Appendix A
Related Work.
Appendix Models Overview.
Appendix
Memory and Computational Constraints.
Appendix
Detailed task descriptions.
Appendix
Discussion on Advanced Model Architectures.
Appendix
Additional Experimental Information.
Appendix
Analysis of Noise 3D Structures.
Appendix Reproduction.
Appendix Additional Results.
A
RELATED WORK
RNA property prediction RNA-specific models remain scarce, likely due to limited specialized datasets. Recent advancements in sequence modeling have shown promise. particularly with founda- tion models like RNA-FM (Chen et al. 2022), UTRBERT (Yang et al. (023), and RINALMO (Penić) al. 2024). which use transformer based architectures pre-trained on large RNA sequence corpora to predict various RNA functions and structures. While RNNs and CNNs have been applied to tasks like RNA methylation and protein binding (Wang al. 2022p, they struggle with long-range depen- dencies (Moskalev et al.) 2024). Hybrid models like RNAdegformer (He et al.) 2023). combining convolutional layers with self-attention, improve predictions by capturing both local and global de- pendencies. Although some efforts integrate 2D structures with transformers, explicit 2D and 3D geometric modeling for RNA remains underexplored, with graph-based models mainly focusing on RNA-protein and RNA-drug interaction tasks rather than property prediction (Krishnan al. 2024
Yan et al. 2020: Arora & Sanguinetti 2022 Zheng et al. 2022).
RNA structure prediction RNA 2D structure prediction has progressed from dynamic program- ming methods like Vienna RNAfold (Hofacker al.) 1994) deep learning- -bsee tools like SPOT- RNA2 (Singh et al. (021)) and UFold (Fu et al., 2022). which enhance accuracy by using neural networks and evolutionary data. Models such as F2Ffold (Chen et al. 2020) and RNA-FM (Chen) et al. 2022) employ transformer architectures to achieve state-of-the-aart results in secondary struc- ture prediction.
RNA 3D structure prediction has progressed through ab initio, template-based, and deep learning ap- proaches. Ab initio methods (e-g.. iFoldRNA (Sharma et al. 2008). SimRNA (Boniecki et al. 2016)) balance detail and efficiency but struggle with non-canonical interactions. Template-based models (c.g., FARNAJFARFAR (Das Baker 2007). 3dRNA (Zhang et al.) (0200) depend on existing structures but limited by available data. Deep learning models like DeepFoldRNA (Pearee al. 2022 RhoFold (Shen et al. 2022), TTAFoINA (Baek et al. (0024), and trRosettaRNA (Wang al.
2023) show promise in predicting 3D structures from sequence data but face challenges with novel RNA families due to RNA'S conformational flexibility (Kulkarni et al.) 2023).
Despite these advances, there is gap in applying 2D and 3D modeling techniques to RNA property prediction. Most works focus on ID representations and overlook the potential of geometric infor- mation from 2D and 3D structures. This study the first to systematically explore the benefits and limitations of incorporating explicit structural data in deep learning-
-aaning-based based RNA property prediction.
B MODELS OVERVIEW
1. Transformer1D: The Transformer model a standard Transformer architecture adopted for RNA sequence processing. It includes an embedding layer to convert input tokens into dense vectors, positional encoding (PE) to retain sequence order and a multi-layer Transformer encoder to capture long-range dependencies within the sequence.
17

---

Published as a conference paper at ICLR 2025
2. Transformer1D22: An adaptation of Transformer that integrates sequence and 2D graph structure information. The model encodes each nucleotide and incorporates BPP features, com- bining standard Transformer with positional encoding and convolutional layer the graph adjacency matrix. This convolutional output is added to the attention matrix, enabling the model capture both sequential and structural dependencies.
3. Graph Convolutional Network (GCN): A basic model in graph learning that aggregates and processes node features from local neighborhoods to capture both node characteristics and graph structure, making it effective for tasks like node classification.
4. Graph Attention Network (GAT): Enhances graph convolutions by assigning different impor- tance to neighboring nodes through local attention mechanisms, allowing the model to focus on more relevant nodes during feature aggregation.
ChebNet: A spectral GNN that utilizes Chebyshev polynomials approximate the graph Lapla- cian, enabling graph convolutions with global structural context. This approach allows ChebNet to approximate global graph features with lower computational complexity.
6. Graph Transformer: This model uses Laplacian positional encoding integrate structural in- formation from the graph's Laplacian into node features, which are then processed by Trans- former layers. This enables aligning the sequential nature of transformer layers with graph topol- ogy.
7. GraphGPS: A hybrid model combining GNNs with transformers to capture both local and global graph information. It uses GNNs for local feature aggregation and transformers for long-range dependencies, making it effective for complex graph tasks requiring both local and global context.
8. SchNet: An SE(3)-invariant network designed for molecular property prediction on geometric graphs of atomic systems. operates by modeling interactions through continuous-files con- volutional layers. Since the continuous filter in Schnet is conditioned on distance features, it maintains invariance to rotations and translations of atom coordinates.
Equivvauvva Graph Neural Network (EGNN): An equivariant network for geometric graphs with rotations, translations, and reflections symmetry. The EGNN operates as non- linear message passing between scalar-invariant and vector equivariant quantities.
10. FAENet:
FAENet, or Frame Averaging Equivariant Network, is a lightweight and scalable Graph Neural Network designed for materials modeling. It introduces Stochastic Frame Averaging to achieve =e)-viquivariaaee through data transformations rather than architectural constraints, en- abling superior accuracy and scalability on tasks like molecular and materials property prediction.
11. GVPGNN:
Geometric Vector Perceptron GNN is a graph-based model designed to learn from 3D molecular structures by integrating geometric and relational information. uses Geometric Vector Perceptrons to encode scalar and vector features, enabling effective modeling of both protein structure and interactions.
12. DimeNet:
Directional Message Passing Neural Network leverages directional message passing to encode both distance and angular information between atoms. It utilizes spherical harmonics and Bessel functions for rotation equivariant message representations, enabling precise modeling of molecular interactions.
13. FastEGNN:
FastEGNN introduces virtual nodes to traditional equivariant GNNs, enabling global message passing while maintaining scalability. These virtual nodes are designed to ap- proximate and enhance real node interactions, ensuring E(3)--quivariance for efficient modeling of large geometric graphs.
C MEMORY AND COMPUTATIONAL CONSTRAINTS
In this section, we compare the models based on run times and GPU memory. Both Trans- formerID2D and 3D models (even with nucleotide pooling) encounter out-of-memory (OOM) is- sues when processing longer sequences, such as those in the Fungal dataset (Table I]. This high- lights the need for optimization to handle longer sequences. Figure shows that TransformerID scales poorly in both runtime and memory due to its expensive attention mechanism, and Trans- former DDD faces additional challenges by processing the sequence and adjacency matrix simulta- neously. In contrast, simpler 2D models like GCN, GAT, and ChebNet more efficient. 3D models
18

---

Published as a conference paper at ICLR 2025
also scale poorly with sequence length due to the increasing number of atoms. Overall, 2D models provide a good balance between computational demands and performance for encoding structural information.
104
105
TrendreeedD
TrandomerrDDD
Trandomer1DDD
GPU memory (MB)
104
10a
Gripn
10
EGNN
EGAN
Time
SonNet
SoINet
10
10a
103
10
101.
102.0
102.5
103.0
103.5
102
101.5
102.0
102.5
103.0
103.5
Sequence length
Sequence length
Figure 7: Running time and memory usage comparison across models. (I) Running time vs. sequence length (left): Transformer and TransformerlD2D scale poorly with sequence length due to expensive attention mechanisms and simultaneous processing of sequences and adjacency matrices, while simpler 2D models like GCN, GAT, and ChebNet are more efficient.
(2) Memory usage vs. sequence length (right): Transformer and 3D models face out-of-memory issues with longer sequences, especially in the largest Fungal dataset, whereas 2D models use memory more efficiently, balancing computational demands and structural encoding.
D DETAILED TASK DESCRIPTIONS
To comprehensively evaluate RNA property prediction models, we define five tasks that simulate real-world experimental challenges. These tasks designed assess the performance, efficiency, and robustness of models under various conditions encountered in RNA research and application. Below, we provide detailed explanations for each task, their scientific motivation, and their relevance to practical RNA property prediction.
D.1 TASK 1: IMPACT OF STRUCTURAL INFORMATION ON PREDICTION PERFORMANCE
Motivation: RNA molecules fold into complex secondary (2D) and tertiary (3D) structures that dic- tate their functional properties. While traditional MI approaches often rely solely the nucleotide sequence (ID representation), it is well-known in chemistry and biology that structural informa- tion influences RNA properties (Schlick Pyle, 2017). However, the extent which geometric representations can improve performance of ML models remains unclear.
Task Definition: We evaluate multiple models across three classes (models described in main text):
1D models:
Utilize only the linear sequence of nucleotides.
2D models:
Incorporate RNA secondary structure, such as base-pairing interactions.
Leverage full tertiary structural information, including spatial arrangements of nu- cleotides.
3D models:
By comparing their performance across diverse datasets, this task quantifies the benefits of using geometric information. Specifically, we aim to determine models with explicit 2D or 3D inputs outperform sequence-only models, thereby highlighting the added value of structural representations in RNA property prediction.
Significance: This analysis addresses the gap in existing literature, where the impact of structural information on RNA property prediction remains underexplored. The findings inform whether ad- ditional computational costs associated with structural modeling are justified by performance gains.
19

---

Published as a conference paper at ICLR 2025
D.2 TASK 2: MODEL EFFICIENCY IN LIMITED TRAINING DATA SETTINGS
High-quality RNA datasets with experimentally measured properties are scarce due the technical difficulty and cost of experimental data acquisition (Byron et al. 2016 Teulel & Sobetzko 2022). In such scenarios, models must be sample-efficient, learning effectively from small datasets.
Motivation:
Let D = {X,Y } represent a dataset, where X is the RNA input and Y is the prop- erty label. For this task, we define subsets Da by sampling fraction a of the original dataset. We evaluate models trained on varying a values (e.g., 25%, 50%, 75%) to assess how performance scales with reduced training data.
Task Definition:
Significance:
This task simulates real-world scenarios where large labeled datasets are unavailable. evaluates the ability of different models to generalize from limited data, providing insights into their efficiency and practical applicability.
D.3 TASK 3: PERFORMANCE WITH PARTIAL SEQUENCE LABELING
Motivation: In nucleotide- RNA datasets, labels for molecular properties (e.g., binding affini- ties or reactivity) are often available only for a subset of nucleotides in sequence (Wayment-Stecld ct al. 2022b). This is due to the high cost of annotating every nucleotide experimentaly.
Task Definition: We consider datasets where only the first é-nucleotide positions in each sequence are labeled, representing partial sequence labeling. Models are trained using these incomplete labels and evaluated their ability to predict properties across the full sequence. Performance is assessed using datasets with varying é-label fractions.
Significance: This task reflects critical real-world challenge where complete annotations available. It assesses a model's ability to generalize from partial labels to unseen portions of RNA sequences, valuable capability for applications with sparse experimental data.
D.4 TASK 4: ROBUSTNESS TO SEQUENCING NOISE
Motivation:
RNA sequencing technologies often introduce noise in the form of random nucleotide
errors (e.g., insertions, deletions, or substitutions) (Ozsolak Milos 201T: Fox et al. 2014). This noise propagates through derived RNA structures (2D and 3D), potentially degrading model perfor- mance. Understanding how well models handle noise is crucial for their reliable deployment.
Task Definition: To simulate realistic sequencing errors, we introduce controlled noise levels into RNA sequences during both training and testing. Models are evaluated for their ability to maintain performance under consistent noise conditions, representing a scenario where noise characteristics are stable across experimental phases.
Significance: This task assesses a model's robustness in practical settings where sequencing noise unavoidable. It provides insights into the resilience of RNA models to variations in input quality.
D.S TASK 5: GENERALIZATION OUT-OF-DISTRRIUTIONN (OOD) DATA
Motivation: RNA models are often trained on high-quality experimental datasets but deployed in conditions where data characteristics differ significantly due to variations in sequencing platforms or protocols (Tran et al. 2020: Tom et al. 2017). This mismatch can lead to performance degradation.
Task Definition:
We evaluate models trained on clean RNA data and tested on datasets with higher levels of noise (representing OOD conditions). Performance metrics analyzed function of increasing noise, quantifying the models' ability to generalize to unseen distributions.
Significance: This task mirrors real-world deployment scenarios where models encounter noisy or biased data. It highlights the limitations of models trained on idcalized datasets and informs strategies for improving generalization under OOD conditions.
20

---

Published as a conference paper at ICLR 2025
E DISCUSSION ON ADVANCED MODEL ARCHITECTURES
E. ENHANCING 3D RNA PREDICTION WITH DEGREE STEERABLE FEATURES
Apart from the 3D models considered in this work, models using high-degree steerable features such as TFN (Thomas et al. 2018). MACE (Batatia et al. 2022) and NequlP (Batzner et al. 2022) repre- sent an important aspect of equivariant models, with potential enhance model expressiveness by incorporating higher-order information. While memory constraints have limited their practical im- plementation in our current work, techniques such as scalarization of high-degree steerable features, as demonstrated in works such as HEGNN (Frank et al.) (024) and SO3krates (Cen et al.) (024), could address this challenge. For instance, HEGNN can improve the expressivity of equivariant GNNs by mitigating expressivity degeneration representations. Although 3D RNA graphs are too complex to be considered symmetrie, the abil- ity to bandle high-degree steerable features in large RNA graphs, with hundreds of nucleotides and thousands of atoms, remains valuable and has the potential to improve performance. Future work explore such methods to improve 3D RNA prediction.
symmetric graphs and leveraging higher-order
E.2 MITIGATING SEQUENCING NOISE
Notably, most studies in this field do not explicitly address sequencing noise common for RNA in model architecture design, prompting a need explore effective strategies for mitigating its impact. An effective way to handle noise can be through ensemble methods. For instance, combining ID, 2D, and 3D models by independently learning representations and integrating them via attention mechanisms that dynamically weigh each modality based on task relevance and noise can leverage the robustness of ID methods while benefiting from the strengths of 2D and 3D approaches and be good direction for future research.
F ADDITIONAL EXPERIMENTAL INFORMATION
DATASET STATISTICS
Here, present the statistics for each dataset used in the paper in Table 2 The datasets are catego- rized as small, medium, or large based on the number of sequences and sequence length. "Target" refers the task the dataset is designed predict, and "# Avg. Atoms" indicates the average number of atoms used in 3D models.
Table 2: The statistics of Tc-Riboswitches, Ribonanza, COVID, and Fungal datasets.
Tc-Riboswitcees
Ribonanza
COVID
Fungal
Dataset Size
Small RNA-level Switching Factor 355
Medium Neeleotide-Ievel
Medium Nucleotide-leve Degradation 4082
Large
Task Level
RNA-level
Target Sequences Sequence Length
Degradation 2260
Expression
7089
66 75
177
107 - 130
150 3063
# Labels
2
3
1
# Avg. Atoms
1531
3791
2598
N/A
(for 3D models)
F.2 COMPARISON OF PARTIAL TRAINING DATA AND PARTIAL SEQUENCE LABELING
To clarify the differences and provide more detailed explanation, we illustrate two experiments: partial training data and partial sequence labeling (Figure 8).
F.3 DETAILS OF NOISY EXPERIMENTS: ROBUSTNESS AND GENERALIZATION
To create the noisy datasets, we vary the noise ratio r across the values (0.05, 0.1, 0.15, 0.2, 0.25, 0.3). For each given noise ratio, independently mutate the nucleotide at each position in
21

---

Published as a conference paper at ICLR 2025
varying fraction of sequence labelling 0.2 1.0
Seq
Seq
train sequences
train sequences
varying fraction of training data 0.25 1.0
validation sequences
validation sequences
test sequences
test sequences
Seq
Seq
(a) Varying fraction of training data.
(b) Varying fraction of sequence labeling.
Figure 8: Comparison of partial training data and partial sequence labeling. The orange arrows indicate the varying components. (a) utilizes full nucleotide labels but trains on varying fractions of RNA sequences (0.25, 0.5, 0.75, 1.0). (b) uses all training sequences but with varying fractions of nucleotide labels (0.2, 0.4, 0.6, 0.8, 1.0). Therefore, (b) is only for nucleotide-level tasks.
sequence with probability r. as illustrated in Figure g The resulting mutated sequence is then passed to the 2D and 3D prediction tools generate the corresponding structures. Figure gives comprehensive illustration of getting noisy ID, 2D, and 3D structures.
For the robustness experiments, all training, validation, and testing are conducted on the noisy datasets. In contrast, for the generalization experiments, the model is trained and validated on clean datasets, and its performance tested on noisy datasets with varying noise ratios.
c
CCCUUCAAA
G
U
U
A
U
A
r
c
Figure 9: Mutation the nucleotide at each position independently with the probability
F.4 ANALYSIS OF TRANSFORMER ID AND TRANSFORMER D2D
To further validate that incorporating structural information contributes to the final results, we an- alyze the attention maps generated by TransformerID and TransformerID2D. Fig. I0 illustrates the average attention maps across all heads before the final output layer for both the models for a randomly selected RNA sequence. The attention maps of Transformer1 exhibit striking similarity to both the adjacency matrix and the BPP matrix, whereas the attention maps from the standard Transformer model seem to suggest that the model does not learn to attend to the structural features. Moreover, attention maps of the models and the true adjacency matrix and BPP for all sequences the COVID dataset. The results, reported in Table B show that MaanffrmerrD2DD achieves much higher sim- ilarity scores compared to the ID Transtormer alone. This reinforces the conclusion that explicit encoding of structural information essential for improved model performance.
quantify this observation by computing the cosine similarity between the
22

---

Published as a conference paper at ICLR 2025
0
100
10
10.
20
20
30
30
10-1
40
40
50
50
60
60
70
70
10-2
80
80
90
100
100
10-3
110
110
0
10
20
30
40
50
60
70
80
90
100
110
10
20
30
40
50
60
70
80
90
100
110
(a) Transformer1D attention matrix.
(b) Transformer ID2D attention matrix.
0
10
10
20
20
30
30
40
40
50
50
60
60
70
70
80
80
90
90
100
100
110
110
10
20
30
40
50
60
70
80
90
100
110
10
20
30
40
50
60
70
0
90
100
110
(c) Basc pair probability matrix.
(d) Graph adjacency matrix.
Figure 10: The heatmaps of matrices in Transformer1D and Transformer1D20) Attention maps from Transformer ID2D exhibit striking resemblance to both the adjacency matrix and BPP matrix, highlighting the model's ability to learn structural features. In contrast, the standard Transformer struggles with this task, as shown by lower cosine similarity scores, reinforcing the conclusion that explicitly encoding structural information is crucial for enhanced model performance.
Table 3: Cosine similarity values for different models. Cosine similarity scores between the attention maps and the true adjacency and BPP matrices for all sequences in the COVID dataset demonstrate that Transformer DDD significantly outperforms the standard Transformer. These sults underscore the importance of explicitly encoding structural information for superior model performance.
Model
Cosine similarity adjacency
Cosine similarity BPP
TransformerID
0.107
0.090
Transformer ID2D
0,448
0.672
G ANALYSIS OF NOISE IN 3D STRUCTURES
As mentioned in the main context, predicted 3D structures consistently exhibit noise. In this section, we analyze this issue from two perspectives: sensitivity to sequence length and variability across different prediction tools.
23

---

Published as a conference paper at ICLR 2025
G.1 IMPACT OF SEQUENCE LENGTH ON 3D STRUCTURE PREDICTION NOISE
To investigate the hypothesis that longer sequences result in greater noise in 3D structure predictions, we randomly selected a COVID and Tc-Riboswitches dataset sequence and generated structures using four state-of- offte--f--hh-arr the-art art 3D structure prediction tools: RhoFold (Shen et al.) (022). RNACom- poser (Xu al. 2014), trRosetta (Baek et al., 2024), and SimRNA (Boniecki et al.) 20T6). High variability among these predicted structures would indicate significant uncertainty in absolute atom positions. We quantified this noise by aligning the structures using the Kabsch algorithm (Kabsch) 1976) and computing the pairwise RMSD values, resulting in a 4x4 matrix showing structural devia- tions between each pair of tools (see Table 4). The observed pairwise RMSD values ranged from 16 to 45 A for the COVID dataset and from 1 to 15 Å for the Tc-Riboswitches dataset, reflecting sub- stantial variability and suggesting considerable noise in the 3D predictions. This level of structural inaccuracy likely contributes the poorer performance of 3D models. However, found that 3D) models outperform ID models for shorter sequences, such as those in the Te-Riboswitches dataset (67 to 73 nuclcotides long). This improved performance is due the lower noise in 3D predictions for shorter sequences, phenomenon supported by previous studies (Nithin et al. 2024; Ponce- Salvatierra al., 2019) and also exhibited by the comparatively smaller RMSD values reported in Table for Rioosbsswst dataset. The reduced complexity of shorter sequences allows 3D mod- els to capture structural details more accurately, thereby enhancing performance and validating that accurate 3D structure encoding can outperform ID models.
Table 4: Pairwise RMSD values (in Å) between 3D structure prediction tools for the COVID dataset (left) and Te-Riboswitches dataset (right). The results indicate larger noise in predictions for longer COVID sequences.
COVID
RhoFold
trRosetta
SimRNA
Composer
RhoFold
0
39.05192
44.76146
45.45994
trRosetta
39.05192
0
22.54974 0
18. 18319
SimRNA
44.76146
22.54974
16.73399
Composer
45.45994
18.57359
16.73399
Te-Ribo
RhoFold
trRosetta
SimRNA
Composer
RhoFold
0
14.338
11.996
15.056
trRosetta
14.338
0
12.932
14.916
SimRNA
11.996
12.932
14.243
Composer
15.056
14.916
14.243
0
G.2 IMPACT OF DIFFERENT 3D PREDICTION TOOLS
In this section, we demonstrate the significant differences in 3D structures predicted by various tools. We compare the 3D structure obtained from RhoFold (Shen method, with those predicted by RNAComposer (Xu et al. 2014). trRosetta (Back et al.) (024), and SimRNA (Boniecki al. 2016). Each structure visualized side by side with RhoFold in Figure) to facilitate more intuitive comparison. As observed, these structures predicted by each tool vary considerably.
2022), which serves as our default
H REPRODUCTION
This section outlines the necessary details to reproduce all experiments discussed in this paper. The code will made publicly available upon acceptance.
H. TRAINING DETAILS
All experiments were conducted on a single NVIDIA A100 GPU. For each baseline, hyperparame- ters were optimized using Optuna (Akiba al. 2019), restricting the search to models with fewer
24

---

Published as a conference paper at ICLR 2025
tholo
Tholol
amna
iRossta
(a) RhoFold vs. RNAComposer
(b) RhoFold vs. SimRNA
(c) RhoFold trRosetta
Figure 11: Comparison of RhoFold against other 3D structure prediction tools on an example se- quence from Te-Riboswitches dataset.
than 10 million parameters that fit within the memory constraints of an 80GB NVIDIA A 100 GPU.
Most baseline implementations were sourced from Py PyTorch Geometric (Fey & Lenssen 2019). The Transformer1D model was adapted to DDD as detailed in the paper. For EGNN, utilized the authors' implementation (Satorras et al., 2021), and for SchNet, the implementation from (Joshi al.] 2023) was used.
H.2 HYPERPARAMETERS
This section provides comprehensive overview of the hyperparameters used in each baseline model, facilitating reproducibility and understanding of the model configurations.
Common hyperparameters across these models include in cwc.chaaeess.... which specifies the number of input features, hi dden, which determines the number of hidden units in each hidden layer, and out out-channels, which defines the number of output features. The L parameter controls the num- ber of layers in the network, and the dropout parameter sets the dropout rate for regularization. The 1r parameter specifies the learning rate, and we wit.tgeeheeeee sets the weight decay for regu- larization of the optimizer. For graph-level tasks, the pool parameter specifies the pooling method, which can be mean, max, add.
Transformer1D is standard Transformer architecture for RNA sequence processing. It in- cludes an embedding layer to convert input tokens into dense vectors, positional encoding (PE) to retain sequence order and multi-layer Transformer encoder to capture complex de- pendencies within the sequence. There are some hyperparameters from the original trans- former (Vaswani et al., 2017). nhead, which defines the number of attention heads in each Transformer layer; num.encoder: layers, which controls the number of encoder layers in the Transformer; modeel,, which determines the dimensionality of the embeddings and the model; fim__feeffooo feeeforwwar d, which sets the dimensionality of the feedforward network model. To shrink search space, we set model and sim.feedfooo oonforrwrdd as the same with a new hyperparameter hidde.
Transformer1D21 is adaptation of Transformeri D that integrates both sequence and 2D graph structure information. In addition encoding each nucleotide, the model incorporates base pair probabilities (BPP) features for each nucleotide. It combines standard Transformer with positional encoding and convolutional layer applied to the graph adjacency matrix. This convolutional out- put is added to the Transformer's attention matrix, allowing the model to incorporate graph structure into its attention mechanism. This design captures both the sequential and structural dependen- cies in RNA data, improving predictive performance. The unique hyperparameter for this model is kernel.st ze, which specifies the size of the convolutional kernel.
GAT includes the unique hyperparameters gat heads, which specify the number of attention
heads in each GAT layer.
25

---

Published as a conference paper at ICLR 2025
ChebNet model has the unique hyperparameter power, which specifies the polynomial order for the Chebyshev convolution.
GraphGPS and Graph Transformer includes heads, which specifies the number of attention heads in each layer, and pe_dim, which defines the dimensionality of positional encoding.
EGNN and SchNet are 3D models that operate layers and nucleotide layers. The two types of layers are connected through nucleotide pooling. Atom layers use atoms nodes, while nucleotide layers use nucleotides as nodes. Both the atom layer and nucleotide layer employ a point cloud setting and calculate edges based on the distance between two nodes. An edge considered to exist if the distance smaller than certain threshold. Therefore, EGNN and SchNet share the following hyperparameters: I_atom, which denotes the number of atom layers; L.nt, which specifies the number of nuclcotide layers; thresholdLat om, which is the threshold for edges in atom layers; and threshold.nt, which is the threshold for edges in nucleotide layers.
granularities within the network: atom
For SchNet, the unique hyperparameters include numfiilers, which refers to the number of filters used convolutional layers, and num.gaussiann which indicates the number Gaussian functions used for radial filters. For more detailed explanation of these hyperparameters, please refer to (Schütt et al. (017).
To ensure a fair comparison, the best hyperparameter configuration for each method was selected based on validation set performance. We report the mean performance and standard deviation across 5 random splits the test set. For the COVID and Ribonanza datasets, performed hyperparam- eter searching only on the COVID dataset and applied the same configuration to Ribonanza, as the two datasets share similar properties. The optimal hyperparameters shown in Table
I ADDITIONAL RESULTS
and in main text. In this section, we present the additional results supporting Figures 2
1.1 IMPACT OF DATA AVAILABILITY
The detailed results of partial training data from Figure 2 are shown in Tables and
1.2 IMPACT OF PARTIAL LABELING
The detailed results of the partial labeling sequence from Figure 3 are shown in Tables 9 and 10
1.3
ROBUSTNESS TO SEQUENCING NOISE
The results of the robustness experiment from Figure 5 are shown in Tables
and 14
1.4 GENERALIZATION TO OOD DATA
The results of the generalization experiment from Figure are shown in Tables 15 16 and
26

---

Published as a conference paper at ICLR 2025
Hyperparameter
COVID & Ribonanza
Te-riboswitches
Fungal
0.001
0.0005
0.001
weight.decay
0
0.0005
0.0005
hidden
128
32
TransformerI
nhead
8
8
8
num_encoder.layers
8
6
pool
mean
mean
0.001
0.005
weight.decay
hidden nhead
256
Transformer1
16
OOM
encoder
8
pool kerneLsize
5
0.001
0.0001
weight decay hidden
1024
512
512
GCN
7
3
dropout pool
0.3
0.1
0.7
/
add
0.001
0.005
0.0001
weight.decay
0.0005
hidden
512
256
256
ChebNet
5
dropout
0.3
0.3
0.3
power
6
2
pool
mean
0.001
0.005
0.0005
weight decay hidden
0.0005
256
1024
256
7
dropout
0.1
0.1
0.3
heads
pool
add
add
0.001
0.005
0.005
weight decay hidden
0.0005
Gruph Transformer
128
256
7
heads pool
2
add
mean
0.001
le-5
0.0005
weight_decay
0
0.0005
0.0005
hidden
256
256
512
GraphGPS
heads
pool
add
max
0.0005
0.001
weight_decay hidden L_atom 1. nt
0
256
256
EGNN
3
OOM
1
threshold atom
1.6
1.6
threshold_nt
22
0.0005
0.001
weight_decay hidden L_atom
0
0
128
128
1
SchNet
I.ntt threshold atom
00M
1.6
1.8
threshold.nt
44
num filters
128
256
num san_gaasssans
Table 5: Best hyperparameters for each model and dataset. Hyperparameters are searched by Op- tuna. COVID and Ribonanza share the same hyperparameters.
27

---

Fungal Published as a conference paper at ICLR 2025
Table Performance (MCRMSE) of different models across various training data (mean standard deviation) fractions on COVID dataset.
COVID
0.25
0.50
0.75
1.00
Transformer I
0.429 + 0.018
0.410 + 0.016
0.375 + 0.018
0.361 + 0.017
Transformer
DDD D2D
0.345 = 0.010
0.330 = 0.011
0.306
0.014
0.305 0.012
GCN
0.389 = 0.008
0.377 = 0.009
0.358 - 0.014
0.359
GAT
0,352 = 0.011
0.342 = 0.009
=
0.006
ChebNet
0.320 + 0.011
0.309
0.009
0.286 + 0.018
0.279 + 0.007
Graph Transformer
0.356 + 0.008
0.344 0.007
0.324 + 0.016
0.318 + 0.008
GraphGPS
0.367 0.018
0.362
0.005
0.344 + 0.010
0.332 0.013
EGNN
0.398 + 0.001
0.391 = 0.013
0.368 + 0.013
0.364 + 0.003
SchNet
0.419 = 0.003
0.414 0.011
0.392 = 0.011
0.390 0.006
FastEGNN
0,460 + 0.018
0.452 = 0.015
0 0.016
=
0.444 + 0.003
Table 7: Performance (MCRMSE) of different models across various fractions of training data (mean standard deviation) on Ribonanza dataset.
Ribonanza
0.25
0.50
0.75
1.00
Transformer
0.777 + 0.014
0.740
0.005
0.739 = 0.001
0.705
0.015
Transformer D2D
DDDD
0.630 + 0.016
0.553
0.015
0.541 + 0,018
0.514 0.004
GCN
0.668 + 0.018
0
0.006
GAT
2.6008
0.553
0.544
0.504
ChebNet
0.537 0.019
0.494
0.499 0.007
0.468 0.002
Graph Transformer
0.567 0.019
0.529
0.013
0.529 + 0.010
0.515 + 0.001
GraphGPS
0.581 + 0.021
0.529
0.015
0.540 + 0.004
0.523 + 0.003
EGNN
0.694 + 0.010
0.650
0.010
0.632 0.015
0.619 0.007
SchNet
0.768 + 0.008
0.724 0.013
0.715 = 0.015
0.685 + 0.006
FastEGNN
0.795 + 0.007
0.788 0.002
0.774 1 0.016
0.753 0.015
Table 8: Performance (MCRMSE) of different models across various fractions of training data
(mean + standard deviation) on Fungal dataset.
0.25
0.50
0.75
1.00
Transformer
1.510 +: 0.006
1.446 + 0.014
1,475 + 0.035
1.417 + 0.005
GCN
1.243 + 0.064
1.244 0.128
1.151 0.077
1.192 0.077
GAT
1.211 = 0.125
1.168 0.033
1.106
1.112 0.035
ChebNet
1.125 = 0.097
1.011 0.010
1.008 = 0.009
0.973 0.003
Graph Transformer
1.415 + 0.014
1.331 0.163
1.306 + 0.127
1.317 + 0.002
GraphGPS
1.289 + 0.071
1.377 + 0.127
1.357 + 0.106
1.025 0.081
Table 9: Performance (MCRMSE) of different models across various fractions of sequence labeling (mean standard deviation) COVID dataset.
COVID
0.2
0.4
0.6
0.8
1.0
TransformerlD
0.654 1: 0.040
0.559 : 0.011
0.480
0.004
0.429 + 0.034
0.361 + 0.017
Transformerl D2D
0.502 + 0.002
0.470 + 0.052
0.374
0.007
0.325 + 0.006
0.305 + 0.012
GCN
0.450 + 0.012
0.416 + 0.012
0.397
0.012
+ +
0.359 + 0.009
GAT
0.411 1 0.010
0.376 - 0.012
0.360 0.012
0.336 1 0.009
0.315 1 0.006
ChebNet
0.380 0.007
0.344 0.008
0.009
0.299 1
0.279 0.007
Graph Transformer
0.415 1 0.012
0.379 1 0.011
0.001
0.338 1 0.004
0.318 1: 0.008
GraphGPS
0.428 - 0.015
0.400 = 0.017
0.016
0,351 + 0.007
0.332 - 0.013
EGNN
0.436 0.014
0.421 = 0.010
0.407
0.004
+.006
0.364 0.003
SchNet
0.442 0.004
0.429
= 0.005
0.413
+
0.407 0.005
0.390 0.006
FastEGNN
0.497 0.004
0.490 + 0.007
0.007 0.466
0.469 + 0.009
0.444 0.003
28

---

Published as a conference paper at ICLR 2025
Table 10: Performance (MCRMSE) of different models across various fractions of sequence labeling (mean + standard deviation) Ribonanza dataset.
Ribonanza
0.2
0.4
0.6
0.8
1.0
Transforeer
1.137 0.163
0.929 + 0.023
0.823
0.018
0.742 1 0.013
0.705 1: 0.015
Transformer D22
0,859 1 0.025
0.638 1 0.013
0.632 1 0.028
0.568 1 0.013
0.514 1: 0.004
GCN
1,191 + 0.031
1,026 = 0.079
1.111 + 0.206
1.070
0.595 + 0.006
GAT
0.703 =
0.560 0.12 0.025
ChebNet
0.614
0.546 = 0.008
0
=
0.468 = 0.002
Graph Transformer
0.719 +: 0.043
0.607 = 0.020
0.584
0.552 + 0.013
0.515 - 0.001
GraphGPS
0.743 + 0.058
0.663 + 0.026
0.627 0.024
0.651 + 0.026
0.523 + 0.003
EGNN
0.882 + 0.010
0.722 # 0.021
0.687 + 0.008
0.665 + 0.013
0.619 + 0.007
SchNet
0,810 = 0.002
0.781 = 0.009
0.750 + 0.009
0.725 = 0.004
0.685 = 0.006
FastEGNN
1.223 1 0.008
0.929 = 0.008
0.860 0.004
0.837 = 0.004
0.753 #0.015
Table 11: Performance (MCRMSE) of various models in robustness experiments on the COVID dataset (mean + standard deviation).
Model
0.00
0.05
0.10
0.15
0.20
0.25
0,30
TransformerID
0.361
0.017
0.386
0.015
0.400 = 0.010
0,409
0.006
0.428 + 0.005
0.455
0.003
0,449
0.011
Transormer DD2D
0.305
0.012
0.373 + 0.007
01403 + 0.007
0.428 + 0.011
0.444 + 0,015
0.457
0.009
0,463
-0.009
GCN
0.009
0,464
0.012
0.501
GAT
0.315 0.006
0.409 0.009
0.448 + 0.011
0.471 + 0.010
0.484 0 0.0012
0.494
0.011
0.500
0.010
ChehNet
0.279 0.007
0.368 0.003
0.423 0.009
0.456 0.007
0.471
0,000
0,481
0,010
0.487
0.008
Graph Transfrmer
0.318 0.008
0.403
0.008
0,41
0.012
0.467
0:011
0.480
0.012
0.487
0.000
0.494
GraphGPS
0.332 0.013
0.408 :: 0.012
0.44] :: 0.010
0.464
1: 0.014
0.475
1 0.012
0.484
1: 0.012
0.487
0.008
EGNN
0.467
0.009
0.009
0.011
0.012
0.511
SchNet
0.390 0.006
0.447
0.012
0.477 + 0.011
0.496 0.009
0.507
0.014
0.513 0.012
0.517 0.010
FaMEGNN
0.444 0.003
0.49283 1 0.008
0.516 0.005
0.516 0.004
0.522 0,002
0.527 0.003
0.540 0.004
Table 12: Performance (MCRMSE) of various models in robustness dataset (mean # standard deviation).
eebbustnesseepprmmeentt experiments on the Ribonanza
Model
0.00
0.05
0.10
0.15
0.20
0.25
0.30
TransformerI
0.705 + 0.015
0.733 + 0.010
0.769
0.014
0.782 + 0.005
0.794 0.010
0.805
00.017
0.823 + 0.005
TransfrmerlDaa
0.514 1. 0.004
0.635
0.004
0.714
1. 0.014
0.763 1 0.008
0.790 0.009
0.811
0.014
0.830 1 0.008
GCN
0,595 1: 0.906
0.750 :: 0.014
0.846
0.008
0.893 : 0.003
0,912 1: 0.005
0.924
0.005
0.929 1 0.005
GAT
0.534 = 0.006
0.691 = 0.015
0,785
0.006
0.850
= 0.003
0.877 + 0.001
0:904
0.007
0.915 + 0.007
ChebNet
0,468 + 0.002
0.611 0.012
0.720
0.006
0.802
0.011
0,841 0.003
0.876
0.007
0.897 0.008
Graph Transformer
0.515 :: 0.001
0.670 :: 0.011
0,768
0.011
0.833 : 0.008
0,870 0.006
0.893
0.010
0.908 1 0.007
GraphGPS
0,523
0.017
0.006
0.004
0.006
EGNN
0.619 + 0.007
0.764
0.003
0.847
0.003
0.889
0.005
0.904 + 0.003
0.917
0.000
0.922
+ 0.002
SchNet
0.685 1. 0.006
0.814
0.006
0.873
0.004
0.897
0.004
0,908
0.004
0.918
0.005
0.922 0.005
FaSEEONN
0.753 : 0.015
0.857 - 0.001
0.884 1 0.005
0.908 = 0.001
0.914 1 0.004
0.920 0.003
0.922 1 0.003
Table 13: Performance (MCRMSE) of various models in robustness experiments on the Fungal
dataset (mean + standard deviation).
Model
0.00
0.05
0.10
0.15
0.20
0.25
0.30
TransfomeerD
LA417 0.005
1.545
0.045
1.546
1546=20.04
1.546
0.046
1.543
0.048
1.543
0.051
1.550 : 0.041
GCN
1.192 + 0.077
1.222 + 0.044
1.255 4: 0.002
1.277 + 0.014
1.269
0.011
1.328
0.026
1.294 + 0.025
GAT
1.112 0.035
1.244 0.074
1.391
0.155
1.334
0.099
1.468
0.056
1.444
0.094
1.446 1 0.092
ChebNet
0.978 1: 0.000
1.031
0.003
1,99
0.007
1.108
1 0.009
1.243
0.005
1210
:: 0.014
1.269 1 0.007
Graph Transformer
1.342 + 0.087
1,267
+ 0.116
1.409
0.046
1.426
4 0.051
1.442 + 0.038
1.413 0.020
1.450
+ 0.018
GraphGPS
1.083 0.131
1048
0.095
1.133
0.057
1.199
0.040
0.071
1256 0.016
1.328 0.041
Table 14: Performance (MCRMSE) of various models in robustness experiments on the Te- riboswitches dataset (mean + standard deviation).
Model
0,00
0.05
0.10
0.15
0.20
0.25
0.30
TransfomeelD
0.705 =: 0.079
0.698 :: 0.071
0.736 - 0.004
0.672 = 0.003
0.739 1: 0.008
0.694 = 0.011
0.675 4 0.047
Transformer
0.633 = 0.001
0.697 = 0.031
0,742 0.003
0.708
= 0.008
0.681
0.000
0.762
0.022
0.738 = 0.016
GCN
0.701 + 0.004
0.758 + 0.003
0.747
0.005
0.744
+ 0.004
0,733
+ 0.013
0.740 - 0.011
0.765 + 0.009
GAT
0,685
0.749 0.017
0.770
0.047
0.734
10021
0.009
0.753
0.747
ChebNet
0.621 + 0.022
0.766 + 0.014
0.754
0.021
0.738 + 0.014
0.763 + 0.099
0.778 = 0.048
0.739 + 0.004
Graph Transformer
0.703 1: 0.054
0.754
0.005
0.754
0.006
0.773 1 0.008
0,810 1 0.087
0.742 0.004
0.754 1 0.005
GraphGPS
0,702 1 0.028
0.785 1 0.053
0.805
0.092
0.750 1 0.006
0.755 1: 0.060
0.769 : 0.031
1.078 1 0.469
EGNN
0.663 : 0.010
0.750
0.001
0.739
2 0.002
0.749 = 0.005
0,749 1 0.001
0.749 0.001
0.756 1 0.013
SchNet
0.655 - 0.038
0.762
0.005
0,742
0.002
0.771
+ 0.037
0.746
+ 0.005
0.791
0.016
0.750 0.016
FastEGNN
0.710 + 0.010
0.733 + 0.007
0,749 + 0.006
0.748 + 0.006
0,752 + 0.008
0.758 + 0.017
0.761 + 0.010
29

---

Published as a conference paper at ICLR 2025
Performance (MCRMSE) of various models in generalization experiments on the COVID dataset (mean + standard deviation).
Table 15:
Model
0.00
0.10
0.15
0.20
0.25
0.30
TransformerID
0.361 1. 0.017
0.382
0,022
0.402 1 0.018
0.436
0.015
0.461
0.021
0.478
0.015
0,494 1. 0.016
TransformerIDDD
0,305 +: 0.012
0.406 2 0.016
0.466
: 0.017
0.513
0.016
0.545 = 0.027
0.581 = 0.025
0.596 1: 0.018
GCN
0.359 = 0.009
0.459 + 0,011
0.508
= 0.011
0,550
0.55000.01
0.014
0.572 0.016
0.001 = 0.014
0.612 0.008
GAT
0.315
0,437 0.013
0.490
0.528 0.008
0.585
0.580 0.015
0.592 0.012
ChehNet
0,279 1. 0.007
0.415 1 0.017
0.483
1 0.023
0.538 0.025
0.571 0.029
0.604 1. 0.030
0.621 1 0.028
Graph Transfommr
0.015
0.501
0.018
0.015
0.014
GraphGPS
0,332 + 0.013
0.443
0.011
0.496
0.006
0.536 = 0.005
0.359 = 0.010
0.586 + 0.007
0.593 : 0.005
EGNN
0.365 0.011
0,458
0.014
0.504 0.018
0.530 0.020
0.549 0.021
0.565 0.022
0.572+0.022
SchNet
0.390 1: 0.006
0.457 + 0.011
0.491 4: 0.008
0.515 0.007
0.531 0.10
0.543
0.009
0.556 1: 0.002
FastEGNN
0.444 + 0.003
0,491 + 0.020
0,51
+ 0,014
0.524 0.009
0.533
0.006
0,541
0.003
0.543+00.00
Table 16: Performance (MCRMSE) of various models in generalization experiments on the Ribo- nanza dataset (mean + standard deviation).
Model
0.00
0.05
0.10
0.15
0.20
0.25
0.30
ID
0,705 1: 0.015
0.747
0.005
0.796
0.006
0.828 1 0.008
0,860
1: 0.013
0.886
0.013
0.899 1: 0.003
DDD
0.514 = 0.004
0.685 = 0.014
0.857 = 0.008
0.986 = 0.015
1.055
0.007
1.142 0.020
1.192 : 0.034
GCN
0.595 0.906
0.857
0.018
0.993 0.012
1.054 0.034
1,094
+194++0.04
1.129
0.061
1.139 + 0.075
GAT
0,534 : 0.006
0.778 = 0.021
0.919 2 0.030
1.003 = 0.056
1.047
1 0.073
1.76 = 0.082
1.093 + 0.091
ChehNNt
0,468 0.002
0.699
0.005
0.881
0.038
1.025 0.095
0.111
1,165
1,200 0.220
Graph Transformer
0,515 + 0.001
0.752
0.005
0.930
0.013
1.067 + 0.036
1.124 + 0.003
1.194 0.080
1.224 + 0.104
GrapbGPS
0.523 J. 0.003
0.771
0.026
0,958
0.068
1.087 1 0.142
1.116 0.195
1.154 0.202
1.165 0.196
EGNN
0,691 :: 0.906
0.815 = 0.004
0.755
0.026
1.138 = 0.078
1.228 1 0.079
1.300 0.173
1.395 1 0.187
SchNet
0.685 = 0.006
0,844
0.006
0,949 0.022
1.06 = 0.035
1.157 + 0.069
1.270 0.049
1,342 + 0.117
FastEGNN
0.753 0.015
0.857
0.001
0912 0.007
0.939 0.011
0,940 + 0.010
0.952 0.008
0.957 + 0.001
Table 17: Performance (MCRMSE) of various models in generalization experiments on the Fungal dataset (mean + standard deviation).
Model
0.00
0.05
TransforeerD
1,417 + 0.008
1.575 + 0.002
1,575
0.002
1.575 + 0.002
1.575 + 0.002
1.575
0.002
1.575 + 0.002
GCN
1.192
0.061
1.256
0.050
1,280
0.052
1,290
1.328
0.046
GAT
1.112 1: 0.035
1.262
:: 0.115
1.284
0.1002
1.312 : 0.092
1.334
1: 0.087
1.364 0.080
1.373 1: 0.072
ChehNet
0,973
0.008
1.257
1.382 = 0.022
0.020
1.686
0.033
Graph Transformer
1.317 1. 0.002
1.407
0.053
1.418
1 0.046
1.427 1. 0.040
1.439
1. 0.033
1.447 0.029
1.456 1. 0.028
GraphGPS
1.025
0.081
1.083
0.011
1.160
0.006
1.217 0.012
1.316
0.026
1.405 0.040
1,463 1 0.052
Table 18: Performance (MCRMSE) of various models in generalization experiments on the Te- riboswitches dataset (mean + standard deviation).
Model
0.00
0.05
0.10
0.15
0.20
0.25
0.30
TransformerID
0,705 :: 0.079
0.711
0.038
0.732
0.007
0.753 :: 0.019
0.815
0.091
0.803
0.062
0.796 +: 0.079
Transfommel
0.633 1. 0.001
0.705
0.007
0.745
0.008
0.749
1 0.017
0,800
0.034
0.766
0.017
0.754 1 0.014
GCN
0.701
0.026
0.051
0.782
0.093
0.845
0.858
0.126
GAT
0.685 1 0.024
0.829 1 0.074
0.958
0.131
0.926
1 0.152
1.037
1 0.297
0.9%
0.213
1.036 1 0.392
ChebhNet
0.621
0.514
Graph Transformer
0.710 = 0.041
0.759
= 0.035
0.770
0.04
0.776
= 0.054
08115
0.101
0.795 = 0.085
0.822 + 0.115
GraphGPS
0.715 0.012
0.751
0.016
0,801
0.023
0.795
+ 0.044
0,833 0.025
0.814 0.022
0,830 + 0.018
EGNN
0.663 1: 0.010
0.760
0.043
0.898
0.134
0.901
1 0.123
0.849
0.053
1.058 0.239
1.192 1 0.453
SchNet
0.655 1 0.038
1,368
0.449
1,099
0.320
1.419 1 0.491
0,900
0.119
1.957
0.858
2,025 1 0.779
FastEGNN
0.710 0.011
0.854
0.079
0.032 0.806
0.863 0.020
0.110
0.849 0.079
0.968 0.131
30


Reviews:
[{"review_id": "vPD9EMICBV", "reviewer": "Reviewer_JSSw", "summary": "This paper introduce a newly curated set of RNA datasets with enhanced 2D and 3D structural annotations, providing a resource for model evaluation on RNA data. The paper reveals that models with explicit geometry encoding generally outperform sequence-based models, and geometry-unaware sequence-based models are more robust under sequencing noise but often require around 2 − 5× training data to match the performance of geometry-aware models. The authors conducted thorough and detailed experiments to support their proposed arguments.", "questions": "1. Predicting the 3D structure of models directly from 1D sequence data can indeed introduce noise, affecting model performance. Have the authors considered finding a 3D RNA dataset, such as the RNAsolo [1]?\n\n2. This paper primarily focuses on predicting RNA properties and utilizes the MCRMSE metric. I would like to know what specific properties are being referred to, and what are the units for the predicted values?\n\n[1] Rnasolo: a repository of cleaned pdb-derived rna 3d structures, Bioinformatics 2022.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The authors investigated the enhancement of RNA property prediction through the utilization of both 2D and 3D data, and explored the performance degradation of corresponding models under various influencing factors, including noisy data and partial label.\n\n2. The authors collected a substantial amount of RNA sequence data across a wide range of nucleotide count intervals, ensuring comprehensive coverage.\n\n3. The authors conducted extensive experiments using various models on the dataset to support their proposed arguments.\n\nOriginality, quality, clarity, and significance:\n\nThe paper is original and provides a comprehensive exploration of the impact of different types of input data on RNA property prediction capabilities for the first time. The writing is clear, and the overall quality is marginally acceptable. This research makes a contribution to the exploration of RNA property prediction.", "weaknesses": "1. The authors utilized a limited number of 3D models for geometric structure modeling, most of which are relatively early models, and neither of the two models (EGNN and SchNet) is specifically designed for 3D RNA structure modeling. Therefore, I believe their performance does not fully reflect the potential improvements offered by geometric information across various datasets. The authors are supposed to validate models specifically designed for RNA 3D structure modeling, such as ARES [1] and PaxNet [2], as well as some classic models in the protein domain (like GVP [3], GearNet [4], and MEAN [5]).\n\n2. The pooling strategy is difficult to classify as an innovative contribution from the authors, as it is relatively simple. Thus, the models employed in the paper are essentially existing models, and the authors have not proposed their own model. Since 3D data encompasses more information than 1D and 2D data, the authors should reflect on how to better utilize 3D information to enhance model performance.\n\n3. As noted in the paper, \"all-atom SchNET and EGNN rely on a limited local neighborhood of adjacent atoms, limiting their receptive fields and preventing them from capturing long-range dependencies\", full-atom modeling indeed incurs a significant computational burden. The authors could consider adopting the approach from methods like MEAN [5], treating each nucleotide as a node in a graph, which would allow for the expansion of local neighborhood relationships. Alternatively, the strategy used in PaxNet [2] could be employed to model the long-range and short-range interactions.\n\n4. The analysis in section 4.2 lacks insights. Some conclusions are quite obvious, such as the enhancement of performance due to increased training data, which is even more pronounced in transformer-like structures. Moreover, the inferior performance of 3D models compared to 2D models can be attributed to the significantly lower prediction accuracy of existing methods for 3D structures compared to 2D structures.\n\n5. The analysis in section 4.3 presents similar issues. Since the authors introduced sequencing noise, this error accumulates greater noise in both 2D and 3D data as a result of using 2D and 3D predictive tools. Therefore, it is expected that the transformer1D, which directly models 1D sequence data, would exhibit stronger performance.\n\n[1] Geometric deep learning of RNA structure, Science 2021.\n\n[2] Physics-aware Graph Neural Network for Accurate RNA 3D Structure Prediction, NIPS 2022 workshop.\n\n[3] Learning from Protein Structure with Geometric Vector Perceptrons, ICLR 2021.\n\n[4] Protein representation learning by geometric structure pretraining, ICLR 2023.\n\n[5] Conditional Antibody Design as 3D Equivariant Graph Translation, ICLR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B7m7ry47XJ", "reviewer": "Reviewer_k6pU", "summary": "The paper provides a curated set of RNA datasets with annotated 2D and 3D structures and investigates RNA property prediction using various deep learning models, focusing on 1D (nucleotide sequences), 2D (graph representations), and 3D (atomic structures) approaches. Key findings reveal that 2D models generally outperform 1D models, while 3D models excel in noise-free scenarios but are sensitive to noise. In contrast, 1D models show greater robustness in noisy and OOD conditions. The authors emphasize the trade-offs of each approach and advocate for future research to integrate the strengths of all models.", "questions": "1. In recent years, many RNA language models have been proposed, applicable to various downstream tasks. Compared to these models, what are the advantages and disadvantages of the methods mentioned in the article?\n2. What are the criteria for selecting secondary structure prediction tools, and is there a detailed analysis and further experimentation?\n3. Please explain the motivation behind the five task settings, as these experiments may make the article appear cumbersome and the key points unclear.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This study provides a thorough comparison of 1D, 2D, and 3D models, showcasing their respective strengths and weaknesses in handling RNA data.\n2. This study provides a comprehensive analysis of various deep learning models, assessing their performance under different conditions, including limited data and labels, different types of sequencing errors, and out-of-distribution scenarios, which is crucial for real-world applications.\n3. The commitment to transparency and reproducibility by making methodologies, datasets, and code publicly available promotes collaborative progress in the field.", "weaknesses": "1. The article lacks methodological innovation, missing deep improvements on existing technologies and novel algorithm designs.\n2. The article merely compares various metrics, and the key points are not sufficiently emphasized.\n3. Both the secondary and tertiary structures are predicted using software, particularly the tertiary structure, which is not very accurate. This can lead to significant uncertainties in further property predictions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7Ls6HhhfLs", "reviewer": "Reviewer_HBUE", "summary": "This paper presents a systematic evaluation of incorporating explicit geometric information into RNA property prediction, considering not only performance but also real-world challenges such as limited data availability, partial labeling, sequencing noise, and computational efficiency. To this end, authors introduce a newly curated set of RNA datasets with enhanced 2D and 3D structural annotations, providing a resource for model evaluation on RNA data.", "questions": "See Weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "For the field of AI for Science, high-quality datasets are very important assistants. This article integrates four datasets, which contain a large number of data samples of various types.", "weaknesses": "> **W1. Lack of explanation for RNA's uniqueness.**\n\nIn section 3, it seems that a task of general sequence molecules is defined, and it is not specifically for RNA. Is there any fundamental difference in methodology between them and other sequenced molecules (such as proteins and DNA), except that the molecular composition may be slightly different?\n\n> **W2. The method used is relatively old.**\n\nThere are many new works for 1D sequences and 2D topological graphs, which are not elaborated here. As for 3D geometric graph neural networks, many latest works are not included in the comparison. In fact, the EGNN used has been pointed out by some literature to have form capacity limitations [a], which makes the conclusions drawn in the article unreliable. More and stronger baselines should be added. You can refer to these surveys [b,c,d], and add baselines such as SaVeNet [e], LEFTNET [f], FAENet [g], TFN [h], NequIP [i], MACE [j], EquiformerV2 [k], EPT [l], MEAN [m], dyMEAN [n], etc.\n\n[a] On the expressive power of geometric graph neural networks\n\n[b] A Survey of Geometric Graph Neural Networks: Data Structures, Models and Applications\n\n[c] A Hitchhiker's Guide to Geometric GNNs for 3D Atomic Systems\n\n[d] Artificial Intelligence for Science in Quantum, Atomistic, and Continuum Systems\n\n[e] SaVeNet: A Scalable Vector Network for Enhanced Molecular Representation Learning\n\n[f] A new perspective on building efficient and expressive 3D equivariant graph neural networks\n\n[g] FAENet: Frame Averaging Equivariant GNN for Materials Modeling\n\n[h] Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds\n\n[i] E(3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials\n\n[j] Mace: Higher order equivariant message passing neural networks for fast and accurate force fields\n\n[k] EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations\n\n[l] Equivariant Pretrained Transformer for Unified Geometric Learning on Multi-Domain 3D Molecules\n\n[m] Conditional Antibody Design as 3D Equivariant Graph Translation\n\n[n] End-to-End Full-Atom Antibody Design", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)