RDesign: Hierarchical Data-efficient Representation Learning for Tertiary Structure-based RNA Design
Authors: Cheng Tan
Yijie Zhang
Zhangyang Gao
Bozhen Hu
Siyuan Li
Zicheng Liu
Stan Z. Li
Venue: ICLR 2024 poster (2024)
Keywords: Bioinformatics
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: While artificial intelligence has made remarkable strides in revealing the relationship between biological macromolecules' primary sequence and tertiary structure, designing RNA sequences based on specified tertiary structures remains challenging. Though existing approaches in protein design have thoroughly explored structure-to-sequence dependencies in proteins, RNA design still confronts difficulties due to structural complexity and data scarcity. Moreover, direct transplantation of protein design methodologies into RNA design fails to achieve satisfactory outcomes although sharing similar structural components. In this study, we aim to systematically construct a data-driven RNA design pipeline. We crafted a large, well-curated benchmark dataset and designed a comprehensive structural modeling approach to represent the complex RNA tertiary structure. More importantly, we proposed a hierarchical data-efficient representation learning framework that learns structural representations through contrastive learning at both cluster-level and sample-level to fully leverage the limited data. By constraining data representations within a limited hyperspherical space, the intrinsic relationships between data points could be explicitly imposed. Moreover, we incorporated extracted secondary structures with base pairs as prior knowledge to facilitate the RNA design process. Extensive experiments demonstrate the effectiveness of our proposed method, providing a reliable baseline for future RNA design tasks. The source code and benchmark dataset are available at https://github.com/A4Bio/RDesign.

Published as a conference paper at ICLR 2024
REPRE- SENTATION LEARNING FOR TERTIARY STRUCTURE- BASED RNA DESIGN
RDESIGN: HIERARCHICAL
Cheng 1.2 Yijie Zhang Zhangyang Gaol.2,, Bozhen Hu².2, Siyuan Li Zicheng Liu Stan Z. Li
!Zhejiang University, Hangzhou, China McGiil University, Montréal, Québec, Canada AA Lab, Research Center for Industries of the Future, Westlake University, Hangzhou, China (tancheng, ggozhangyangJevest...no
cn; yj zhang@mail wtt.mmaam..mmm...mmmmmmm ca
ABSTRACT
While artificial intelligence has made remarkable strides in revealing the relation- ship between biological macromolecules primary sequence and tertiary structure, designing RNA sequences based on specified tertiary structures remains chal- lenging. Though existing approaches in protein design have thoroughly explored structure-to-seqqeenc dependencies in proteins, RNA design still confronts diffi- culties due to structural complexity and data scarcity. Moreover, direct transplan- tation of protein design methodologies into RNA design fails to achieve satisfac- tory outcomes although sharing similar structural components. In this study, we aim to systematically construct a data-driven RNA design pipeline. We crafted large, well-curated benchmark dataset and designed comprehensive structural modeling approach to represent the complex RNA tertiary structure. More impor- tantly, we proposed hierarchical data-efficient representation learning framework that learns structural representations through contrastive learning at both cluster- level and sample-level to fully leverage the limited data. By constraining data representations within limited hyperspherical space, the intrinsic relationships between data points could be explicitly imposed. Moreover, we incorporated ex- tracted secondary structures with base pairs as prior knowledge to facilitate the RNA design process. Extensive experiments demonstrate the effectiveness of our proposed method, providing a reliable baseline for future RNA design tasks. The source code and benchmark dataset are available at gigihbb.com//BBii/DDessg
INTRODUCTION
Ribonucleic acid (RNA) is fundamental polymer composed ribonucleotides, serving as a vital biological macromolecule that regulates a plethora of cellular functions (Kaushik et al. [0188] Guo
al. 2010 Sloma & Mathews, 2016 Warner al. 2018). Non-coding RNA strands exhibit structures, which are essential for their biological activities (Feingold & Pachter 2004; Gsttrr et al.] 2014). The complex geometries of RNA molecules empower them to
tricate
exccute irreplaceable functions crucial cellular processes (Crick) 1970). encompassing but not limited to mRNA translation (Roth & Breaker 2009), RNA splicing (Runge et al. 2018: Wanrooi]
al. 2010 Kortmann & Narberhaus] 2012), and gene regulation (Meyer al. 2016).
Specifically, the primary structure of RNA refers to its linear sequence of ribonucleotides (Hofacker et al. 1994 Rother et al.] 2011 Kagaya al. 2023). The primary structure then folds into
secondary structure with canonical base pairs, forming stems and loops (Nicholas & Zuker 2008
Yang al. 2017 Liu et al. 2022). Tertiary interactions between secondary structural elements subsequently give rise to the three-dimensional structure (Qin et al.] 2022) Wang & Dokholyan
2022 Yesselman Das 2015 Das al. (2100). Figure Ligure[illue illustrates an example of the hierarchical
Tolding of RNA primary, secondary, and tertiary structures. Gaining comprehensive understanding
"Equal contribution. Corresponning author.

---

Published as a conference paper at ICLR 2024
of RNA structure is fundamental to figuring out biological mysteries and holds tremendous promise for biomedical applications. However, solving RNA structures through experimental techniques remains challenging due to their structural complexity and transient nature. Computational modeling of RNA structure and dynamics has thus become particularly valuable and urgent.
GGCUUCAACA ACCOCGUAGG LUGGGCCGAA AGGCAGCGAA
UCUACUGGAG CC
(a) RNA primary structure
(b) RNA secondary structure
(c) RNA tertiary structure
Figure I: The schematic diagrams of RNA primary, secondary and tertiary structures.
Recent years have witnessed the emergence and rapid advancement of data-driven computational modeling of RNA (Angermueller et al. 2016: Xiong et al.] 202T /Singh et al.] 2021] Cao et al. 2024). In particular, algorithms for RNA secondary structure prediction have been extensively de- veloped, yielding impressive results through leveraging large datasets of known secondary struc- tures (Singh et al.) | et al., 2019b Fu al. 2022 Tan et al. (022). However, knowledge of RNA tertiary structures, which is crucial for thoroughly understanding RNA functional mechanisms and discovering RNA-targeted therapies (Warner al., 2018: Churkin et al. 2018), remains lim- ited (Townshend et al. 2021). The success of protein structure prediction (Jumper et al.) 2021] Baak al. (202]) has motivated researchers tackle the even more challenging problem of RNA tertiary structure prediction, leading to the development of RNA tertiary structure folding algorithms such as DeepFoldRNA (Pearce et al. 2022). TTAFoldAA (Baek et al. 2022). and RhoFold (Chen) et al. 2022 Shen et al. 2022). While predicting RNA tertiary structures from primary sequences can leverage abundant sequence data (Chen et al., 2022), its inverse problem, designing RNA sequences that reliably fold into specified tertiary structure, remains largely underexplored.
Chen
The key reasons that RNA tertiary structure modeling lags far behind protein tertiary structure mod- eling stem from two main aspects: (1) RNA demonstrates greater structural intricacy and flexibility than proteins, posing formidable hurdles for structure prediction and design (Townshend et al,, 2021 Bernstein al. 2012)] Berman et al., 2000). The less constrained structure space of RNA leads to intractable challenges in modeling the RNA tertiary structure. (2) High-resolution RNA tertiary structures scarce compared to proteins due to their conformational dynamics and insta- bility(Rother et al. (201I). The quantity of available RNA structures constitutes less than 1% that for proteins (Adamezyk et al. 2022b Kalvari et al. 2021). To deal with the above problem, pro- pose a thorough pipeline aiming at data-driven tertiary structure-basee RNA design tasks. In detail, we first compile a large-scale RNA tertiary structure dataset based on extant high-quality structure data from Protein Data Bank (PDB) (Berman et al. 2000) and RNAsolo (Adamczyk et al. 2022b). Then, regarding the deficiency in RNA tertiary structure modeling and the unsatisfying transferable capability of conventional protein structure modeling techniques to the RNA field, we propose comprehensive RNA tertiary structure modeling. To optimize the use of the limited data, we in- troduce a hierarchical and data-efficient representation learning framework that applies contrastive learning at both the cluster and sample levels. We can explicitly impose intrinsic relationships between the data by constraining the data representations within a limited hyperspherical space. Moreover, we provide a strategy that utilizes extracted secondary structure as prior information to guide the RNA design inspired by the correlation between RNA secondary and primary structures.
The main contributions of this work are summarized as follows:
We propose a formal formulation of the tertiary structure- RNA design problem. To establish fair benchmark for tertiary sructtre-based RNA design, we compile a large dataset of RNA tertiary structures and provide a fundamental data split based on both struc- tural similarity and sequence length distribution.
2

---

Published as a conference paper at ICLR 2024
We propose comprehensive structural modeling approach for the complex RNA tertiary structure and design an RNA design framework called RDesign, which is composed of hierarchical representation learning scheme and secondary structure imposing strategy.
Through extensive experiments across standard RNA design benchmarks and generaliza- tion ability assessments, we demonstrate the efficacy of our proposed method. This pro- vides reliable pipeline for future research in this important and promising field.
2 RELATED WORK
2.1 BIOMOLECULAR ENGINEERING
In recent decades, the rapid advancements in biophysics, biochemistry, and chemical engineering have enabled plethora novel applications (Nggamune 2017). including engineering enzymes for industrial biocatalysis (Pugh al. 2018), tailoring antibodies for precision cancer therapies (Jeschek
et al. 2016), developing trackable fluorescent proteins for biological imaging (Rosenbaum) (017)). and optimizing polymerases for forensic DNA analysis (Martell al., (016). RNA design of particular interest among them due to the diverse functions that RNA can fulfill, ranging from trans- lation and gene expression catalysis (Ellefson et al. 2016). This multifunctionality
ascribed to the structural diversity of RNA (Andronescu et al. 2004). In this work, we focus on tertiary structure- based RNA design to uncover the relationships between RNA structure and sequence.
2.2 PROTEIN DESIGN
RNA and protein essential components of cells. Despite having different chemical constituents, their higher-order structures can be described similarly (Rother et al. 2011). Early works on com- putational protein design (Wang et al. 2018 Chen
2019a) utilize multi-layer perceptron
(MLP), and convolutional neural (eN)) to predict residue types from protein structure. 3D CNNs have enabled tertiary structure- based design such as ProDCoNN (Zhang et al., (020) and
DenseCPD QQi & Zhang 2020). Graph (Ingraham et al. 2019) combines attention (Vaswani jet al. 2017) and auto-regressive decoding to generate protein sequences from graphs, inspiring
series of recent advancing approaches (Jing et al.) 2020 Dauparas et al. 2022, Hsu et al. [022] Tan
ctt al. 2023] Gao et al.] 2022b] 2023 2024 TTa et al. 2024). While insights from protein research
have illuminated RNA biology, RNA studies have trailed due scarcity of available data and
complex structure modeling (Gan et al. 2003).
2.3
RNA DESIGN
The computational design of RNA sequences aims to generate nucleic acid strands that will fold into targeted secondary tertiary structure. Secondary structure-basee based RNA design first introduced by Vienna (Hofacker et al.] 1994). Early works solved the RNA design problem through stochastic optimization and energy minimization with thermodynamic parameters, such as RNAfold (Lorenz et al.] 2011). Mfold (Zuker) (003)), UNAFold (Nicholas & Zuker 2008), and RNAStructure (Mathews] 2014). Probabilistic models and posterior decoding were employed to solve this problem (Sato et al. 2009). Other works that operate on a single sequence and try to find solution by changing a few nucleotides include RNAInverse (Holacker et al. 1994), RNA- SSD (Andronescu et al. 2004), INFO-RNA (Busch & Backofen (00kk) and NUPACK (Zadeh et al., 2011. There are global searching methods, including antaRNA (Kleinkaff al. 2015). aRNAque
MMerleau & Smerlak 2022). eM2dRNAs (Rubio-Largo et al.) (023) and RNA (Yang et al. 2017). Reinforcement learning-bant methods have also been developed (Runge et al. 2018).
Although numerous approaches have been studied for engineering RNA secondary structures, RNA design based on the tertiary structure still challenging due to the lack of high-resolution structural data Yesselman & Das 2015 Das et al.) 2010). Although structure prediction algorithms (Liu et al.)
2022 Qin et al.) 2022; Wang & Dokholyan, 2022) can utilize abundant RNA primary sequence information, the progress of RNA design has been hindered by the scarcity of determined RNA 3D structures. To alleviate the difficulty, we explore the uncharted areas tertiary structure-centric
RNA design systematically and propose complete pipeline to address this challenge.
3

---

Published as a conference paper at ICLR 2024
3 METHODS
3.1
PRELIMINARIES
For an RNA sequence in its primary structure, we assume it comprises N nucleotide bases selected from the set of nucleotides: A (Adenine), U (Uracil), C (Cytosine), and G (Guanine). Therefore, the sequence can be represented as:
Nucleotides :- {A, U.
(1)
S {st E Nucleotides i n2),
The formation of the tertiary structure requires the folding of this sequence in three-dimensional space, which can be denoted as:
Atoms (P, O5', C5', C4',
X { [ E R i nz, E Atoms);
where the Atoms set denotes the six atoms that comprise the RNA backbone.
A
(n) RNA Secondary Structure
(b) RNA Sequence
(c) Dot-bracket Representation
Figure 2: Brief view of RNA sequence and secondary structure.
We incorporate secondary structure information using dot-bracket notation. Unpaired nucleotides are represented by dots, and paired nucleotides are represented by brackets, shown in Figure 2
A = {a, E f (, }} i [L),
where a; is dot if the nucleotide is unpaired, or matching bracket otherwise. Finally, the tertiary structure-bart based RNA design problem could be formulated as:
Fe: : N + S.V.
(4)
such that A = (xx* satisfics the pairing rules,
where Fe is learnable mapping with parameters O and g(.) is function that extracts the sec- ondary structure denoted by dot-bracket notation from the tertiary structure. Namely, Fe denotes the mapping from X to SNN which means from the tertiary structure the primary structure. illustrates that while we map the tertiary structure X' of RNA to the primary sequence S, we ensure that the predicted sequence aligns with the pairing rules associated with the secondary structure A.
3.2 COMPREHENSIVE RNA TERTIARY STRUCTURE MODELING
We construct a local coordinate system Q: for the i-th nucleotide in the RNA tertiary structure. The detailed procedure for defining the local coordinate system is in Appendix D While studies on protein design have achieved considerable success using only the Co atoms to model backbone geometry, this approach does not readily translate to RNA.
0%
phosphorse stor
pocgen soxn
carbos atoms
directoont
04
CA'
C1',
RNA exhibits diversity of backbone conforma- tions and base- Base-pairing geometries that cannot be sufficiently captured by such modeling. The com- plexity and plasticity of RNA structure necessitate comprehensive treatment.
CX)
0st
p2)
Figure 3: An example of RNA structure.
4

---

Published as a conference paper at ICLR 2024
To adequately capture the complex structural information inherent in the direediiensionall folding of RNA molecules, we propose a general approach for modeling RNA tertiary structure. We repre- sent RNA tertiary structure as attributed graph G (V, Æ)) comprising node attributes V and edge attributes E. The graph is constructed by identifying the K nearest neighbors in 3D space for each node; each node has set of K neighbors denoted N K). Specifically, V E Nx contains fa fa-dimensional node attributes for N nodes, and E' E I v xfm contains fm Finddimesional edge attributes for each node's K neighbors. By default, we K 30.
We outline the attributes used in our modeling approach along with their corresponding illustrations [able which includes two levels of attributes: (i) intra-nucleotide level attributes describing the local geometry of each nucleotide as the node attribute V. and (ii) inter-nucleotide level attributes describing the relative geometry between nucleotides as the edge attribute F.
Intra-nucleotide level (1) The dihedral angles, shown as red arrows in Figure 3 calculated. We represent the dihedral angles of the RNA backbone using sin and cos functions. (2) The spatial distances between the other intra-nucleotie aloms and the atom P. are encoded into radial basis functions (RBFs). (3) The directions of the other intra-nucleotide atoms relative to the atom P, are calculated with respect to the local coordinate system Q,-
Inter-nucleotide level (1) An orientation encoding q(-) is calculated from the quaternion repre- sentation the spatial rotation matrix Q Qj- (2) The spatial distances between inter-nucleotide atoms from neighboring nucleotides and the atom P. are encoded into radial basis functions (RBFs). (3) The directions of the other nuclooiee atoms relative to the atom P. are calculated.
Table I: The feature construction of RNA tertiary structure modeling.
Level
Feature
Illustration
Dihedral Angle
sin, cos } x
nuclooice
Distance
RBF((K: - P:ll)
w E (05', C,', C3', C,,,,300) 081}
Direction
wi Q
(05', C5', C3',
Orientation
q(QTQ,)
Inter nucleotide
Distance
[BB[(W P.I)) {Qf
((, (05', C5', C1', C3', 03')
Direction
Poy-P: j E K), e (05', C5', C3', C3',
-
3.3
HIERARCHICAL DATA-RFFICIEN REPRESENTATION LEARNING
Now that RNA tertiary structure has been adequately modeled, the remaining challenge is how to leamn from scarce data in data-cfficient manner. The key motivation is explicitly imposing the inherent data relationships based on prior knowledge. We first use L layers of message-passing neural networks (MPNNs) learn the node representation. Specifically, the I-th hidden layer of the i-th nucleotide is defined as follows:
(1) h
(1-1) MPNN ( KE. RV.
Σ
ny D.
(1-1))
(5)
JEN(K.)
where O hg are the embeddings of the nucl-nucee and inter-nucleotide level features from the tertiary structure modeling, respectively. When generating the RNA sequence, a fully connected layer maps the node representation h!? to the RNA sequence space: f(h(/)).
To enable data-efficient representation learning, we obtain the graph-level representation through the average pooling of the node representations hG g(ha) by the projection g R S that projects the Euclidean space into the hyperspherical space.
D ((f and the corresponding projection
We propose a hierarchical representation learning framework comprising cluster-level and confidence-aware sample-level representation learning, shown in Figure 4 The cluster-level rep-
5

---

Published as a conference paper at ICLR 2024
resentation learning utilizes topological similarity between RNA structures. We obtain RNA struc- ture clusters based on TM-score, which indicates structural similarity Aniiarity(Zhhnn & Skolnick 2005). We define positive pairs as RNA data with similar topologiaal structures that attract each other in the emhedding space, while negative pairs with dissimilar topological structures repel each other. The cluster-level representation learning is defined as follows:
exp(ep 9q/T)
Celuster
Σ log Eg, Eg,, exp(Ip 9k/TT)
(6)
PED
where (gp. 9q) is positive pair that comes from the same structural cluster, K. is of negative samples for 9p identified by the cluster they belong to, and D is the data set. We denote Ip as the graph representation projection of the p-th RNA sample for notational convenience.
The confidence-aware sample-level representation learning is designed to capture the microscopic intrinsic properties of RNA structures. The positive pairs are defined given RNA structure sam- ple and its random perturbed structures. The perturbed structures are obtained by adding Gaussian noise to the experimentally determined coordinates. To prevent excessive deviation, we filter out the perturbed structures with low structural similarity (TM-score I 0.8) and high structural deviation (RMSD 2 1.0). The RMSD also evaluates the confidence level of the perturbed data. Formally, the sample-level representation learning can be formulated as:
exp(yr sp/T)
Lsample
Σ To.p'' log E (og)(,, exp(yp 9g//T) PED
where p' is the perturbed structure of the p-th RNA structure, and K, is simply defined as other samples apart from 9p- The confidence score Ypppp is defined exp -RMSD(P-")) that when RMSD(p.P') + 0, the confidence approaches 1.
The cluster-level representation provides coarse- grarse-graindd embedding, capturing the global topolog- ical similarity between RNA structures. The confidence-awarr sample-level representation provides intrinsic knowledge that is robust minor experimental deviations. As shown in Figure 4 by con- straining the limited data into the restricted hyperspherical space with imposed prior knowledge, the intrinsic relationships between data are explicitly modeled.
(a) Cluster level
(b) Sample level
(c) Multi-level representation leamning
Figure 4: The different levels of hierarchical representation learning. Green arrows denote positive pairs tend to attract each other, and the black arrow denotes negative pairs tend to repel each other.
3.4 SECONDARY STRUCTURE IMPOSING STRATEGY
With the given tertiary structure, we derive the corresponding secondary structure using the notation shown in Figure 2 The secondary structure is represented using parentheses to indicate paired nucleotides, with unpaired sites represented by one of the four RNA nucleotides (A, U, C. G). Paired sites are denoted by two nucleotides placed simultaneously in one of the following pairs: {CG, GC, AU, UA, UG. GU). When a pair of positions (i,j) in the predicted primary sequence and its corresponding secondary structure are given, we can calculate the confidence score for each position i based on the predicted letter at that position and the known secondary structure constraint. We then choose the position with the higher confidence score as the "reference" (say position i). We correct the predicted letter at position j so that the letters at and j form the allowed pairs.
Specifically. if position is selected as the reference, and modify the predicted letter at j to satisfy the base pairing constraint. We then update the pre- dicted primary sequence. By leveraging the information from the known secondary structure,
maintain the predicted letter at i unchanged

---

Published as a conference paper at ICLR 2024
rectify and refine the initially predicted primary sequence. The refinement helps enhance the accuracy of RNA 3D structure prediction. In the training phase, we compel the model to sharpen the confidence of the nucleotides in the paired positions. The supervised loss is defined as:
Crup
Σ
((CE(s,,((i)) (( + (CE(S), 5(h (/)))/)])++
E
(S f (11 (8)
(f.))
lef Pairs
where Pairs encompasses all the paired position indices given by the secondary structure, r' the temperature that is set as 0.5 by default to sharpen the confidence of paired nucleotides, are true sequence labels.
The training objective is the linear combination of representation learning loss and supervised loss:
C = Sup + X A(Cluster + ssample);
(9)
where we set the weight parameter À as 0.5 by default.
4 EXPERIMENTS
We evaluate RDesign the tertiary structure-based RNA design task by comparing it with four categories baseline models: (i) sequence-based models (SeqRNN and SeqLSTM) that not uti- lize any structural information and could be viewed the performance reference for RNA design; (ii) A tertiary structure- based model (StructMLP) that exploits structural features while ignoring the graph topological structure; (iii) Tertiary structure- based models (StructGNN and GraphTrans, (In- graham et al.) 2019) and PiFold (Gao et al., 2022h) that incorporate the graph topological struc- ture; (iv) secondary arnecturebbsed RNA sequence design models (MCTS-RNA (Yang | (217])
LEARNA (Runge et al. 2018), e.M2RRAs (Rubio-Large et al. 2023). aRNAque (Merleau & Smer-
lak
2022
The detailed experimental settings and dataset descriptions are shown in Appendix A
and
Appendix
B
respectively.
4.1 STANDARD TERTIARY STRUCTURE- BAASED BASED
RNA DESIGN
Using our carefully curated benchmark dataset, we trained the model using the training set. Then, we evaluated the performance of the model on the testing set by selecting the model with the lowest loss on the validation set. Given that RNA sequences of varying lengths may impact prediction results, we stratified the testing set into three groups based on RNA length: (i) Short, RNA samples less than equal to 50 nucleotides; (ii) Medium RNA samples greater than 50 nucleotides but less than or equal to 100 mucleotides; (iii) Long RNA samples greater than 100 nucleotides. To gain a thorough understanding of the relationship between RNA length and model accuracy, reported both the recovery and Macro-F1 metrics for Short, Medium, and Long testing samples separately, in addition to overall testing set performance.
Table 2: The recovery on the benchmark dataset. The best results are highlighted in bold.
Recovery (%) f
Method
Short
Medium
Long
All
SeqRNN (h=
26.52.11.07
24.86.10.82
27.31110.41
26.23
SeqRNN (h=256)
27.61+1.85
27.16+00.63
28.71+0.14
28.2440.46
SeqLSTM (h=128)
23.48++1007
26.32++0.05
26.78
24.70+0.64
SeqLSTM (h=256)
25.00++0.00
26.89+40.35
28.55+0.13
26.93+4.93
StructMLP
25.72-10.51
25.03.11.39
25.38
25.35
StruciGNN
27.55
28.78-10.87
28.23
28.23
GraphTrans
26.15
23.78
23.80
24.73
PiFold
24.81+2.01
1..90
23.55+4.13
1..4
RDesign
37.22-11.14
44.89.11.67
43.06.10.08
41.53
As presented in Table 2 the baseline models achieved suboptimal recovery scores, with performance
ranging from 24-28%. Unexpectedly, tertiary structure- based models like StructMLP, StructGNN, and GraphTrans attained comparable results sequence- based models. This indicates that directly applying protein design techniques to RNA is misguided and fails to capture the intricacies RNA structures. Moreover, StructMLPP and GraphTrans achieved higher recovery scores for short RNA

---

Published as a conference paper at ICLR 2024
sequences but struggled longer, namely complex RNA structures. Their struggles the long dataset stem from the inability to learn from more intricate RNA structures and their low gen- eralization capabilityy In contrast.
RDesign model outperforms all baseline methods the covery metric, achieving substantial gains. RDesign's strong performance, particularly on medium and long sets, indicates that it learn intrinsic RNA structural properties.
Table 3: The Macro-FI on the benchmark dataset. The score multiplied by 100 for aesthetics.
Macro (x 100) f
Method
Short
Medium
Long
SeqRNN (h=128)
17.22:1..99
17.2011..11
8.441:2.70
1..74=1.59
SeqRNN (h=256)
12.5442.94
13.64445.24
8.85++2.41
13.6442.69
SCqLSTM (h=128)
9.89+t0557
10,44+1142
10.71+22.3
10.28=00611
ScqLSTM (h=256)
9.26+1.16
9.48++0.74
7.14+0.00
10.93=0.15
StructMLP
17.46.12.39
18.57
17.53.18.43
18.88
StrucIGNN
24.01
22.15:14.67
26.05.16.43
24.87
Graph Trans
16.34.12.67
16.39.L4.74
18.6717.116
17.1813.81
PiFold
17.4841.22
18.10.1..76
14.06.13.53
17.45
RDesign
38.25
40.41
41.48
40.89 40.89-10.49
It could be seen from Table that there exist large gaps between the recovery metrics and Macro-F1 scores of most baseline models, which suggests those models tend to predict the high-frequency nucleotide letters instead of reflecting the actual tertiary structure. Among them, only StructGNN achieved consistent results in its Macro-F1 score and recovery metric but with unsatisfying per- formance. Our proposed RDesign consistently outperformed all other models these metrics, demonstrating its effectiveness.
4.2 EVALUATE GENERALIZATION
RFAM AND RNA-PUZZLER
To assess the generalization capability of our model, evaluated our model and the baseline meth- ods on the Rfam (Kalvari et al. 2021 and RNA-Puzzles (Miao al. 2020 datasets using the model pre-trained our benchmark training set. We presented the results in Table 4 The performance remained consistent with that of our benchmark dataset. Specifically, StruetGNN, which effectively learned certain tertiary structure information, achieved relatively small gap between the recovery metric and Macro Fl score. contrast, the other baselines that learned little structural information performed sub-optimally. Our proposed RDesign model demonstrated superior generalization both datasets and outperformed all the basclines.
notable that the results reported here were generated by directly assessing pretrained models on the entire training dataset, mirroring real-world scenarios. Furthermore, in Appendix H. we have included the results of the pretrained models assessed on a training set with similar data removed.
Table 4: The overall recovery and Macro-F1 scores on the Rfam and RNA-Puzzles datasets.
Recovery (%) f
Macro F1 (x 10000)
Method
Rfam
RNA-Puzzles
Rfam
RNA-Puzzles
SegRNN (h=128)
27.9911.221
28.994
15.79
16.06:1:..2
SeqRNN (h=256)
30.94-10.41
31.25-10.72
13.07
13.24 1.25
SeqLSTM (h=128)
24.96-10.46
25.78:1:0:3
10.13+1.24
10.39
SeqLSTM (h=256)
31.45+:0.01
31.62:+0.20
11.76:10.00
12.22=0.221
StructMLP
2..40+1..3
24.22+11.28
16.7914.01
16.40=3.28
StructGNN
27.6411.31
27.9643.008
24.3543.45
22.7613.19
GraphTrans
+3.81
22.21+2.998
17.32+5.28
17.04
PiFold
22.55+44.13
23.78+6.522
16.08+2.34
16.20+3.49
MCTS-RNA
31.74
32.06+1.87
23.82+4.60
24.12+3.447
LEARNA
31.92 12.37
30.9414.15
24.02.13.73
22.75 1.17
aRNAque
30.01 13.26
31.07-22.22
22.84.1..00
23.30
eM2dRNAs
33.34 11.02
37.10-13.24
24.80
26.91 12.32
RDesign
56.12.1.03
50.12
53.27
49.24 1.07
8

---

Published as a conference paper at ICLR 2024
4.3 ABLATION STUDY
We conducted an ablation study of RDesign and presented the results in Table Firstly, we replaced our tertiary structure modeling approach with the classical modeling from protein design, which led to a significant decrease in performance. Secondly, removing the hierarchical representation learning also resulted in a performance drop. indicating importance. Replacing the hyperspherical space with Euclidean space led to substantial reduction in performance, indicating its impact on data- efficient learning- In contrast, removing the secondary structure constraints provided relatively small decrease in performance because RDesign itself could accurately generate RNA sequences. Additionally, we further tested the capability that our designed sequences could fold into desired tertiary structures, which is the final aim of the RNA sequences design problem. However, due to the lack of reliable RNA tertiary structure prediction tools, we are only able to conduct this experiment in qualitative way. Results of three example structures from each length category have been reported and analyzed in Appendix S
Table 5: The ablation study of our model on three datasets.
Recovery (%) f Rfam
Macro Fl (x 100) 1 RNA-Puzzles
Method
Ours
RNA-Puzzaa
Ours
Rfam
RDesign
41.53
56.12
50.12
40.89
53.27
49.24
w/o our modeling
36.45
53.19
44.93
36.33
48.95
43.88
w/o reprsentation learning
37.12
52.17
46.88
36.52
49.22
46.36
w/o hyperspherical space
30.69
36.33
36.00
30.67
30.34
33.57
w/o secondary structure
38.55
54.83
47.69
38.77
52.85
47.62
EVALUATION THE CAPABILITY OF FOLDING FOR DESIGNED SEQUENCES
We used RhoFold (Shen et al. 2022) to predict the structures of RNA sequences designed by RDe- sign. Figure 5 shows three visualization examples: (a) a short sequence reconstructed by RDesign; (b) a long sequence that was designed with a similar structure and low structure deviation; (c) a complicated sequence that was designed with a similar structure but failed to achieve low structure deviation. These visualization examples demonstrate the effectiveness of our RDesign model in designing RNA sequences with structures similar to the target structure.
RVSD-8.69
RMSD-0.86
RM5D=3.22 22
Ground trath:: USUIA AUA
Ground truth: GGAGC G60CS GBUUC GABUC CCSUC cauuc
Ground truth:
CrosE csGal saccs CAUN caric
CROSE aGUiA 4005S cuscr CC
Generated:
USUAA
Generated: GGGGG CGGCG GBIUG AAUUC CCGUC GOUCC
Generated:
AARUA
ADOCC AAUSS EaGAS OCCOS
(a) 7CGK
(b) GUGI
(c) 7D7V
Figure 5: Visualization of RDesign's designed examples.
6 CONCLUSION AND LIMITATIONS
In this work, we investigate the challenging task of designing RNA tertiary structures. We compile a benchmark dataset to systematically assess the performance of various computational models on this task. While existing protein design methods cannot be directly applied, we propose hierarchi- cal data-efficient representation learning framework. Our framework explicitly captures the intrinsic relationships within the data while constraining the limited data to restricted hyperspherical space. We also introduce a secondary structure constraining strategy to leverage extra structural informa- tion. Extensive experiments demonstrate the effectiveness of our proposed RDesign model. We hope this work provides a new perspective on tertiary structure-bre based RNA design. A limitation is that our method is currently limited to in silico design leave wet-lab validation to future work.

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGEMENTS
We thank the anonymous reviewers for their constructive and helpful reviews. This work was sup- ported by National Science and Technology Major Project (No. 20222D01 National Natural Science Foundation of China Project (No. U21A20427), the Center of Synthetic Biology and Inte- grated Bioengineering of Westlake University and Integrated Bioengineering Westlake University Project (No. WU2022A009) and the Westlake University Industries of the Future Research Funding Project (No. WU2023C019).
REFERENCES
Bartosz Adamczyk, Maciej Antezak, and Marta Szachniuk. RNAsolo: repository of cleaned PDB- derived RNA 3D structures. Bioinformatics, 38(14):3668-3670, 06 2022a. ISSN 1367-4803. doi:
10.009//boinfomaaccc/taa.....
Bartosz Adamczyk, Maciej Antezak, and Marta Szachniuk. Rnasolo: repository of cleaned pdb- derived ma structures. Bioinformatics, 38( (5(14):3668-3670 2022b.
Mirela Andronescu, Anthony P Fejes, Frank Hutter, Holger H Hoos, and Anne Condon. A new algorithm for ma secondary structure design. Journal of molecular biology, 336(3):607-624, 2004.
Christof Angermueller, Tanel Pärnamaa, Leopold Parts, and Oliver Stegle. Deep learning for com- putational biology. Molecular systems biology, 12(7):878, 2016.
Minkyung Back, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang, Qian Cong. Lisa N Kinch, R Dustin Schaeffer, et al. Accurate prediction of protein structures and interactions using three-track neural network. Science, 373(6557):871- 876, 2021.
Minkyung Back, Ryan McHugh, Ivan Anishchenko, David Baker, and Frank DiMaio. Accurate prediction of nucleie acid and protein-nucleic acid complexes using rosettafoldna. bioRxiv, 2022.
Protein Data Bank. Protein data bank. Nature New Biol, 233:223, 1971.
Helen Berman, John Westbrook, Zukang Feng, Gary Gilliland, Talapady N Bhat, Helge Weissig, Ilya N Shindyalov, and Philip E Bourne. The protein data bank. Nucleic acids research, 28(1): 235- 242, 2000.
BE Bernstein, E Birney, Dunham, ED Green, C Gunter, and M Snyder. Project consortium encode. An integrated encyclopedia of DNA elements in the human genome. Nature, 489:57-74, 2012.
Anke Busch and Rolf Backofen. Info-rna-a fast approach to inverse ma folding. Bioinformatics, 1823-13183-183111 2006.
Hanqun Cao, Cheng Tan, Zhangyang Gao, Yilun Xu, Guangyong Chen, Pheng-Ann Heng, and Stan Li. A survey on generative diffusion models. IEEE Transactions on Knowledge and Data Engineering, 2024.
Jiayang Chen, Zhihang Hu, Siqi Sun, Qingxiong Tan, Yixuan Wang, Qinze Yu, Licheng Zong, Liang Hong, Jin Xiao, Tao Shen, et al. Interpretable ma foundation model from unannotated data for highly accurate ma structure and function predictions. bioRxiv, 2022.
Sheng Chen, Zhe Sun, Lihua Lin, Zifeng Liu, Xun Liu, Yutian Chong, Yutong Lu, Huiying Zhao, and Yuedong Yang. To improve protein sequence profile prediction through image captioning on pairwise residue distance map. Journal of chemical information and modeling, 60(1):391-33) 2019a.
Xinshi Chen, Yu Li, Ramzan Umarov, Xin Gao, and Le Song. Rna secondary structure prediction by learning unrolled algorithms. In International Conference Learning Representations, 201 9b.
10

---

Published as a conference paper at ICLR 2024
Alexander Churkin, Matan Drory Retwitzer, Vladimir Reinharz, Yann Ponty, Jérôme Waldispühh, and Danny Barash. Design mas: comparing programs for inverse ma folding. Briefings in bioinformatics, 19(2):350-333 358, 2018.
Francis Crick. Central dogma of molecular biology. Nature, 22765258):511-563, 1970.
Rhiju Das, John Karanicolas, and David Baker. Atomic accuracy in predicting and designing non-
canonical ma structure. Nature methods, 7(4):291-294, 2010.
Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas Milles, Basile IM Wicky, Alexis Courbet, Rob J de Haas, Neville Bethel, et al. Robust deep learning- based protein sequence design using proteinmpan. Science, 378(6615):49 2022.
Jared W Ellefson, Jimmy Gollihar, Raghav Shroff, Haridha Shivram, Vishwanath R Iyer, and An- drew D Ellington. Synthetic evolutionary origin of proofreading reverse transcriptase. Science, 352(6293); 1590-1593, 2016.
EA Feingold and Pachter. The encode (encyclopedia of dna elements) project. Science, 306 (5696):636-66) 2004.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint 19X3vv/903.02440 2019.
Laiyi Fu, Yingxin Cao, Jie Wu, Qinke Peng, Qing Nie, and Xiaohui Xie. Ufold: fast and accurate ma secondary structure prediction with deep learning. Nucleic acids research, 50())e14-e14, 2022.
Hin Hark Gan, Samuela Pasquali, and Tamar Schlick. Exploring the repertoire of ma secondary motifs using graph theory; implications for raa design. Nucleic acids research, 31(11):2926- 2943, 2003.
Zhangyang Gao, Cheng Tan, Stan Li, et al. Alphadesign: A graph protein design method and benchmark on alphafolddb. arXiv preprint arXiv:2202.01079 2022a.
Zhangyang Gao, Cheng Tan, and Stan Z Li. Pifold: Toward effective and efficient protein inverse folding. arXiv preprint arXiv:2209. 12643, 2022b.
Zhangyang Gao, Cheng Tan, and Stan Li. Knowledge-design: Pushing the limit of protein deign via knowledge refinement. arXiv preprint arXiv:2305. 2023.
Zhangyang Gao, Cheng Tan, Yijie Zhang, Xingran Chen, Lirong Wu, and Stan Z Li. Proteinin- vbench: Benchmarking protein inverse folding on diverse tasks, models, and metrics, Advances in Neural Information Processing Systems, 36, 2024.
Paul P Gardner, Jennifer Daub, John G Tate, Eric Nawrocki, Diana L Kolbe, Stinus Lindgreen, Adam C Wilkinson, Robert D Finn, Sam Griffiths-Jones, Sean R Eddy, et al. Rfam: updates to the ma families database. Nucleic acids research, 37(suppl 1):D136-D140, 2009.
Ronald Gstír, Simon Schafferer, Marcel Scheideler, Matthias Misslinger, Matthiss Griehl, Nina Daschil, Christian Humpel, Gerald Obermair, Claudia Schmuckermair, Joerg Striessnig, et al. Generation of neuro-specifi, microarray reveals novel differentially expressed noncoding mas in mouse models for neurodegenerative diseases. Rna, 20(12))1929-1944 2014.
Peixuan Guo, Oana Coban, Nicholas M Snead, Joe Trebley, Steve Hoeprich, Songchuan Guo, and Yi Shu. Engineering ma for targeted sirna delivery and medical application. Advanced drug delivery reviews, 6266)66006666, 2010.
Ivo L Hofacker, Walter Fontana, Peter F Stadler, L Sebastian Bonhoeffer, Manfred Tacker, and Peter Schuster. Fast folding and comparison of ma secondary structures. Monatshefte für Chemie/Chemical Monthly, 127 1994.
Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie. Tom Sereu, Adam Lerer, and Alexan- der Rives. Learning inverse folding from millions of predicted structures. bioRxiv, 2022.

---

Published as a conference paper at ICLR 2024
John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph- based protein design. Advances in neural information processing systems, 32, 2019.
Markus Jeschek, Raphael Reuter, Tillmann Heinisch, Christian Trindler, Juliane Klehr, Sven Panke, and Thomas R Ward. Directed evolution of artificial metalloenzymes for in vivo metathesis. Nature, 637(7622):661-6635 2016.
Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael John Lamarre Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. International Conference on Learning Representations, 2020.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger,
Kathryn Tunyasuvunakool, Russ Bates, Augustin Zídek, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873) 2021.
Yuki Kagaya, Zicong Zhang, Nabil Ibtehaz, Xiao Wang, Tsukasa Nakamura, David Huang, and Daisuke Kihara. Nufold: novel tertiary ma structure prediction method using deep learning with flexible nucleobase center representation. bioRxiv, 2023.
Ioanna Kalvari, Eric P Nawrocki, Nancy Ontiveros-Palacion Joanna Argasinska, Kevin Lamkiewicz, Manja Marz, Sam Griffiths-Jones, Claire Toffano-Nioche, Daniel Gautheret, Za- sha Weinberg, et al. Rfam 14: expanded coverage of metagenomic, viral and microrna families. Nucleic Acids Research, 40(11))D122-D200, 2021.
Kriti Kaushik, Ambily Sivadas, Shamsudheen Karuthedath Vellarikkal, Ankit Verma, Rijith Jayara- jan, Satyaprakash Pandey, Tavprithesh Sethi, Souvik Maiti, Vinod Scaria, and Sridhar Sivasubbu. Rna secondary structure profiling in zebrafish reveals unique regulatory features. BMC genomics, 19(1):1-17, 2018.
Robert Kleinkauf, Torsten Houwaart, Rolf Backofen, and Martin Mann. antarna-multi-obiee objective inverse folding of pseudoknot ma using ant-colony optimization. BMC bioinformatics, 16(1): 2015.
Jens Kortmann and Franz Narberhaus. Bacterial ma thermometers: molecular zippers and switches. Nature reviews microbiology, 10(4):255-265, 2012.
Zhixiu Li, Yuedong Yang, Eshel Faraggi, Jian Zhan, and Yaoqi Zhou. Direct prediction of profiles of sequences compatible with protein structure by neural networks with fragment-based local and energy-based nonlocal profiles. Proteins: Structure, Function, and Bioinformatics, 82(10): 2565-2573, 2014.
Zhendong Liu, Yurong Yang, Dongyan Li, Xinrong Lv, Xi Chen, and Qionghai Dai. Prediction of the ma tertiary structure based on random sampling strategy and parallel mechanism. Frontiers in Genetics, pp. 2521, 2022.
Ronny Lorenz, Stephan H Bernhart, Christian Höner zu Siederdissen, Hakim Tafer, Christoph Flamm, Peter F Stadler, and Ivo L Hofacker. Viennarna package 2.0. Algorithums for molecu- lar biology, 6(1):1-14, 2011.
Jeffrey D Martell, Masahito Yamagata, Thomas J Deerinck, Séhastien Phan, Carolyn G Kwa, Mark H Ellisman, Joshua R Sanes, and Alice Y Ting. split horseradish peroxidase for the detection of intercellular protein-protein interactions and sensitive visualization of synapses. Na- ture biotechnology, 34(7):774-780, 2016.
David H Mathews. Rna secondary structure analysis using rnastructure. Current Protocols in Bioin- formatics, 46(1):12-6, 2014.
Nono SC Merleau and Matteo Smerlak. arnaque: an evolutionary algorithm for inverse pseudoknot- ted ma folding inspired by lévy flights. BMC bioinformatics, 23(1):335, 2022.
Sarai Meyer, James Chappell, Sitara Sankar, Rebecca Chew, and Julius B Lucks. Improving fold activation of small transcription activating mas (stars) with rational ma engineering strategies. Biotechnology and bioengineering, 113(1):21-2225, 2016.
12

---

Published as a conference paper at ICLR 2024
Zhichao Miao, Ryszard W Adamiak, Maciej Antezak, Michat J Boniecki, Janusz Bujnicki, Shi-Jie Chen, Clarence Yu Cheng, Yi Cheng, Fang- Chich Chou, Rhiju Das, et al. Rua-puzles round iv: 3d structure predictions of four ribozymes and two aptamers. RNA, 26(8):982 995, 2020.
Teruyuki Nagamune. Biomolecular engineering for nanoiiorbionanotechnollyy, Nano convergence, 4(1):1-56, 2017.
Eric Nawrocki, Sarah W Burge, Alex Bateman, Jennifer Daub, Ruth Y Eberhardt, Sean R Eddy, Evan W Floden, Paul P Gardner, Thomas A Jones, John Tate, et al. Rfam 12.0: updates to the ma families database. Nucleic acids research, 43(D1):D130-D133 2015.
RM Nicholas and M Zuker. Unafold: Software for nucleic acid folding and hybridization. Bioinfor- maticy, 453:3-31, 2008.
James O' Comnell, Zhixiu Li, Jack Hanson, Rhys Heffernan, James Lyons, Kuldip Paliwal, Abdollah Dehzangi. Yuedong Yang, and Yaoqi Zhou. Spin2: Predicting sequence profiles from protein structures using deep neural networks. Proteins: Structure, Function, and Bioinformatics, 86(6): 629-633, 2018.
Robin Pearce, Gilbert S Omenn, and Yang Zhang. De novo ma tertiary structure prediction at atomic resolution using geometric potentials from deep learning. bioRxiv, 2022.
Genevieve C Pugh, Jonathan R Burns, and Stefan Howorka. Comparing proteins and nucleic acids for next-generation biomolecular engineering, Nature Reviews Chemistry, 2(7):113-130, 2018
Yifei Qi and John ZH Zhang. Denseepd: improving the accuracy of neural-network-basee compu- tational protein sequence design with densenet. Journal of chemical information and modeling, 1055::224 2020.
Mengying Qin, Zhendong Liu, Dongyan Li, Xi Chen, Xinrong Lv, Xiaofeng Li, Junxue Zhou, and Hui Wang. Improved predictive algorithm ma tertiary structure based on gnn. In 2022 18th International Conference Computational Intelligence and Security (CIS), pp. 117-121. IEEE, 2022.
Lisa Rosenbaum. Tragedy, perseverance, and chance the story of car-t therapy. Engl J Med, 377 (14):1313-133 1315, 2017.
Adam Roth and Ronald R Breaker. The structural and functional diversity of metabolite-binding riboswitches. Annual review of biochemistry, 78:305, 2009.
Kristian Rother, Magdalena Rother, Michat Boniecki, Tomasz Puton, and Janusz Bujnicki. Rna and protein 3d structure modeling: similarities and differences. Journal of molecular modeling. 17:2325-2336, 2011.
Álvaro Rubio-Largo, Nuria Lozano-Gareía, José
Sranado-Criadee and Miguel A Vega- Rodríguez. Solving the ma inverse folding problem through target structure decomposition and multiobjective evolutionary computation. Applied Soft Computing, pp. 110779, 2023.
Frederic Runge, Danny Stoll. Stefan Falkner, and Frank Hutter. Learning to design ma. In Interna- tional Conference on Learning Representations, 2018.
Kengo Sato, Michiaki Hamada, Kiyoshi Asai, and Toutai Mituyama. Centroidfold: a web server for ma secondary structure prediction. Nucleic acids research. 31(suppl.)::2277-vv 2009.
Tao Shen, Zhihang Hu, Zhangzhi Peng. Jiayang Chen, Peng Xiong, Liang Hong, Liangzhen Zheng, Yixuan Wang, Irwin King, Sheng Wang, et al. E2efold-3d: End-to-end deep learning method for accurate de novo ma 3d structure prediction. arXiv preprint arXiv:2207.01586 2022.
Jaswinder Singh, Jack Hanson, Kuldip Paliwal, and Yaoqi Zhou. Rna secondary structure predic- tion using an ensemble of two-dimensional deep neural networks and transfer learning. Nature communications, 10(1):1-13, 2019.
Jaswinder Singh, Kuldip Paliwal, Jaspreet Singh, and Yaoqi Zhou. Rna backbone torsion and pseu- dotorsion angle prediction using dilated convolutional neural networks. Journal of Chemical Information and Modeling, 6166):260--262) 2021.
13

---

Published as a conference paper at ICLR 2024
Michael F Sloma and David H Mathews. Exact calculation of loop formation probability identifies folding motifs in ma secondary structures. RNA, (281)): 1818, 2016.
Cheng Tan, Zhangyang Gao, and Stan Z Li. Rfold: Towards simple yet effective ma secondary structure prediction. arXiv preprint arXiv:2212.14041, 2022.
Cheng Tan, Zhangyang Gao, Jun Xia, Bozhen Hu, and Stan Li. Global-context aware generative protein design. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Cheng Tan, Zhangyang Gao, Lirong Wu, Jun Xia, Jiangbin Zheng, Xihong Yang, Yue Liu, Bozhen Hu, and Stan Z. Li. Cross-gate mlp with protein complex invariant embedding is one-shot antibody designer, 2024.
Raphael JL Townshend, Stephan Eismann, Andrew M Watkins, Ramya Rangan, Maria Karelina, Rhiju Das, and Ron O Dror. Geometric deep learning of ma structure. Science, 373(6558): 1047-1051, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention you need. Advances in neural informa- fion processing systems, 30, 2017.
Jian Wang and Nikolay V Dokholyan. Application of ma tertiary structure prediction tool ifoldma in ma nanotechnology. In RNA Nanotechnology and Therapeutics, pp. 101-109. CRC Press, 2022.
Jingxue Wang, Huali Cao, John ZH Zhang, and Yifei Qi. Computational protein design with deep learning neural networks. Scientific reports, 8(1):1-9, 2018.
Paulina H Wanrooij, Jay Uhler, Tomas Simonsson, Maria Falkenberg, and Claes M Gustafsson. G-quadruplex structures in ma stimulate mitochondrial transcription termination and primer for- mation. Proceedings of the National Academy of Sciences, 107037):1072-1167777 2010.
Katherine Deigan Warner, Christine E Hajdin, and Kevin M Weeks. Principles for targeting ma with drug-like small molecules. Nature reviews Drug discovery, 17(8)5547-588 2018.
Peng Xiong, Ruibo Wu, Jian Zhan, and Yaoqi Zhou. Pairing a high-resolution statistical poten- tial with a nucleobase-centrie sampling algorithm for improving ma model refinement. Nature communications, 12(1):1-11, 2021.
Xiufeng Yang, Kazuki Yoshizoe, Akito Taneda, and Koji Tsuda. Rna inverse folding using monte carlo tree search. BMC bioinformatics, 18(1):1-12,
2017 2017.
Joseph D Yesselman and Rhiju Das. Rna-redesign: web server for fxxedbbakboone 3d design of ma. Nucleic Acids Research, 43(W1): 2015.
Joseph N Zadeh, Conrad D Steenberg, Justin S Bois, Brian R Wolfe, Marshall B Pierce, Asif R Khan, Robert M Dirks, and Niles A Pierce. Nupack: Analysis and design of nucleic acid systems. Journal of computational chemistry, 32(1): 173, 2011.
Yang Zhang and Jeffrey Skolnick. Tm-align: a protein structure alignment algorithm based on the tm-score. Nucleic acids research, 33(7):2302-2399, 2005.
Yuan Zhang, Yang Chen, Chenran Wang, Chun-Chao Lo, Xiuwen Liu, Wei Wu, and Jinfeng Zhang. Prodconn: Protein design using convolutional neural network. Proteins: Structure, Function, and Bioinformatics, 88(7))819-829 2020.
Michael Zuker. Mfold web server for nucleic acid folding and hybridization prediction. Nucleic acids research, (1(3))33406 3415, 2003.
14

---

Published as a conference paper at ICLR 2024
A EXPERIMENTAL SETTING
Datasets We evaluate method on two tasks using three datasets: (i We train and assess perfor- mance on our proposed RNA structure benchmark dataset which aggregates and cleans data from RNAsolo (Adamczyk al.) 2022b) and the Protein Data Bank (PDB) (Bank) 1971, Berman al.) 2000). The benchmark dataset consists of 2218 RNA tertiary structures, which are divided into training (1774 structures), testing (223 structures), and validation (221 structures) sets based on their structural similarity. The RNA sequence length is highly consistent within each subset, mini- mizing negative effects from distribution shifts. A detailed illustration of the creation and features of our dataset is shown in Appendix B (ii) To test the generalization ability,, we apply pre-trained models to the Rfam (Gardner et al. 2009. Nawrocki et al., 2015) and RNAPuzzle (Miao et al. 2020) datasets that contain non-overlapping structures. Rfam and RNAPuzzle are common datasets that contain novel RNA structures, which have 105 and 17 tertiary structures, respectively. The detailed description is in Appendix C
Metries Previous protein design approaches usually use sequence perplexity and recovery metrics
to evaluate the performance (Li et al., 2014; Ingraham et al. 2019 Wang et al.) 2018 Jing al.) 2020 Tan et al. 2022] Gao et al. 2022at JO Connell et al.] 2018). However, since RNA sequences only contain four letters, the perplexity metric traditionally used in natural language processing may not accurately reflect nucleotide-level prediction accuracy. Therefore, we use the recovery score to assess nucleotide-leve accuracy by measuring the proportion of target RNA sequences that our model regenerates correctly. We also introduce the Macro-Fl score to evaluate performance by con- ceptualizing the RNA design task as a multi-class classification problem. The detailed illustrations of recovery and Macro-F1 scores are provided in Appendix E. We conducted each experiment three times with different random seeds and reported the mean and standard deviation values.
Implementation Details We trained the model for 200 epochs using the Adam optimizer with a learning rate of 0.001. The batch size was set as 64. The model was implemented based on the standard PyTorch Geometric (Fey & Lenssen 2019) library using the Py PyTorch 1.11 library. We ran the models on Intel(R) Xeon(R) Gold 6240R CPU 2.40GHz CPU and NVIDIA A100 GPU.
The model's encoder and decoder each had three layers. With a dropout rate of 1. it considered 30 nearest neighbors and a vocabulary size matching RNA's four alphabets. For the baseline models, the loss function is the negative log-likelihood loss 1 Σi= log p(y:). where p(y) denotes the predicted probability of the true class. Notably, p(y) are presented in logarithmic form. With the exception of SeqRNN (h=256) and SeqLSTM (b=256), which feature hidden size of 256, the remaining models were configured with a hidden size of 128. In our study, we have repurposed StructGNN and GraphTTran Trans, initially designed for protein design as described in Ingraham et al.'s work (Ingraham et al.) 2019). These models have been adapted for RNA design, harnessing the capabilities of graph neural networks and the transformer architecture, respectively.
N
Baseline Models We explored six baseline models that could categorized into four classes based on the level of structural information utilized: (i) Sequence- based models (SeqRNN and SeQLSTM) that do not utilize any structural information and could be viewed as the performance reference for RNA design; (ii) A tertiary structure-based model (StructMLP) that exploits structural features while ignoring the graph topological structure; (iii) Tertiary structure-basse
models (StructGNN and GraphTrans) (Ingraham et al., 2019) and PiFold (Gao et al., (002b) that incorporate the graph topological structure; (iv) several state-of of-te--f-hh-arr the--rt art RNA sequence design models based on secondary structure (MCTS-RNA (Yang et al. (2077). LEARNA (Runge al. 2018), eM2dRNAs (Rubio-
Largo et al. 2023). aRNAque (Merleau & Smerlak 2022)):
B BENCHMARK DATASET DETAILS
B. OVERVIEW
We meticulously gathered RNA tertiary structure data from two principal sources: RNAsolo (Adam- ezyk et al., 2022a) and the Protein Data Bank (PDB) (Berman al. 2000). RNAsolo is an RNA 3D structure database that provides free online access for downloading. After the initial collection, cleaned the structures by removing non-RNA data. We then annotated and assigned them to
15

---

Published as a conference paper at ICLR 2024
equivalent classes defined by pairwise analysis of structural redundancy. We refined the abnormal samples that had inconsistent sequences and structures according to the PDB database.
Regarding the sequence length distribution, provided a quantitative analysis shown in Table 6 From the original RNA solo dataset, 95.79% of sequences are below 500 nucleotides in length, with sequences exceeding this length being sporadically dispersed between 500 and 4000 nucleotides.
Table 6: Sequence length distribution in RNA solo dataset.
Range
Number of Sequences
Percentage (Cumulative)
0-50
1875
69.81%
50-100
483
87.79%
100-500
215
95.79%
500-1000
22
96.61%
1000-2000
53
98.59%
2000-3000
24
99.48% %
3000-4000
14
100.00%
Incorporating such extremely long sequences would result in the selected features being over- whelmed by superfluous padding, thereby significantly escalating the computational resources needed. Therefore, we filtered out those with more than 500 nucleotides. We provide a runtime and memory usage analysis in Table
Table 7: Runtime and memory usage evaluation of RDesign.
Sequence Length
Memory (MB)
Runtime (Second)
500
7705 (1.00 x)
7.5 (1.00 x)
2000
16499 (2.14 x)
33.6 (4.48 x)
4000
48913 (6.35 x)
64.0 (8.53 x)
After filtering out the sequences with more than 500 nucleotides, we collected a total of 2218 high- quality representative RNA structures, each with resolution of less than 4.0 Å.
B.2 DATA CLUSTERING
We used the TM-score (Zhang & Skolnick, 2005) to assess structural similarity among RNA struc- tures. To group samples with similar structures, employed graph-based clustering approach. Each structure is represented as a node, and the TM-Score between every pair of structures is com- puted. If the TM-Score for a pair exceeds 0.45, edge drawn between them, indicating their high similarity. After processing all pairs in this manner, clusters are identified as connected components within the graph. From our dataset of 2218 RNA structures, we yielded 987 distinct clusters.
B.3 DATA SPLITTING
How to split biological datasets in way that avoids information leakage and provides consistent evaluation an important topic in Al for science. To achieve this, we split the collected data based on two principles: (i) avoiding similar structures in different sets and (ii) maintaining similar length distributions across sets. After obtaining 987 clusters, we added each to the training, validation, and testing datasets, maintaining vigilance prevent similar structures from appearing in divergent sets. Specifically, we calculated the average length of all sequences and sequentially allocated clusters to the respective train/validation/test sets, managing align each cluster closely with the global average length. Consequently, we maintained a consistent sequence length distribution across all sets, depicted in F
In this way, we allocated entire clusters- rather than individual samples-an to specific sets, eliminat- ing the possibility of data leakage between sets. Of the 987 total clusters, 838 were designated the
16

---

Published as a conference paper at ICLR 2024
0.014
Train Dalaset
Test Dalaset
0.012
Val idititiion Dalaset
0,010
D 0008
0.036
0.004
0.002
0,000
100
200
300
500
RNA sequence length
Figure 6: Evaluation on sequence length distribution in the train, test, and validation set.
training set, 68 the validation set, and 81 to the test set. The cluster size distribution in each is shown in Table It is worth noting that clusters containing large number of samples (size >> 30) are placed in the training set rather than the validation and test sets to prevent bias from dominant clusters when evaluating model performance.
Table 8: Size distribution of clusters in each dataset
Dataset
Size S
2 Size < 10
10 < Size S 30
Size > 30
Train
810
25
0
3
Validation
26
0
Test
53
26
2
0
Finally, we partitioned the original dataset into training (1774 structures), testing (223 structures), and validation (221 structures) sets based structural similarity and RNA sequence length. The evaluations of the baseline models were conducted on this split to ensure fair comparison.
B.4 DATA AUGMENTATION
For the purpose of sample-level representation learning, we employed method generate pseudo RNA structures by introducing slight perturbations to the coordinates. This approach takes into account the possibility of minor errors experimental data. Initially, we obtained the coordinates of each atom within the RNA chains and introduced Gaussian noise, with the magnitude of noise determined by the scale of each coordinate value.
Subsequently, we assessed the structural fidelity of the generated data by calculating two key met- rics: the Root Mean Square Deviation (RMSD) and the TM-score in comparison to the ground truth structure. We excluded any generated data with an RMSD exceeding 1.0 Á or TM-score lower than 0.8. This stringent criterion was applied to ensure that the augmented samples maintained structural characteristics closely resembling those of the original data.
C REAM AND
DATASET DETAILS
C.I RFAM DATABASE
The Rfam database Kalvari et al. 202I) is a collection of RNA families, each represented by multiple sequence alignment, consensus secondary structure, and covariance models that describe
17

---

Published as a conference paper at ICLR 2024
the consensus structure and base-pairing constraints of the RNA family, In the 14.7 release, 122 usable RNA sequences are aligned with their 3D structure stored in PDB format.
C.2 RNA-PUZZLES DATASET
The RNA-puzzles dataset (Mio et al. 2020 is a public dataset that consists a series of RNA structure prediction challenges, designed to evaluate and improve the accuracy of computational methods for predicting the structure of RNA molecules. Each RNA-puzzles challenge provides a of experimental data, such X-ray crystallography measurements stored in PDB format, along with a target sequence and structure for the RNA molecule. It has been widely used to benchmark and compare different computational methods for RNA structure prediction.
D LOCAL COORDINATE SYSTEMS UTILIZED IN OUR PAPER
Generally, the conventional modeling mentioned is the common protein tertiary structural modeling approach from (Ingraham et al. 2019). It considers dihedral angles node features and uses Co atoms to build the local coordinate system. While it is sufficient for extracting features from protein structures, they may be suitable for the more complex RNA structures. We propose a compre- hensive approach to modeling RNA tertiary structures that are specifically designed for RNA.
Our baseline models StructGNN and GraphTrans are from (Ingraham et al., 2019). which uses the conventional modeling approach. We adjusted this protein modeling for RNA by replacing its three dihedral angles with RNA's six and using the P atom instead of the Co atom.
We construct a local coordinate system for each nucleotide i. This system is defined as:
Q [b;, n; b, n:);
(10)
where b, is the negative bisector of angles between the rays of contiguous coordinates (æ4-1,P, 2..P)) and (I,+1,P, P). and n, a unit vector normal to that plane. Formally, b, and n, are:
Xi
X
u. 02+1
u. X 0i+1
b
(11)
x; x,
u.41
[u u,+1
Ui+1
#1+1
b;
C Ci++
Normalize
Di
Di
U1
20;
T;-1
n2
7;
U4+1
b;
b;
bi n
Figure Procedure of building the local coordinates.
As illustrated Figure 7 constructing the local coordinate system involves calculating the bond vectors between successive P atoms and using these to determine the bisector b, and normal n, Since this coordinate system defined relative to a fixed set atoms within each nucleotide, is invariant to overall rotations and translations of the RNA structure.
18

---

Published as a conference paper at ICLR 2024
E EVALUATION METRICS
E. RECOVERY
To rigorously assess the nucleotide-level prediction accuracy of RNA sequence model, adopt the following recovery metric:
N Σ
Recovery(S Xx))
1 S = arg max P(S 12))
(12)
N
where P(S |X Xes)) represents the probability of predicting nucleotide S. at position given the input structure X. and [ is the indicator function. This recovery metric evaluates the fraction of nucleotides in the RNÁ sequence that correctly predicted the most probable by the model, averaged over all N positions. thus rigorously assesses the model's ability to accurately predict the sequence at the nucleotide level based on the inputs. This metric provides a nuanced evaluation of the predictive accuracy achieved by the RNA sequence model at the nucleotide resolution.
E.2 MACRO-F1 SCORE
We introduce the Macro-Fl score to evaluate performance by conceptualizing the RNA design task as a multi-class classification problem and adopt the Macro-F1 score metric to comprehensively evaluate the model's ' performance on different classes. It is calculated as the average of the Fl scores for each class of letters.
Macro-Fl
Σ
Fle:
(13)
|C|
cE(A,U,C.C)
where |C| is the number of letters and F1, is the FI score for letter c. The F1 score is the harmonic
mean of precision and recall. It considers both the proportion of correctly predicted instances of letter and the proportion of instances predicted letter that belong to letter
Procision, X Rocall.
F1c = 2 Precision. Recall.
(14)
Therefore, the Macro-F1 score comprehensively reflects the model's prediction performance dif- ferent letters by evaluating the Fl score for each letter. Compared to simply calculating the average precision or recall over all letters, can more comprehensively evaluate the model's ability iden- tify instances of each letter, especially in cases where there is an imbalance between the number of instances different letters. It examines the proportion of correctly classified instances at the letter level while balancing precision and recall.
E.3 PERPLEXITY
Perplexity is a common metric used in various fields including natural language processing and bioinformatics. In the context of RNA design, perplexity can be used as measure to evaluate the uncertainty of RNA sequences, reflecting how well a model can predict a given RNA sequence.
Perplexity(8", X = exp(
Σ S log P(S )),
(15)
N
Where P(S, ( is the output probability from the model.
While perplexity crucial metric in natural language processing (NLP) and protein design, its suitability is significantly less in RNA design due to the inherent simplicity and structural constraints of RNA sequences. RNA sequences, with only four letters A, U, C. G, lack the compositional complexity found in natural languages, making perplexity less discriminative and revealing model evaluation metric in this context. Considering these limitations, we chose not to prioritize this metric in main text. We show the results in Table 9 It can be seen that the outcomes are not aligned with other metrics, and notably, sequence- methods consistently outperform structure-based methods, which is phenomenon contrary common sense.
19

---

Published as a conference paper at ICLR 2024
Table 9: The perplexity metric on the benchmark dataset.
Perplexity (%)
4
Method
Short
Medium
Long
SeqRNN (h=128)
4.02.10.02
4.01110.1
4.00.10.01
4.01
SeqRNN (h=256)
4.00++0.01
3.99+0..00
3.99++0.00
3.99+0.001
SeqLSTM (h=128)
4.00=0.01
4.00:10.00
3.99-0.011
4.00+:0.01
SeqLSTM (h=256)
4.00.10.00
3.99110.00
3.98
3.99
StruetMI.P
10.83=1.32
7.78+2244
7.93+:2.74
8.51+2.35
StructGNN
0.37
4.87:0044
5.12++0.43
4.95:L0.43
GraphTrans
1.79
6.32+2.23
6.56+2.26
6,25+2.07
RDesign
4.28+0.14
3.98+00.10
3.97=-0.04
4.10+00.99
F HYPERSPHERICAL SPACE OF REPRESENTATION LEARNING
In contrastive representation learning, mapping features to hyperspherical space is technique used better represent similarity. In this supervised learning algorithm, the similarity between two samples is represented as the distance or similarity between their feature vectors. To better represent this similarity, the features are mapped hyperspherical space. Specifically, the feature vector is mapped to hyperspherical space by dividing it by its norm ||/|], resulting in unit vector h ||h||" This unit vector is the representation in hyperspherical space. The projection head comprises an MLP and feature normalization. The advantage of this method is that normalizes feature vectors to the same length, avoiding the influence of scale changes in the feature space similarity measurement. Additionally, since hyperspherical space is convex space, similarity measurement this space is convenient and efficient.
Using hypersphere for embedding data points is advantageous because it allows each data point to cover larger expressive space compared to confining all points within small region of Euclidean space. This enables the model to maximize the expressive power and learnability of each individ- ual data point within the limited dataset. Conversely, data distributions are controlled, the representations may collapse and cluster in a small portion of the space, limiting their individual expressiveness. The hypersphere provides a structured way to distribute data representations evenly. This uniform assignment of limited data to hypersphere space is thus crucial for ensuring every data point can be maximally informative during contrastive representation learning. The design choice stems directly from the need to learn efficiently from scarce RNA structure data.
G MODEL DETAILS
G.1 RDESIGN MODEL
Here we present levels of the system paradigm of our model. The brief pipeline is shown in Figureß] which serves as a comparison with the baselines proposed in our paper. Another detailed pipeline presented Figure 9 which draws comprehensive view of our model, indicating the function of each part, and illustrating the formulation of the loss function.
RNA Tantas Souctmn
Hertayy
Secondary Structure
Compraaann
Consraining
Streaur Moicling
Decoder
AND
ANN
Model Input
Featurizer
Encoder
Decoder
Structure Constraining
Figure 8: The pipeline RDesign model.
20

---

Published as a conference paper at ICLR 2024
PEA MAD
122
Secndday Shuctwe
Desrrizaie RNA Strnktn Modking
Constrining
Senatual
Desster
Input
Featurizer
Decoder
Straccure Constraining
Superwsed
Loss
Contassie
Sample LoSs
M.Tinmaer Lesnale
Loss Measuring
Cluster Spliting
Sample Spliting
Contrrtow
Figure 9: Detailed system pipeline of our model.
G.2 BASELINE MODELS
In this work, we employed five baseline models perform as the reference to our RDesign model. For each of them, we provide brief introduction and summarize the pipeline in the figure below. In experiment, both the encoder and decoder contain three layers by default.
SeqRNN SeqRNN is context-aware recurrent neural network model that is suitable for sequence generation tasks. Based on recurrent networks, it can utilize its memory to process the input se- quences and learn the distribution of the sequences by adjusting the weight coefficients. The output of SeqRNN a sequence with the same length as the inputs. The pipeline shown in Figure I0
RNA Sequence
Lineur Layer
Embedding
Sequenial
RNN
RNN
Model Input
Embedding
RNN
Model Output
Figure 10: The pipeline of SeqRNN model.
SeqLSTM Similar to SeqRNN, SeqLSTM is an LSTM-based context-aware suitable for sequence generation tasks. It can extract the sequence order features into embeddings. Then the output quence could obtained by passing the embeddings through a linear layer. The pipeline of the model is shown in Figure
21

---

Published as conference paper at ICLR 2024
RNA Soquence
Linear Layer
Embedding
Sequeniai
Model Input
Embedding
LSTM
Model Output
Figure 11: The pipeline of SeqLSTM model.
StructMLP StructMLP represents a structure-based RNA sequence prediction model. This ap- proach harnesses the structural characteristies RNA molecules through Multilayer Percep- tron (MLP) architecture, allowing for the structural insights to produce sequence predictions. The schematic representation the model's workflow illustrated in Figure 12
RNA Tertirry
Conventional Modeling
Structural
Decoder
Model Inpul
Featurizer
Encoder
Decoder
Figure 12: The pipeline of StructMLP model.
StructGNN contrast to StructMLP, StructGNN employs a Graph Neural Network (GNN) as its feature encoder. This choice equips StructGNN with the capacity capture and learn the intricate graph structures inherent in RNA tertiary structures, a capability beyond the reach of StructMLP. The model's workflow is visually depieted in Figure
Conventional Modeling
Tertiory
Stractural
Decocler
CNW
GNN
Model Input
Featurizer
Encoder
Decoder
Figure 13: The pipeline of StructGNN model.
GraphTrans The primary distinction between GraphTram and StruetGNN lies in the feature coder employed. GraphTrans leverages the Transformer architecture for this purpose. To illustrate the model's workflow, please refer to Figure 14
22

---

Published as a conference paper at ICLR 2024
Conventional Modeling
RNA Tariary
Transformer
Transformz
Strnetuml
Decoder
Model Input
Featurizer
Encoder
Decoder
Figure 14: The pipeline of Graph Trans model.
H EVALUATION ON EXTERNAL DATASETS
In the main text, we used the pretrained model trained on the entire training set to evaluate models on the external datasets Rfam and RNA-Puzzles. We show the structural similarity distribution between our training data and the three test sets Figure T5| This distribution demonstrates that the majority of structural similarities similarity. In our benchmark dataset, the structural similarity between all test data and any training sample remains below 0.45. However, in the Rfam and RNA-Puzzles datasets, there are still a few instances similar samples. We directly evaluated the models pretrained on the complete training set these two datasets, as this approach closely mirrors real-world application scenarios.
concentrated around TM-score of 0.1, which suggests low structural
Rtam RNAA-puzzles
Test
830
Figure 15: The pairwise structural similarity between our training and the three test sets.
To evaluate the model's generalization ability on dissimilar samples, we implemented a filtering process the training set. While evaluating models the Rfam dataset, we removed all samples from the training set that exhibited a structural similarity exceeding 0.45 with any sample in Rfam. Likewise, during the evaluation on the RNA-Puzzles dataset, we excluded all samples from the training set that possessed structural similarity scores surpassing 0.45 with any sample in RNA- Puzzles. The distributions of pairwise structural similarity between the training set and three test sets are visualized in Figure 16
The results on the Rfam and RNA-Puzzles with filtered training sets are shown Table 10 It can be seen that RDesign still outperforms other baseline models.
I COMPARISON ON AUROC AND AUPRC METRICS
We conduct comparisons with baseline models using the AUROC and AUPRC metrics. For our benchmark test set, the results for the AUROC and AUPRC metrics presented in Tables and
23

---

Published as a conference paper at ICLR 2024
RNA-puzzles
8.00
3.50
TM-Scare
Figure 16: The pairwise structural similarity between our training set and the three test sets after removing similar structures in Rfam and RNA-Puzzles.
Table 10: The recovery and Macro-Fi scores on the Rfam and RNA-Puzzles datasets with filtered training sets.
Recovery (%)
Macro F1 (x 100) 1
Method
Rfam
RNA-Puzzles
Rfam
RNA-Puzzles
SeqRNN (h=128)
31.05-0.511
31.51
11.92.10.17
12.11
SeqRNN (h=256)
31.04=0.50
31.53++0.04
11.93+0.116
12.12+00.02
SeqLSTM (h=128)
30.28=0.200
31.35:10.26
12.36
12.40:t0.15
SeqLSTM (h=256)
31.45-10.08
31.79.10.44
11.76110.09
12.07-10.00
StructMLP
26.77-3.38
27.06113.81
16.22-12.43
16.72-12.53
StructGNN
1..41
20.68++0.70
14.54
12.70++2.60
GraphTrans
27.50=4.115
25.69
20.66:2..51
10.17:t0.14
RDesign
41.28-0035
43.99+1..67
38.76-+0.82
43.11+0.38
12
respectively. Regarding the Rfam and RNA-Puzzles datasets, the AUROC and AUPRC results
are displayed in Tables 13 14 15 and 16 respectively. RDesign consistently outperforms the other baseline models in these datasets.
Table 11: AUROC on our benchmark test set.
Method
C
G
SeqRNN (h=128)
49.40 0.18
50.29 0.58
51.01
0.60
49.48 4 1.21
SeqRNN (h=256)
51.91 0.57
49.77 0.51
51.58 + 0.32
50.88 + 0.41
SeqLSTM (b=128)
49.38 + 0.30
49.00
+
SeqLSTM (h=256)
50.06 0.95
51.48
47.65
50.44
StructMIPP
48.82 1.17
50.10
51.26
49.44
StrictGNN
53.17 1.02
49.00 2.58
47.92
50.59 + 1.42
GraphTrans
53.06 + 0.44
49.19
4..44
52.08 + 2.62
PiFold
49.09 1 1.60
49,97 + 2.70
50.51 1 4.13
51.21 1 0.74
RDesign
66.16 0.27
63.22 0.38
65.58 0.41
67.31 0.21
24

---

Published as a conference paper at ICLR 2024
Table 12: AUPRC on our benchmark test set.
Method
U
C
G
SeqRNN (h=128)
24.61 0.23
25.97
0.44
23.36 1 0.66
27.45
0.34
ScqRNN (h=256)
26.90 + 0.46
25.64 + 0.34
23.83 + 0.34
28.97 + 0.24
SeqLSTM (h=128)
24.49 + 0.23
25.56 1.60
22.72
27.55 1.49
ScqLSTM (h=256)
24.78 0.57
26.40
0.16
28.29 0.65
StructMLP
24.72 + 0.57
25.17
23.47
27.18 1.30
StructGNN
27.18 0.49
24.63 1.68
2..14
28.33 1 0.84
GraphTrans
26.65 1 0.25
24.88 4:88 2.87
22.07
1 1.84
28.66 1.62 1.62
PiFold
24.30 1 0.84
25.01 1.0111700
23.41
2.28
28.62 1 0.61
RDesign
40.37 0.08
34.60 0.89
34.91
0.89
45.17 0.81
Table 13: AUROC Rfam dataset.
Method
A
U
C
G
ScqRNN (h=122 128)
50.60 1.14
50.06 1.65
52.43 + 1.04
49.33 + 1.10
ScqRNN (h=256)
51.60 + 0.71
50.11 0.29
51.62 + 0.67
52.32 1.64
ScqLSTM (h=128)
49.24 1.47
50.31 1.52
50.59 2.35
47.58 = 1.56
SeqLSTM (h=256)
50.04 + 1.36
50.01 0.88
50.34 0.85
50.00 1.41
StructMLP
46.97 2:20
$2.09 2.96
+ 49.77
47.56
StructGNN
53.18 0.85
50,93 + 1.69
48.10 2.37
51.58 + 0.37
GranhTrans
51.38 + 2.72
48.82 + 1.57
47.51 2.07
53.04 + 1.72
PiFold
49.34 :
48.86 1.78
3,07
51.57 - 0.92
RDesign
67.89 0.24
64.87 0.25
66.42 + 0.57
66.78 + 0.06
Table 14: AUPRC on Rfam dataset.
Method
A
U
C
G
SeqRNN (h=128)
22.79 0.66
21.10 0.96
27.61 + 0.47
31.32 + 1.18
SegRNN (h=256)
23.66 + 0.40
20.81 - 0.40
27.37 +
33,87 + 1.24
SeqL.TM (h=128)
22.11 + 0.66
21.46
26.71 1.67
30.08 1.37
Seq..TM (h=256)
22.80 + 0.75
20.86
26.12
31.61 1.75
StructMI.P
21.55 1.37
22.31
25.53
29.07 + 0.92
StructGNN
25.45 0.67
21.37 + 1.56
1.474
31.99 1 1.03
GranhTrans
24.09 + 1.40
20.14 1144:120
25.16:1.1
32.92 1 1.24
PiFold
22.83 1 1.53
20.31 1 1.33
25.72 1.70
32.63 1 0.68
RDesign
40.40 1 0.85
31.65 0.15
42.36
47.36 0.24
Table 15: AUROC on RNA-Puzzles.
Method
A
U
C
G
ScqRNN (h=128)
49.25 1.01
49.70
0.31
50.80 1.40
49.03 + 0.87
ScqRNN (h=256)
49.29 + 0.68
49.31 + 1.45
50.69 + 0.53
51.78 + 0.78
ScqL.TMM (h=128)
48.53 + 0.81
50.39 + 1.90
49.62
47.57 1.76
SeqLSTM (h=256)
50.63 1.43
49.94 1.15
48.44
49.13 1 2.33
StructMLP
44.98 2.31
3.67
51.1115.11
46.92 1.91
StructGNN
$3.46 1.59
48.81 1. 2.88
46.97 3.26
50.62 1 1.35
GraphTrans
51.99 + 2.02
47.48 + 0.33
46.14.1.991
51.84 + 3.13
PiFold
49.94 + 2.69
48.68
= 3.67
49.55 6.09
52.49 0.57
RDesign
71.43 0.77
70.69 1.01
70.56 0.78
0.59
25

---

Published as a conference paper at ICLR 2024
Table 16: AUPRC on RNA-Puzzles.
Method
A
U
C
G
SeqRNN (h=128)
22.15 0.53
20.13
0.39
25.90 0.87
32.74 0.92
SeqRNN (h=256)
22.50 + 0.15
18.90 + 0.41
26.10 0.65
34,50 0.23
SeqLSTM (h=128)
21.88 + 0.59
19.74 1.28
25.64 +
32.00 + 1.75
SeqLSTM (h=256)
22.91 + 1.20
19.62 = 0.40
2.700
31,82 1.98
StructMLP
20.32 1.06
21.40
33227
29.76
StructGNN
25.85 2.14
19.18 1118:200 2.00
24.12
33,30 - 0.95
GraphTrams
24.82 + 1.08
18.53 + 0.19
24.07 1.70
33.08 + 3.34
PiFold
23.58 + 1.34
18.49 + 1.52
25.44
34.43 + 1.13
RDesign
48.93 0.58
37.74
1.87
46.44
53.22
0.12
Comparison
Precisio-Reccal Compatison
Delaset
SeaRRAL 2122 AUROOC 6:401
SeaPARUhCE INAUREC
SeaPANL 1255 MLAIROC 6.53
NIZ2 IKAUFAC 627)
SeaLSTI,
431
Stueich AUROC 3.551
SuvciieN AUTIC - 2
Cradithrrai 0
Craptiura AUTICC
AURIC
KDwsgg LLAUPRE 0A2)
Tho Fobtle 5200
05
Positive
Figure 17: The ROC/PRC curve comparison on the base A of our benchmark test set.
Comparson
Prectsion Recail
Seatirt 1124 AUBOr 0.5
Sestie JUD B1 A.Pee
Seatsi
Seatii 1034 in susec 992
Mruare Di mene del
SeruttEE mLL 0.28
Pikiss ju mapec 0.25)
nDnaig AUROC
#Ceelgi 0.3)
Thus Rostave 8000
Palse Posttive Fabe
Rocall
Figure 18: The ROC/PRC curve comparison on the base U our benchmark test set.
26

---

Published as a conference paper at ICLR 2024
ROC Curve for Test Oatazet
Prezision Rocall Curve Compartron Test Ostaoet
VTM 5125 AIRRE 2:241
WeSESTTE AUAOC 47!
VOSA AIPRE 6.1
GrapiTTaaic IC A 221
CrpptTura IC AURRC 0.241
AUADC
AATIC 0.231
Thuc Poobive Ratc
36-
False Fositive Rate
Rocall
Figure 19: The ROC/PRC curve comparison on the base C of our benchmark test set.
Camparison
Procision Recall Comparison
Dataset
SeafRRL 5123 12 A2AAR0
Segiin 3320 AJFIC
SeafANl 1255 12 AUA0: 531
Seat F226 AUFIC 622)
SeaLSTN 10
6.22
Strcinr MIROC 2.55)
SinutM 10 AUFREC 0.211
Streich AJROC 0.50
StueAGUM IG AUTII - 301
Cresithrrai
Creptitern AUTIIC
Kordgi MIROC 6677
Aundign NUMAE 6.50
Tuo Posithre R350
66
Positive
Recall
Figure 20: The ROC/PRC curve comparison on the base G of our benchmark test set.
Comparison
Precision Recall Curve Comparison
SAafRRL 1125 AIROC 6.52
SuStROL/CZE AUFII 024
SenLSTH 3258 AUnDC $51
SUILITH AUPII
Mtr CA AUMOC 3:41
StrvciALL LA AUPRC 0.211
GraahTTaan AUAOC
Grapiiiane AAPRIC
CIA AUROC 4.4
Fills MARRIC 0.22)
Roudgn AUROC
KGuegg GA2)
mhe Restave R0te
Falsa Positive Rabe
Recall
Figure 21: The ROC/PRC curve comparison on the base A Rfam dataset.
27

---

Published as a conference paper at ICLR 2024
ROC Curve forpptttonntrUU RR&M Datoset
PrssioonRecall Curve Comparison for REAN Dotee
SesESTTL 3O28 AUAOC MM
SOIINT MIRRI 8:211
SeSESTTE AINOC 44
SOVISTT 6.211
GrepiiTurs IU RUROC- E.12
CrgghTera TU AMFRIC 2.22
IREGIU AUAOC- E.77
JARRIC
Thuc Poodive Ratc
36-
False Fositive Rato
Rocall
Figure 22: The ROC/PRC curve comparison on the base U of Rfam dataset.
BOC Comparison
Dataset
Precision Recall Curve Comparison in Dataset
SeafFRL 5120 AUnOC 0.52
SeafRNL 1255 AUROC 0.521
Suatenl hod6 ACA.I 622)
SeaLSTN A 541
ICAMEC 6.2)
Strcin AUnOC 252)
StruMALI IC ALFRE 0.21
Streich AIDOC 2553
Seudicin IC AUTIC - 221
Cresithrrii - AM;
Crapithur AUTIIC
Kordgn i.. ABRO 043))
Kividgn IC ALPAE 6.30)
Tho Poritrre R350
Positive
Recall
Figure 23: The ROC/PRC curve comparison on the base C of Rfam dataset.
Curve Compartion
Predstoo Reciilnn-Reall Comparison
SAsfRRL X123 AUno: 4B:
Seattal J326 IC AUFIC - 030)
SeatiV) 2235 12
Seattin
SenLSTH 3258 12 AUnoc 2/40
SeaLSTN Jiz6 15 AUMII 0.301
Str IC AUROE 2 2,40
StruMMPP 70 AUPRC 0.501
GraahTaanss HUROC
Grephliuen AMPRIC 2.22
PIO AUROC - 6.221
Wilid AMRIIC 0.35)
Roedgn tG 667
Adselggn AJRAC G47)
mhu Restavv Rote
Falsa Positive Rabe
Recall
Figure 24: The ROC/PRC curve comparison on the base G of Rfam dataset.
28

---

Published as a conference paper at ICLR 2024
ROC Curve Comparison for Puzzl Ostsoet
Predzion Secall Curve Comparsen A Puzzle Dataret
SeSESTTL F028 ALAIAE wn da:
STM V1D0 KIRRI 2-221
NeSESTTE ILADIAE 111
GrepiTTuTs 1A - 222
CraptTura AUFRIC
AATRIC
Thuc Poobive Ratc
36
False Fositive Rato
Rocall
Figure 25: The ROC/PRC curve comparison on the base A of RNA-Puzzles dataset.
Comparison
Dataset
Precision-Secall Curve Compar son
SeafRRL 1120 AUROC 0.51
Seattn /3.0 AJFRI
SeafANL 1255 AUROC 0.501
Soottel Jcd6 AUFfI 01.0)
SeaLSTi 4U10c
Strciee2 AUNOC 2.48
SeousMith 10 AUFRE 0121
StreicW AUAOC 652
Soudicia tu AUTTIC - 221
Cresiihurs
Kordgn di ABROC
Kivedgn (t AURAE 0.200
Tho Posttre R350
Positive
Recall
Figure 26: The ROC/PRC curve comparison on the base U of RNA-Puzzles dataset.
Comparison Puzzle
Preditoon Secall Comparison
SAafRRL 1123 AUROC 840t
EustAOLIO 1028 AUPSI 025)
Sestii) 0.51
SenLSTH 3258 AUnoc 4t
SUaLITH IC AUMIC 0.251
Mtr AUnOC 3se)
StruatLi 1C AUPRC 0
GraahTaaanic AUAOC
Grapiihane AMPIC 0.34)
PIS AUROC 0.46
Pilass AMRIC 0.24)
Roudgn AUROC
Köeelgn AUPAC GAl))
mhe Rostave Rote
Falsa Positive Rabe
Recall
Figure 27: The ROC/PRC curve comparison on the base C of RNA-Puzzles dataset.
29

---

Published as a conference paper at ICLR 2024
Compar
Dataset
Precisio-ftecal Curve Comparison
Dataset
SesFRRL 5120 12 12AURDC 00:0
Segtin, 3320 IC AJFIC
SeaFANL 1255 12 AUA00 0
Seatin 3226 AUFIC 635
SeaLSTN 1228 10
0.341
Stucin MIROC
Scuut 10 AUFRE 0.311
Strecc AJREC 0.50
StuetGal to AUTIC - 221
Cresihhrrai
Creptihers AUTIC
Kordgi MiROC - 6.71)
Auxdign AUMAE - 05.)
Tuo Positrre R350
66
Positive
Figure 28: The ROC/PRC curve comparison on the base G of RNA-Puzzles dataset.
30


Reviews:
[{"review_id": "ObVoyCh4o9", "reviewer": "Reviewer_cP6M", "summary": "This manuscript proposes several methods for identifying the primary structure, or nucleic acid sequence, of an RNA molecule given its tertiary structure, or three-dimensional structure. RNA structure prediction, for reasons that will be discussed, is studied much less than protein structure prediction. This discrepancy presents two major issues: Firstly, there are not enough high-quality RNA structures to build models from, and secondly, there are not as many methods developed to predict RNA structure. Therefore, the authors initially compile and refine a dataset, carefully dividing it into training and testing sets, and then develop new algorithms to address this problem. Since there are no existing methods to predict the RNA sequence from its structure, the authors not only propose a novel method — inspired from Ingraham _et al._ (2019) — but also introduce five baseline methods for comparison. Finally, they test their models on two significant, completely independent datasets: Rfam and RNA-Puzzles.", "questions": "I decided to move some of the perceived weaknesses to the questions:\n\n- The base atoms are masked; however, purines (R or A/G) and pyrimidines (Y or C/T) differ significantly in size, and the model could have simply learned to predict the base using the spacing between the backbone atoms of neighboring nucleotides. This is not as significant of an issue in proteins, as there are twenty different side chains that do not canonically pair with each other. One way to test this is to evaluate performance by converting all A/G nucleotides to R and all C/T nucleotides to Y, then report recovery scores based on R and Y only. If this score is substantially higher than the original one, then the model may simply be learning whether bases are purines or pyrimidines.\n\n- When constructing the neighborhood graph, wouldn't it be more intuitive to use a ball around each nucleotide? Could enforcing a fixed K result in some edges that are too distant in space?\n\n- Can't this method be iterative? That is, determine the primary sequence from the structure, then compute secondary and tertiary structures from the inferred primary sequence, identify discrepancies, and predict a new primary sequence, repeating this process until a certain condition is met.\n\n- Related to the previous question, is the \"recovery score\" the best metric? Is the goal to predict THE primary sequence or A primary sequence that gives rise to a very similar tertiary structure? Recovery scores, in general, tend to be low, and reporting predicted RMSD (e.g., using RhoFold) might be helpful in this regard.\n\n- RNA structure is not entirely rigid, and some techniques like NMR can capture an ensemble of structures. Would all such structures in the bundle get the same sequence?\n\n- Approximately 90% of RNA sequences are short (under 100 nt), and a Bayesian Optimization method might work well for this problem. It involves obtaining the tertiary structure, computing the secondary structure, starting with a random sequence, computing its secondary structure, and using the model to propose a new sequence until a close enough solution is found.\n\n- Can Cohen's Kappa be used instead of or in addition to Macro F1?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The authors developing this method — the first of its kind — have done an impressive job on multiple fronts:\n\n- Although the proposed method is an extension of the \"protein structure to sequence\" method proposed in Ingraham et al. (2019), RNA molecules are different from protein molecules, featuring more dihedral angles and a different local geometry. This necessitates the development of new encoding schemes.\n\n- The attention and care the authors have devoted to compiling a clean RNA structure dataset are noteworthy. The training and testing splits are carried out in a structurally-informed manner, which is crucial in structural biology. Additionally, they have maintained similar length distributions.\n\n- Given the limited availability of RNA structures, the authors employ a hierarchical representation learning scheme to address this challenge. This approach groups similar clusters together, much like protein fold families. They also utilize data augmentation by making slight alterations to RNA structures and ensuring their proximity in the latent space.\n\n- The ablation study is informative and serves to justify the modeling decisions made by the authors.\n\n- Impressively, the authors have also developed five baseline models.", "weaknesses": "Proteins are arguably not structurally simpler than RNAs, and a reader might get this impression from reading this manuscript. Most RNAs lack a defined structure, and the majority of RNAs do not function primarily through their structures. Even a large number of small non-coding RNAs function through base pairing, such as miRNAs. I believe the authors should demonstrate a tangible application for their proposed method. Is it intended for designing RNA aptamers? If so, how and why would it be superior to SELEX methods?\n\nMinor Issues:\n- Figure 4 could be made more clear; the green arrows appear too small. Additionally, clusters and samples within clusters should be better aligned.\n- The last sentence of the second paragraph on page 2 is incomplete.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YwXXGYNrjI", "reviewer": "Reviewer_J4ia", "summary": "This paper presents a method for RNA inverse folding (i.e. given a RNA molecule’s 3D structure, predict the corresponding sequence). Coordinate information about an input molecule’s backbone is fed into a GNN which predicts a nucleotide for each node. Late representations from the GNN are also fed to contrastive losses which contrast structures with slightly perturbed structures, as well as structures with other structures of different folds. The authors demonstrate that this method outperforms other graph-learning methods, and also perform an ablation study to show the usefulness of each loss component (among other attributes).", "questions": "- Is there a constraint that makes the learned representations more evenly distributed on the hypersphere after projection? The supplement claims that having evenly distributed points on the hypersphere allows the model to better leverage a limited dataset. The projection to hypersphere space just normalizes the magnitude of the representations, but I don’t see how that allows for the distribution of directions on the hypersphere to be uniform.\n\n- Are the dataset splits identical for all baselines?\n\n- Would sharpening confidence be helpful in non-paired residues, as well? It seems like that regularization was added to the paired residues, and the ablation study showed that it was helpful. Would it also be helpful if applied to all residues (not just the paired ones)?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "### Interesting and sound architectural design choices\n\nThe authors started with a fairly standard GNN architecture, but several modifications and additions on it were made, which reasonably can be expected to improve RNA inverse folding. They adapted the coordinate system from Ingraham, et. al. 2019 to model RNA instead of proteins. The application of contrastive losses at multiple levels (i.e. the cluster-level loss and the sample-level loss) is well-motivated. Additionally, the imposition of secondary-structure constraints via confidence sharpening is a good idea. The ablation study presented also provides good evidence to the usefulness of these additions.\n\n### Overall well-written with many details\n\nGenerally, the explanations were clear. Many details were included in the main text or in the supplement which made this paper straightforward to read and understand, which is very appreciated. There were only a few places which I found to be less clear, which will be detailed below. But overall, the clarity and presentation were very above par.", "weaknesses": "### Lack of comparison against other RNA inverse-folding methods\n\nI consider this to be the most critical and crucial analysis that is missing. Although the paper does compare the proposed architecture (with its multi-level contrastive losses and secondary-structure loss) to other architectures like LSTMs or graph transformers, there aren’t any comparisons to any other methods which directly tackle the RNA inverse-folding problem. There are _many_ such methods, including several that are as recent as this year (e.g. MCTS-RNA, LEARNA, aRNAque, eM2dRNAs, etc.). Not having these comparisons really holds this paper back, because there is no understanding of how this method (which is being presented as a RNA inverse-folding method) compares to any other RNA inverse-folding method.\n      \n### Performance metrics are somewhat limited or measure less important aspects\n\nAlthough there is some justification provided for the performance metrics focused on in the paper (i.e. recovery and Macro-F1), these metrics are still somewhat limited.\n\nRecovery is simply “accuracy” here, and although this is a great metric to report, some bases will naturally be harder than others to predict. For example, bases participating in certain secondary structures, or those that interact with other residues in the 3D structure, may be easier or harder to classify. Thus, instead of reporting recovery/accuracy pooled over the entire macromolecule, it would be useful to see these numbers stratified by different secondary structures or tertiary attributes.\n\nMacro-F1 is also not particularly clear. It summarizes precision and recall across the 4 different bases and averages them. Since the problem of inverse folding is effectively a classification problem, other more understandable metrics can be used, such as auROC and auPRC (or visualizing the ROC curve and PR curve across different methods). Again, these can be stratified by secondary structure or tertiary attributes.\n\nAdditionally, it would be useful to have a more systematic or global analysis of the accuracy of predicted secondary or tertiary structures. Figure 5 is very useful, but those are only a few examples. Having a more global analysis across the entire dataset would be very informative. \n\n### Potential leakage of RNA folds between training and test sets\n\nThe process of allocating the training/validation/test sets certainly took into account structural similarity by ensuring that different clusters of structures were allocated to entirely different sets. However, the clusters were generated pretty conservatively (i.e. TM score < 0.45, which is already a very low cut-off). This definition of clusters might be useful for contrastive learning, but it might still let a lot of similar structures to exist in both the training and test set. Instead of reusing the same clusters, it would be good to ensure that the TM scores between training/test sets are much more significantly different (compared to connected components of TM < 0.45), or splitting the training/test sets based on Rfam families to ensure distinct folds. An analysis that shows minimal cross-contamination of folds/structures/scaffolds between the training and test sets would help put this concern to rest.\n\n### Some minor areas to clear up in writing\n\n- The edge features should be $E\\in\\mathbb{R}^{N\\times K\\times f_m}$\n- Equation 5 is not clear, and does not seem to match typical definitions of MPNNs; for example, it is unclear what the brackets mean and what $h_{E_{ij}}$ is doing on the left; also, it should be $\\mathcal{N}(i, K)$ (capital $K$)\n- Backwards quote around “reference” in Section 3.4\n- In Section 3.4, the confidence scores $c_i$ are not used anywhere; are they equivalent to the model’s output probability? It should also be clarified that $s_i$ in Equation 8 are the true sequence labels", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LNSLToUCYR", "reviewer": "Reviewer_kDxF", "summary": "This paper presents an approach to RNA design that leverages a large, well-curated benchmark dataset and a hierarchical data-efficient representation learning framework.", "questions": "My questions correspond to the weaknesses I mentioned above. Additionally, regarding the base pairs mentioned in Section 3.4, apart from the four mentioned, aren't GU and UG pairs also common?\n\n**Minor Concern:**\n1. In the \"Inter-nucleotide level\" section, there is a missing space between the second point and the following content.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "1. The research problem addressed in the article is highly meaningful.\n2. The method is easy to follow.", "weaknesses": "1. Some parts of the presentation are unclear. For instance, in Section 3.2, it isn't explicitly mentioned until the fourth page that each node represents a nucleotide base and not an atom. Although it becomes clear after reading the entire section, the author should clarify this earlier.\n2. The section on \"Biomolecular Engineering\" in the related work part mainly discusses the significance of RNA design. I believe this content should be integrated into the introduction section rather than being placed under related work. Therefore, the organization of the article could be improved.\n3. I find the article lacks novelty; the core multi-level representation learning relies on self-supervised methods from computer vision, which is not novel.\n4. The baseline comparisons are incomplete. While the article compares some protein inverse folding methods, it fails to include recent protein inverse folding methods for comparison, such as ProteinMPNN and PiFold, which are mentioned in the related work section.\n5. The dataset used in the experiments is not publicly available, making it impossible to directly compare the results with those in previously published works.\n6. The author mentions collecting data to create a dataset, which is divided by sequence length. However, this approach is widely used in the protein domain and lacks innovation. Additionally, the author does not cite articles from the protein domain as references here.\n7. The author claims to have aggregated and cleaned data from RNAsolo and PDB to create a new benchmark. I find this innovation to be insufficient, and the author does not clearly indicate the limitations of existing datasets or what constitutes \"clean data\". The motivation behind creating this new benchmark is not convincingly explained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)