Title: gRNAde: Geometric Deep Learning for 3D RNA inverse design
Venue: NeurIPS 2024
Accepted: no
Keywords: RNA Structure

RNA Design

Inverse Folding

Geometric Deep Learning

Graph Neural Networks
Abstract: Computational RNA design tasks are often posed as inverse problems, where sequences are designed based on adopting a single desired secondary structure without considering 3D geometry and conformational diversity. We introduce gRNAde, a geometric RNA design pipeline operating on 3D RNA backbones to design sequences that explicitly account for structure and dynamics. Under the hood, gRNAde is a multi-state Graph Neural Network that generates candidate RNA sequences conditioned on one or more 3D backbone structures where the identities of the bases are unknown. On a single-state fixed backbone re-design benchmark of 14 RNA structures from the PDB identified by Das et al. [2010], gRNAde obtains higher native sequence recovery rates (56% on average) compared to Rosetta (45% on average), taking under a second to produce designs compared to the reported hours for Rosetta. We further demonstrate the utility of gRNAde on a new benchmark of multi-state design for structurally flexible RNAs, as well as zero-shot ranking of mutational fitness landscapes in a retrospective analysis of a recent RNA polymerase ribozyme structure. Open source code and tutorials are available at: https://anonymous.4open.science/r/geometric-rna-design

Abstract

Computational RNA design tasks are often posed as inverse problems, where sequences are designed based on adopting a single desired secondary structure without considering 3D geometry and conformational diversity. We introduce gRNAde , a g eometric RNA de sign pipeline operating on 3D RNA backbones to design sequences that explicitly account for structure and dynamics. Under the hood, gRNAde is a multi-state Graph Neural Network that generates candidate RNA sequences conditioned on one or more 3D backbone structures where the identities of the bases are unknown. On a single-state fixed backbone re-design benchmark of 14 RNA structures from the PDB identified by Das et al. [2010], gRNAde obtains higher native sequence recovery rates (56% on average) compared to Rosetta (45% on average), taking under a second to produce designs compared to the reported hours for Rosetta. We further demonstrate the utility of gRNAde on a new benchmark of multi-state design for structurally flexible RNAs, as well as zero-shot ranking of mutational fitness landscapes in a retrospective analysis of a recent RNA polymerase ribozyme structure. Open source code and tutorials are available at: anonymous.4open.science/r/geometric-rna-design

1 Introduction

Why RNA design? Historical efforts in computational drug discovery have focussed on designing small molecule or protein-based medicines that either treat symptoms or counter the end stages of disease processes. In recent years, there is a growing interest in designing new RNA-based therapeutics that intervene earlier in disease processes to cut off disease-causing information flow in the cell [Damase et al., 2021, Zhu et al., 2022]. Notable examples of RNA molecules at the forefront of biotechnology today include mRNA vaccines [Metkar et al., 2024] and CRISPR-based genomic medicine [Doudna and Charpentier, 2014]. Of particular interest for structure-based design are ribozymes and riboswitches in the untranslated regions of mRNAs [Mandal and Breaker, 2004, Leppek et al., 2018]. In addition to coding for proteins (such as the spike protein in the Covid vaccine), naturally occurring mRNAs contain riboswitches that are responsible for cell-state dependent protein expression of the mRNA. Riboswitches act by 'switching' their 3D structure from an unbound conformation to a bound one in the presence of specific metabolites or small molecules. Rational design of riboswitches will enable translation to be dependent on the presence or absence of partner molecules, essentially acting as 'on-off' switches for highly targeted mRNA therapies in the future [Felletti et al., 2016, Mustafina et al., 2019, Mohsen et al., 2023].

Challenges of RNA modelling. Despite the promises of RNA therapeutics, proteins have instead 33 been the primary focus in the 3D biomolecular modelling community. Availability of a large number 34 of protein structures from the PDB combined with advances in deep learning for structured data 35 [Bronstein et al., 2021, Duval et al., 2023] have revolutionized protein 3D structure prediction [Jumper 36

Multi-state

Network

Encoder

Sequence

Decoder

RNA Conformational

Ensemble

Set of Backbone

Extract

Backbones

GAGCGU...

RNA

Sequence

Fixed backbone

re-design

3D roto-translations

node order

conformation order

Equivariant to:

et al., 2021] and rational design [Dauparas et al., 2022, Watson et al., 2023]. Applications of deep 37 learning for computational RNA design are underexplored compared to proteins due to paucity of 38 3D structural data [Schneider et al., 2023]. Most tools for RNA design primarily focus on secondary 39 structure without considering 3D geometry [Churkin et al., 2018] and use non-learnt algorithms for 40 aligning 3D RNA fragments [Han et al., 2017, Yesselman et al., 2019], which can be restrictive due 41 to the hand-crafted nature of the heuristics used. 42

In addition to limited 3D data for training deep learning models, the key technical challenge is that 43 RNA is more dynamic than proteins. The same RNA can adopt multiple distinct conformational 44 states to create and regulate complex biological functions [Ganser et al., 2019, Hoetzel and Suess, 45 2022, Ken et al., 2023]. Computational RNA design pipelines must account for both the 3D geometric 46 structure and conformational flexibility of RNA to engineer new biological functions. 47

48

49

50

51

52

Our contributions. This paper introduces gRNAde , a g eometric deep learning-based pipeline for RNA inverse de sign conditioned on 3D structure, analogous to ProteinMPNN for proteins [Dauparas et al., 2022]. As illustrated in Figure 1, gRNAde generates candidate RNA sequences conditioned on one or more backbone 3D conformations, enabling both single- and multi-state fixed-backbone sequence design.

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

We demonstrate the utility of gRNAde for the following design scenarios:

Improved performance and speed over Rosetta. We compare gRNAde to Rosetta [Leman et al., 2020], the state-of-the-art physically based tool for 3D RNA inverse design, for singlestate fixed backbone design of 14 RNA structures of interest from the PDB identified by Das et al. [2010]. We obtain higher native sequence recovery rates with gRNAde (56% on average) compared to Rosetta (45% on average). Additionally, gRNAde is significantly faster than Rosetta for inference; e.g. sampling 100+ designs in 1 second for an RNA of 60 nucleotides on an A100 GPU, compared to the reported hours for Rosetta.

Enables multi-state RNA design , which was previously not possible with Rosetta. gRNAde with multi-state GNNs improves sequence recovery over an equivalent single-state model on a benchmark of structurally flexible RNAs, especially for surface nucleotides which undergo positional or secondary structural changes.

Zero-shot learning of RNA fitness landscape. In a retrospective analysis of mutational fitness landscape data for an RNA polymerase ribozyme [McRae et al., 2024], we show how gRNAde's perplexity, the likelihood of a sequence folding into a backbone structure, can be used to rank mutants based on fitness in a zero-shot/unsupervised manner and outperforms random mutagenesis for improving fitness over the wild type in low throughput scenarios.

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

2 The gRNAde pipeline

2.1 The 3D RNA inverse folding problem

Figure 1 illustrates the RNA inverse folding problem: the task of designing new RNA sequences conditioned on a structural backbone. Given the 3D coordinates of a backbone structure, machine learning models must generate sequences that are likely to fold into that shape. The underlying assumption behind inverse folding (and rational biomolecule design) is that structure determines function [Huang et al., 2016]. To the best of our knowledge, gRNAde is the first explicitly multi-state inverse folding pipeline, allowing users to design sequences for backbone conformational ensembles (a set of 3D backbone structures) as opposed to a single structure.

2.2 RNA conformational ensembles as geometric multi-graphs

Featurization. The input to gRNAde is an RNA to be re-designed. For instance, this could be a set of PDB files with 3D backbone structures for the given RNA (a conformational ensemble) and the corresponding sequence of n nucleotides. As shown in Appendix Figure 11, gRNAde builds a geometric graph representation for each input structure:

We start with a 3-bead coarse-grained representation of the RNA backbone, retaining the coordinates for P, C4', N1 (pyrimidine) or N9 (purine) for each nucleotide [Dawson et al., 2016]. This 'pseudotorsional' representation describes RNA backbones completely in most cases while reducing the size of the torsional space to prevent overfitting [Wadley et al., 2007].

Each nucleotide i is assigned a node in the geometric graph with the 3D coordinate ⃗ x i ∈ R 3 corresponding to the centroid of the 3 bead atoms. Random Gaussian noise with standard deviation 0.1Å is added to coordinates during training to prevent overfitting on crystallisation artifacts, following Dauparas et al. [2022]. Each node is connected by edges to its 32 nearest neighbours as measured by the pairwise distance in 3D space, ∥ ⃗ x i -⃗ x j ∥ 2 .

Nodes are initialized with geometric features analogous to the featurization used in protein inverse folding [Ingraham et al., 2019, Jing et al., 2020]: (a) forward and reverse unit vectors along the backbone from the 5' end to the 3' end, ( ⃗ x i +1 -⃗ x i and ⃗ x i -⃗ x i -1 ); and (b) unit vectors, distances, angles, and torsions from each C4' to the corresponding P and N1/N9.

Edge features for each edge from node j to i are initialized as: (a) the unit vector from the source to destination node, ⃗ x j -⃗ x i ; (b) the distance in 3D space, ∥ ⃗ x j -⃗ x i ∥ 2 , encoded by 32 radial basis functions; and (c) the distance along the backbone, j -i , encoded by 32 sinusoidal positional encodings.

Multi-graph representation. As described in the previous section, given a set of k (conformer) structures in the input conformational ensemble, each RNA backbone is featurized as a separate geometric graph G ( k ) = ( A ( k ) , S ( k ) , ⃗ V ( k ) ) with the scalar features S ( k ) ∈ R n × f , vector features ⃗ V ( k ) ∈ R n × f ′ × 3 , and A ( k ) , an n × n adjacency matrix. For clear presentation and without loss of generality, we omit edge features and use f , f ′ to denote scalar/vector feature channels.

The input to gRNAde is thus a set of geometric graphs {G (1) , . . . , G ( k ) } which is merged into what we term a 'multi-graph' representation of the conformational ensemble, M = ( A , S , ⃗ V ) , by stacking the set of scalar features { S (1) , . . . , S ( k ) } into one tensor S ∈ R n × k × f along a new axis for the set size k . Similarly, the set of vector features { ⃗ V (1) , . . . , ⃗ V ( k ) } is stacked into one tensor ⃗ V ∈ R n × k × f ′ × 3 . Lastly, the set of adjacency matrices { A (1) , . . . , A ( k ) } are merged via a union ∪ into one single joint adjacency matrix A .

2.3 Multi-state GNN for representation learning on conformational ensembles

The gRNAde model, illustrated in Appendix Figure 12, processes one or more RNA backbone graphs via a multi-state GNN encoder which is equivariant to 3D roto-translation of coordinates as well as to the ordering of conformers, followed by conformer order-invariant pooling and sequence decoding. We describe each component in the following sections.

Multi-state GNN encoder. When representing conformational ensembles as a multi-graph, each node feature tensor contains three axes: (#nodes, #conformations, feature channels). We perform

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

message passing on the multi-graph adjacency to independently process each conformer, while maintaining permutation equivariance of the updated feature tensors along both the first (#nodes) and second (#conformations) axes. This works by operating on only the feature channels axis and generalising the PyTorch Geometric [Fey and Lenssen, 2019] message passing class to account for the extra conformations axis; see Appendix Figure 14 and the pseudocode for details.

We use multiple rotation-equivariant GVP-GNN [Jing et al., 2020] layers to update scalar features s i ∈ R k × f and vector features ⃗ v i ∈ R k × f ′ × 3 for each node i :

where MSG , UPD are Geometric Vector Perceptrons, a generalization of MLPs to take tuples of scalar and vector features as input and apply O (3) -equivariant non-linear updates. The overall GNN encoder is SO (3) -equivariant due to the use of reflection-sensitive input features (dihedral angles) combined with O (3) -equivariant GVP-GNN layers.

Our multi-state GNN encoder is easy to implement in any message passing framework and can be used as a plug-and-play extension for any geometric GNN pipeline to incorporate the multi-state inductive bias. It serves as an elegant alternative to batching all the conformations, which we found required major alterations to message passing and pooling depending on downstream tasks.

Conformation order-invariant pooling. The final encoder representations in gRNAde account for multi-state information while being invariant to the permutation of the conformational ensemble. To achieve this, we perform a Deep Set pooling [Zaheer et al., 2017] over the conformations axis after the final encoder layer to reduce S ∈ R n × k × f and ⃗ V ∈ R n × k × f ′ × 3 to S ′ ∈ R n × f and ⃗ V ′ ∈ R n × f ′ × 3 :

A simple sum or average pooling does not introduce any new learnable parameters to the pipeline and is flexible to handle a variable number of conformations, enabling both single-state and multi-state design with the same model.

Sequence decoding and loss function. We feed the final encoder representations after pooling, S ′ , ⃗ V ′ , to autoregressive GVP-GNN decoder layers to predict the probability of the four possible base identities (A, G, C, U) for each node/nucleotide. Decoding proceeds according to the RNA sequence order from the 5' end to 3' end. gRNAde is trained in a self-supervised manner by minimising a cross-entropy loss (with label smoothing value of 0.05) between the predicted probability distribution and the ground truth identity for each base. During training, we use autoregressive teacher forcing [Williams and Zipser, 1989] where the ground truth base identity is fed as input to the decoder at each step, encouraging the model to stay close to the ground-truth sequence.

Sampling. When using gRNAde for inference and designing new sequences, we iteratively sample the base identity for a given nucleotide from the predicted conditional probability distribution, given the partially designed sequence up until that nucleotide/decoding step. We can modulate the smoothness or sharpness of the probability distribution by using a temperature parameter.

2.4 Evaluation metrics for designed sequences

In principle, inverse folding models can be sampled from to obtain a large number of designed sequences for a given backbone structure. Thus, in-silico metrics to determine which sequences are useful and which ones to prioritise in wet lab experiments are a critical part of the overall pipeline. We currently use the following metrics to evaluate gRNAde's designs, visualised in Appendix Figure 13:

Native sequence recovery , which is the average percentage of native (ground truth) nucleotides correctly recovered in the sampled sequences. Recovery is the most widely used metric for biomolecule inverse design [Dauparas et al., 2022] but can be misleading in the case of RNAs where alternative nucleotide base pairings can form the same structural patterns.

Secondary structure self-consistency score , where we 'forward fold' the sampled sequences using a secondary structure prediction tool (we used EternaFold [Wayment-Steele et al., 2022])

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

and measure the average Matthew's Correlation Coefficient (MCC) to the groundtruth secondary structure, represented as a binary adjacency matrix. MCC values range between -1 and +1, where +1 represents a perfect prediction, 0 an average random prediction and -1 an inverse prediction. This measures how well the designs recover base pairing patterns.

Tertiary structure self-consistency scores , where we 'forward fold' the sampled sequences using a 3D structure prediction tool (we used RhoFold [Shen et al., 2022]) and compute the average RMSD, TM-score and GDT_TS to the groundtruth C4' coordinates to measure how well the designs recover global structural similarity and 3D conformations.

Perplexity , which can be thought of as the average number of bases that the model is selecting from for each nucleotide. Formally, perplexity is the average exponential of the negative log-likelihood of the sampled sequences. A perfect model would have perplexity of 1, while a perplexity of 4 means that the model is making random predictions (the model outputs a uniform probability over 4 possible bases). Perplexity does not require a ground truth structure to calculate, and can also be used for ranking sequences as it is the model's estimate of the compatibility of a sequence with the input backbone structure.

Significance and limitations. Self-consistency metrics, termed 'designability' (eg. scRMSD ≤ 2Å), as well as perplexity have been found to correlate with experimental success in protein design [Watson et al., 2023]. While precise designability thresholds are yet to be established for RNA, pairs of structures with TM-score ≥ 0.45 or GDT_TS ≥ 0.5 are known to correspond to roughly the same fold [Zhang et al., 2022]. Another major limitation for in-silico evaluation of 3D RNA design compared to proteins is the relatively worse state of structure prediction tools [Schneider et al., 2023].

3 Experimental Setup

3D RNA structure dataset. We create a machine learning-ready dataset for RNA inverse design using RNASolo [Adamczyk et al., 2022], a novel repository of RNA 3D structures extracted from solo RNAs, protein-RNA complexes, and DNA-RNA hybrids in the PDB. We used structures at resolution ≤ 4.0Å resulting in 4,223 unique RNA sequences for which a total of 12,011 structures are available (RNASolo date cutoff: 31 October 2023). Dataset statistics are available in Appendix Figure 15, illustrating the diversity of our dataset in terms of sequence length, number of structures per sequence, as well as structural variations among conformations per sequence.

Structural clustering. In order to ensure that we evaluate gRNAde's generalization ability to novel RNAs, we cluster the 4,223 unique RNAs into groups based on structural similarity. We use US-align [Zhang et al., 2022] with a similarity threshold of TM-score >0.45 for clustering, and ensure that we train, validate and test gRNAde on structurally dissimilar clusters (see next paragraph). We also provide utilities for clustering based on sequence homology using CD-HIT [Fu et al., 2012], which leads to splits containing biologically dissimilar clusters of RNAs.

Splits to evaluate generalization. After clustering, we split the RNAs into training ( ∼ 4000 samples), validation and test sets (100 samples each) to evaluate two different design scenarios:

Single-state split. This split is used to fairly evaluate gRNAde for single-state design on a set of RNA structures of interest from the PDB identified by Das et al. [2010], which mainly includes riboswitches, aptamers, and ribozymes. We identify the structural clusters belonging to the RNAs identified in Das et al. [2010] and add all the RNAs in these clusters to the test set (100 samples). The remaining clusters are randomly added to the training and validation splits.

Multi-state split. This split is used to test gRNAde's ability to design RNA with multiple distinct conformational states. We order the structural clusters based on median intra-sequence RMSD among available structures within the cluster 1 . The top 100 samples from clusters with the highest median intra-sequence RMSD are added to the test set. The next 100 samples are added to the validation set and all remaining samples are used for training.

Validation and test samples come from clusters with at most 5 unique sequences, in order to ensure diversity. Any samples that were not assigned clusters are directly appended to the training set. We

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

ViennaRNA

(2D only)

FARNA

Rosetta

gRNAde

0.00

0.25

0.50

0.75

1.00

Native sequence recovery

0.269

0.321

0.450

0.568

(a) gRNAde outperforms Rosetta.

0.00

0.25

0.50

0.75

1.00

Rosetta seq. recovery

0.00

0.25

0.50

0.75

1.00

gRNAde seq. recovery

1.1

1.2

1.3

1.4

1.5

1.6

gRNAde perplexity

(b) gRNAde's perplexity correlates with recovery.

also directly add very large RNAs (> 1000 nts) to the training set, as it is unlikely that we want to design very large RNAs. We exclude very short RNA strands (< 10 nts).

Evaluation metrics. For a given data split, we evaluate models on the held-out test set by designing 16 sequences (sampled at temperature 0.1) for each test data point and computing averages for each of the metrics described in Section 2.4: native sequence recovery, structural self-consistency scores and perplexity. We employ early stopping by reporting test set performance for the model checkpoint for the epoch with the best validation set recovery. Standard deviations are reported across 3 consistent random seeds for all models.

Hyperparameters. All models use 4 encoder and 4 decoder GVP-GNN layers, with 128 scalar/16 vector node features, 64 scalar/4 vector edge features, and drop out probability 0.5, resulting in 2,147,944 trainable parameters. All models are trained for a maximum of 50 epochs using the Adam optimiser with an initial learning rate of 0.0001, which is reduced by a factor 0.9 when validation performance plateaus with patience of 5 epochs. Ablation studies of key modelling decisions are available in Appendix Table 1.

4 Results

4.1 Single-state RNA design benchmark

We set out to compare gRNAde to Rosetta, a state-of-the-art physically based toolkit for biomolecular modelling and design [Leman et al., 2020]. We reproduced the benchmark setup from Das et al. [2010] for Rosetta's fixed backbone RNA sequence design workflow on 14 RNA structures of interest from the PDB, which mainly includes riboswitches, aptamers, and ribozymes (full listing in Table 2). We trained gRNAde on the single-state split detailed in Section 3, explicitly excluding the 14 RNAs as well as any structurally similar RNAs in order to ensure that we fairly evaluate gRNAde's generalization abilities vs. Rosetta.

gRNAde improves sequence recovery over Rosetta. In Figure 2, we compare gRNAde's native sequence recovery for single-state design with numbers taken from Das et al. [2010] for Rosetta, FARNA (a predecessor of Rosetta), and ViennaRNA (the most popular 2D inverse folding method). gRNAde has higher recovery of 56% on average compared to 45% for Rosetta, 32% for FARNA, and 27% for ViennaRNA. See Appendix Table 2 for per-RNA results.

gRNAde is significantly faster than Rosetta. In addition to superior sequence recovery, gRNAde is significantly faster than Rosetta for high-throughout design pipelines. Training gRNAde from scratch takes roughly 2-6 hours on a single A100 GPU, depending on the exact hyperparameters. Once trained, gRNAde can design hundreds of sequences for backbones with hundreds of nucleotides

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

Design 1:

perplexity: 1.310

recovery: 0.591 (27 edits)

sc2D = 0.923, scRMSD = 1.384

scTM = 0.831, scGDT = 0.830

Design 2: perplexity: 1.382 recovery: 0.409 (37 edits) sc2D = 0.922, scRMSD = 2.125 scTM = 0.687, scGDT = 0.678

Design 3:

perplexity: 1.425

recovery: 0.515 (30 edits)

sc2D = 0.923, scRMSD = 3.213

scTM = 0.512, scGDT = 0.526

in ∼ 1 second with GPU acceleration. On the other hand, Rosetta takes order of hours to produce a single design due to performing expensive Monte Carlo optimisations 2 . Deep learning methods like gRNAde are arguably easier to use since no expert customization is required and setup is easier compared to Rosetta [Dauparas et al., 2022], potentially making RNA design more broadly accessible.

gRNAde's perplexity correlates with sequence and structural recovery. In Figure 2b, we plot native sequence recovery per sample for Rosetta vs. gRNAde, shaded by gRNAde's average perplexity for each sample. Perplexity is an indicator of the model's confidence in its own prediction (lower perplexity implies higher confidence) and appears to be correlated with native sequence recovery. Additionally, visualisations of gRNAde's designs for a riboswitch in Figure 3 show that perplexity is also correlated with structural self-consistency scores. In the subsequent Section 4.3, we further demonstrate the utility of gRNAde's perplexity for zero-shot ranking of RNA fitness landscapes.

4.2 Multi-state RNA design benchmark

Structured RNAs often adopt multiple distinct conformational states to perform biological functions [Ken et al., 2023]. For instance, riboswitches adopt at least two distinct functional conformations: a ligand bound (holo) and unbound (apo) state, which helps them regulate and control gene expression [Stagno et al., 2017]. If we were to attempt single-state inverse design for such RNAs, each backbone structure may lead to a different set of sampled sequences. It is not obvious how to select the input backbone as well as designed sequence when using single-state models for multi-state design. gRNAde's multi-state GNN, descibed in Section 2.3, directly 'bakes in' the multi-state nature of RNA into the architecture and designs sequences explicitly conditioned on multiple states.

In order to evaluate gRNAde's multi-state design capabilities, we trained equivalent single-state and multi-state gRNAde models on the multi-state split detailed in Section 3, where the validation and test sets contain progressively more structurally flexible RNAs as measured by median RMSD among multiple available states for an RNA.

Multi-state gRNAde boosts sequence recovery. In Figure 4a, we compared a single-state variant of gRNAde with otherwise equivalent multi-state models (with up to 3 and 5 states, respectively) in terms of native sequence recovery. Multi-state variants show marginal improvements, overall. As a caveat, it is worth noting that multi-state models consume more GPU memory than an equivalent single-state model during mini-batch training (approximate peak GPU usage for max. number of states = 1: 12GB, 3: 28GB, 5: 50GB on a single A100 with at most 3000 total nodes in a mini-batch).

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

1 state

3 states

5 states

0.00

0.25

0.50

0.75

1.00

Native sequence recovery

0.455

0.469

0.484

0.0

0.5

1.0

Nucleotide Paired Probability

0.00

0.25

0.50

0.75

1.00

Native sequence recovery

0

200

400

Nucleotide SASA

0

10

20

gRNAde model

1 state

3 states

5 states

(a) Per-sample sequence recovery

Improved recovery in structurally flexible regions. In Figure 4b, we evaluated gRNAde's multi-state sequence recovery at a fine-grained, per-nucleotide level. Multi-state GNNs improve sequence recovery over the single-state variant on structurally flexible nucleotides, as characterised by undergoing changes in base pairing/secondary structure, higher average RMSD between 3D coordinates across states, and larger solvent accessible surface area.

4.3 Zero-shot ranking of RNA fitness landscape

Lastly, we explored the use of gRNAde as a zero-shot ranker of mutants in RNA engineering campaigns. Given the backbone structure of a wild type RNA of interest as well as a candidate set of mutant sequences, we can compute gRNAde's perplexity of whether a given sequence folds into the backbone structure. Perplexity is inversely related to the likelihood of a sequence conditioned on a structure, as described in Section 2.4. We can then rank sequences based on how 'compatible' they are with the backbone structure in order to select a subset to be experimentally validated in wet labs.

Retrospective analysis on ribozyme fitness landscape. A recent study by McRae et al. [2024] determined a cryo-EM structure of a dimeric RNA polymerase ribozyme at 5Å resolution 3 , along with fitness landscapes of ∼ 75K mutants for the catalytic subunit 5TU and ∼ 48K mutants for the scaffolding subunit t1. We design a retrospective study using this data of (sequence, fitness value) pairs where we simulate an RNA engineering campaign with the aim of improving catalytic subunit fitness over the wild type 5TU sequence.

We consider various design budgets ranging from hundreds to thousands of sequences selected for experimental validation, and compare 4 unsupervised approaches for ranking/selecting variants: (1) random choice from all ∼ 75,000 sequences; (2) random choice from all 449 single mutant sequences; (3) random choice from all single and double mutant sequences (as sequences with higher mutation order tend to be less fit); and (4) negative gRNAde perplexity (lower perplexity is better). For each design budget and ranking approach, we compute the expected maximum change in fitness over the wild type that could be achieved by screening as many variants as allowed in the given design budget. We run 10,000 simulations to compute confidence intervals for the 3 random baselines.

gRNAde outperforms random baselines in low design budget scenarios. Figure 5 illustrates the results of our retrospective study. At low design budgets of up to hundreds of sequences, which are relevant in the case of a low throughput fitness screening assay, gRNAde outperforms all random baselines in terms of the maximum change in fitness over the wild type. The top 10 mutants as ranked by gRNAde contain a sequence with 4-fold improved fitness, while the top 200 leads to a 5-fold improvement. Note that gRNAde is used zero-shot here, i.e. it was not fine-tuned on any assay data.

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

1st best (fit.: 2.88)

2nd best (fit.: 2.40)

5th best (fit.: 2.23)

10th best (fit.: 1.96)

20th best (fit.: 1.61)

50th best (fit.: 1.35)

wildtype

1

10

50

100

200

449

1500

5000

10493

Selected sequences for assaying

0x

1x

0x

2x

4x

6x

8x

10x

12x

14x

16x

18x

Expected 'max' fold change over WT

Max Fitness by Sample Size and Condition (n=74,943; simulations=10,000)

Condition

random

n_mut==1

n_mut<=2

gRNAde

0.00

0.69

1.39

1.79

2.08

2.30

2.48

2.64

2.77

2.89

Fitness

Perspective. Overall, it is promising that gRNAde's perplexity correlates with experimental fitness measurements out-of-the-box (zero-shot) and can be a useful ranker of mutant fitness in our retrospective study. In realistic design scenarios, improvements could likely be obtained by fine-tuning gRNAde on a low amount of experimental fitness data. For example, latent features from gRNAde may be finetuned or used as input to a prediction head with supervised learning on fitness landscape data. This study acts as a sanity check before committing to wet lab validation of gRNAde designs. We see random mutagenesis and directed evolution-based approaches as complementary to de-novo design and inverse folding approaches like gRNAde. Random mutagenesis can be thought of as local exploration around a wild type sequence, optimising fitness within an 'island' of activity. Structure-based design approaches are akin to global jumps in sequence space, with the potential to find new islands further away from the wild type [Huang et al., 2016].

5 Conclusion

We introduce gRNAde, a geometric deep learning pipeline for RNA sequence design conditioned on one or more 3D backbone structures. gRNAde is superior to the physically based Rosetta for 3D RNA inverse folding in terms of performance, inference speed, and ease of use. Further, gRNAde enables explicit multi-state design for structurally flexible RNAs which was previously not possible with Rosetta. gRNAde's perplexity correlates with native sequence and structural recovery, and can be used for zero-shot ranking of mutants in RNA engineering campaigns. To the best of our knowledge, gRNAde is also the first geometric deep learning architecture for multi-state biomolecule representation learning; the model is generic and can be repurposed for other learning tasks on conformational ensembles, including multi-state protein design.

Limitations. Key avenues for future development of gRNAde include supporting multiple interacting chains, accounting for partner molecules with RNAs, and supporting negative design against undesired conformations. We discuss practical tradeoffs to using gRNAde in real-world RNA design scenarios in Appendix B, including limitations due to the current state of 3D RNA structure prediction tools.

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

References

Adamczyk, M. Antczak, and M. Szachniuk. Rnasolo: a repository of cleaned pdb-derived rna 3d structures. Bioinformatics , 2022. (Cited on page 5)

Baek, F. DiMaio, I. Anishchenko, J. Dauparas, S. Ovchinnikov, G. R. Lee, J. Wang, Q. Cong, L. N. Kinch, R. D. Schaeffer, et al. Accurate prediction of protein structures and interactions using a three-track neural network. Science , 2021. (Cited on page 13)

Baek, R. McHugh, I. Anishchenko, H. Jiang, D. Baker, and F. DiMaio. Accurate prediction of protein-nucleic acid complexes using rosettafoldna. Nature Methods , 2024. (Cited on page 13)

Bonnet, P. Rzazewski, and F. Sikora. Designing rna secondary structures is hard. Journal of Computational Biology , 2020. (Cited on page 13)

M. Bronstein, J. Bruna, T. Cohen, and P. Velickovic. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint , 2021. (Cited on page 1)

Chen, Z. Hu, S. Sun, Q. Tan, Y. Wang, Q. Yu, L. Zong, L. Hong, J. Xiao, T. Shen, et al. Interpretable rna foundation model from unannotated data for highly accurate rna structure and function predictions. arXiv preprint , 2022. (Cited on page 13)

Churkin, M. D. Retwitzer, V. Reinharz, Y. Ponty, J. Waldispühl, and D. Barash. Design of rnas: comparing programs for inverse rna folding. Briefings in bioinformatics , 2018. (Cited on page 2, 13)

R. Damase, R. Sukhovershin, C. Boada, F. Taraballi, R. I. Pettigrew, and J. P. Cooke. The limitless future of rna therapeutics. Frontiers in bioengineering and biotechnology , 2021. (Cited on page 1)

Das, J. Karanicolas, and D. Baker. Atomic accuracy in predicting and designing noncanonical rna structure. Nature methods , 2010. (Cited on page 1, 2, 5, 6, 13, 14, 15, 19)

Dauparas, I. Anishchenko, N. Bennett, H. Bai, R. J. Ragotte, L. F. Milles, B. I. Wicky, et al. Robust deep learning based protein sequence design using proteinmpnn. Science , 2022. (Cited on page 2, 3, 4, 7, 14)

K. Dawson, M. Maciejczyk, E. J. Jankowska, and J. M. Bujnicki. Coarse-grained modeling of rna 3d structure. Methods , 2016. (Cited on page 3)

Didi, F. Vargas, S. Mathis, V. Dutordoir, E. Mathieu, U. J. Komorowska, and P. Lio. A framework for conditional diffusion modelling with applications in motif scaffolding for protein design. In NeurIPS 2023 Machine Learning for Structural Biology Workshop , 2023. (Cited on page 13)

A. Doudna and E. Charpentier. The new frontier of genome engineering with crispr-cas9. Science , 2014. (Cited on page 1)

Duval, S. V. Mathis, C. K. Joshi, V. Schmidt, S. Miret, F. D. Malliaros, T. Cohen, P. Lio, Y . Bengio, and M. Bronstein. A hitchhiker's guide to geometric gnns for 3d atomic systems. arXiv preprint , 2023. (Cited on page 1)

Felletti, J. Stifel, L. A. Wurmthaler, S. Geiger, and J. S. Hartig. Twister ribozymes as highly versatile expression platforms for artificial riboswitches. Nature communications , 2016. (Cited on page 1)

Fey and J. E. Lenssen. Fast graph representation learning with pytorch geometric. ICLR 2019 Representation Learning on Graphs and Manifolds Workshop , 2019. (Cited on page 4)

Fu, B. Niu, Z. Zhu, S. Wu, and W. Li. Cd-hit: accelerated for clustering the next-generation sequencing data. Bioinformatics , 2012. (Cited on page 5)

R. Ganser, M. L. Kelly, D. Herschlag, and H. M. Al-Hashimi. The roles of structural dynamics in the cellular functions of rnas. Nature reviews Molecular cell biology , 2019. (Cited on page 2)

Han, X. Qi, C. Myhrvold, B. Wang, M. Dai, S. Jiang, M. Bates, Y. Liu, B. An, F. Zhang, et al. Single-stranded dna and rna origami. Science , 2017. (Cited on page 2, 13)

He, R. Huang, J. Townley, R. C. Kretsch, T. G. Karagianes, D. B. Cox, H. Blair, D. Penzar, V . Vyalt378 sev, E. Aristova, et al. Ribonanza: deep learning of rna structure through dual crowdsourcing. 379 bioRxiv , 2024. (Cited on page 13) 380

381

382

Hoetzel and B. Suess. Structural changes in aptamers are essential for synthetic riboswitch engineering. Journal of Molecular Biology , 2022. (Cited on page 2)

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

P.-S. Huang, S. E. Boyken, and D. Baker. The coming of age of de novo protein design. Nature , 2016. (Cited on page 3, 9)

Ingraham, V. Garg, R. Barzilay, and T. Jaakkola. Generative models for graph-based protein design. NeurIPS , 2019. (Cited on page 3, 20)

B. Ingraham, M. Baranov, Z. Costello, K. W. Barber, W. Wang, A. Ismail, V. Frappier, D. M. Lord, C. Ng-Thow-Hing, E. R. Van Vlack, et al. Illuminating protein space with a programmable generative model. Nature , 2023. (Cited on page 13)

Jing, S. Eismann, P. Suriana, R. J. L. Townshend, and R. Dror. Learning from protein structure with geometric vector perceptrons. In International Conference on Learning Representations , 2020. (Cited on page 3, 4, 20)

K. Joshi, C. Bodnar, S. V. Mathis, T. Cohen, and P. Lio. On the expressive power of geometric graph neural networks. In International Conference on Machine Learning , 2023. (Cited on page 17)

Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Zidek, A. Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature , 2021. (Cited on page 1, 13)

L. Ken, R. Roy, A. Geng, L. R. Ganser, A. Manghrani, B. R. Cullen, U. Schulze-Gahmen, D. Herschlag, and H. M. Al-Hashimi. Rna conformational propensities determine cellular activity. Nature , 2023. (Cited on page 2, 7)

K. Leman, B. D. Weitzner, S. M. Lewis, J. Adolf-Bryfogle, N. Alam, R. F. Alford, M. Aprahamian, D. Baker, K. A. Barlow, P. Barth, et al. Macromolecular modeling and design in rosetta: recent methods and frameworks. Nature methods , 2020. (Cited on page 2, 6)

Leppek, R. Das, and M. Barna. Functional 5' utr mrna structures in eukaryotic translation regulation and how to find them. Nature reviews Molecular cell biology , 2018. (Cited on page 1)

Li, S. Moayedpour, R. Li, M. Bailey, S. Riahi, L. Kogler-Anele, M. Miladi, J. Miner, D. Zheng, J. Wang, et al. Codonbert: Large language models for mrna design and optimization. bioRxiv , 2023a. (Cited on page 13)

Li, C. Zhang, C. Feng, R. Pearce, P. Lydia Freddolino, and Y. Zhang. Integrating end-toend learning with deep geometrical potentials for ab initio rna structure prediction. Nature Communications , 2023b. (Cited on page 13)

Mandal and R. R. Breaker. Gene regulation by riboswitches. Nature reviews Molecular cell biology , 2004. (Cited on page 1)

K. McRae, C. J. Wan, E. L. Kristoffersen, K. Hansen, E. Gianni, I. Gallego, J. F. Curran, J. Attwater, P. Holliger, and E. S. Andersen. Cryo-em structure and functional landscape of an rna polymerase ribozyme. Proceedings of the National Academy of Sciences , 2024. (Cited on page 2, 8, 9, 19)

Metkar, C. S. Pepin, and M. J. Moore. Tailor made: the art of therapeutic mrna design. Nature Reviews Drug Discovery , 2024. (Cited on page 1)

G. Mohsen, M. K. Midy, A. Balaji, and R. R. Breaker. Exploiting natural riboswitches for aptamer engineering and validation. Nucleic Acids Research , 2023. (Cited on page 1)

Mustafina, K. Fukunaga, and Y. Yokobayashi. Design of mammalian on-riboswitches based on tandemly fused aptamer and ribozyme. ACS Synthetic Biology , 2019. (Cited on page 1)

J. Penic, T. Vlasic, R. G. Huber, Y. Wan, and M. Sikic. Rinalmo: General-purpose rna language 424 models can generalize well on structure prediction tasks. arXiv preprint , 2024. (Cited on page 13) 425

426

427

Runge, D. Stoll, S. Falkner, and F. Hutter. Learning to design RNA. In ICLR , 2019. (Cited on page 13)

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

Schneider, B. A. Sweeney, A. Bateman, J. Cerny, T. Zok, and M. Szachniuk. When will rna get its alphafold moment? Nucleic Acids Research , 2023. (Cited on page 2, 5)

Shen, Z. Hu, Z. Peng, J. Chen, P. Xiong, L. Hong, L. Zheng, Y. Wang, I. King, S. Wang, et al. E2efold-3d: End-to-end deep learning method for accurate de novo rna 3d structure prediction. arXiv preprint , 2022. (Cited on page 5)

Stagno, Y. Liu, Y. Bhandari, C. Conrad, S. Panja, M. Swain, L. Fan, G. Nelson, C. Li, D. Wendel, et al. Structures of riboswitch rna reaction states by mix-and-inject xfel serial crystallography. Nature , 2017. (Cited on page 7)

Tan, Y. Zhang, Z. Gao, H. Cao, and S. Z. Li. Hierarchical data-efficient representation learning for tertiary structure-based rna design. arXiv preprint , 2023. (Cited on page 13, 14)

J. Townshend, S. Eismann, A. M. Watkins, R. Rangan, M. Karelina, R. Das, and R. O. Dror. Geometric deep learning of rna structure. Science , 2021. (Cited on page 13)

Vicens and J. S. Kieft. Thoughts on how to think (and talk) about rna structure. Proceedings of the National Academy of Sciences , 2022. (Cited on page 13, 17)

M. Wadley, K. S. Keating, C. M. Duarte, and A. M. Pyle. Evaluating and learning from rna pseudotorsional space: quantitative validation of a reduced representation for rna structure. Journal of molecular biology , 2007. (Cited on page 3)

Wang, C. Feng, R. Han, Z. Wang, L. Ye, Z. Du, H. Wei, F. Zhang, Z. Peng, and J. Yang. trrosettarna: automated prediction of rna 3d structure with transformer network. Nature Communications , 2023. (Cited on page 13)

Ward, E. Courtney, and E. Rivas. Fitness functions for rna structure design. Nucleic Acids Research , 2023. (Cited on page 13)

M. Watkins, R. Rangan, and R. Das. Farfar2: improved de novo rosetta prediction of complex global rna folds. Structure , 2020. (Cited on page 13)

L. Watson, D. Juergens, N. R. Bennett, B. L. Trippe, J. Yim, H. E. Eisenach, W. Ahern, A. J. Borst, R. J. Ragotte, L. F. Milles, et al. De novo design of protein structure and function with rfdiffusion. Nature , 2023. (Cited on page 2, 5, 13)

K. Wayment-Steele, W. Kladwang, A. I. Strom, J. Lee, A. Treuille, A. Becka, E. Participants, and R. Das. Rna secondary structure packages evaluated and improved by high-throughput experiments. Nature methods , 2022. (Cited on page 4)

J. Williams and D. Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation , 1989. (Cited on page 4)

D. Yesselman, D. Eiler, E. D. Carlson, M. R. Gotrik, A. E. d'Aquino, A. N. Ooms, W. Kladwang, P. D. Carlson, X. Shi, D. A. Costantino, et al. Computational design of three-dimensional rna structure and function. Nature nanotechnology , 2019. (Cited on page 2, 13, 14)

Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and A. J. Smola. Deep sets. NeurIPS , 2017. (Cited on page 4, 20)

Zhang, M. Shine, A. M. Pyle, and Y. Zhang. Us-align: universal structure alignments of proteins, nucleic acids, and macromolecular complexes. Nature methods , 2022. (Cited on page 5)

Zhu, L. Zhu, X. Wang, and H. Jin. Rna-based therapeutics: An overview and prospectus. Cell Death & Disease , 2022. (Cited on page 1)

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

A Related Work

We attempt to briefly summarise recent developments in RNA structure modelling and design, with an emphasis on deep learning-based approaches.

RNA inverse folding. Most tools for RNA inverse folding focus on secondary structure without considering 3D geometry [Churkin et al., 2018, Runge et al., 2019] and approach the problem from the lens of energy optimisation [Ward et al., 2023]. Rosetta fixed backbone re-design [Das et al., 2010] is the only energy optimisation-based approach that accounts for 3D structure. Deep neural networks such as gRNAde can incorporate 3D structural constraints and are orders of magnitude faster than optimisation-based approaches; this is particularly attractive for high-throughput design pipelines as solving the inverse folding optimisation problem is NP hard [Bonnet et al., 2020].

RNA structure design. Inverse folding models for protein design have often been coupled with backbone generation models which design structural backbones conditioned on various design constraints [Watson et al., 2023, Ingraham et al., 2023, Didi et al., 2023]. Current approaches for RNA backbone design use classical (non-learnt) algorithms for aligning 3D RNA motifs [Han et al., 2017, Yesselman et al., 2019], which are small modular pieces of RNA that are believed to fold independently. Such algorithms may be restricted by the use of hand-crafted heuristics and we plan to explore data-driven generative models for RNA backbone design in future work.

RNA structure prediction. There have been several recent efforts to adapt protein folding architectures such as AlphaFold2 [Jumper et al., 2021] and RosettaFold [Baek et al., 2021] for RNA structure prediction [Li et al., 2023b, Wang et al., 2023, Baek et al., 2024]. A previous generation of models used GNNs as ranking functions together with Rosetta energy optimisation [Watkins et al., 2020, Townshend et al., 2021]. None of these architectures aim at capturing conformational flexibility of RNAs, unlike gRNAde which represents RNAs as multi-state conformational ensembles. Neither can structure prediction tools be used for RNA design tasks as they are not generative models.

RNA language models. Self-supervised language models have been developed for predictive and generative tasks on RNA sequences, including general-purpose models such as RNA FM [Chen et al., 2022] and RiNaLMo [Penic et al., 2024] as well as mRNA-specific CodonBERT [Li et al., 2023a]. RNA sequence data repositories are orders of magnitude larger than those for RNA structure (eg. RiNaLMo is trained on 36 million sequences). However, standard language models can only implicitly capture RNA structure and dynamics through sequence co-occurence statistics, which can pose a chellenge for designing structured RNAs such as riboswitches, aptamers, and ribozymes. RibonanzaNet [He et al., 2024] represents a recent effort in developing structure-informed RNA language models by supervised training on experimental readouts from chemical mapping, although RibonanzaNet cannot be used for RNA design. Inverse folding methods like gRNAde are language models conditioned on 3D structure, making them a natural choice for structure-based design.

Comparison to contemporaneous work. Concurrently, Tan et al. [2023] also developed a deep learning-based 3D RNA inverse folding model. We want to emphasize that this is independent work, but for completeness we include a discussion on key differences to gRNAde:

Methodology:

-New capabilities : gRNAde enables explicit multi-state design to generate sequences conditioned on multiple backbone structures, which is not possible with Rosetta nor Tan et al. [2023]'s approach. We have also demonstrated the utility of gRNAde's perplexity for zero-shot ranking of mutants in RNA engineering campaigns.

-Decoding : gRNAde uses an autoregressive decoder with rotation-equivariant GNN layers, while Tan et al. [2023] use a non-autoregressive (one-shot) decoder with rotation-invariant layers. In our ablation study (Appendix D), we found autoregressive decoding to show significantly higher 2D and 3D self-consistency scores than non-autoregressive decoding, even though non-autoregressive decoding lead to higher sequence recovery. Autoregressive decoding is more expressive and can condition predictions at each decoding step on past predictions, while one-shot decoders sample from independent probability distributions for each nucleotide. We find autoregressive decoding to be a better inductive bias for predicting base pairing and base stacking interactions that are drivers of RNA structure [Vicens and Kieft, 2022]. For instance, G-C and A-U pairs can often be swapped for one another, but non-autoregressive decoding does not capture such paired constraints.

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

Evaluation:

-Evaluation metrics : Tan et al. [2023] focus on measuring native sequence recovery, only. We have additionally introduced structural self-consistency metrics at the 2D and 3D level, which have been shown to better correlate with experimental success in protein design.

-Perplexity : We found gRNAde's perplexity to be correlated with sequence and structural recovery, as well as demonstrated its utility for zero-shot ranking of mutants in RNA engineering. On the other hand, Tan et al. [2023] do not report perplexity and claim that perplexity is an unsuitable metric for RNA design.

-Data splitting : While both studies use structural clustering to evaluate generalisation to structurally dissimilar RNAs, Tan et al. [2023]'s test splits are determined randomly. Our experiments use currated test splits from Das et al. [2010] to fairly compare gRNAde to physically based Rosetta, as well as split based on structural flexibility to benchmark multi-state design.

Usage and reproducibility: We release open source training and inference code as well as model checkpoints to enable complete reproducibility. We also release Colab notebooks and detailed tutorials to make gRNAde broadly applicable and useful in real-world RNA design campaigns. At present, it is not possible to reproduce the results in Tan et al. [2023] or compare to gRNAde directly as no training code is available.

B FAQs on using gRNAde

How to chose the number of states to provide as input to gRNAde? In general, this would depend on the design objective. For instance, designing riboswitches may necessitate multi-state design, while a single-state pipeline may be more sensible for locking an aptamer into its bound conformation [Yesselman et al., 2019]. Note that it may be possible to benefit from multi-state gRNAde models even when performing single-state design by using slightly noised variations of the same backbone structure as an input conformational ensemble.

How to prioritise or chose amongst designed sequences? We have currently provided 3 types of evaluation metrics: native sequence recovery, structural self-consistency scores and perplexity, towards this end. We suspect that recovery may not be the ideal choice, except for design scenarios where we require certain regions of the RNA sequence to be conserved or native-like. Self-consistency scores may provide an overall more holistic evaluation metric as they accounts for alternative base pairings which still lead to similar structures as well as better capture the recovery of structural motifs responsible for functionality. However, structural self-consistency scores inherit the limitations of the structure prediction methods used as part of their computation. For instance, computing the selfconsistency score between an RNA backbone and its own native sequence provides an upper bounds on the maximum score that designs can obtain under a given structure prediction method. Lastly, gRNAde's perplexity estimates the likelihood of a sequence given a backbone and can be useful for ranking designs and mutants in RNA engineering campaigns (especially for design scenarios where structure prediction tools are not performant).

In real-world design scenarios, we can pair gRNAde with another machine learning model (an 'oracle') for ranking or predicting the suitability of designed sequences for the objective (for instance, binding affinity or some other notion of fitness). We hope to conduct further experimental validation of gRNAde designs in the wet lab in order to better understand these tradeoffs.

Why not average single-state logits over multiple states for multi-state design? ProteinMPNN [Dauparas et al., 2022] proposes to average logits from multiple backbones for multi-state protein design. Here is a simple example to highlight issues with such an approach: Consider two states A and B, and choice of labels X, Y, and Z. For state A: X, Y, Z are assigned probabilities 75%, 20%, 5%. For state B: X, Y, Z are assigned probabilities 5%, 20%, 75%. Logically, label Y is the only one that is compatible with both states. However, averaging the probabilities would lead to label X or Z being more likely to be sampled in designs. As an alternative, gRNAde is based on multi-state GNNs which can take as input one or more backbone structures and generate sequences conditioned on the conformational ensemble directly.

C 3D Visualisation of gRNAde Designs 574

Guanine

Riboswitch

aptamer

Tetrahymena

Ribozyme

P4-P6 domain

binding

aptamer

0.00

0.25

0.50

0.75

1.00

scTM (RhoFold)

0.0

0.2

0.4

0.6

0.8

1.0

Perplexity (normalized)

(a) self-consistency TM-score

Guanine

Riboswitch

aptamer

Tetrahymena

Ribozyme

P4-P6 domain

binding

aptamer

0.00

0.25

0.50

0.75

1.00

scGDT (RhoFold)

0.0

0.2

0.4

0.6

0.8

1.0

Perplexity (normalized)

(b) self-consistency GDT_TS

Design 1:

GGGGCUCCGGCGACGCAGUCGAAAG

CCCAGCAGUACCAAGCCUCAGGGGA

AACUUUGAGGUGGCCUAACAAAGGA

UACGGUAAUAAGCUGCGGGAAAAGG

UUGUAAGCCGGAGCGAAGACCUAAG

GCACCGCUUUUGGCGGUGCUAUGGU

UGAAGUUAA

perplexity: 1.2462

recovery: 0.7170 (44 edits)

sc2D = 0.8301

scRMSD = 5.4562

scTM = 0.6481

scGDT = 0.4465

Design 2: GGGGUACCGGCGACGCAGUCGAAUG CCCUGUGGUACCAAGCCCCGGGGGA AACUUCGGGGUGGCCUUACCAAGGA CACGGUAAUAAGCCACGGGAAAUGG UUGUAAGCCGGUCCGAAGCCCUAAG GCCGCGCUUUGGGCGCGGCUAUGGG UGAAGGCAA

perplexity: 1.3273

recovery: 0.6226 (58 edits)

sc2D = 0.6896

scRMSD = 6.7239

scTM = 0.6300

scGDT = 0.4513

Design 3: GAGGCCACGGCAACGCAGUCUAACG CCCUGUGGUACCAAGUCUUAGGAGA AAUUUUAAGAUGGCCUAAUAAAGGA UAUGGUAAUAAGCCACGGGAAAAGG UUGUAAGACGUGACGAAGUCCUAAG GCCACAGUUUUGCUGUGGCUAUGGA UGGAGUACA

perplexity: 1.3204

recovery: 0.7044 (45 edits)

sc2D = 0.7922

scRMSD = 8.8211

scTM = 0.4582

scGDT = 0.2909

Design 1: GUCAAACGCAGCCGAAA GCGCGAUAGUCCCAGGAA perplexity = 1.6237 recovery = 0.4571 (16 edits) sc2D = -0.0074 scRMSD = 3.9505 scTM = 0.2597 scGDT = 0.4786

Design 2: GGCAAACGCGGCCGAAA GCGCGUGAGUCCCCGGAC perplexity = 1.6630 recovery = 0.4857 (16 edits) sc2D = -0.0099 scRMSD = 3.3549 scTM = 0.2526 scGDT = 0.5000

Design 3: CGUAGUCGGAGCCGAAG GGCCGUUAGUCCCAGGAG perplexity = 1.7020 recovery = 0.4000 (17 edits) sc2D = 0.4035 scRMSD = 16.4102 scTM = 0.0319 scGDT = 0.0571

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

D Ablation Study

Table 1 presents an ablation study as well as aggregated benchmark for various configurations of gRNAde. Key takeaways are highlighted below. Note that all results in the main paper are reported for models trained on the maximum length of 5000 nucleotides using autoregressive decoding and rotation-equivariant GNN layers, as this lead to the lowest perplexity values.

Max. train RNA length Limiting the maximum length of RNAs used for training can be seen as ablating the use of ribosomal RNA families (which are thousands of nucleotides long and form complexes with specialised ribosomal proteins). We find that training on only short RNAs fewer than 1000s of nucleotides leads to worse sequence recovery and 3D self-consistency scores, even though it improves 2D self-consistency across both evaluation splits. This suggests that tertiary interactions learnt from ribosomal RNAs can generalise to other RNA families to some extent (large ribosomal RNAs were excluded from test sets).

GNN We ablated whether the internal representations of the GVP-GNN are rotation invariant or equivariant. Equivariant GNNs are theoretically more expressive [Joshi et al., 2023] and we do find them more capable at fitting the training distribution (as shown by lower perplexity). However, we do not find significant differences in terms of other performance metrics across different GNN layers.

Model 'AR' implies autoregressive decoding (described in Section 2.3, uses 4 encoder and 4 decoder layers), while 'NAR' implies non-autoregressive, one-shot decoding using an MLP (uses 8 encoder layers). Across both evaluation splits, AR models show significantly higher self-consistency scores than NAR, even though NAR lead to higher sequence recovery. AR is more expressive and can condition predictions at each decoding step on past predictions, while one-shot NAR samples from independent probability distributions for each nucleotide. Thus, AR is a better inductive bias for predicting base pairing and base stacking interactions that are drivers of RNA structure [Vicens and Kieft, 2022]. For instance, G-C and A-U pairs can often be swapped for one another, but non-autoregressive decoding does not capture such paired constraints.

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

Max. #states We evaluate the impact of increasing the maximum number of states as input to gRNAde. Multi-state models marginally improve native sequence recovery as well as structural self-consistency scores over an equivalent single state variant, even for the single-state benchmark where the multi-state model is being used with only one state as input. This suggests that seeing multiple states during training can be useful for gRNAde's performance even for single-state design tasks.

Non-learnt baselines. We report the performance of two non-learnt baselines to contextualise gRNAde's performance: for each test sample, simply predicting the groundtruth sequence back and predicting a random sequence. Structural self-consistency scores for the Groundtruth baseline provides a rough upper bounds on the maximum score that any gRNAde designs can theoretically obtain given the current state of 2D/3D structure predictors being used. gRNAde always performs better than the random baseline and often reaches 2D self-consistency scores close to the upper bound. Both 2D and 3D self-consistency scores are inherently limited by the performance of the structure prediction methods used.

2D inverse folding baseline. We additionally report results for ViennaRNA's 2D-only inverse folding method to further demonstrate the utility of 3D inverse folding. ViennaRNA has improved 2D self-consistency scores over gRNAde but fails to capture tertiary interactions in its designs, as evident by poor recovery and 3D self-consistency scores similar to the random baseline.

Split. Single- and multi-state splits are described in Section 3; the multi-state split is relatively harder than the single-state split based on overall reduced performance for all baselines and models. Models trained on 'All data' use all RNASolo samples for training, solely for the purpose of releasing the best possible gRNAde checkpoints for real-world usage. Evaluation metrics for 'All data' are reported on the single-state test set.

E Additional Results 623

Overall recovery:

0.27

0.32

0.45

Max Fitness by Sample Size and Condition (n=47,504; simulations=10,000)

1st best (fit.: 3.41)

3rd best (fit.: 3.16)

10th best (fit.: 2.67)

50th best (fit.: 2.27)

200th best (fit.: 1.94)

wildtype

1

10

50

100

200

403

1500

5000

17027

Selected sequences for assaying

0x

1x

0x

3x

6x

9x

12x

15x

18x

21x

24x

27x

30x

Expected 'max' fold change over WT

Condition

random

n_mut==1

n_mut<=2

gRNAde

0.00

1.10

1.79

2.20

2.48

2.71

2.89

3.04

3.18

3.30

3.40

Fitness

0.5682

F Additional Figures 624

Figure 14: Multi-graph tensor representation of RNA conformational ensembles.

P

O5'

C5'

C4'

C3'

C2'

C1'

O4'

O3'

P

P

P

C4'

RNA backbone atoms

Coarse-grained features

Node (nucleotide)

Backbone chain

3D neighbourhood

3x distances

3x angles

3x torsions

Ribose

sugar

Base

3-bead

representation

(P, C4', N1/N9)

N1/N9

5'

3'

GVP-GNN

encoder layer

GVP-GNN

encoder layer

Backbone k

Backbone 1

Autoregressive

decoder

+

...

A G C U

Per-node

logits

Pooling

x L

x L

GAGCG_

Sampling

Partial

sequence

Backbone

graph

Sampled

sequences

Predicted

structures

True

sequence

...

gRNAde

2D: EternaFold

3D: RhoFold

...

True

structure

Sequence recovery

Structural self-consistency scores

2D

: MCC,

3D

: RMSD, TM, GDT

Set of RNA Conformations

Multi-graph

tensor

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

Permute nodes

Permute conformers

Rotate 3D features

1

2

3

4

5

6

Update with

multi-GNN layer

1

2

3

4

5

6

0

1000

2000

3000

4000

Sequence length

0

200

400

600

800

1000

1200

1400

Frequency

Histogram of sequence lengths

Distribution: 684.9 ± 1072.8, Max: 4455, Min: 11

0

50

100

150

200

0

100

200

300

(a) Sequence length. The dataset is long-tailed in terms of RNA sequence length, with many short sequences including aptamers, riboswitches, ribozymes, and tRNAs (fewer than 200 nucleotides). The dataset also includes several longer ribosomal RNAs (thousands of nucleotides).

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

Avg. pairwise RMSD among structures per sequence (Å)

0

50

100

150

200

250

300

350

400

Frequency

Histogram of avg. pairwise RMSD per sequence

Distribution: 1.33Å ± 1.89, Max: 18.35Å, Min: 0.00Å

0

1

2

3

4

5

0

20

40

60

(c) Average pairwise RMSD per sequence. For 1,547 sequences with multiple structures, there is significant structural diversity among conformations. On average, the pairwise C4' RMSD among the set of structures for a sequence is greater than 1Å.

0

10

20

30

40

50

Number of structures per sequence

0

500

1000

1500

2000

2500

Frequency

Histogram of no. of structures per unique sequence

Distribution: 2.84 ± 9.39, Max: 267, Min: 1

5

10

15

20

0

200

400

600

800

Sequences with >1 structure

(b) Number of structures per sequence. The dataset covers a wide range of RNA conformation ensembles, with on average 3 structures per sequence. There are multiple structures available for 1,547 sequences. The remaining 2,676 sequences have one corresponding structure.

10 1

10 2

10 3

Sequence length (log scale)

0

1

2

3

4

5

6

7

Avg. pairwise RMSD among structures (Å)

(d) Bivariate distribution for sequence length vs. avg. RMSD. The joint plot illustrates how structural diversity (measured by avg. pairwise RMSD) varies across sequence lengths. We notice similar structural variations regardless of sequence length.

631

632

633

634

635

636

637

638

639

640

641

642

643

644

645

646

647

648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

666

667

668

669

670

671

672

673

674

675

676

677

678

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Yes, the main claims made in the abstract are that the proposed 3D RNA design method, gRNAde, improves sequence recovery over Rosetta, is capable of multi-state design, and can be useful for zero-shot ranking of RNA fitness landscapes. All the claims are supported by empirical results and expanded upon in detail in the rest of the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Yes, we have discussed limitations at several places, including the conclusion section, an FAQ section, as well as a detailed ablation study in the appendix.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All code, data, and pretrained models are publicly available, along with detailed instructions on installation, reproducing results, and real-world usage.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All data and code is publicly available, including the exact commands and environments needed to access the raw data, process the data, and reproduce the results.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experimental setup is presented in detail as a dedicated section in the main text, as well as extensively documented in the code.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All results are accompanied by error bars and confidence intervals, along with the factors of variability that the error bars capture.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes] 679

680

681

Justification: Yes, the paper and code provide information on the computer resources used for this work. However, we currently do not have estimes on the total compute used.

682

683

684

685

686

687

688

689

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conforms with the NeurIPS Code of Ethics.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes] 690

Justification: We hope that our tools contributes to the development of RNA-based therapeutics 691

692

towards improving health outcomes. We have attempted to make gRNAde as convinient to use

693

694

695

696

697

698

as possible towards this end. We do not foresee any immediate negative societal impact of our work.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA] 699

700

Justification: The paper poses no such risks.

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Original owners of any assets used as part of our stidy are appropriately credited.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Our datasets, code, and model checkpoints are publicly available under the permissive MIT License.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such 721 risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or 722 an equivalent approval/review based on the requirements of your country or institution) were 723 obtained? 724

Answer: [NA] 725

Justification: The paper does not involve crowdsourcing nor research with human subjects. 726

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\ngRNAde is a graph neural network designed to address the RNA reverse folding problem, a significant challenge due to the potential of RNA as therapeutic modalities and their unique data properties. RNA molecules have lower thermodynamic stability compared to proteins, resulting in fewer training samples, and their increased flexibility means multiple final states are possible. gRNAde addresses these issues by proposing a custom multi-graph representation and extending message passing to operate independently on each conformer while sharing an adjacency graph. The authors thoroughly explore evaluation techniques, comparing their model performance against Rosetta by assessing the percentage of native sequence recovery on held-out sequence families. Additionally, they demonstrate slightly improved performance when utilizing multiple conformers for model training. The authors also conduct an interesting zero-shot ranking analysis on mutation data providing a refreshing evaluation against random baselines.\n\nSTRENGTHS:\n- The paper is exceptionally well written presenting a thorough overview of the challenges within modality, broader field, and the importance of the problem.\n- The experimental validation assesses the utility of design choices and although the improved performance in the presence of multiple conformers isn't large (the authors don't report statistical significance), the approach is promising. \n- The authors conduct a variant effect evaluation assessing whether their model is capable of learning impact of single or double mutant sequences demonstrating convincing improvement over random baselines.\n\nWEAKNESSES:\n- I find the arguments regarding gRNAde perplexity being correlated with recovery to have limited support in the current presentation. In figure 2 (b) color denotes perplexity instead of one of the axis making it very challenging to assess the correlation. In addition the authors don't report a correlation value or its significance. \n- The authors only use random baselines for the retrospective variant effect analysis. Including another reverse folding model or a metric from Rosetta similar to gRNAde's perplexity could strengthen the evaluation.\n\nQUESTIONS:\n-  Can the authors conduct a study assessing the impact of the quantity of training data on gRNAde? As structural data increases, it will be useful to understand how gRNAde's performance scales with data abundance.\n- What are the other possible bottlenecks currently inhibiting further performance improvements of multi-conformational models? Do the authors believe further advancements in architectural designs are required to demonstrate larger improvements between models capturing a single vs multi-conformer information?\n\nLIMITATIONS:\n- The authors effectively discuss the current evaluation limitations and the difficulty of assessing the novelty and ground truth recovery of generated sequences.\n- There is limited discussion on the data limitations in the current field. With only 4000 sequences, training points are very few, presenting a major challenge.\n- The authors could include a brief statement on the broader impacts, such as the potential design of harmful molecules. As these models improve, the dual-use concern becomes legitimate.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-05"}

comments: SUMMARY:
This paper proposes a geometric RNA design model. Specifically, it introduces multi-stage GNN to encode multiple conformations and aggregate these candidates, and further feed decoder to predict probabilities of a set of candidate sequences.

STRENGTHS:
1.	This work creates a new dataset for RNA inverse design, with diverse properties such as sequence length, number of structures, and structural variations.
2.	This work designs a multi-state RNA reverse design model, distinct from existing methods.

WEAKNESSES:
1.	The technical contribution appears to be somewhat weak. The backbone used is from existing GNN models for equivariant design.
2.	Some technical details are not claimed. For example, are the comparison baselines retained on the new dataset or simply tested on their released model.

QUESTIONS:
1.	It is nice that they compared several SOTA baselines. However, I'm not sure if they 're retraining these baselines on the new dataset or just testing their released model.
2.	If I understand the claimed multi-state concept correctly, they are multiple conformations of a single RNA as input. It seems that multi-state design is an ensemble fusion of candidate representation. So, for a specific RNA with multiple conformations, does multi-state design always achieve better metrics than single-state design?
3.	I am still concerned about the technical insights of this paper as NIPS is highlight on the technical contributions. And I note that this paper has been accepted in ICML23 workshop.

LIMITATIONS:
This paper has no negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduce gRNAde, a geometric deep learning pipeline for RNA sequence design conditioned on one or more 3D backbone structures. gRNAde is superior to the physically based Rosetta for 3D 320 RNA inverse folding in terms of performance, inference speed, and ease of use. The method demonstrates significant superiority across various experiments.\n\nSTRENGTHS:\n1. The authors introduce gRNAde, the first work to consider multi-state biomolecule representation. This study explores the feasibility and specific experimental results of using multi-state biomolecule representation, providing new ideas for researchers in the field.\n2. The authors conduct extensive experiments and analyses on multiple datasets and experimental settings, demonstrating the model's effectiveness from various perspectives, especially regarding the \"Zero-shot ranking of RNA fitness landscape\" experiment, which is currently lacking in this field.\n3. The authors present various experimental details using numerous visualizations and data tables, making the paper easier for readers to understand.\n\nWEAKNESSES:\n1. The model architecture proposed by the author lacks innovation. The core structure of gRNAde is directly stacked using GVP-GNN, and the handling of multi-state conformations is merely simple stacking. Additionally, the 3-beads representation method is very common in traditional RNA 3D structure modeling, which is also not an innovation by the author. Therefore, I believe the model design is lacking.\n2. The baselines compared by the author in various experiments are either outdated or too simple, such as \"Rosetta(2020)\" and the \"random baseline\" in the Zero-shot experiment. This makes it difficult to demonstrate the actual performance of gRNAde. Some recent works using deep learning to model RNA 3D structures can serve as baselines, such as [1-3].\n3. The author mentions that gRNAde has a significant speed improvement over Rosetta, but the author did not run the Rosetta code themselves and instead directly cited the original Rosetta paper. I believe this point is debatable because the model's running speed is also limited by GPU computational performance. The author uses an A100, whereas the GPU used by Rosetta four years ago is obviously inferior to the A100. Therefore, the author needs to rerun the Rosetta program on the A100 to provide accurate model inference times.\n\n\n\n\n\n\n\n[1] Geometric deep learning of RNA structure, Science 2021\n\n[2] Physics-aware Graph Neural Network for Accurate RNA 3D Structure Prediction, NIPS workshop 2022\n\n[3] RDesign: Hierarchical Data-efficient Representation Learning for Tertiary Structure-based RNA Design, ICLR 2024\n\nQUESTIONS:\n1. The second experiment results indicate that multi-state biomolecule representation did not significantly improve performance. What does the author believe is the reason for this?\n2. The RNA used in the first experiment are older and limited in number. Has the author considered testing with more recently published and more numerous RNA?\n\nLIMITATIONS:\nThe authors discuss practical tradeoffs to using gRNAde in real-world RNA design scenarios 330 in Appendix B, including limitations due to the current state of 3D RNA structure prediction tools.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work designed gRNAde, a geometric deep learning pipeline for RNA sequence design conditioned on one or more 3D backbone structures. To achieve this, the authors created single-state and multi-state 3D RNA structure datasets, built a geometric graph representation, and proposed an architecture consisting of a multi-state GNN encoder, a pooling layer, and a autoregressive decoder. The single-state RNA design, multi-state RNA design, and zero-shot rank experiments were conducted and results show that gRNAde outperformed all previous methods including Rosetta.\n\nSTRENGTHS:\n1.\tThe datasets were carefully designed. Only structures with high resolution were maintained. Two kinds of clusters were used to split train, validate, and test sets where the hard samples were split into test sets.\n\n2.\tThe model architecture makes use of information from multiple conformations. This is achieved by sum or average pooling.\n\n3.\tThe experiments were conducted fairly. Datasets were split carefully. The results were averaged on 16 sampled sequences across 3 random seeds.\n\n4.\tThe inference speed is much faster than traditional methods. This makes it possible to be used in High throughput screening. The zero-shot ranking ability is also an advantage.\n\nWEAKNESSES:\n1.\tThe model architecture has no novelty. All components are token from previous work and the overall structure is very similar to that of ProteinMPNN. The multiple conformations are processed independently and the representations are simply averaged or summed, which may not grasp all information.\n\n2.\tThe model is trained on only about 4 thousand RNA sequences. These sequences are too few to cover the entire space. RNAs with no 3D structures should be exploited, as done in alphafold3.\n\n3.\tThe results about single-state RNA design were reported on only 14 samples. More samples should be used to test the model. For example, the results on test sets with 100 samples should be reported.\n\n4.\tThe improvements on multi-state RNA design task are limited. The native sequence recovery is obviously lower than the results of single-state design task.\n\nQUESTIONS:\n1.\tLots of the RNA conformations come from protein-RNA complexes and DNA-RNA hybrids, which means these conformations do not appear alone. Since the model encoder cannot consider other molecules, the extracted representations are biased. How about adding a scalar feature to indicate if a nucleotide is on the interface with another molecule?\n\n2.\tWhy is the decoder only designed for autoregressive decoding? The arbitrary decoding in ProteinMPNN is useful when part of the sequence is already known.\n\nLIMITATIONS:\nThe representation ability of gRNAde remains to be verified. Usually, the representation is extracted when all nucleotides are known, so the architecture for inverse folding problem may not suitable for representation learning.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduced a multi-state geometric graph neural network for the RNA inverse folding problems. Experiments are conducted on carefully splited structural datasets that avoid data leakage. The results have shown convincing performance improvement over the physics-based methods such has FARFAR and Rosetta which are commonly used for RNAs.\n\nSTRENGTHS:\n- This paper is well written and pleasing to read. Explanations on key biological concepts related to RNAs, and how they motivate the model design as well as experiment setup are adequate and above all, clear.\n- Evaluation metrics are well considered. The self-consistent scores on the secondary and tertiary levels are meaningful.\n    - limitations on self-consistence scores are acknowledged in the main text. For RNAs, many challenges are unique especially when they are compared to proteins. Therefore, clarifications and precautions are particularly needed for RNA related tasks. I appreciate the authors’ effort for stating these limitation clearly in the main text.\n\nWEAKNESSES:\n- Comparison to contemporary deep learning models for RNA inverse folding is limiting\n    - RDesign (https://openreview.net/forum?id=RemfXx7ebP) for example is a recent deep learning based method for 3D RNA inverse design\n    - For inverse design on the secondary structure level there are many more options — a lot of them are better than RNAinverse from ViennaRNA. I would suggest checking out this survey (Design of RNAs: comparing programs for inverse RNA folding) and include a few other more competitive baselines.\n- For the self-consistency scores, I personally doubt if RhoFold (also called e2efold-3d) is reliable software for RNA tertiary structure prediction, since it is from the same group that published e2efold which is a spectacularly awful RNA secondary structure predictor (I personally would avoid using any of their tools; checkout its Github issues, and also followup works on RNA secondary structure predictions that have compared with e2efold). Have the authors used more recent folding softwares such as RosettaFoldNA and AlphaFold3?\n    - Would using different structure predictors significantly impact the results? This also includes EternaFold. How would gRNAde hold up against the baselines when RNAfold or LinearFold is used to compute the self-consistency scores on the secondary level?\n- Data splits (train, validation and test) are carefully constructed so that the evaluation is not contaminated by data leak. But I wonder if the TM-score cut-off at 0.45 is too lenient? Is it still possible to have similar structures between training and test sets under this threshold?\n\nLine 258 to 259. The argument would be more compelling if the inverse design operates at the quaternary level which would include information about ligand structures.\n\nQUESTIONS:\n- Why would a perfect model has a perplexity of 1 (i.e. mapping each structure to a single sequence)? The solution for RNA inverse folding should be an one to many mapping, because for a target backbone structure there are potentially many sequences that can fold into the same structure. Likewise, a perplexity of 4 doesn’t mean the predictions are nonsense. For unfolded RNAs (e.g. a linear chain), they very likely correspond to random RNA sequences.\n- Conformers would indicate that the structures are the local energy minimizers on the potential energy surface. How would you know if these multi-state conformations are really conformers? Where are multi-state conformations from?\n\nLIMITATIONS:\n- Comparison to contemporary models for RNA inverse folding is a bit hollow. It would be more meaningful if some deep learning based baselines can be included into the comparison.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}