gRNAde: Geometric Deep Learning for 3D RNA inverse design

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 13:05:24

Abstract: Computational RNA design tasks are often posed as inverse problems, where sequences are designed based on adopting a single desired secondary structure without considering 3D geometry and conformational diversity. We introduce gRNAde, a geometric RNA design pipeline operating on 3D RNA backbones to design sequences that explicitly account for structure and dynamics. Under the hood, gRNAde is a multi-state Graph Neural Network that generates candidate RNA sequences conditioned on one or more 3D backbone structures where the identities of the bases are unknown. On a single-state fixed backbone re-design benchmark of 14 RNA structures from the PDB identified by Das et al. [2010], gRNAde obtains higher native sequence recovery rates (56% on average) compared to Rosetta (45% on average), taking under a second to produce designs compared to the reported hours for Rosetta. We further demonstrate the utility of gRNAde on a new benchmark of multi-state design for structurally flexible RNAs, as well as zero-shot ranking of mutational fitness landscapes in a retrospective analysis of a recent RNA polymerase ribozyme structure. Open source code and tutorials are available at: https://anonymous.4open.science/r/geometric-rna-design

Main paper: 
 introduction: 
 Why RNA design? Historical efforts in computational drug discovery have focussed on designing small molecule or protein-based medicines that either treat symptoms or counter the end stages of disease processes. In recent years, there is a growing interest in designing new RNA-based therapeutics that intervene earlier in disease processes to cut off disease-causing information flow in the cell [Damase et al., 2021 , Zhu et al., 2022] . Notable examples of RNA molecules at the forefront of biotechnology today include mRNA vaccines [Metkar et al., 2024] and CRISPR-based genomic medicine [Doudna and Charpentier, 2014] . Of particular interest for structure-based design are ribozymes and riboswitches in the untranslated regions of mRNAs [Mandal and Breaker, 2004, Leppek et al., 2018] . In addition to coding for proteins (such as the spike protein in the Covid vaccine), naturally occurring mRNAs contain riboswitches that are responsible for cell-state dependent protein expression of the mRNA. Riboswitches act by 'switching' their 3D structure from an unbound conformation to a bound one in the presence of specific metabolites or small molecules. Rational design of riboswitches will enable translation to be dependent on the presence or absence of partner molecules, essentially acting as 'on-off' switches for highly targeted mRNA therapies in the future [Felletti et al., 2016 , Mustafina et al., 2019 , Mohsen et al., 2023] .Challenges of RNA modelling. Despite the promises of RNA therapeutics, proteins have instead been the primary focus in the 3D biomolecular modelling community. Availability of a large number of protein structures from the PDB combined with advances in deep learning for structured data [Bronstein et al., 2021 , Duval et al., 2023] have revolutionized protein 3D structure prediction [Jumper Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute.Network EncoderSequence DecoderSet of Backbone Geometric Graphs Extract Backbones GAGCGU...Fixed backbone re-design 3D roto-translations node order conformation order Equivariant to:Figure 1 : The gRNAde pipeline for 3D RNA inverse design. gRNAde is a generative model for RNA sequence design conditioned on backbone 3D structure(s). gRNAde processes one or more RNA backbone graphs (a conformational ensemble) via a multi-state GNN encoder which is equivariant to 3D roto-translation of coordinates as well as conformer order, followed by conformer order-invariant pooling and autoregressive sequence decoding.et al., 2021] and rational design [Dauparas et al., 2022 , Watson et al., 2023] . Applications of deep learning for computational RNA design are underexplored compared to proteins due to paucity of 3D structural data [Schneider et al., 2023] . Most tools for RNA design primarily focus on secondary structure without considering 3D geometry [Churkin et al., 2018] and use non-learnt algorithms for aligning 3D RNA fragments [Han et al., 2017 , Yesselman et al., 2019] , which can be restrictive due to the hand-crafted nature of the heuristics used.In addition to limited 3D data for training deep learning models, the key technical challenge is that RNA is more dynamic than proteins. The same RNA can adopt multiple distinct conformational states to create and regulate complex biological functions [Ganser et al., 2019 , Hoetzel and Suess, 2022 , Ken et al., 2023] . Computational RNA design pipelines must account for both the 3D geometric structure and conformational flexibility of RNA to engineer new biological functions.Our contributions. This paper introduces gRNAde, a geometric deep learning-based pipeline for RNA inverse design conditioned on 3D structure, analogous to ProteinMPNN for proteins [Dauparas et al., 2022] . As illustrated in Figure 1 , gRNAde generates candidate RNA sequences conditioned on one or more backbone 3D conformations, enabling both single-and multi-state fixed-backbone sequence design.We demonstrate the utility of gRNAde for the following design scenarios:• Improved performance and speed over Rosetta. We compare gRNAde to Rosetta [Leman et al., 2020] , the state-of-the-art physically based tool for 3D RNA inverse design, for singlestate fixed backbone design of 14 RNA structures of interest from the PDB identified by Das et al. [2010] . We obtain higher native sequence recovery rates with gRNAde (56% on average) compared to Rosetta (45% on average). Additionally, gRNAde is significantly faster than Rosetta for inference; e.g. sampling 100+ designs in 1 second for an RNA of 60 nucleotides on an A100 GPU, compared to the reported hours for Rosetta.• Enables multi-state RNA design, which was previously not possible with Rosetta. gRNAde with multi-state GNNs improves sequence recovery over an equivalent single-state model on a benchmark of structurally flexible RNAs, especially for surface nucleotides which undergo positional or secondary structural changes.• Zero-shot learning of RNA fitness landscape. In a retrospective analysis of mutational fitness landscape data for an RNA polymerase ribozyme [McRae et al., 2024] , we show how gRNAde's perplexity, the likelihood of a sequence folding into a backbone structure, can be used to rank mutants based on fitness in a zero-shot/unsupervised manner and outperforms random mutagenesis for improving fitness over the wild type in low throughput scenarios.2 The gRNAde pipeline 2.1 The 3D RNA inverse folding problemFigure 1 illustrates the RNA inverse folding problem: the task of designing new RNA sequences conditioned on a structural backbone. Given the 3D coordinates of a backbone structure, machine learning models must generate sequences that are likely to fold into that shape. The underlying assumption behind inverse folding (and rational biomolecule design) is that structure determines function [Huang et al., 2016] . To the best of our knowledge, gRNAde is the first explicitly multi-state inverse folding pipeline, allowing users to design sequences for backbone conformational ensembles (a set of 3D backbone structures) as opposed to a single structure. 
rna conformational ensembles as geometric multi-graphs: 
 Featurization. The input to gRNAde is an RNA to be re-designed. For instance, this could be a set of PDB files with 3D backbone structures for the given RNA (a conformational ensemble) and the corresponding sequence of n nucleotides. As shown in Appendix Figure 11 , gRNAde builds a geometric graph representation for each input structure:1. We start with a 3-bead coarse-grained representation of the RNA backbone, retaining the coordinates for P, C4', N1 (pyrimidine) or N9 (purine) for each nucleotide [Dawson et al., 2016] .This 'pseudotorsional' representation describes RNA backbones completely in most cases while reducing the size of the torsional space to prevent overfitting [Wadley et al., 2007] .2. Each nucleotide i is assigned a node in the geometric graph with the 3D coordinate ⃗x i ∈ R 3corresponding to the centroid of the 3 bead atoms. Random Gaussian noise with standard deviation 0.1Å is added to coordinates during training to prevent overfitting on crystallisation artifacts, following Dauparas et al. [2022] . Each node is connected by edges to its 32 nearest neighbours as measured by the pairwise distance in 3D space, ∥⃗ x i -⃗ x j ∥ 2 .3. Nodes are initialized with geometric features analogous to the featurization used in protein inverse folding [Ingraham et al., 2019 , Jing et al., 2020] Multi-graph representation. As described in the previous section, given a set of k (conformer) structures in the input conformational ensemble, each RNA backbone is featurized as a separate geometric graph  and A (k) , an n × n adjacency matrix. For clear presentation and without loss of generality, we omit edge features and use f , f ′ to denote scalar/vector feature channels.G (k) = (A (k) , S (k) , ⃗ V (k) ) with the scalar features S (k) ∈ R n×f , vector features ⃗ V (k) ∈ R n×f ′ ×3 ,The input to gRNAde is thus a set of geometric graphs {G (1) , . . . , G (k) } which is merged into what we term a 'multi-graph' representation of the conformational ensemble, M = (A, S, ⃗ V ), by stacking the set of scalar features {S (1) , . . . , S (k) } into one tensor S ∈ R n×k×f along a new axis for the set size k. Similarly, the set of vector features{ ⃗ V (1) , . . . , ⃗ V (k) } is stacked into one tensor ⃗ V ∈ R n×k×f ′ ×3 .Lastly, the set of adjacency matrices {A (1) , . . . , A (k) } are merged via a union ∪ into one single joint adjacency matrix A.The gRNAde model, illustrated in Appendix Figure 12 , processes one or more RNA backbone graphs via a multi-state GNN encoder which is equivariant to 3D roto-translation of coordinates as well as to the ordering of conformers, followed by conformer order-invariant pooling and sequence decoding.We describe each component in the following sections.Multi-state GNN encoder. When representing conformational ensembles as a multi-graph, each node feature tensor contains three axes: (#nodes, #conformations, feature channels). We perform message passing on the multi-graph adjacency to independently process each conformer, while maintaining permutation equivariance of the updated feature tensors along both the first (#nodes)and second (#conformations) axes. This works by operating on only the feature channels axis and generalising the PyTorch Geometric [Fey and Lenssen, 2019] message passing class to account for the extra conformations axis; see Appendix Figure 14 and the pseudocode for details.We use multiple rotation-equivariant GVP-GNN [Jing et al., 2020] layers to update scalar featuress i ∈ R k×f and vector features ⃗ v i ∈ R k×f ′ ×3for each node i:EQUATIONEQUATIONwhere MSG, UPD are Geometric Vector Perceptrons, a generalization of MLPs to take tuples of scalar and vector features as input and apply O(3)-equivariant non-linear updates. The overall GNN encoder is SO(3)-equivariant due to the use of reflection-sensitive input features (dihedral angles) combined with O(3)-equivariant GVP-GNN layers.Our multi-state GNN encoder is easy to implement in any message passing framework and can be used as a plug-and-play extension for any geometric GNN pipeline to incorporate the multi-state inductive bias. It serves as an elegant alternative to batching all the conformations, which we found required major alterations to message passing and pooling depending on downstream tasks.Conformation order-invariant pooling. The final encoder representations in gRNAde account for multi-state information while being invariant to the permutation of the conformational ensemble. To achieve this, we perform a Deep Set pooling [Zaheer et al., 2017] over the conformations axis after the final encoder layer to reduce S ∈ R n×k×f and ⃗EQUATIONA simple sum or average pooling does not introduce any new learnable parameters to the pipeline and is flexible to handle a variable number of conformations, enabling both single-state and multi-state design with the same model.Sequence decoding and loss function. We feed the final encoder representations after pooling, S ′ , ⃗ V ′ , to autoregressive GVP-GNN decoder layers to predict the probability of the four possible base identities (A, G, C, U) for each node/nucleotide. Decoding proceeds according to the RNA sequence order from the 5' end to 3' end. gRNAde is trained in a self-supervised manner by minimising a cross-entropy loss (with label smoothing value of 0.05) between the predicted probability distribution and the ground truth identity for each base. During training, we use autoregressive teacher forcing [Williams and Zipser, 1989] where the ground truth base identity is fed as input to the decoder at each step, encouraging the model to stay close to the ground-truth sequence.Sampling. When using gRNAde for inference and designing new sequences, we iteratively sample the base identity for a given nucleotide from the predicted conditional probability distribution, given the partially designed sequence up until that nucleotide/decoding step. We can modulate the smoothness or sharpness of the probability distribution by using a temperature parameter.In principle, inverse folding models can be sampled from to obtain a large number of designed sequences for a given backbone structure. Thus, in-silico metrics to determine which sequences are useful and which ones to prioritise in wet lab experiments are a critical part of the overall pipeline. We currently use the following metrics to evaluate gRNAde's designs, visualised in Appendix Figure 13 :• Native sequence recovery, which is the average percentage of native (ground truth) nucleotides correctly recovered in the sampled sequences. Recovery is the most widely used metric for biomolecule inverse design [Dauparas et al., 2022] but can be misleading in the case of RNAs where alternative nucleotide base pairings can form the same structural patterns.• Secondary structure self-consistency score, where we 'forward fold' the sampled sequences using a secondary structure prediction tool (we used EternaFold [Wayment-Steele et al., 2022] ) and measure the average Matthew's Correlation Coefficient (MCC) to the groundtruth secondary structure, represented as a binary adjacency matrix. MCC values range between -1 and +1, where +1 represents a perfect prediction, 0 an average random prediction and -1 an inverse prediction. This measures how well the designs recover base pairing patterns.• Tertiary structure self-consistency scores, where we 'forward fold' the sampled sequences using a 3D structure prediction tool (we used RhoFold [Shen et al., 2022] ) and compute the average RMSD, TM-score and GDT_TS to the groundtruth C4' coordinates to measure how well the designs recover global structural similarity and 3D conformations.• Perplexity, which can be thought of as the average number of bases that the model is selecting from for each nucleotide. Formally, perplexity is the average exponential of the negative log-likelihood of the sampled sequences. A perfect model would have perplexity of 1, while a perplexity of 4 means that the model is making random predictions (the model outputs a uniform probability over 4 possible bases). Perplexity does not require a ground truth structure to calculate, and can also be used for ranking sequences as it is the model's estimate of the compatibility of a sequence with the input backbone structure.Significance and limitations. Self-consistency metrics, termed 'designability' (eg. scRMSD≤2Å), as well as perplexity have been found to correlate with experimental success in protein design [Watson et al., 2023] . While precise designability thresholds are yet to be established for RNA, pairs of structures with TM-score≥0.45 or GDT_TS≥0.5 are known to correspond to roughly the same fold [Zhang et al., 2022] . Another major limitation for in-silico evaluation of 3D RNA design compared to proteins is the relatively worse state of structure prediction tools [Schneider et al., 2023] .3 Experimental Setup 3D RNA structure dataset. We create a machine learning-ready dataset for RNA inverse design using RNASolo [Adamczyk et al., 2022] , a novel repository of RNA 3D structures extracted from solo RNAs, protein-RNA complexes, and DNA-RNA hybrids in the PDB. We used structures at resolution ≤4.0Å resulting in 4,223 unique RNA sequences for which a total of 12,011 structures are available (RNASolo date cutoff: 31 October 2023). Dataset statistics are available in Appendix Figure 15 , illustrating the diversity of our dataset in terms of sequence length, number of structures per sequence, as well as structural variations among conformations per sequence.Structural clustering. In order to ensure that we evaluate gRNAde's generalization ability to novel RNAs, we cluster the 4,223 unique RNAs into groups based on structural similarity. We use US-align [Zhang et al., 2022] with a similarity threshold of TM-score >0.45 for clustering, and ensure that we train, validate and test gRNAde on structurally dissimilar clusters (see next paragraph). We also provide utilities for clustering based on sequence homology using CD-HIT [Fu et al., 2012] , which leads to splits containing biologically dissimilar clusters of RNAs.Splits to evaluate generalization. After clustering, we split the RNAs into training (∼4000 samples), validation and test sets (100 samples each) to evaluate two different design scenarios:1. Single-state split. This split is used to fairly evaluate gRNAde for single-state design on a set of RNA structures of interest from the PDB identified by Das et al. [2010] , which mainly includes riboswitches, aptamers, and ribozymes. We identify the structural clusters belonging to the RNAs identified in Das et al. [2010] and add all the RNAs in these clusters to the test set (100 samples). The remaining clusters are randomly added to the training and validation splits.2. Multi-state split. This split is used to test gRNAde's ability to design RNA with multiple distinct conformational states. We order the structural clusters based on median intra-sequence RMSD among available structures within the cluster1 . The top 100 samples from clusters with the highest median intra-sequence RMSD are added to the test set. The next 100 samples are added to the validation set and all remaining samples are used for training.Validation and test samples come from clusters with at most 5 unique sequences, in order to ensure diversity. Any samples that were not assigned clusters are directly appended to the training set. We 2 . also directly add very large RNAs (> 1000 nts) to the training set, as it is unlikely that we want to design very large RNAs. We exclude very short RNA strands (< 10 nts).Evaluation metrics. For a given data split, we evaluate models on the held-out test set by designing 16 sequences (sampled at temperature 0.1) for each test data point and computing averages for each of the metrics described in Section 2.4: native sequence recovery, structural self-consistency scores and perplexity. We employ early stopping by reporting test set performance for the model checkpoint for the epoch with the best validation set recovery. Standard deviations are reported across 3 consistent random seeds for all models.Hyperparameters. All models use 4 encoder and 4 decoder GVP-GNN layers, with 128 scalar/16vector node features, 64 scalar/4 vector edge features, and drop out probability 0.5, resulting in 2,147,944 trainable parameters. All models are trained for a maximum of 50 epochs using the Adam optimiser with an initial learning rate of 0.0001, which is reduced by a factor 0.9 when validation performance plateaus with patience of 5 epochs. Ablation studies of key modelling decisions are available in Appendix Table 1 . 
single-state rna design benchmark: 
 We set out to compare gRNAde to Rosetta, a state-of-the-art physically based toolkit for biomolecular modelling and design [Leman et al., 2020] . We reproduced the benchmark setup from Das et al.[2010] for Rosetta's fixed backbone RNA sequence design workflow on 14 RNA structures of interest from the PDB, which mainly includes riboswitches, aptamers, and ribozymes (full listing in Table 2 ). We trained gRNAde on the single-state split detailed in Section 3, explicitly excluding the 14 RNAs as well as any structurally similar RNAs in order to ensure that we fairly evaluate gRNAde's generalization abilities vs. Rosetta.gRNAde improves sequence recovery over Rosetta. In Figure 2 , we compare gRNAde's native sequence recovery for single-state design with numbers taken from Das et al. [2010] for Rosetta, FARNA (a predecessor of Rosetta), and ViennaRNA (the most popular 2D inverse folding method).gRNAde has higher recovery of 56% on average compared to 45% for Rosetta, 32% for FARNA, and 27% for ViennaRNA. See Appendix Table 2 for per-RNA results.gRNAde is significantly faster than Rosetta. In addition to superior sequence recovery, gRNAde is significantly faster than Rosetta for high-throughout design pipelines. Training gRNAde from scratch takes roughly 2-6 hours on a single A100 GPU, depending on the exact hyperparameters.Once trained, gRNAde can design hundreds of sequences for backbones with hundreds of nucleotides gRNAde's perplexity correlates with sequence and structural recovery. In Figure 2b , we plot native sequence recovery per sample for Rosetta vs. gRNAde, shaded by gRNAde's average perplexity for each sample. Perplexity is an indicator of the model's confidence in its own prediction (lower perplexity implies higher confidence) and appears to be correlated with native sequence recovery.Additionally, visualisations of gRNAde's designs for a riboswitch in Figure 3 show that perplexity is also correlated with structural self-consistency scores. In the subsequent Section 4.3, we further demonstrate the utility of gRNAde's perplexity for zero-shot ranking of RNA fitness landscapes.Structured RNAs often adopt multiple distinct conformational states to perform biological functions [Ken et al., 2023] . For instance, riboswitches adopt at least two distinct functional conformations: a ligand bound (holo) and unbound (apo) state, which helps them regulate and control gene expression [Stagno et al., 2017] Improved recovery in structurally flexible regions. In Figure 4b , we evaluated gRNAde's multi-state sequence recovery at a fine-grained, per-nucleotide level. Multi-state GNNs improve sequence recovery over the single-state variant on structurally flexible nucleotides, as characterised by undergoing changes in base pairing/secondary structure, higher average RMSD between 3D coordinates across states, and larger solvent accessible surface area.Lastly, we explored the use of gRNAde as a zero-shot ranker of mutants in RNA engineering campaigns. Given the backbone structure of a wild type RNA of interest as well as a candidate set of mutant sequences, we can compute gRNAde's perplexity of whether a given sequence folds into the backbone structure. Perplexity is inversely related to the likelihood of a sequence conditioned on a structure, as described in Section 2.4. We can then rank sequences based on how 'compatible' they are with the backbone structure in order to select a subset to be experimentally validated in wet labs.Retrospective analysis on ribozyme fitness landscape. A recent study by McRae et al. [2024] determined a cryo-EM structure of a dimeric RNA polymerase ribozyme at 5Å resolution3 , along with fitness landscapes of ∼75K mutants for the catalytic subunit 5TU and ∼48K mutants for the scaffolding subunit t1. We design a retrospective study using this data of (sequence, fitness value) pairs where we simulate an RNA engineering campaign with the aim of improving catalytic subunit fitness over the wild type 5TU sequence.We consider various design budgets ranging from hundreds to thousands of sequences selected for experimental validation, and compare 4 unsupervised approaches for ranking/selecting variants: (1)random choice from all ∼75,000 sequences; (2) random choice from all 449 single mutant sequences;(3) random choice from all single and double mutant sequences (as sequences with higher mutation order tend to be less fit); and (4) negative gRNAde perplexity (lower perplexity is better). For each design budget and ranking approach, we compute the expected maximum change in fitness over the wild type that could be achieved by screening as many variants as allowed in the given design budget.We run 10,000 simulations to compute confidence intervals for the 3 random baselines.gRNAde outperforms random baselines in low design budget scenarios. et al., 2024] , we retrospectively analyse how well we can rank variants at multiple design budgets using random selection vs. gRNAde's perplexity for mutant sequences conditioned on the backbone structure (catalytic subunit 5TU). Note that gRNAde is used zero-shot here, i.e. it was not fine-tuned on any assay data. For stochastic strategies, bars indicate median values, and error bars indicate the interquartile range estimated from 10,000 simulations per strategy and design budget. At low throughput design budgets of up to ∼500 sequences, selecting mutants using gRNAde outperforms random baselines in terms of the expected maximum improvement in fitness over the wild type. In particular, gRNAde performs better than single site saturation mutagenesis, even when all single mutants are explored (total of 449 single mutants, 10,493 double mutants for the catalytic subunit 5TU in McRae et al. [2024] ). See Appendix Figure 10 for results on scaffolding subunit t1.Perspective. Overall, it is promising that gRNAde's perplexity correlates with experimental fitness measurements out-of-the-box (zero-shot) and can be a useful ranker of mutant fitness in our retrospective study. In realistic design scenarios, improvements could likely be obtained by fine-tuning gRNAde on a low amount of experimental fitness data. For example, latent features from gRNAde may be finetuned or used as input to a prediction head with supervised learning on fitness landscape data. This study acts as a sanity check before committing to wet lab validation of gRNAde designs. We see random mutagenesis and directed evolution-based approaches as complementary to de-novo design and inverse folding approaches like gRNAde. Random mutagenesis can be thought of as local exploration around a wild type sequence, optimising fitness within an 'island' of activity.Structure-based design approaches are akin to global jumps in sequence space, with the potential to find new islands further away from the wild type [Huang et al., 2016] . 
conclusion: 
 We introduce gRNAde, a geometric deep learning pipeline for RNA sequence design conditioned on one or more 3D backbone structures. gRNAde is superior to the physically based Rosetta for 3D RNA inverse folding in terms of performance, inference speed, and ease of use. Further, gRNAde enables explicit multi-state design for structurally flexible RNAs which was previously not possible with Rosetta. gRNAde's perplexity correlates with native sequence and structural recovery, and can be used for zero-shot ranking of mutants in RNA engineering campaigns. To the best of our knowledge, gRNAde is also the first geometric deep learning architecture for multi-state biomolecule representation learning; the model is generic and can be repurposed for other learning tasks on conformational ensembles, including multi-state protein design.Limitations. Key avenues for future development of gRNAde include supporting multiple interacting chains, accounting for partner molecules with RNAs, and supporting negative design against undesired conformations. We discuss practical tradeoffs to using gRNAde in real-world RNA design scenarios in Appendix B, including limitations due to the current state of 3D RNA structure prediction tools. 
 
 Appendix: We attempt to briefly summarise recent developments in RNA structure modelling and design, with an emphasis on deep learning-based approaches.RNA inverse folding. Most tools for RNA inverse folding focus on secondary structure without considering 3D geometry [Churkin et al., 2018 , Runge et al., 2019] and approach the problem from the lens of energy optimisation [Ward et al., 2023] . Rosetta fixed backbone re-design [Das et al., 2010] is the only energy optimisation-based approach that accounts for 3D structure. Deep neural networks such as gRNAde can incorporate 3D structural constraints and are orders of magnitude faster than optimisation-based approaches; this is particularly attractive for high-throughput design pipelines as solving the inverse folding optimisation problem is NP hard [Bonnet et al., 2020] .RNA structure design. Inverse folding models for protein design have often been coupled with backbone generation models which design structural backbones conditioned on various design constraints [Watson et al., 2023 , Ingraham et al., 2023 , Didi et al., 2023] . Current approaches for RNA backbone design use classical (non-learnt) algorithms for aligning 3D RNA motifs [Han et al., 2017 , Yesselman et al., 2019] , which are small modular pieces of RNA that are believed to fold independently. Such algorithms may be restricted by the use of hand-crafted heuristics and we plan to explore data-driven generative models for RNA backbone design in future work.RNA structure prediction. There have been several recent efforts to adapt protein folding architectures such as AlphaFold2 [Jumper et al., 2021] and RosettaFold [Baek et al., 2021] for RNA structure prediction [Li et al., 2023b , Wang et al., 2023 , Baek et al., 2024] . A previous generation of models used GNNs as ranking functions together with Rosetta energy optimisation [Watkins et al., 2020 , Townshend et al., 2021] . None of these architectures aim at capturing conformational flexibility of RNAs, unlike gRNAde which represents RNAs as multi-state conformational ensembles. Neither can structure prediction tools be used for RNA design tasks as they are not generative models.RNA language models. Self-supervised language models have been developed for predictive and generative tasks on RNA sequences, including general-purpose models such as RNA FM [Chen et al., 2022] and RiNaLMo [Penic et al., 2024] as well as mRNA-specific CodonBERT [Li et al., 2023a] . RNA sequence data repositories are orders of magnitude larger than those for RNA structure (eg. RiNaLMo is trained on 36 million sequences). However, standard language models can only implicitly capture RNA structure and dynamics through sequence co-occurence statistics, which can pose a chellenge for designing structured RNAs such as riboswitches, aptamers, and ribozymes.RibonanzaNet [He et al., 2024] represents a recent effort in developing structure-informed RNA language models by supervised training on experimental readouts from chemical mapping, although RibonanzaNet cannot be used for RNA design. Inverse folding methods like gRNAde are language models conditioned on 3D structure, making them a natural choice for structure-based design.Comparison to contemporaneous work. Concurrently, Tan et al. [2023] also developed a deep learning-based 3D RNA inverse folding model. We want to emphasize that this is independent work, but for completeness we include a discussion on key differences to gRNAde:• Methodology:-New capabilities: gRNAde enables explicit multi-state design to generate sequences conditioned on multiple backbone structures, which is not possible with Rosetta nor Tan et al. [2023] 's approach. We have also demonstrated the utility of gRNAde's perplexity for zero-shot ranking of mutants in RNA engineering campaigns.-Decoding: gRNAde uses an autoregressive decoder with rotation-equivariant GNN layers, while Tan et al. [2023] use a non-autoregressive (one-shot) decoder with rotation-invariant layers. In our ablation study (Appendix D), we found autoregressive decoding to show significantly higher 2D and 3D self-consistency scores than non-autoregressive decoding, even though non-autoregressive decoding lead to higher sequence recovery. Autoregressive decoding is more expressive and can condition predictions at each decoding step on past predictions, while one-shot decoders sample from independent probability distributions for each nucleotide. We find autoregressive decoding to be a better inductive bias for predicting base pairing and base stacking interactions that are drivers of RNA structure [Vicens and Kieft, 2022] . For instance, G-C and A-U pairs can often be swapped for one another, but non-autoregressive decoding does not capture such paired constraints.-Evaluation metrics: Tan et al. [2023] focus on measuring native sequence recovery, only.We have additionally introduced structural self-consistency metrics at the 2D and 3D level, which have been shown to better correlate with experimental success in protein design.-Perplexity: We found gRNAde's perplexity to be correlated with sequence and structural recovery, as well as demonstrated its utility for zero-shot ranking of mutants in RNA engineering. On the other hand, Tan et al. [2023] do not report perplexity and claim that perplexity is an unsuitable metric for RNA design.-Data splitting: While both studies use structural clustering to evaluate generalisation to structurally dissimilar RNAs, Tan et al. [2023] 's test splits are determined randomly. Our experiments use currated test splits from Das et al. [2010] to fairly compare gRNAde to physically based Rosetta, as well as split based on structural flexibility to benchmark multi-state design.• Usage and reproducibility: We release open source training and inference code as well as model checkpoints to enable complete reproducibility. We also release Colab notebooks and detailed tutorials to make gRNAde broadly applicable and useful in real-world RNA design campaigns.At present, it is not possible to reproduce the results in Tan et al. [2023] or compare to gRNAde directly as no training code is available.How to chose the number of states to provide as input to gRNAde? In general, this would depend on the design objective. For instance, designing riboswitches may necessitate multi-state design, while a single-state pipeline may be more sensible for locking an aptamer into its bound conformation [Yesselman et al., 2019] . Note that it may be possible to benefit from multi-state gRNAde models even when performing single-state design by using slightly noised variations of the same backbone structure as an input conformational ensemble.How to prioritise or chose amongst designed sequences? We have currently provided 3 types of evaluation metrics: native sequence recovery, structural self-consistency scores and perplexity, towards this end. We suspect that recovery may not be the ideal choice, except for design scenarios where we require certain regions of the RNA sequence to be conserved or native-like. Self-consistency scores may provide an overall more holistic evaluation metric as they accounts for alternative base pairings which still lead to similar structures as well as better capture the recovery of structural motifs responsible for functionality. However, structural self-consistency scores inherit the limitations of the structure prediction methods used as part of their computation. For instance, computing the selfconsistency score between an RNA backbone and its own native sequence provides an upper bounds on the maximum score that designs can obtain under a given structure prediction method. Lastly, gRNAde's perplexity estimates the likelihood of a sequence given a backbone and can be useful for ranking designs and mutants in RNA engineering campaigns (especially for design scenarios where structure prediction tools are not performant).In real-world design scenarios, we can pair gRNAde with another machine learning model (an 'oracle') for ranking or predicting the suitability of designed sequences for the objective (for instance, binding affinity or some other notion of fitness). We hope to conduct further experimental validation of gRNAde designs in the wet lab in order to better understand these tradeoffs.Why not average single-state logits over multiple states for multi-state design? ProteinMPNN [Dauparas et al., 2022] proposes to average logits from multiple backbones for multi-state protein design. Here is a simple example to highlight issues with such an approach: Consider two states A and B, and choice of labels X, Y, and Z. For state A: X, Y, Z are assigned probabilities 75%, 20%, 5%. For state B: X, Y, Z are assigned probabilities 5%, 20%, 75%. Logically, label Y is the only one that is compatible with both states. However, averaging the probabilities would lead to label X or Z being more likely to be sampled in designs. As an alternative, gRNAde is based on multi-state GNNs which can take as input one or more backbone structures and generate sequences conditioned on the conformational ensemble directly. Table 1 : Ablation study and aggregated benchmark results for gRNAde. We report metrics averaged over 100 test sets samples and standard deviations across 3 consistent random seeds. The percentages reported in brackets for the 3D self-consistency scores are the percentage of designed samples within the 'designability' threshold values (scRMSD≤2Å, scTM≥0.45, scGDT≥0.5).Max Table 1 presents an ablation study as well as aggregated benchmark for various configurations of gRNAde. Key takeaways are highlighted below. Note that all results in the main paper are reported for models trained on the maximum length of 5000 nucleotides using autoregressive decoding and rotation-equivariant GNN layers, as this lead to the lowest perplexity values.Max. train RNA length Limiting the maximum length of RNAs used for training can be seen as ablating the use of ribosomal RNA families (which are thousands of nucleotides long and form complexes with specialised ribosomal proteins). We find that training on only short RNAs fewer than 1000s of nucleotides leads to worse sequence recovery and 3D self-consistency scores, even though it improves 2D self-consistency across both evaluation splits. This suggests that tertiary interactions learnt from ribosomal RNAs can generalise to other RNA families to some extent (large ribosomal RNAs were excluded from test sets).GNN We ablated whether the internal representations of the GVP-GNN are rotation invariant or equivariant. Equivariant GNNs are theoretically more expressive [Joshi et al., 2023] and we do find them more capable at fitting the training distribution (as shown by lower perplexity). However, we do not find significant differences in terms of other performance metrics across different GNN layers.Model 'AR' implies autoregressive decoding (described in Section 2.3, uses 4 encoder and 4 decoder layers), while 'NAR' implies non-autoregressive, one-shot decoding using an MLP (uses 8 encoder layers). Across both evaluation splits, AR models show significantly higher self-consistency scores than NAR, even though NAR lead to higher sequence recovery. AR is more expressive and can condition predictions at each decoding step on past predictions, while one-shot NAR samples from independent probability distributions for each nucleotide. Thus, AR is a better inductive bias for predicting base pairing and base stacking interactions that are drivers of RNA structure [Vicens and Kieft, 2022] . For instance, G-C and A-U pairs can often be swapped for one another, but non-autoregressive decoding does not capture such paired constraints.Max. #states We evaluate the impact of increasing the maximum number of states as input to gRNAde. Multi-state models marginally improve native sequence recovery as well as structural self-consistency scores over an equivalent single state variant, even for the single-state benchmark where the multi-state model is being used with only one state as input. This suggests that seeing multiple states during training can be useful for gRNAde's performance even for single-state design tasks.Non-learnt baselines. We report the performance of two non-learnt baselines to contextualise gRNAde's performance: for each test sample, simply predicting the groundtruth sequence back and predicting a random sequence. Structural self-consistency scores for the Groundtruth baseline provides a rough upper bounds on the maximum score that any gRNAde designs can theoretically obtain given the current state of 2D/3D structure predictors being used. gRNAde always performs better than the random baseline and often reaches 2D self-consistency scores close to the upper bound.Both 2D and 3D self-consistency scores are inherently limited by the performance of the structure prediction methods used.2D inverse folding baseline. We additionally report results for ViennaRNA's 2D-only inverse folding method to further demonstrate the utility of 3D inverse folding. ViennaRNA has improved 2D self-consistency scores over gRNAde but fails to capture tertiary interactions in its designs, as evident by poor recovery and 3D self-consistency scores similar to the random baseline. Using the backbone structure and mutational fitness landscape data from an RNA polymerase ribozyme [McRae et al., 2024] , we retrospectively analyse how well we can rank variants at multiple design budgets using random selection vs. gRNAde's perplexity for mutant sequences conditioned on the backbone structure (scaffolding subunit t1). gRNAde performs better than single site saturation mutagenesis, even when all single mutants are explored (total of 403 single mutants, 17,027 double mutants for the scaffolding subunit t1 in McRae et al. [2024] ). See Section 4.3 for results on catalytic subunit 5TU and further discussions. [Jing et al., 2020] to build latent representations of the local 3D geometric neighbourhood of each nucleotide within each state. Representations from multiple states for each nucleotide are then pooled together via permutation invariant Deep Sets [Zaheer et al., 2017] , and fed to an autoregressive decoder to predict a probabilities over the four possible bases (A, G, C, U). The probability distribution can be sampled to design a set of candidate sequences. During training, the model is trained end-to-end by minimising a cross-entropy loss between the predicted probability distribution and the true sequence identity. In-silico evaluation metrics for gRNAde designed sequences. We consider (1) sequence recovery, the percentage of native nucleotides recovered in designed samples, (2) self-consistency scores, which are measured by 'forward folding' designed sequences using a structure predictor and measuring how well 2D and 3D structure are recovered (we use EternaFold and RhoFold for 2D/3D structure prediction, respectively). We also report (3) perplexity, the model's estimate of the likelihood of a sequence given a backbone.Multi-graph tensor Figure 14 : Multi-graph tensor representation of RNA conformational ensembles, and the associated symmetry groups acting on each axis. We process a set of k RNA backbone conformations with n nodes each into a tensor representation. Each multi-state GNN layer updates the tensor while being equivariant to the underlying symmetries; pseudocode is available in Listing 1. Here, we show a tensor of 3D vector-type features with shape n × k × 3. As depicted in the equivariance diagram, the updated tensor must be equivariant to permutation S n of n nodes for axis 1, permutation S k of k conformers for axis 2, and rotation SO(3)/O(3) of the 3D features for axis 3. Listing 1: PyG-style pseudocode for a multi-state GVP-GNN layer. We update node features for each conformer independently while maintaining permutation equivariance of the updated feature tensors along both the first (no. of nodes) and second (no. of conformations) axes. (c) Average pairwise RMSD per sequence. For 1,547 sequences with multiple structures, there is significant structural diversity among conformations. On average, the pairwise C4' RMSD among the set of structures for a sequence is greater than 1Å. For each RNA sequence, we compute the pairwise C4' RMSD among all available structures. We then compute the median RMSD across all sequences within each structural cluster.While we have not run Rosetta ourselves, we note that its documentation states that "runs on RNA backbones longer than ∼ten nucleotides take many minutes or hours".This RNA was not present in gRNAde's training data, which contains structures at ≤4.0Å resolution. 


Decision: The paper introduces gRNAde,  to tackle the RNA inverse folding problem, which is a significant challenge due to RNA's potential as a therapeutic modality and its unique structural properties. The proposed method, a multi-state geometric graph neural network (GNN), constructs custom multi-graph representations of RNA and leverages an architecture consisting of a multi-state GNN encoder, a pooling layer, and an autoregressive decoder.

Strengths:
+ The authors introduce gRNAde, the first method to consider multi-state biomolecule representation.
+ The evaluation metrics are thoughtfully selected, and the experiments are conducted with fairness and rigor. The careful dataset splitting further enhances the validity of the results.
+ The paper is exceptionally well-written, with clear and thorough explanations.

Weaknesses:
- The model architecture proposed by the authors lacks innovation. The core structure of gRNAde is primarily based on a straightforward stacking of GVP-GNN layers, and its approach to handling multi-state conformations is simplistic. All components are derived from previous work, with the overall structure closely resembling that of ProteinMPNN.
- The comparison with contemporary deep learning models for RNA inverse folding is limited. The baselines used in various experiments are either outdated or overly simplistic, which may undermine the strength of the comparative analysis.
- The experimental settings could be improved. For example, the self-consistency scores and data splits could be more rigorously defined. 

In their rebuttal, the authors provided additional explanations regarding the architectural novelty of their work and offered more details about the experimental settings. They also conducted new experimental comparisons with RDesign, a recently published and contemporaneous model. The reviewers appreciate the efforts made by the authors to address the concerns raised.

However, despite these efforts, Reviewer E2ka and Reviewer h3v6 remain unconvinced of the method's architectural novelty, arguing that the contribution of this paper appears to be more of an extension of existing architectures applied to RNA-related tasks. Specifically, Reviewer E2ka believes that it would be more appropriate to retrain the model architecture by RDesign on the dataset used in this paper, rather than directly testing the model using a pre-existing checkpoint.

Given that RDesign was published at ICLR 2024 and was accessible on OpenReview during the review period, it would be reasonable to explicitly present the comparison with it in both the Introduction and Methodology sections, rather than relegating it to the appendix. Since addressing the discussion and experimental comparison with RDesign would require substantial modifications, this paper is not ready for publication in its current form. The authors are strongly encouraged to consider the reviewers' feedback and resubmit to the next conference.

Reviews: review: paper_summary: paper_summary: This paper introduced a multi-state geometric graph neural network for the RNA inverse folding problems. Experiments are conducted on carefully splited structural datasets that avoid data leakage. The results have shown convincing performance improvement over the physics-based methods such has FARFAR and Rosetta which are commonly used for RNAs.
strength_weakness: strength_weakness: - Comparison to contemporary deep learning models for RNA inverse folding is limiting
    - RDesign (https://openreview.net/forum?id=RemfXx7ebP) for example is a recent deep learning based method for 3D RNA inverse design
    - For inverse design on the secondary structure level there are many more options — a lot of them are better than RNAinverse from ViennaRNA. I would suggest checking out this survey (Design of RNAs: comparing programs for inverse RNA folding) and include a few other more competitive baselines.
- For the self-consistency scores, I personally doubt if RhoFold (also called e2efold-3d) is reliable software for RNA tertiary structure prediction, since it is from the same group that published e2efold which is a spectacularly awful RNA secondary structure predictor (I personally would avoid using any of their tools; checkout its Github issues, and also followup works on RNA secondary structure predictions that have compared with e2efold). Have the authors used more recent folding softwares such as RosettaFoldNA and AlphaFold3?
    - Would using different structure predictors significantly impact the results? This also includes EternaFold. How would gRNAde hold up against the baselines when RNAfold or LinearFold is used to compute the self-consistency scores on the secondary level?
- Data splits (train, validation and test) are carefully constructed so that the evaluation is not contaminated by data leak. But I wonder if the TM-score cut-off at 0.45 is too lenient? Is it still possible to have similar structures between training and test sets under this threshold?

Line 258 to 259. The argument would be more compelling if the inverse design operates at the quaternary level which would include information about ligand structures.
questions: questions: - Why would a perfect model has a perplexity of 1 (i.e. mapping each structure to a single sequence)? The solution for RNA inverse folding should be an one to many mapping, because for a target backbone structure there are potentially many sequences that can fold into the same structure. Likewise, a perplexity of 4 doesn’t mean the predictions are nonsense. For unfolded RNAs (e.g. a linear chain), they very likely correspond to random RNA sequences.
- Conformers would indicate that the structures are the local energy minimizers on the potential energy surface. How would you know if these multi-state conformations are really conformers? Where are multi-state conformations from?
limitations: limitations: - Comparison to contemporary models for RNA inverse folding is a bit hollow. It would be more meaningful if some deep learning based baselines can be included into the comparison.
paper_summary: paper_summary: This work designed gRNAde, a geometric deep learning pipeline for RNA sequence design conditioned on one or more 3D backbone structures. To achieve this, the authors created single-state and multi-state 3D RNA structure datasets, built a geometric graph representation, and proposed an architecture consisting of a multi-state GNN encoder, a pooling layer, and a autoregressive decoder. The single-state RNA design, multi-state RNA design, and zero-shot rank experiments were conducted and results show that gRNAde outperformed all previous methods including Rosetta.
strength_weakness: strength_weakness: 1.	The model architecture has no novelty. All components are token from previous work and the overall structure is very similar to that of ProteinMPNN. The multiple conformations are processed independently and the representations are simply averaged or summed, which may not grasp all information.

2.	The model is trained on only about 4 thousand RNA sequences. These sequences are too few to cover the entire space. RNAs with no 3D structures should be exploited, as done in alphafold3.

3.	The results about single-state RNA design were reported on only 14 samples. More samples should be used to test the model. For example, the results on test sets with 100 samples should be reported.

4.	The improvements on multi-state RNA design task are limited. The native sequence recovery is obviously lower than the results of single-state design task.
questions: questions: 1.	Lots of the RNA conformations come from protein-RNA complexes and DNA-RNA hybrids, which means these conformations do not appear alone. Since the model encoder cannot consider other molecules, the extracted representations are biased. How about adding a scalar feature to indicate if a nucleotide is on the interface with another molecule?

2.	Why is the decoder only designed for autoregressive decoding? The arbitrary decoding in ProteinMPNN is useful when part of the sequence is already known.
limitations: limitations: The representation ability of gRNAde remains to be verified. Usually, the representation is extracted when all nucleotides are known, so the architecture for inverse folding problem may not suitable for representation learning.
paper_summary: paper_summary: This paper introduce gRNAde, a geometric deep learning pipeline for RNA sequence design conditioned on one or more 3D backbone structures. gRNAde is superior to the physically based Rosetta for 3D 320 RNA inverse folding in terms of performance, inference speed, and ease of use. The method demonstrates significant superiority across various experiments.
strength_weakness: strength_weakness: 1. The model architecture proposed by the author lacks innovation. The core structure of gRNAde is directly stacked using GVP-GNN, and the handling of multi-state conformations is merely simple stacking. Additionally, the 3-beads representation method is very common in traditional RNA 3D structure modeling, which is also not an innovation by the author. Therefore, I believe the model design is lacking.
2. The baselines compared by the author in various experiments are either outdated or too simple, such as "Rosetta(2020)" and the "random baseline" in the Zero-shot experiment. This makes it difficult to demonstrate the actual performance of gRNAde. Some recent works using deep learning to model RNA 3D structures can serve as baselines, such as [1-3].
3. The author mentions that gRNAde has a significant speed improvement over Rosetta, but the author did not run the Rosetta code themselves and instead directly cited the original Rosetta paper. I believe this point is debatable because the model's running speed is also limited by GPU computational performance. The author uses an A100, whereas the GPU used by Rosetta four years ago is obviously inferior to the A100. Therefore, the author needs to rerun the Rosetta program on the A100 to provide accurate model inference times.







[1] Geometric deep learning of RNA structure, Science 2021

[2] Physics-aware Graph Neural Network for Accurate RNA 3D Structure Prediction, NIPS workshop 2022

[3] RDesign: Hierarchical Data-efficient Representation Learning for Tertiary Structure-based RNA Design, ICLR 2024
questions: questions: 1. The second experiment results indicate that multi-state biomolecule representation did not significantly improve performance. What does the author believe is the reason for this?
2. The RNA used in the first experiment are older and limited in number. Has the author considered testing with more recently published and more numerous RNA?
limitations: limitations: The authors discuss practical tradeoffs to using gRNAde in real-world RNA design scenarios 330 in Appendix B, including limitations due to the current state of 3D RNA structure prediction tools.
paper_summary: paper_summary: This paper proposes a geometric RNA design model. Specifically, it introduces multi-stage GNN to encode multiple conformations and aggregate these candidates, and further feed decoder to predict probabilities of a set of candidate sequences.
strength_weakness: strength_weakness: 1.	The technical contribution appears to be somewhat weak. The backbone used is from existing GNN models for equivariant design.
2.	Some technical details are not claimed. For example, are the comparison baselines retained on the new dataset or simply tested on their released model.
questions: questions: 1.	It is nice that they compared several SOTA baselines. However, I'm not sure if they 're retraining these baselines on the new dataset or just testing their released model.
2.	If I understand the claimed multi-state concept correctly, they are multiple conformations of a single RNA as input. It seems that multi-state design is an ensemble fusion of candidate representation. So, for a specific RNA with multiple conformations, does multi-state design always achieve better metrics than single-state design?
3.	I am still concerned about the technical insights of this paper as NIPS is highlight on the technical contributions. And I note that this paper has been accepted in ICML23 workshop.
limitations: limitations: This paper has no negative societal impact.
paper_summary: paper_summary: gRNAde is a graph neural network designed to address the RNA reverse folding problem, a significant challenge due to the potential of RNA as therapeutic modalities and their unique data properties. RNA molecules have lower thermodynamic stability compared to proteins, resulting in fewer training samples, and their increased flexibility means multiple final states are possible. gRNAde addresses these issues by proposing a custom multi-graph representation and extending message passing to operate independently on each conformer while sharing an adjacency graph. The authors thoroughly explore evaluation techniques, comparing their model performance against Rosetta by assessing the percentage of native sequence recovery on held-out sequence families. Additionally, they demonstrate slightly improved performance when utilizing multiple conformers for model training. The authors also conduct an interesting zero-shot ranking analysis on mutation data providing a refreshing evaluation against random baselines.
strength_weakness: strength_weakness: - I find the arguments regarding gRNAde perplexity being correlated with recovery to have limited support in the current presentation. In figure 2 (b) color denotes perplexity instead of one of the axis making it very challenging to assess the correlation. In addition the authors don't report a correlation value or its significance. 
- The authors only use random baselines for the retrospective variant effect analysis. Including another reverse folding model or a metric from Rosetta similar to gRNAde's perplexity could strengthen the evaluation.
questions: questions: -  Can the authors conduct a study assessing the impact of the quantity of training data on gRNAde? As structural data increases, it will be useful to understand how gRNAde's performance scales with data abundance.
- What are the other possible bottlenecks currently inhibiting further performance improvements of multi-conformational models? Do the authors believe further advancements in architectural designs are required to demonstrate larger improvements between models capturing a single vs multi-conformer information?
limitations: limitations: - The authors effectively discuss the current evaluation limitations and the difficulty of assessing the novelty and ground truth recovery of generated sequences.
- There is limited discussion on the data limitations in the current field. With only 4000 sequences, training points are very few, presenting a major challenge.
- The authors could include a brief statement on the broader impacts, such as the potential design of harmful molecules. As these models improve, the dual-use concern becomes legitimate.

Comments: title: Are all your concerns addressed?
Summary of rebuttal discussions
Got RDesign to work; results are very poor
Direct comparison to RDesign checkpoint
Using RNAFold instead of EternaFold for 2D self-consistency
Thank you for responding
Further issues with RDesign
Now trying to run RDesign
Thank you for responding
Thank you for responding
Thank you for responding
Thank you for responding
comment: Thank you for your response!

Firstly, regarding the first point, when modeling 3D structural data, equivariant neural networks generally outperform invariant neural networks, which aligns with the experimental results for the multi-state scenario in your table. However, this is not the case for the single-state scenario, and this discrepancy might require a more detailed explanation from you.

Regarding the RDesign paper, the authors established strict dataset partitions to prevent potential data leakage. I would like you to retrain their model architecture on your dataset, rather than directly testing using their checkpoint.

On the second point: Since RDesign was published in ICLR 2024, there has been a considerable time gap before the NeurIPS 2024 submission deadline. Therefore, your study is not the first to address the RNA inverse folding problem. I do agree with the insights and related experiments you've conducted utilizing pre-existing model architecture on the multi-state problem, but I believe this alone may not be sufficient to support a top-tier conference paper.

For the third point, thank you for your response. I believe my concerns have been adequately addressed.

Finally, on the fourth point, I would like to revisit RDesign. This method first categorizes RNA structures based on their similarity and then partitions the dataset based on these categories. In my opinion, evaluating RDesign’s performance reasonably requires retraining their model architecture on your dataset. Directly using the checkpoint files is not appropriate.
Dear Reviewer Hi6M,

As you stated that your concerns are addressed, would you consider revising your score to reflect that?

Best regards,

Authors
Dear Reviewers and AC,

As the discussion period will close in a few hours, we have summarized all the questions+weaknesses identified by the reviewers and how we have addressed all of them. We hope this will be useful for making your decision. Please write to us ASAP if there are any outstanding concerns.

---

**Direct comparison to RDesign public checkpoint** (Reviewers vioU, E3ka): 
- We [ran inference with RDesign](https://openreview.net/forum?id=Fm4FkfGTLu&noteId=RdkCCdrrqx) and found that it **significantly underperforms** gRNAde as well as Rosetta. This is despite RDesign being trained on a random split --> test data leakage compared to gRNAde being evaluated o.o.d.
- We uncovered serious [reproducibility issues](https://openreview.net/forum?id=Fm4FkfGTLu&noteId=q3oYMrQ5Os) and [technical flaws](https://openreview.net/forum?id=Fm4FkfGTLu&noteId=6R0Gjy0EXH) with RDesign code. 
- We have gone to great lengths to do this comparison to what we believe is concurrent work that beat us to publication. We hope the reviewers are happy with this.

**Apples-to-apples comparison to RDesign's architectural components**: Our ablation study already ablates key architectural choices that differ between RDesign and gRNAde; see [this comment](https://openreview.net/forum?id=Fm4FkfGTLu&noteId=OsoSako8HF) for details.

**Architectural novelty vs. applied work** (Reviewers h3v6, E2ka, Hi6M): 
- We think papers not introducing new architectures are well within the scope of NeurIPS.
- We think our main technical contributions are rigorous experiments and evaluation + new insights to a novel application area. We think that it is valuable to the community to understand the capabilities of our best models on new problems. 
- This is the **first work in deep learning to tackle multi-state design** and introduce multi-state representation learning for biomolecules. We also did experiments on more expressive set pooling methods (see rebuttal PDF), but found that they do not generalize any better than Deep Sets o.o.d.
- This is also the **first work to use deep learning for RNA mutant ranking given structural context**.
- We believe the gRNAde tool and codebase will unlock new capabilities for RNA designers and help machine learners build better models in the future.

---

**How the data was split** (Reviewer vioU): We used [US-align criteria](https://openreview.net/forum?id=Fm4FkfGTLu&noteId=RA6umvZYnO), which is the standard in the structure modelling/prediction community and was used in the latest CASP. There is strong precedent and this should not be counted as a weakness.

**Using different structure predictors for evaluation** (Reviewers vioU): We did [an experiment](https://openreview.net/forum?id=Fm4FkfGTLu&noteId=OmvMVAfYZC) on whether changing from EternaFold to RNAfold impacts our evaluation metrics and relative performance of models: it has no significant impact, only marginal differences within standard deviation.

**Using AlphaFold3 to evaluate or ideas from AlphaFold3 to train our models** (Reviewers vioU): AF3 came out days prior to the NeurIPS deadline and [its code or checkpoints are not available](https://openreview.net/forum?id=Fm4FkfGTLu&noteId=WPrtt5CXnl) -- we think it is not fair to ask us to use it in experiments/use ideas from it.

**Use of word 'conformer'** (Reviewer vioU): We will revise this to conformational state.

---

**Trained on small dataset** (Reviewers h3v6, QQ46): We have trained on ALL publicly available RNA 3D structures and we believe our model is useful for real-world applications. Inverse folding problem is token level, unlike structure prediction (global). We feel confident enough that gRNAde designs have been sent to experimental collaborators for testing.

**Evaluated on only 14 samples** (Reviewer h3v6): We have evaluated on 100+ RNAs across multiple different splits. This is a misunderstanding.

**Arbitrary decoding order** (Reviewer h3v6): We did an experiment in the rebuttal PDF showing that gRNAde can be trained with arbitrary decoding order with minimal performance loss.

---

**Speed claims over Rosetta** (Reviewer E2ka): 
- We agree that its not possible to directly compare CPU vs GPU speed and will revise the claims. A lot of deep learning papers have made the same claim in the past -- there is strong precedent that our tools are more broadly accessible than Rosetta. Nobody can use Rosetta RNA recipes in latest builds.
- gRNAde inference on CPU is also <10s to produce designs, compared to O(hours) for Rosetta.

---

**Did we retrain baselines** (Reviewer Hi6M): This was a misunderstanding, we gave a detailed reply.

---

**Perplexity correlation** (Reviewer QQ46): We measured how well perplexity correlates with sequence and structure recovery in the rebuttal PDF.

**Ablate quantity of training data** (Reviewer QQ46): We already did this in the appendix.

**Potential negative social impact** (Reviewer QQ46): We will add this.

----

Best regards,

gRNAde Authors
After a lot of effort, we figured out how to make the RDesign code perform inference.

We have made direct comparisons to RDesign in this comment: https://openreview.net/forum?id=Fm4FkfGTLu&noteId=RdkCCdrrqx - RDesign underperforms gRNAde and Rosetta.
Dear reviewers,

We realise that 3 of you are still concerned that we have not compared directly to RDesign. (Please see our response for why we think it was justified + it is concurrent work but beat us to the race of publication.)

We have been working hard to get the RDesign codebase to work. It has been very challenging as it is impossible to reproduce the results or re-train their model, please see [this comment](https://openreview.net/forum?id=Fm4FkfGTLu&noteId=q3oYMrQ5Os) and [this comment](https://openreview.net/forum?id=Fm4FkfGTLu&noteId=6R0Gjy0EXH), or just visit their codebase and find the complete lack of documentation. After spending 6 hours, we have managed to load their checkpoint and run inference with it. Note that they also do not use any sampling (*which makes their model practically useless*), but we implemented sampling in the same way as we implemented it for our Non-autoregressive variant of gRNAde.

**Overall, RDesign significantly underperforms gRNAde on all 14 of the high-quality RNAs of interest identified by Das et al.**

**gRNAde has improved sequence recovery over Rosetta, and Rosetta has improved sequence recovery over RDesign.**

Please also note that the RDesign checkpoint almost surely has data leakage because it was trained on a random split, so we expect performance to drop further if we are able to re-train it fairly and make apples-to-apples comparison.

---

Here are the results:

Average native sequence recovery, in the same format as Figure 2(a):
- gRNAde: 0.568
- Rosetta: 0.450
- RDesign: 0.429

Full results, in the same format as Appendix Table 2:

|PDB ID|Description                               |Rosetta Recovery|RDesign Recovery|Rdesign Perplexity|RDesign 2D self-cons.|gRNAde Recovery|gRNAde Perplexity|gRNAde 2D self-cons.|
|------|------------------------------------------|----------------|----------------|------------------|---------------------|---------------|-----------------|--------------------|
|1CSL  |RRE high affinity site                    |0.44            |0.4455          |2.4220            |0.1742               |0.5720         |1.2813           |0.8645              |
|1ET4  |Vitamin B12 binding RNA aptamer           |0.44            |0.3929          |2.5460            |0.1301               |0.6250         |1.3458           |-0.0136             |
|1F27  |Biotin-binding RNA pseudoknot             |0.37            |0.3013          |2.2294            |0.2488               |0.3438         |1.6204           |0.4524              |
|1L2X  |Viral RNA pseudoknot                      |0.48            |0.3727          |2.5163            |0.3289               |0.4721         |1.3181           |0.5693              |
|1LNT  |RNA internal loop of SRP                  |0.53            |0.5556          |1.9740            |0.4169               |0.5844         |1.4337           |0.1380              |
|1Q9A  |Sarcin/ricin domain from E.coli 23S rRNA  |0.41            |0.4417          |2.3934            |0.1669               |0.5045         |1.3412           |0.0597              |
|4FE5  |Guanine riboswitch aptamer                |0.36            |0.4112          |2.6697            |0.5121               |0.5301         |1.3825           |0.9116              |
|1X9C  |All-RNA hairpin ribozyme                  |0.5             |0.3967          |2.0729            |0.3285               |0.5000         |1.3905           |0.6630              |
|1XPE  |HIV-1 B RNA dimerization initiation site  |0.4             |0.3834          |2.5047            |0.1582               |0.7037         |1.2177           |0.7768              |
|2GCS  |Pre-cleavage state of glmS ribozyme       |0.44            |0.4518          |2.4815            |0.2849               |0.5078         |1.3054           |0.4062              |
|2GDI  |Thiamine pyrophosphate-specific riboswitch|0.48            |0.3523          |2.2182            |0.3205               |0.6500         |1.2363           |-0.0252             |
|2OEU  |Junctionless hairpin ribozyme             |0.37            |0.5000          |2.4634            |0.2101               |0.9519         |1.0914           |0.7769              |
|2R8S  |Tetrahymena ribozyme P4-P6 domain         |0.53            |0.5641          |2.4050            |0.1095               |0.5689         |1.1881           |0.7281              |
|354D  |Loop E from E. coli 5S rRNA               |0.55            |0.4458          |2.3081            |0.1723               |0.4411         |1.4939           |0.0431              |

---

**Please consider revising your assessment of the paper, as we have worked hard to address all the concerns raised.**

We've tried to do honest work and release high-quality code and resources that we think will be valued by the community. We are not chasing architectural novelty; we want to build a system that is useful for a real-world problem and rigorously benchmark it.

Best regards,

Authors of gRNAde
> for the 2D self-consistency scores, I think it really doesn't hurt for you to give RNAfold or LinearFold a try. I doubt it would generate any significant impact on the results paper, but having these additional checks would greatly strength this paper.

We just got results for this experiment: Replacing EternaFold with RNAFold as the 2D structure predictor lead to **unchanged results** and **did not modify the relative rankings of the models.**

Model parameters | EternaFold scMCC | RNAFold scMCC
---|---|---
AR, 1 state, Equivariant GNN | 0.5903 +- 0.0147 | 0.6051 +- 0.0185
NAR, 1 state, Equivariant GNN | 0.4337 +- 0.0324 | 0.4352 +- 0.0361

The results are for the single-state test split, and we reported results for 3 different models trained with identical random seeds (following the same protocols as the main paper).

We would like to add these results in the revised version of our paper, too.

We have worked hard to alleviate your concerns as much as possible. Please let us know how to proceed.
Thank you for responding.

Re. RDesign
- As you noted, 2 other reviewers were concerned with the comparison to RDesign. Short of all options, we have been trying to use RDesign but we simply cannot reproduce its data format or run inference with their model, at present (eg. no installation instructions, dataset files seem corrupted, model does not have an actual sampling mode). **We hope you would agree that the community needs good open source training code, datasets, and reproducibility in order to further develop this new research direction (RNA design with deep learning).**
- Also, please see our ablation study for apples-to-apples comparisons to RDesign's architectural ideas vs. gRNAde's (majorly, non-autoregressive/one-shot vs autoregressive), benchmarked fairly on our split and experimental settings. We think this brings insights to readers, b/c the question of autoregressive vs. one-shot models is also relevant for other biomolecule design tasks.

Re. publication date: 
- The decision notifications for ICLR were released in January privately to authors. We would imagine that the actual publication date is when a camera-ready version of a paper is **released to the public audience** beyond Authors, Reviewers, ACs, PCs (which was April).

Re. consistency of self-consistency metrics: That's a great point and we do agree that these metrics are not perfect. These metrics (as with most metrics for generative models) are supposed to be proxies for actually evaluating these designs in real world experiments. So maybe they are not perfect, but the way we actually use them for RNA design is as filters where really poor self-consistency means that the design is very likely a poor one (but good self-consistency does not automatically mean its a good design). We have actually discussed this and **caveated our results** at several instances in the paper.

Re. AlphaFold3's server: 
- We are restricted to **20 jobs per day**, and it is **not possible to submit batched jobs**.
- Given these restrictions, and the fact that AF3 server came out after the NeurIPS submission deadline, would you agree that actually using AF3 for our evaluations is impossible?
    - Here is a rough scale of how many times a structure predictor is called during our evaluation protocol:
        - For example, take 100 test RNA backbones.
        - Design 16 sequences for each backbone.
        - Fold the 16 x100 designed sequences = 1600 calls per model evaluation (per one entry in the table in Appendix D)
    - Assuming there are on average 5 authors in our team, evaluating one variant of our model would take 16 days via AF3 server. We have run experiments on ~25 models (not including further results we will be adding after rebuttals to other reviewers). 
    - Also note that there is no programmatic way to run AF3 server -- we would have been manually doing data entry.

Re. 2D structure predictors: Okay, we will now be rushing to try one of these out and will report back the results honestly. **Please stay tuned.**
I have read and appreciate the author's response.

While I appreciate the paper's writing and its clear biological motivation and analysis, I had hoped to see a more comprehensive comparison with competitive baselines. This seems to be a shared concern among other reviewers as well. 
- According to [the conference policy](https://neurips.cc/Conferences/2024/PaperInformation/NeurIPS-FAQ), "Papers appearing less than two months before the submission deadline are generally considered concurrent to NeurIPS submissions.", so I am not sure if [RDesign](https://openreview.net/forum?id=RemfXx7ebP) satisfies that criterion, since it was published in Jan 16 2024 (although it was modified later in Apr). 
- But anyways, I understand that RDesign lacks sufficient utility or means of reproduction. I appreciate any efforts the authors have put into running RDesign.

I was also hoping to see some sanity checks, in particular regarding the "consistency" of the 2D/3D self-consistency tests. Although these provide important insights into the model's performance, they are essentially based on imperfect computational models, so it is expected there will be variance in these self-consistency scores. 
- This is the reason why, in my opnion, using a set of relatively good and reliable computational predictor for these self-consistency would provide more reliable performance evaluation.
- It is true that AlphaFold 3 is not publicly released but they have an online server where you can submit jobs. If they accept batch jobs, and if time permits, then I would suggest giving it a try...
- Now for the 2D self-consistency scores, I think it really doesn't hurt for you to give RNAfold or LinearFold a try. I doubt it would generate any significant impact on the results paper, but having these additional checks would greatly strength this paper.
Another issue that makes direct comparison with RDesign impossible is that **their model does not implement sampling** during inference.

- Their model class contains a method called `sample()`: https://github.com/A4Bio/RDesign/blob/master/model/rdesign_model.py#L76 -- however, this method does not actually implement any sampling from the probability distribution outputted by the model. It just directly outputs the probability distribution.

- This would explain why RDesign's perplexity numbers are non-sensical (Appendix E.3 of their paper, as well as just the training logs for their model).

- Not being able to perform sampling means that the model will be useless in real design scenarios (especially as it is non-autoregressive/one-shot independent decoding per token), because it will keep outputting the same probability distribution and inherently cannot be used to generate diverse sequences as a result.

Is the reviewer somewhat convinced now that:
1. Our work brings something new to the community and is a valuable contribution?
2. Direct comparison to RDesign is not possible given the limitations of both their experimental methodology as well as their reproducibility?
We also wanted to add an additional note: After your comment, we are now trying to run RDesign to see if we can load the datasets and checkpoint, as well as run inference with it.

1. There are **no installation instructions** available in the repository: https://github.com/A4Bio/RDesign/

2. The dataset files released by them are **corrupted** (we tried loading them on a Macbook, a linux server, and on Github Codespaces). You can even try it quickly on your end on your browser:
    - Open a GitHub Codespaces on their repository.
    - Download the dataset files they have released: https://github.com/A4Bio/RDesign/releases/tag/data
    - Upload any of the files to the Codespaces.
    - Open a terminal or create a new jupyter notebook in the Codespaces and type the following:
```
import torch
torch.load("<path-to-data>/<data_file>.pt")  # or torch.load("<path-to-data>/<data_file>.pt", map_location='cpu')
```
...which will always lead to the following error:
```
RuntimeError                              Traceback (most recent call last)
Cell In[3], line 1
----> 1 torch.load("val_data.pt", map_location='cpu')

File ~/.local/lib/python3.10/site-packages/torch/serialization.py:1040, in load(f, map_location, pickle_module, weights_only, mmap, **pickle_load_args)
   1038     except RuntimeError as e:
   1039         raise pickle.UnpicklingError(UNSAFE_MESSAGE + str(e)) from None
-> 1040 return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)

File ~/.local/lib/python3.10/site-packages/torch/serialization.py:1264, in _legacy_load(f, map_location, pickle_module, **pickle_load_args)
   1262 magic_number = pickle_module.load(f, **pickle_load_args)
   1263 if magic_number != MAGIC_NUMBER:
-> 1264     raise RuntimeError("Invalid magic number; corrupt file?")
   1265 protocol_version = pickle_module.load(f, **pickle_load_args)
   1266 if protocol_version != PROTOCOL_VERSION:

RuntimeError: Invalid magic number; corrupt file?
```

This happens for every data file they have provided.

We will keep you posted if we manage to run the code, but these two points should already tell you why direct apples-to-apples comparisons are so difficult in this situation. Without being able to load the processed dataset files, and without any documentation in the README or within the code as to what is the expected format of input data to their model, it is not possible to re-train or do inference with this model.
Thank you for acknowledging the rebuttal and wanting to engage in discussion.

> On point 1
- We have in fact ablated the architecture of the GVP GNN used within our model. Please see Appendix D, line 587 onwards and the results in green. We compared using a rotation invariant GNN vs. rotation equivariant GNN, which we believe is the **fundamental concept when building geometric 3D GNNs** that should bring insight to readers (as opposed to what specific choice of layers one makes which differs a lot from paper to paper).
- On RDesign, we disagree -- we have seen repeatedly in structural biology that **using random data splits will give the impression that models are working well, but that their o.o.d. performance will be greatly over-exagerated** (examples: [1](https://arxiv.org/abs/2308.05777) [2](https://openreview.net/forum?id=A8pqQipwkt) [3](https://arxiv.org/abs/2206.12411) -- it is very simple to delude ourselves with random splits).
    - RDesign seems overfit on its training data based on [the logs](https://github.com/A4Bio/RDesign/blob/master/checkpoints/log.log): their training perplexity is order of magnitude lower than validation perplexity even on random splits...
    - Running RDesign on our splits is guaranteed to have data leakage.
- Also, **we did ablate the fundamental conceptual difference between RDesign and gRNAde** -- autoregressive vs. non-autoregressive decoding -- in Appendix D, line 591 onwards. We find that autoregressive decoding is better for structural self-consistency, which we care more about in real-world design scenarios. We think this finding will be super interesting to the community working on other types of biomolecules, too, as their is debate on the two approaches to decoding.
- Finally, if you really want to compare numbers, our model's recovery rates are in the 50s for the single-state split. RDesign's recovery rates are in the 40s in their Table 2. 

> On point 2
- Adapting existing architectures to new problems, doing the experiments rigorously, setting the first benchmarks for a field, and releasing open-source resources so others can build upon them is a valuable contribution. This is stated in the NeurIPS Call for Papers, too.
- It is valuable for the community to know how to apply our best tools work to new problems. Doing applied work well requires strong understanding of the application domain as well as the deep learning architectures and evaluation, but may not always involve new architecture ideas.
- It would obviously have been nice for the novelty of our paper if we could have proposed a fancy multi-state fusion method. However, we benchmarked many ideas rigorously and did not find them to bring improvements over Deep Set. We should not be penalized for this.

> On point 3
- The point of those experiments is to demonstrate that deep learning tools perform better and are **more broadly accessible** that Rosetta RNA (its latest build cannot even run RNA design...). We tried to compare as fairly as possible to Rosetta by being careful about the splits.
- There is a lot of precedent for deep learning papers claiming superior speed and performance to Rosetta in the same manner as us for other tasks, such as inverse folding, structure generation, rotamer optimization, etc. (Examples [1](https://www.science.org/doi/10.1126/science.add2187), [2](https://www.mit.edu/~vgarg/GenerativeModelsForProteinDesign.pdf), [3](https://www.pnas.org/doi/full/10.1073/pnas.2216438120) and many structure prediction papers rely on this claim)
- We do take your point though and we will be more careful about our Rosetta speed claim. We propose to revise it to add caveats everywhere in our paper, for instance:
    - Line 60: "gRNAde is significantly faster than Rosetta for inference; e.g. sampling 100+ designs in 1 second for an RNA of 60 nucleotides on an A100 GPU, compared to the reported hours for Rosetta **on CPUs, making gRNAde more broadly accessible**.
    - Line 246: "Rosetta takes order of hours to produce a single design due to performing expensive Monte Carlo optimisations **on a CPU**."

> On point 4
- RDesign and our work were developed concurrently but **they beat us to publication**. These kinds of situations are very difficult for authors and we have tried to provide an honest and detailed comparison between the two works in line 507 onwards as well as the ablation study experiments we already pointed you to.
- RDesign did not release training code -- this makes it impossible to reproduce their study.
- RDesign's evaluation makes use of random splits (we already highlighted why this is has issues), does not compute any metrics beyond recovery, and does not compare to Rosetta or physics-based tools.
- We think our study brings significant new insights to the community and can hopefully be complementary to the RDesign paper. We believe these are concurrent works and one being published first should not be the reason to stop the publication of the other.
Thank you for your detailed response. However, I still have the following concerns:

1. In fact, a comparison with the three papers I mentioned is feasible, even though the tasks they address differ from yours. You use a stacked GVP architecture, but theoretically, the GVP could be replaced with the architectures from those three papers. The fundamental difference lies in the models used to represent the RNA 3D structure. As for RDesign, the lack of training code and the specifics of the data split do not prevent the transfer of the model architecture. Therefore, I believe that simply comparing with a random baseline is not persuasive.

2. Regarding the use of DSS, it essentially applies an existing method (DSS) to your dataset, but this does not constitute an innovation in your model architecture. In my view, this paper still lacks sufficient innovation in terms of the modeling approach.

3. Concerning the use of Rosetta, while it is true that most methods within Rosetta rely on MCMC sampling and run on CPUs, this means that you cannot claim faster performance in comparison, as that would be unfair. You should compare your methods with other deep learning methods which run on GPU.

4. For the 3D RNA inverse design task, you are not the first to define or provide a dataset for this problem. RDesign also addresses this task and presents a more novel modeling approach. The main difference lies in the use of the multi-state (please correct me if I misunderstood).

If I have misunderstood any of the above points, I would greatly appreciate it if you could clarify. If you can provide more detailed answers to these questions, I would be happy to consider raising my score.
Thank you for acknowledging the rebuttal -- we are happy to hear your main concerns are now addressed.

Would you consider increasing your score to reflect that?
My main concerns are addressed. Thanks for your responses.
Thank you for acknowledging the rebuttal. We hope you will reconsider, given that we put significant effort to address your review and we think it did improve the paper overall. 

Adapting existing architectures to new problems, doing the experiments rigorously, setting the first benchmarks for a field, and releasing open-source resources so others can build upon them **is a valuable contribution**. The NeurIPS Call for Papers states: “We invite submissions presenting new and original research on topics including but not limited to the following: … **Applications**, **Machine learning for sciences**.” Doing applied work well requires strong understanding of the application domain as well as the deep learning architectures and evaluation, but may not *always* involve new architecture ideas.

We addressed as many of your weaknesses and questions as we could via the rebuttal.
- Weakness 1: We tried more expressive set pooling methods, but it often happens in structural biology applications that complex architectural ideas **do not generalize** to o.o.d. test sets. It would obviously have been nice for the novelty of our paper if we could have proposed an exciting, new multi-state fusion method. However, we benchmarked many ideas rigorously and did not find them to bring real improvements.
- Weakness 2: We basically processed and prepared ML-ready datasets from **all the structured RNAs available publicly** on the internet; we couldn't use the AlphaFold3 self-distillation idea b/c it was not available when we did the work. We hope this is not counted as a weakness in your assessment.
- Weakness 3: We actually did use a lot more test samples and reported results with confidence intervals in the appendix. We hope this is also not counted as a weakness in your assessment.
- We can use our models with arbitrary decoding order, too. We did this experiment that you asked for, too.
Thanks a lot for taking the time and effort to answer my questions, but I want to keep my neutral score. Although the performance seems ok, I still think the contribution of this paper is more like an extension of existing architecture on RNA-related tasks.This neutral score means that if other reviewers and AC lean to accept this work, I would not oppose it.
Thank you for responding to the rebuttal. We're happy to hear that it further addressed your concerns.
Thank you to the authors for additional experiments and explanations. I think the paper is very well written, and expect that the additional resources in the form of notebooks and other code provided by the authors, can spur further innovation on this important problem.

It's interesting that across all the author evaluations, there is a sharp performance increase at perplexity < 1.2. This is a great work and I recommend acceptance. I have increased the overall score