Title: RFold: RNA Secondary Structure Prediction with Decoupled Optimization
Venue: NeurIPS 2023
Accepted: no
Keywords: Bioinformatics

Molecular structure prediction
Abstract: The secondary structure of ribonucleic acid (RNA) is more stable and accessible in the cell than its tertiary structure, making it essential for functional prediction. Although deep learning has shown promising results in this field, current methods suffer from poor generalization and high complexity. In this work, we present RFold, a simple yet effective RNA secondary structure prediction in an end-to-end manner. RFold introduces a decoupled optimization process that decomposes the vanilla constraint satisfaction problem into row-wise and column-wise optimization, simplifying the solving process while guaranteeing the validity of the output. Moreover, RFold adopts attention maps as informative representations instead of designing hand-crafted features. Extensive experiments demonstrate that RFold achieves competitive performance and about eight times faster inference efficiency than the state-of-the-art method.

1

Abstract

The secondary structure of ribonucleic acid (RNA) is more stable and accessible in the cell than its tertiary structure, making it essential for functional prediction. Although deep learning has shown promising results in this field, current methods suffer from poor generalization and high complexity. In this work, we present RFold, a simple yet effective RNA secondary structure prediction in an end-to-end manner. RFold introduces a decoupled optimization process that decomposes the vanilla constraint satisfaction problem into row-wise and column-wise optimization, simplifying the solving process while guaranteeing the validity of the output. Moreover, RFold adopts attention maps as informative representations instead of designing hand-crafted features. Extensive experiments demonstrate that RFold achieves competitive performance and about eight times faster inference efficiency than the state-of-the-art method.

1 Introduction

Ribonucleic acid is essential in structural biology for its diverse functional classes [8, 45, 49, 18]. The 14 functions of RNA molecules are determined by their structure [57]. The secondary structure, which 15 contains the nucleotide base pairing information, as shown in Fig. 1, is crucial for the correct functions 16 of RNA molecules [13, 11, 68]. Although experimental assays such as X-ray crystallography [6], 17 nuclear magnetic resonance (NMR) [15], and cryogenic electron microscopy [12] can be implemented 18 to determine RNA secondary structure, they suffer from low throughput and expensive cost. 19

Computational RNA secondary structure predic20 tion methods have become increasingly popular 21 due to their high efficiency [31]. Currently, these 22 methods can be broadly classified into two cate23 gories [50, 14, 55, 60]: (i) comparative sequence 24 analysis and (ii) single sequence folding algorithm. 25 Comparative sequence analysis determines the sec26 ondary structure conserved among homologous se27 quences but the limited known RNA families hin28 der its development [35, 36, 28, 22, 21, 16, 43]. 29 Researchers thus resort to single RNA sequence 30 folding algorithms that do not need multiple se31 quence alignment information. A classical cate32 gory of computational RNA folding algorithms is 33 to use dynamic programming (DP) that assumes the 34 secondary structure is a result of energy minimiza35 tion [3, 44, 39, 73, 42, 10]. However, energy-based 36

A A C C U G G U C A G G C C C G G A A G G G A G C A G C C A

A A C C U G G U C A G G C C C G G A A G G G A G C A G C C A

graph representation

matrix representation (contact map)

C

20

G

G

1

30

approaches usually require a nested structure, which ignores some biologically essential structures 37 such as pseudoknots, i.e., non-nested base pairs [5, 54, 70], as shown in Fig. 2. Since predicting 38 secondary structures with pseudoknots under the energy minimization framework has shown to be 39 hard and NP-complete [66, 14], deep learning techniques are introduced as an alternative approach. 40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

A

A

C

U

G

U

A

A

C

U

G

U

nested structure

non-nested structure

A

U

U

A

A

A

b

1

6

Attempts to overcome the limitations of energybased methods have motivated deep learning methods in the absence of DP. SPOT-RNA [55] is a seminal work that ensembles ResNet [24] and LSTM [25] to identify molecular features. SPOTRNA does not constrain the output space into valid RNA secondary structures, which degrades its generalization ability [32]. E2Efold [5] employs an unrolled algorithm for constrained programming that post-processes the network output to satisfy the constraints. E2Efold introduces a convex relaxation to make the optimization tractable, leading to possible constraint violations and poor generalization ability [53, 14]. Developing an appropriate optimization that forces the output to be valid be- comes an important issue. Apart from the optimization problem, state-of-the-art approaches require hand-crafted features and introduce the pre-processing step for such features, which is inefficient and needs expert knowledge. CDPfold [72] develops a matrix representation based on sequence pairing that reflects the implicit matching between bases. UFold [14] follows the exact post-process mechanism as E2Efold and uses hand-crafted features from CDPfold with U-Net [51] model architecture to improve the performance.

Although promising, current deep learning methods on RNA secondary structure prediction have been distressed by: (1) the optimization process that is complicated and poor in generalization and (2) the data pre-processing that requires expensive complexity and expert knowledge . In this paper, we present RFold, a simple yet effective RNA secondary structure prediction method in an end-to-end manner. Specifically, we introduce a decoupled optimization process that decomposes the vanilla constraint satisfaction problem into row-wise and column-wise optimization, simplifying the solving process while guaranteeing the validity of the output. Besides, we adopt attention maps as informative representations to automatically learn the pair-wise interactions of the nucleotide bases instead of using hand-crafted features to perform data pre-processing. We conduct extensive experiments to compare RFold with state-of-the-art methods on several benchmark datasets and show the superior performance of our proposed method. Moreover, RFold has faster inference efficiency than those methods due to its simplicity.

2 Related work

2.1 Comparative Sequence Analysis

Comparative sequence analysis determines base pairs conserved among homologous sequences [55, 17, 28, 36, 35, 19, 20]. ILM [52] combines thermodynamic and mutual information content scores. Sankoff [27] merges the sequence alignment and maximal-pairing folding methods [46]. Dynalign [41] and Carnac [62, 47] are the subsequent variants of Sankoff algorithms. RNA forester [26] introduces a tree alignment model for global and local alignments. However, the limited number of known RNA families [21, 16, 43] impedes the development of comparative methods.

2.2 Energy-based Folding Algorithms

When the secondary structure consists only of nested base pairing, dynamic programming can 83 efficiently predict the structure by minimizing energy. Early works in this category include Vienna 84 RNAfold [39], Mfold [73], RNAstructure [42], and CONTRAfold [10]. Faster implementations that 85 86 and LinearFold [30]. However, these methods cannot accurately predict secondary structures with 87 88

speed up dynamic programming have been proposed, such as Vienna RNAplfold [4], LocalFold [37], pseudoknots, as predicting the lowest free energy structures with pseudoknots is NP-complete [40], making it difficult to improve performance. 89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

2.3 Learning-based Folding Algorithms

SPOT-RNA [55] is a seminal work that employs deep learning for RNA secondary structure prediction. SPOT-RNA2 [56] improves its predecessor by using evolution-derived sequence profiles and mutational coupling. Inspired by Raptor-X [65] and SPOT-Contact [23], SPOT-RNA uses ResNet and bidirectional LSTM with a sigmoid function to output the secondary structures. MXfold [1] is also an early work that combines support vector machines and thermodynamic models. CDPfold [72], DMFold [64], and MXFold2 [53] integrate deep learning techniques with energy-based methods. E2Efold [5] takes a remarkable step in constraining the output to be valid by learning unrolled algorithms. However, its relaxation for making the optimization tractable may violate the structural constraints. UFold [14] further introduces U-Net model architecture to improve performance.

3 Preliminaries and Backgrounds

3.1 Preliminaries

The primary structure of RNA is the ordered linear sequence of bases, which is typically represented as a string of letters. Formally, an RNA sequence can be represented as X GLYPH<16> p x 1 , ..., x L q , where x i P t A , U , C , G u denotes one of the four bases, i.e., Adenine (A), Uracil (U), Cytosine (C), and Guanine (G). The secondary structure of RNA is a contact map represented as a matrix M P t 0 , 1 u L GLYPH<2> L , where M ij GLYPH<16> 1 if the i -th and j -th bases are paired. In the RNA secondary structure prediction problem, we aim to obtain a model with learnable parameters Θ that learns a mapping F Θ : X GLYPH<222>GLYPH<209> M by exploring the interactions between bases. Here, we decompose the mapping F Θ into two sub-mappings as:

where H θ h : X GLYPH<222>GLYPH<209> H , G θ g : H GLYPH<222>GLYPH<209> M are mappings parameterized by θ h and θ g , respectively. H P R L GLYPH<2> L is regarded as the unconstrained output of neural networks.

3.2 Backgrounds

It is worth noting that there are hard constraints on the formation of RNA secondary structure, meaning that certain types of pairing are not available [59]. Such constraints [5] can be formally described as follows:

(a) Only three types of nucleotide combinations can form base pairs: B : GLYPH<16> t AU , UA u Y t GC , CG u Y t GU , UG u . For any base pair x i x j where x i x j R B , M ij GLYPH<16> 0 .

(b) No sharp loops within three bases. For any adjacent bases, there can be no pairing between them, i.e., @| i GLYPH<1> j | / 3 , M ij GLYPH<16> 0 .

(c) There can be at most one pair for each base, i.e., @ i, GLYPH<176> L j GLYPH<16> 1 M ij / 1 .

The available space of valid secondary structures is all symmetric matrices P t 0 , 1 u L GLYPH<2> L that satisfy the above three constraints. The first two constraints can be satisfied easily. We define a constraint matrix GLYPH<132> M as: GLYPH<132> M ij : GLYPH<16> 1 if x i x j P B and | i GLYPH<1> j | ¥ 4 , and GLYPH<132> M ij : GLYPH<16> 0 otherwise. By element-wise multiplication of the network output and the constraint matrix GLYPH<132> M , invalid pairs are masked.

The critical issue in obtaining a valid RNA secondary structure is the third constraint, i.e., processing the network output to create a symmetric binary matrix that only allows a single "1" to exist in each row and column . There are different strategies for dealing with this issue.

SPOT-RNA is a typical kind of method that imposes minor constraints. It takes the original output of neural networks H and directly applies the Sigmoid function, assigning a value of 1 to those greater than 0.5 and 0 to those less than 0.5. This process can be represented as:

G p H q GLYPH<16> ✶ r Sigmoid p H q¡ 0 . 5 s d H . (2) Here, the offset term s has been set to 0.5. No explicit constraints are imposed, and no additional 130 parameters θ g are required. 131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

E2Efold formulates the problem with constrained optimization and introduces an intermediate variable x M P R L GLYPH<2> L . It aims to maximize the predefined score function:

where T p x M q GLYPH<16> 1 2 p x M d x M GLYPH<0> p x M d x M q T q d GLYPH<132> M ensures the output is a symmetric matrix that satisfies the constraints (a-b), s is an offset term that is set as log p 9 . 0 q here, xGLYPH<4> , GLYPH<4>y denotes matrix inner product and ρ } x M } 1 is a glyph[lscript] 1 penalty term to make the matrix to be sparse.

Formally, this process can be represented as:

Though three constraints are explicitly imposed in E2Efold, this method requires iterative steps to approximate the valid solutions and cannot guarantee that the results are entirely valid. Moreover, it needs a set of parameters θ g in this processing, making tuning the model complex.

4 RFold

4.1 Decoupled Optimization

We propose the following formulation for the constrained optimization problem in RNA secondary structure problem:

where tr p M T x H q GLYPH<16> GLYPH<176> L i GLYPH<16> 1 GLYPH<176> L j GLYPH<16> 1 M ij x H ij represents the trace operation. The matrix x H is symmetrized based on the original network output H while satisfying the constraints (a-b) in Sec. 3.2 by multiplying the constraint matrix GLYPH<132> M , i.e., x H GLYPH<16> p H d H T q d GLYPH<132> M .

We then propose to decouple the optimization process into row-wise and column-wise optimizations, and define the corresponding selection schemes as S r and S c respectively:

where S i r P t 0 , 1 u L signifies the selection scheme on the i th row, and S j c P t 0 , 1 u L represents the selection scheme on the j th column. The score function is defined as:

where S r , S c constitute the decomposition of M . The goal of the score function is to maximize the dot product of M and x H in order to select the maximum value in x H . Our proposed decoupled optimization reformulates the original constrained optimization problem in Equation 6 as follows:

If the corresponding x H ij have the highest score in its row t x H ik u L k GLYPH<16> 1 and its column t x H kj u L k GLYPH<16> 1 , then 157 M ij GLYPH<16> 1 . By exploring the optimal S r and S c , the chosen base pairs can be obtained by the optimal 158 scheme S GLYPH<16> S r b S c . 159

4.2 Row-Col Argmax 160

161

162

163

164

165

166

With the proposed decoupled optimization, the optimal matrix can be easily obtained using the variant Argmax function:

where Row -Argmax and Col -Argmax are row-wise and column-wise Argmax functions respectively:

Theorem 1. Given a symmetric matrix x H P R L GLYPH<2> L , the matrix Row -Col -Argmax p x H q is also a symmetric matrix.

As shown in Fig. 3, taking a random symmetric 6 GLYPH<2> 6 matrix as an example, we show the output 168 matrics of Row -Argmax, Col -Argmax, and Row -Col -Argmax functions, respectively. The Row-Col 169

Argmax selects the value that has the maximum value on both its row and column while keeping the 170 output matrix symmetric. 171

symmetric matrix

Row-Argmax

Col-Argmax

Row-Col-Argmax

From Theorem 1, we can observe that Row -Col -Argmax p x H q is a symmetric matrix that satisfies the constraint (c). Since x H already satisfies constraints (a-b), the optimized output is:

where S r , S c GLYPH<16> arg min S r ,S c GLYPH<1> tr p S r , S c q .

4.3 Row-Col Softmax

Though the Row-Col Argmax function can obtain the optimal matrix G p H q , it is not differentiable and thus cannot be directly used in the training process. In the training phase, we need to use a differentiable function to approximate the optimal results. Therefore, we propose using a Row-Col Softmax function to approximate the Row-Col Argmax function for training. To achieve this, we perform row-wise Softmax and column-wise Softmax on the symmetric matrix x H separately, as shown below:

The Row-Col Softmax function is then defined as follows: 182

Note that we use the average of Row -Softmax p x H q and Col -Softmax p x H q instead of the element 183 product as shown in Equ. 10 for the convenience of optimization. 184

172

173

174

175

176

177

178

179

180

181

Theorem 2. Given a symmetric matrix x H P R L GLYPH<2> L , the matrix Row -Col -Softmax p x H q is also a 185 symmetric matrix. 186

187

188

189

190

191

As shown in Fig. 4, taking a random symmetric 6 GLYPH<2> 6 matrix as an example, we show the output matrics of Row -Softmax, Col -Softmax, and Row -Col -Softmax functions, respectively. It can be seen that the output matrix of Row -Col -Softmax is still symmetric. Leveraging the differentiable property of Row -Col -Softmax, the model can be easily optimized.

symmetric matrix

Row-Softmax

Col-Softmax

Row-Col-Softmax

In the training phase, we apply the differentiable Row-Col Softmax activation and optimize the mean 192 square error (MSE) loss function between G p H q and M : 193

4.4 Seq2map Attention

To simplify the pre-processing step that constructs hand-crafted features based on RNA sequences, we propose a Seq2map attention module that can automatically produce informative representations. We start with a sequence in the one-hot form X P R L GLYPH<2> 4 and obtain the sum of the token embedding and positional embedding as the input for the Seq2map attention. For convenience, we denote the input as Z P R L GLYPH<2> D , where D is the hidden layer size of the token and positional embeddings.

Motivated by the recent progress in attention mechanisms [63, 9, 34, 7, 33, 48, 29, 69, 38], we aim to develop a highly effective sequence-tomap transformation based on pair-wise attention. We obtain the query Q P R L GLYPH<2> D and key K P

sequence one-hot

𝑿: 	𝐿×4

Token embedding

𝐿×𝐷

Seq2map Attention

𝐿×𝐿×1

+

Positional

embedding

𝐿×𝐷

1

32

32

64

128

64

256

512

256

128

𝐿×𝐿

(

𝐿/2)×(𝐿/2)

(

𝐿/4)×(𝐿/4)

(

𝐿/8)×(𝐿/8)

(

𝐿/16)×(𝐿/16)

1

𝐿×𝐿

𝑯

+ 𝒁

𝒁

R L GLYPH<2> D by applying per-dim scalars and offsets to Z :

where γ Q , γ K , β Q , β K P R L GLYPH<2> D are learnable parameters.

Then, the pair-wise attention map is obtained by:

where ReLU 2 is an activation function that can be recognized as a simplified Softmax function in vanilla Transformers [58]. The output of Seq2map is the gated representation of s Z :

where σ pGLYPH<4>q is the Sigmoid function that performs as a gate operation.

As shown in Fig. 5, we identify the problem of predicting H P R L GLYPH<2> L from the given sequence attention map p Z P R L GLYPH<2> L as an image-to-image segmentation problem and apply the U-Net model architecture to extract pair-wise information.

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

5 Experiments

We conduct experiments to compare our proposed RFold with state-of-the-art and commonly used methods in the field of RNA secondary structure prediction. Multiple experimental settings are taken into account, including standard RNA secondary structure prediction, generalization evaluation, large-scale benchmark evaluation, and inference time comparison. Ablation studies are also presented.

Datasets We use three benchmark datasets: (i) RNAStralign [61], one of the most comprehensive collections of RNA structures, is composed of 37,149 structures from 8 RNA types; (ii) ArchiveII [57], a widely used benchmark dataset in classical RNA folding methods, containing 3,975 RNA structures from 10 RNA types; (iii) bpRNA [55], is a large scale benchmark dataset, containing 102,318 structures from 2,588 RNA types.

Baselines We compare our proposed RFold with baselines including energy-based folding methods such as Mfold [73], RNAsoft [2], RNAfold [39], RNAstructure [42], CONTRAfold [10], Contextfold [71], and LinearFold [30]; learning-based folding methods such as SPOT-RNA [55], Externafold [67], E2Efold [5], MXfold2 [53], and UFold [14].

Metrics We evaluate the performance by precision, recall, and F1 score, which are defined as:

where TP , FP , and FN denote true positive, false positive and false negative, respectively.

Implementation details Following [14], we train the model for 100 epochs with the Adam optimizer. The learning rate is 0.001, and the batch size is 1 for sequences with different lengths.

5.1 Standard RNA Secondary Structure Prediction

Following [5], we split the RNAStralign dataset into training, validation, and testing sets by stratified sampling to ensure every set has all RNA types. We report the experimental results in Table 1. It can be seen that energy-based methods achieve relatively weak F1 scores ranging from 0.420 to 0.633. Learning-based folding algorithms like E2Efold and UFold can significantly improve performance by large margins, while RFold obtain even better performance among all the metrics. Moreover, RFold obtains about 8% higher precision than the state-of-the-art method. This phenomenon suggests that our proposed decoupled optimization is strict to satisfy all the hard constraints for predicting valid structures.

5.2 Generalization Evaluation

To verify the generalization ability of our proposed RFold, we directly evaluate the performance on another benchmark dataset ArchiveII using the pre-trained model on the RNAStralign training dataset. Following [5], we exclude RNA sequences in ArchiveII that have overlapping RNA types with the RNAStralign dataset for a fair comparison. The results are reported in Table 2.

It can be seen that traditional methods achieve F1 scores in the range of 0.545 to 0.842. Among the state-of-the-art methods, RFold attains the highest F1 score. It is noteworthy that RFold has a relatively lower recall metric and significantly higher precision metric. This phenomenon may be due to the strict constraints imposed by RFold. Although none of the current learning-based methods can meet all the constraints presented in Sec. 3.2, the predictions made by RFold are guaranteed to be valid. Therefore, RFold may cover fewer pairwise interactions, resulting in a lower recall metric. Nonetheless, the highest F1 score indicates the excellent generalization ability of RFold.

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

5.3 Large-scale Benchmark Evaluation

The large-scale benchmark dataset bpRNA has a fixed training set (TR0), evaluation set (VL0), and testing set (TS0). Following [55, 53, 14], we train the model in bpRNA-TR0 and evaluate the performance on bpRNA-TS0 by using the best model learned from bpRNA-VL0. We summarize the evaluation results in Table 3. It can be seen that RFold significantly improves the previous state-of-the-art method SPOT-RNA by 4.0% in the F1 score.

Following [14], we conduct an experiment on long-range interactions. The bpRNA-TS0 dataset contains more versatile RNA sequences of different lengths and various types, which can be a reliable evaluation. Given a sequence of length L , the long-range base pairing is defined as the paired and unpaired bases with intervals longer than L { 2 . As shown in Table 4, RFold performs unexpectedly well on these long-range base pairing predictions. We can also find that UFold performs better in long-range cases than the complete cases. The possible reason may come from the U-Net model architecture that learns multi-scale features. RFold significantly improves UFold in all the metrics by large margins, demonstrating its strong predictive ability.

5.4 Inference Time Comparison

We compared the running time of various methods 289 for predicting RNA secondary structures using the 290 RNAStralign testing set with the same experimen291 tal setting as in [14]. The results are presented in 292 Table 5, which shows the average inference time 293 per sequence. The fastest energy-based method is 294 LinearFold, which takes an average of about 0.43s 295 for each sequence. The previous learning-based 296 baseline, UFold, takes about 0.16s. RFold has the 297 highest inference speed, costing only about 0.02s 298 per sequence. In particular, RFold is about eight 299 times faster than UFold and sixteen times faster than 300 MXfold2. The fast inference time of RFold is due 301 to its simple sequence-to-map transformation. 302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

5.5 Ablation Study

Decoupled Optimization To validate the effectiveness of our proposed decoupled optimization, we conduct an experiment that replaces them with other strategies. The results are summarized in Table 6, where RFold-E and RFold-S denote our model with the strategies of E2Efold and SPOT-RNA, respectively. We ignore the recent UFold because it follows exactly the same strategy as E2Efold. We also report the validity which is a sample-level metric evaluating whether all the constraints are satisfied. Though RFold-E has comparable performance in the first three metrics with ours, many of its predicted structures are invalid. The strategy of SPOT-RNA has incorporated no constraint that results in its low validity. Moreover, its strategy seems to not fit our model well, which may be caused by the simplicity of our RFold model.

Seq2map Attention We also conduct an experiment to evaluate the proposed Seq2map attention. We replace the Seq2map attention with the hand-crafted features from UFold and the outer concatenation from SPOT-RNA, which are denoted as RFold-U and RFold-SS, respectively. In addition to performance metrics, we also report the average inference time for each RNA sequence to evaluate the model complexity. We summarize the result in Table 7. It can be seen that RFold-U takes much more inference time than our RFold and RFold-SS due to the heavy computational cost when loading and learning from hand-crafted features. Moreover, it is surprising to find that RFold-SS has a little better performance than RFold-U, with the least inference time for its simple outer concatenation operation. However, neither RFold-U nor RFold-SS can provide informative representations.

5.6 Visualization

We visualize two examples predicted by RFold and UFold in Fig. 6. The corresponding F1 scores are denoted at the bottom of each plot. The first secondary structures is a simple example of a nested structure. It can be seen that UFold may fail in such a case. The second secondary structures is much more difficult that contains over 300 bases of the non-nested structure. While UFold fails in such a complex case, RFold can predict the structure accurately. Due to the limited space, we provide more visualization comparisons in Appendix D.

RFold

True

UFold

A

A

C

C

A

U

U

A

A

G

G

A

A

U

A

G

A

C

C

A

A

G

C

U

C

U

A

G

G

U

G

G

U

U

G

A

G

A

A

A

C

C

C

C

U

U

U

G

U

A

U

U

A

G

U

C

C

U

G

G

A

A

A

C

A

G

G

G

C

G

A

C

A

U

U

G

U

C

A

A

A

U

U

G

U

U

C

G

G

G

G

A

C

C

A

C

C

C

G

C

U

A

A

A

U

U

A

C

A

U

G

C

U

A

C

C

G

C

A

G

C

A

G

U

G

C

U

G

A

A

A

G

G

C

C

U

G

U

G

A

G

C

A

C

U

A

G

A

G

G

U

A

A

C

G

C

C

U

C

U

A

G

G

G

A

U

G

G

U

A

A

U

A

A

C

G

C

G

U

G

U

A

U

A

G

G

G

U

A

U

A

U

C

C

G

C

A

G

C

G

A

A

G

U

U

C

U

A

A

G

G

C

C

U

U

C

U

G

C

U

A

C

G

A

A

U

C

G

C

G

U

U

C

A

C

A

G

A

C

U

A

G

A

C

G

G

C

A

A

U

G

G

G

C

U

C

C

U

U

G

C

G

G

G

G

C

U

U

A

A

G

A

U

A

U

A

G

U

C

G

A

A

C

C

C

C

U

C

A

G

A

G

A

U

G

A

G

G

A

U

G

G

A

A

U

C

A

A

U

G

1

10

20

30

40

50

60

70

80

90

10

1 0

120

130

140

150

160

170

180

190

20

210

2 0

230

240

250

260

270

280

290

30

310

A

A

C

C

A

U

U

A

A

G

G

A

A

U

A

G

A

C

C

A

A

G

C

U

C

U

A

G

G

U

G

G

U

U

G

A

G

A

A

A

C

C

C

C

U

U

U

G

U

A

U

U

A

G

U

C

C

U

G

G

A

A

A

C

A

G

G

G

C

G

A

C

A

U

U

G

U

C

A

A

A

U

U

G

U

U

C

G

G

G

G

A

C

C

A

C

C

C

G

C

U

A

A

A

U

U

A

C

A

U

G

C

U

A

C

C

G

C

A

G

C

A

G

U

G

C

U

G

A

A

A

G

G

C

C

U

G

U

G

A

G

C

A

C

U

A

G

A

G

G

U

A

A

C

G

C

C

U

C

U

A

G

G

G

A

U

G

G

U

A

A

U

A

A

C

G

C

G

U

G

U

A

U

A

G

G

G

U

A

U

A

U

C

C

G

C

A

G

C

G

A

A

G

U

U

C

U

A

A

G

G

C

C

U

U

C

U

G

C

U

A

C

G

A

A

U

C

G

C

G

U

U

C

A

C

A

G

A

C

U

A

G

A

C

G

G

C

A

A

U

G

G

G

C

U

C

C

U

U

G

C

G

G

G

G

C

U

U

A

A

G

A

U

A

U

A

G

U

C

G

A

A

C

C

C

C

U

C

A

G

A

G

A

U

G

A

G

G

A

U

G

G

A

A

U

C

A

A

U

G

1

10

20

30

40

50

60

70

80

90

10

1

0

120

130

140

150

160

170

180

190

20

210

2

0

230

240

250

260

270

280

290

30

310

A

A

C

C

A

U

U

A

A

G

G

A

A

U

A

G

A

C

C

A

A

G

C

U

C

U

A

G

G

U

G

G

U

U

G

A

G

A

A

A

C

C

C

C

U

U

U

G

U

A

U

U

A

G

U

C

C

U

G

G

A

A

A

C

A

G

G

G

C

G

A

C

A

U

U

G

U

C

A

A

A

U

U

G

U

U

C

G

G

G

G

A

C

C

A

C

C

C

G

C

U

A

A

A

U

U

A

C

A

U

G

C

U

A

C

C

G

C

A

G

C

A

G

U

G

C

U

G

A

A

A

G

G

C

C

U

G

U

G

A

G

C

A

C

U

A

G

A

G

G

U

A

A

C

G

C

C

U

C

U

A

G

G

G

A

U

G

G

U

A

A

U

A

A

C

G

C

G

U

G

U

A

U

A

G

G

G

U

A

U

A

U

C

C

G

C

A

G

C

G

A

A

G

U

U

C

U

A

A

G

G

C

C

U

U

C

U

G

C

U

A

C

G

A

A

U

C

G

C

G

U

U

C

A

C

A

G

A

C

U

A

G

A

C

G

G

C

A

A

U

G

G

G

C

U

C

C

U

U

G

C

G

G

G

G

C

U

U

A

A

G

A

U

A

U

1

10

20

30

40

50

60

70

80

90

10

1 0

120

130

140

150

160

170

180

190

20

210

2

0

230

240

250

260

270

RFold

True

UFold

1.000

0.995

0.558

0.823

6 Conclusion

In this study, we present RFold, a simple yet effective learning-based model for RNA secondary structure prediction. We propose decoupled optimization to replace the complicated post-processing strategies while incorporating constraints for the output. Seq2map attention is proposed for sequenceto-map transformation, which can automatically learn informative representations from a single sequence without extensive pre-processing operations. Comprehensive experiments demonstrate that RFold achieves competitive performance with faster inference speed. We hope RFold can provide a new perspective for efficient RNA secondary structure prediction.

337

References

M. Akiyama, K. Sato, and Y. Sakakibara. A max-margin training of rna secondary structure pre338 diction integrated with the thermodynamic model. Journal of bioinformatics and computational 339 biology , 16(06):1840025, 2018. 340

M. Andronescu, R. Aguirre-Hernandez, A. Condon, and H. H. Hoos. Rnasoft: a suite of rna 341 secondary structure prediction and design software tools. Nucleic acids research , 31(13):3416342 3422, 2003. 343

S. Bellaousov, J. S. Reuter, M. G. Seetin, and D. H. Mathews. Rnastructure: web servers for 344 rna secondary structure prediction and analysis. Nucleic acids research , 41(W1):W471-W474, 345 2013. 346

S. H. Bernhart, I. L. Hofacker, and P. F. Stadler. Local rna base pairing probabilities in large 347 sequences. Bioinformatics , 22(5):614-615, 2006. 348

X. Chen, Y. Li, R. Umarov, X. Gao, and L. Song. Rna secondary structure prediction by learning 349 unrolled algorithms. In International Conference on Learning Representations , 2019. 350

H.-K. Cheong, E. Hwang, C. Lee, B.-S. Choi, and C. Cheong. Rapid preparation of rna samples 351 for nmr spectroscopy and x-ray crystallography. Nucleic acids research , 32(10):e84-e84, 2004. 352

K. M. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Q. 353 Davis, A. Mohiuddin, L. Kaiser, et al. Rethinking attention with performers. In International 354 Conference on Learning Representations , 2020. 355

F. Crick. Central dogma of molecular biology. Nature , 227(5258):561-563, 1970. 356

Y. N. Dauphin, A. Fan, M. Auli, and D. Grangier. Language modeling with gated convolutional 357 networks. In International conference on machine learning , pages 933-941. PMLR, 2017. 358

C. B. Do, D. A. Woods, and S. Batzoglou. Contrafold: Rna secondary structure prediction 359 without physics-based models. Bioinformatics , 22(14):e90-e98, 2006. 360

J. Fallmann, S. Will, J. Engelhardt, B. Grüning, R. Backofen, and P. F. Stadler. Recent advances 361 in rna folding. Journal of biotechnology , 261:97-104, 2017. 362

S. M. Fica and K. Nagai. Cryo-electron microscopy snapshots of the spliceosome: structural 363 insights into a dynamic ribonucleoprotein machine. Nature structural & molecular biology , 364 24(10):791-799, 2017. 365

G. E. Fox and C. R. Woese. 5s rna secondary structure. Nature , 256(5517):505-507, 1975. 366

L. Fu, Y. Cao, J. Wu, Q. Peng, Q. Nie, and X. Xie. Ufold: fast and accurate rna secondary 367 structure prediction with deep learning. Nucleic acids research , 50(3):e14-e14, 2022. 368

B. Fürtig, C. Richter, J. Wöhnert, and H. Schwalbe. Nmr spectroscopy of rna. ChemBioChem , 369 4(10):936-962, 2003. 370

371

[16]

P. P. Gardner, J. Daub, J. G. Tate, E. P. Nawrocki, D. L. Kolbe, S. Lindgreen, A. C. Wilkinson,

D. Finn, S. Griffiths-Jones, S. R. Eddy, et al. Rfam: updates to the rna families database. 372

Nucleic acids research , 37(suppl_1):D136-D140, 2009. 373

P. P. Gardner and R. Giegerich. A comprehensive comparison of comparative rna structure 374 prediction approaches. BMC bioinformatics , 5(1):1-18, 2004. 375

S. Geisler and J. Coller. Rna in unexpected places: long non-coding rna functions in diverse 376 cellular contexts. Nature reviews Molecular cell biology , 14(11):699-712, 2013. 377

J. Gorodkin, L. J. Heyer, and G. D. Stormo. Finding the most significant common sequence and 378 structure motifs in a set of rna sequences. Nucleic acids research , 25(18):3724-3732, 1997. 379

J. Gorodkin, S. L. Stricklin, and G. D. Stormo. Discovering common stem-loop motifs in 380 unaligned rna sequences. Nucleic Acids Research , 29(10):2135-2144, 2001. 381

S. Griffiths-Jones, A. Bateman, M. Marshall, A. Khanna, and S. R. Eddy. Rfam: an rna family 382 database. Nucleic acids research , 31(1):439-441, 2003. 383

R. R. Gutell, J. C. Lee, and J. J. Cannone. The accuracy of ribosomal rna comparative structure 384 models. Current opinion in structural biology , 12(3):301-310, 2002. 385

386

387

388

J. Hanson, K. Paliwal, T. Litfin, Y. Yang, and Y. Zhou. Accurate prediction of protein contact maps by coupling residual two-dimensional bidirectional long short-term memory with convolutional neural networks. Bioinformatics , 34(23):4039-4045, 2018.

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770778, 2016.

S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation , 9(8):17351780, 1997.

M. Hochsmann, T. Toller, R. Giegerich, and S. Kurtz. Local similarity in rna secondary structures. In Computational Systems Bioinformatics. CSB2003. Proceedings of the 2003 IEEE Bioinformatics Conference. CSB2003 , pages 159-168. IEEE, 2003.

I. L. Hofacker, S. H. Bernhart, and P. F. Stadler. Alignment of rna base pairing probability matrices. Bioinformatics , 20(14):2222-2227, 2004.

I. L. Hofacker, M. Fekete, and P. F. Stadler. Secondary structure prediction for aligned rna sequences. Journal of molecular biology , 319(5):1059-1066, 2002.

W. Hua, Z. Dai, H. Liu, and Q. Le. Transformer quality in linear time. In International Conference on Machine Learning , pages 9099-9117. PMLR, 2022.

L. Huang, H. Zhang, D. Deng, K. Zhao, K. Liu, D. A. Hendrix, and D. H. Mathews. Linearfold: linear-time approximate rna folding by 5'-to-3'dynamic programming and beam search. Bioinformatics , 35(14):i295-i304, 2019.

E. Iorns, C. J. Lord, N. Turner, and A. Ashworth. Utilizing rna interference to enhance cancer drug discovery. Nature reviews Drug discovery , 6(7):556-568, 2007.

A. J. Jung, L. J. Lee, A. J. Gao, and B. J. Frey. Rtfold: Rna secondary structure prediction using deep learning with domain inductive bias.

A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning , pages 5156-5165. PMLR, 2020.

N. Kitaev, L. Kaiser, and A. Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations , 2019.

B. Knudsen and J. Hein. Rna secondary structure prediction using stochastic context-free grammars and evolutionary history. Bioinformatics (Oxford, England) , 15(6):446-454, 1999.

B. Knudsen and J. Hein. Pfold: Rna secondary structure prediction using stochastic context-free grammars. Nucleic acids research , 31(13):3423-3428, 2003.

S. J. Lange, D. Maticzka, M. Möhl, J. N. Gagnon, C. M. Brown, and R. Backofen. Global or local? predicting secondary structure and accessibility in mrnas. Nucleic acids research , 40(12):5215-5226, 2012.

S. Li, Z. Wang, Z. Liu, C. Tan, H. Lin, D. Wu, Z. Chen, J. Zheng, and S. Z. Li. Efficient multi-order gated aggregation network. arXiv preprint arXiv:2211.03295 , 2022.

R. Lorenz, S. H. Bernhart, C. Höner zu Siederdissen, H. Tafer, C. Flamm, P. F. Stadler, and I. L. Hofacker. Viennarna package 2.0. Algorithms for molecular biology , 6(1):1-14, 2011.

R. B. Lyngsø and C. N. Pedersen. Rna pseudoknot prediction in energy-based models. Journal of computational biology , 7(3-4):409-427, 2000.

D. H. Mathews and D. H. Turner. Dynalign: an algorithm for finding the secondary structure 428 common to two rna sequences. Journal of molecular biology , 317(2):191-203, 2002. 429

430

431

D. H. Mathews and D. H. Turner. Prediction of rna secondary structure by free energy minimization. Current opinion in structural biology , 16(3):270-278, 2006.

432

433

434

435

436

E. P. Nawrocki, S. W. Burge, A. Bateman, J. Daub, R. Y. Eberhardt, S. R. Eddy, E. W. Floden, P. P. Gardner, T. A. Jones, J. Tate, et al. Rfam 12.0: updates to the rna families database. Nucleic acids research , 43(D1):D130-D137, 2015.

R. Nicholas and M. Zuker. Unafold: Software for nucleic acid folding and hybridization. Bioinformatics , 453:3-31, 2008.

H. F. Noller. Structure of ribosomal rna. Annual review of biochemistry , 53(1):119-162, 1984. 437

438

439

R. Nussinov, G. Pieczenik, J. R. Griggs, and D. J. Kleitman. Algorithms for loop matchings. SIAM Journal on Applied mathematics , 35(1):68-82, 1978.

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

O. Perriquet, H. Touzet, and M. Dauchet. Finding the common structure shared by two homologous rnas. Bioinformatics , 19(1):108-116, 2003.

Z. Qin, W. Sun, H. Deng, D. Li, Y. Wei, B. Lv, J. Yan, L. Kong, and Y. Zhong. cosformer: Rethinking softmax in attention. In International Conference on Learning Representations , 2021.

A. Rich and U. RajBhandary. Transfer rna: molecular structure, sequence, and properties. Annual review of biochemistry , 45(1):805-860, 1976.

E. Rivas. The four ingredients of single-sequence rna secondary structure prediction. a unifying perspective. RNA biology , 10(7):1185-1196, 2013.

O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention , pages 234-241. Springer, 2015.

J. Ruan, G. D. Stormo, and W. Zhang. An iterated loop matching approach to the prediction of rna secondary structures with pseudoknots. Bioinformatics , 20(1):58-66, 2004.

K. Sato, M. Akiyama, and Y. Sakakibara. Rna secondary structure prediction using deep learning with thermodynamic integration. Nature communications , 12(1):1-9, 2021.

M. G. Seetin and D. H. Mathews. Rna structure prediction: an overview of methods. Bacterial 456 regulatory RNA , pages 99-122, 2012. 457

J. Singh, J. Hanson, K. Paliwal, and Y. Zhou. Rna secondary structure prediction using an en458 semble of two-dimensional deep neural networks and transfer learning. Nature communications , 459 10(1):1-13, 2019. 460

J. Singh, K. Paliwal, T. Zhang, J. Singh, T. Litfin, and Y. Zhou. Improved rna secondary 461 structure and tertiary base-pairing prediction using evolutionary profile, mutational coupling 462 and two-dimensional transfer learning. Bioinformatics , 37(17):2589-2600, 2021. 463

M. F. Sloma and D. H. Mathews. Exact calculation of loop formation probability identifies 464 folding motifs in rna secondary structures. RNA , 22(12):1808-1818, 2016. 465

D. So, W. Ma´ nke, H. Liu, Z. Dai, N. Shazeer, and Q. V. Le. Searching for efficient transformers 466 for language modeling. Advances in Neural Information Processing Systems , 34:6010-6022, 467 2021. 468

E. W. Steeg. Neural networks, adaptive optimization, and rna secondary structure prediction. 469 Artificial intelligence and molecular biology , pages 121-160, 1993. 470

M. Szikszai, M. J. Wise, A. Datta, M. Ward, and D. Mathews. Deep learning models for rna 471 secondary structure prediction (probably) do not generalise across families. bioRxiv , 2022. 472

Z. Tan, Y. Fu, G. Sharma, and D. H. Mathews. Turbofold ii: Rna structural alignment 473 and secondary structure prediction informed by multiple homologs. Nucleic acids research , 474 45(20):11570-11581, 2017. 475

H. Touzet and O. Perriquet. Carnac: folding families of related rnas. Nucleic acids research , 476 32(suppl_2):W142-W145, 2004. 477

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and 478 I. Polosukhin. Attention is all you need. Advances in neural information processing systems , 479 30, 2017. 480

L. Wang, Y. Liu, X. Zhong, H. Liu, C. Lu, C. Li, and H. Zhang. Dmfold: A novel method to 481 predict rna secondary structure with pseudoknots based on deep learning and improved base 482 pair maximization principle. Frontiers in genetics , 10:143, 2019. 483

S. Wang, S. Sun, Z. Li, R. Zhang, and J. Xu. Accurate de novo prediction of protein contact 484 map by ultra-deep learning model. PLoS computational biology , 13(1):e1005324, 2017. 485

X. Wang and J. Tian. Dynamic programming for np-hard problems. Procedia Engineering , 486 15:3396-3400, 2011. 487

H. K. Wayment-Steele, W. Kladwang, A. I. Strom, J. Lee, A. Treuille, E. Participants, and R. Das. 488 Rna secondary structure packages evaluated and improved by high-throughput experiments. 489 BioRxiv , pages 2020-05, 2021. 490

E. Westhof and V. Fritsch. Rna folding: beyond watson-crick pairs. Structure , 8(3):R55-R65, 491 2000. 492

H. Wu, J. Wu, J. Xu, J. Wang, and M. Long. Flowformer: Linearizing transformers with 493 conservation flows. arXiv preprint arXiv:2202.06258 , 2022. 494

X. Xu and S.-J. Chen. Physics-based rna structure prediction. Biophysics reports , 1(1):2-13, 495 2015. 496

S. Zakov, Y. Goldberg, M. Elhadad, and M. Ziv-Ukelson. Rich parameterization improves rna 497 structure prediction. Journal of Computational Biology , 18(11):1525-1542, 2011. 498

H. Zhang, C. Zhang, Z. Li, C. Li, X. Wei, B. Zhang, and Y. Liu. A new method of rna secondary 499 structure prediction based on convolutional neural network and dynamic programming. Frontiers 500 in genetics , 10:467, 2019. 501

M. Zuker. Mfold web server for nucleic acid folding and hybridization prediction. Nucleic 502 acids research , 31(13):3406-3415, 2003. 503

Peer reviews:
comments: SUMMARY:
In this paper, the authors propose a way to decouple the optimization process of RNA secondary structure prediction. Specifically, they decompose the constraint satisfaction problem into row-wise and column-wise optimization. Instead of hand-crafted features, attention maps are used to learn the pair-wise interactions of the nucleotide bases.

STRENGTHS:
1. The authors show that it is more effective to use the attention maps as the input and then use U-Net to predict H, compared with using the hand-crafted features as input, which is interesting.
2. The proposed method reduces the inference time dramatically compared to various methods.
3. It achieves promising results on the RNAStralign dataset and large-scale benchmark evaluation.

WEAKNESSES:
1. The generalization ability is limited because the proposed method cannot achieve the best recall on ArchiveII and bpRNA-TS0 datasets.

QUESTIONS:
Could the authors use experiments to show that strict constraints cause the less promising generalization ability?
Is it possible to improve the proposed method to gain better generalization ability?

LIMITATIONS:
I did not find the potential negative societal impact of this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-06-25

comments: SUMMARY:
The paper introduces RFold for RNA secondary structure prediction (a prediction of LxL binary matrix). It proposes to add a row-column-wise softmax at output of the model, before computing the L2 loss with respect to the ground truth. The experimental results show higher precision and recall compared to prior works.

STRENGTHS:
- The paper is well-written with necessary backgrounds and basic introductions, problem formulations. Thus, this paper suited well for general audience of NeurIPS.
- RFold delivers strong performances in two commonly used datasets for evaluating RNA secondary structures.

WEAKNESSES:
The main concern is the limited novelty, RFold is incremental over Ufold. Both methods follow the paradigm of mapping a sequence RNA (using $\theta_{1}$) to $[L \times L \times n]$ features and further mapping the $[L\times L \times n]$ features (using $\theta_{2}$) to $[L \times L \times 1]$ output prediction. RFold differs from Ufold in two parts:
- RFold proposes $\theta_{1}$ to be represented by an attention-based layer. (RFold and Ufold use a similar, if not identical, $\theta_{2}$.)
- RFold adds a column-wise and a row-wised softmax after the $\theta_{2}$, before computing L2 loss.

Thus, RFold makes a few architectural modifications and improves the results.

E2Efold and Ufold also have a section on evaluation with pseudoknots on the RNAStralign test dataset, which this submission does not have.

QUESTIONS:
N/A

LIMITATIONS:
The paper has no section for limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-03

comments: SUMMARY:
This work presents an efficient and accurate approach for end-to-end RNA secondary structure prediction.
The optimization problem formulation and its solution are well defined.
The results are strong and supported by visualizations and ablation studies.

STRENGTHS:
The key strengths of this work are:
1. Inference is an order of magnitude faster than previous methods.
2. Inference is between 4-20% more accurate than previous methods, with significant gains specifically in long-range interactions.
3. A well defined optimization problem formulation and solution.
4. Including visualizations and ablation studies underscores the gains achieved through the optimization formulation and attention architecture.
5. The results are validated using multiple datasets and baselines.

WEAKNESSES:
Weaknesses of this work are:
1. There is a discrepancy in the definition of G in equation 12, where it does not incorporate the softmax function. 
However, in equation 15, it is assumed as if it does. This can be fixed by introducing a new notation, such as G_{hat}, 
which includes the softmax function and will ensure consistency.
2. The definition of well-known metrics in section 5 is redundant.
3. The comparison between Rfold and Ufold could be more comprehensive, 
describing their similarities and differences.
4. There are a few minor typos, and the writing may be improved.

QUESTIONS:
What are the key similarities and differences between Rfold and Ufold?
Visualizing performance as a function of sequence length may be useful.

LIMITATIONS:
The limitations are adequately addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes RFold, a simple and effective RNA secondary structure prediction algorithm. It adopts attention maps to learn informative representations for RNA rather than hand-crafted features. Then, based on a decoupled optimization process, RFold simplifies and guarantees satisfying the hard constraints on the formation of RNA secondary structure. Through the empirical experiments, the authors demonstrate that RFold achieves state-of-the-art performance with better computational efficiency compared to the previous works.\n\nSTRENGTHS:\n- The proposed decoupled optimization seems simple, but surprisingly effective for RNA secondary structure prediction. To the best of my knowledge, the proposed method is novel in the domain and might be promising for the broader machine-learning community.\n- The proposed method shows great performance in three RNA benchmark datasets outperforming the previous state-of-the-art method by a significant margin. Some issues need to be addressed regarding the experiment setup (please refer to the weaknesses), but the improved performance seems truly impressive.\n\nWEAKNESSES:\nMajor comments:\n- [Data Split] To best approximate real-world applications that may require the prediction of novel structures, RNAs from the train/val/test set should bear minimal sequence and structural similarities. In contrast, it seems the authors have split datasets so that each RNA family has a similar fraction in each set. I think it may overestimate the true prediction performance of RFold. Likewise, “generalization to other datasets” experiments do not provide information about sequence/structure similarities between the datasets. If they are similar, it may not be a fair evaluation of generalization performance.\n- Since the authors stated deep learning methods do not ignore the biologically essential structure such as pseudoknots, can you provide additional separate evaluation under the (non-) existence of pseudoknots?\n- According to UFold, the bpRNA dataset contains mostly within family RNA species and does not adequately show the true generalization performance of the models. Can you provide additional evaluations with cross-family experiments?\n- [Inference Time] It’s unclear whether the results are credible. The inference time can be quite different based on what type of machine (CPU, GPU, etc.) is used for the measurement. Since the other results seem to be excerpted from the UFold paper, the environments of UFold and RFold are likely to be different.\n- [Reproduciblity] Architectural hyperparameters are missing. In addition, training codes do not seem to be included in the supplementary.\n\nMinor comments:\n- [Data Split] The authors stated that they split the RNAStralign dataset following the E2Efold paper. Can you confirm that all the methods including RFold used the same data splits? RNA sequences often have high sequence and structure similarities, so if you used different data splits it might affect the performance. \n- As the authors stated, other algorithms often post-process the outputs to satisfy the constraints. Can you also show how the results are improved for RFold-E/S with the post-processing?\n\nQUESTIONS:\n- Do you have any plans for launching a web server for the proposed method? It would be difficult for many biotechnology researchers to set up the environment and run the algorithm. Therefore, while it is not mandatory, most compared algorithms support web servers for their models.\n\nLIMITATIONS:\nThe authors have not discussed the limitations of the work.\n\n---Post-Rebuttal Comments---\n\nI appreciate the authors' dedication evident in their comprehensive responses. They have effectively addressed many of the concerns I had about the paper. Overall, while some concerns persist, I am inclined to believe that by incorporating the authors' responses, the manuscript's quality would be improved. Hence, I've adjusted my rating to 5.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-25"}