Title: Generative modeling for RNA splicing code predictions and design
Venue: NeurIPS 2023
Accepted: no
Keywords: RNA Splicing

Computational Biology

Transformers

Deep Learning

Bayesian Optimization

RNA Design
Abstract: Alternative splicing (AS) of pre-mRNA splicing is a highly regulated process with diverse phenotypic effects ranging from changes in AS across tissues to numerous diseases. The ability to predict or manipulate AS has therefore been a long time goal in the RNA field with applications ranging from identifying novel regulatory mechanisms to designing therapeutic targets. Here we take advantage of generative model architectures to address both the prediction and design of RNA splicing condition-specific outcome. First, we construct a predictive model, TrASPr, which combines multiple transformers along with side information to predict splicing in a tissue specific manner. Then, we exploit TrASPr as on Oracle to produce labeled data for a Bayesian Optimization (BO) algorithm with a costume loss function for RNA splicing outcome design. We demonstrate TrASPr significantly outperforms recently published models and that it can identify relevant regulatory features which are also captured by the BO generative process.

Abstract

Alternative splicing (AS) of pre-mRNA splicing is a highly regulated process with diverse phenotypic effects ranging from changes in AS across tissues to numerous diseases. The ability to predict or manipulate AS has therefore been a long time goal in the RNA field with applications ranging from identifying novel regulatory mechanisms to designing therapeutic targets. Here we take advantage of generative model architectures to address both the prediction and design of RNA splicing condition-specific outcome. First, we construct a predictive model, TrASPr, which combines multiple transformers along with side information to predict splicing in a tissue specific manner. Then, we exploit TrASPr as on Oracle to produce labeled data for a Bayesian Optimization (BO) algorithm with a custom loss function for RNA splicing outcome design. We demonstrate TrASPr significantly outperforms recently published models and that it can identify relevant regulatory features which are also captured by the BO generative process.

1 Introduction

Alternative splicing (AS) occurs when multiple unique mRNA isoforms are produced from a single gene, each including or excluding different pre-mRNA exonic or intronic segments. AS greatly increases transcriptome complexity such that a single gene can encode many mRNA isoforms, each of which include a different subset of pre-mRNA segments. Over 90% of human genes undergo AS, with a conservative estimate that at least 35% of human genes switch their dominant isoform across 16 adult tissues [32, 44, 16]. Changes in the produced isoforms can have significant phenotypic effects: Defects in splicing have been associated with numerous diseases [38] while at the molecular level, AS has been shown to change protein function by, for example, removing a nuclear localization signal, affecting an RNA or DNA binding domain of the encoded protein, or regulating gene expression by introducing a poison exon that leads to nonsense mediated decay (NMD) [40, 28].

Following the discovery of RNA splicing in 1977 [2, 7], decades of work has identified hundreds 25 of RNA Binding Proteins (RBPs) that regulate splicing outcome. These RBPs have been shown to 26 bind exons and proximal introns, typically up to a few hundred bases away from proximal exons, to 27 regulate splicing in a condition specific manner [12]. Consequently, in an influential 2008 review 28 article, Chris Burge and Zefeng Wang set a long-term goal for the RNA community to construct a 29 predictive 'splicing code' that will be able, given a genomic sequence and cellular condition, predict 30 the splicing outcome [45]. Splicing outcomes are typically measured as the ratio of isoforms that 31 include or exclude a specific RNA segment (e.g., an exon). This ratio is commonly referred to as 32 'percent spliced in' (PSI, or Ψ ∈ [0 , 1] ), and changes in splicing between cellular conditions or due to 33 genetic mutations are expressed as dPSI ( ∆Ψ ∈ [ -1 , 1] ). In this work, we consider two tasks related 34 to splicing: splicing prediction, and splicing sequence design. 35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

Splicing prediction. Software to predict splicing was first introduced in 2010, formalizing splicing codes as a supervised learning problem for exon e with differential inclusion/exclusion/no-change in a specific cellular condition (e.g., tissue) c . Subsequent work defines the learning tasks as a prediction task for Ψ e,c or ∆Ψ e,c,c ′ using a variety of modeling approaches, including Bayesian NNs, autoencoders, MLPs, CNNs, and RNNs [48, 6, 49, 25]. Importantly for the work described here, the best performing tissue specific splicing prediction models to date use hand crafted expert derived regulatory features from the genomic sequence of interest, such as RBP binding based on sequence motifs, secondary structure, and conservation values [21]. Subsequent models used the genomic sequence directly as input ( e.g. , CNN models) but mostly focused on predicting the effect of genetic mutations [6, 50] with only moderate success in tissue specific predictions.

Sequence design. Sequence design for RNA splicing is a new task, similar to ones involving the design of untranslated regions (UTR) in mRNA vaccines for optimal expression [5, 26] or the design of alternative polyadenylation [3]. Similar to the latter, we formulate RNA splicing design as a constraint optimization problem, where we are required to generate a genomic sequence that would have specific splicing outcome characteristics, such as increased exon inclusion in brain. In addition, the generated sequence can also be constrained such that it involves for example no more than M mutations in N locations compared to the given starting sequence. Applications of such a design task can vary from optimizing therapeutics for correcting splicing defects to synthetic biology.

This work offers several contributions. First, we propose TrASPr, a new multi-Transformer based splicing code model, demonstrating it can achieve state of the art results for tissue-specific splicing prediction. Second, we formulate RNA splicing design as an optimization problem involving a deep generative model such that Bayesian Optimization (BO) techniques can be utilized for it. Our BO algorithm for splicing (BOS), uses TrASPr as an Oracle to optimize a VAE under sequence and splicing outcome constraints. We first test TrASPr on RNA splicing data from both mouse and human tissues, demonstrating it achieves state-of-the-art prediction accuracy. Then we show TrASPr detects condition specific regulatory elements using ENCODE data involving three RBP Knockdown (KD) in two human cell lines, and data for tissue-specific regulatory elements from a mini-gene reporter assay. Finally, we demonstrate BOS can effectively mutate a given sequence under a limited number of mutations to achieve a pre-defined tissue specific splicing outcome.

2 Background

2.1 Quantifying AS events

Splicing codes require training data in the form of quantified AS events across diverse conditions. Such AS quantification nowadays is mostly derived from Illumina RNA sequencing reads. Each experiments includes millions of these ∼ 100bp long reads that are mapped back to the genome using dedicated tools ( e.g. , STAR). Dedicated splicing analysis algorithms are then used to first detect the AS events, typically from reads spanning across RNA segments, then quantify those in terms of Ψ or ∆Ψ as described above. Here we applied the commonly used MAJIQ algorithm [42] to quantify AS as it has been shown to compare well to other tools [27] and carries several additional benefit important for the task at hand. Specifically, MAJIQ allows for the detection of unannotated splice site, splice junctions, exons, and intron retention events. Furthermore, MAJIQ can capture complex AS events involving multiple alternative splice junctions. These characteristics are key for creating a high-quality train and test samples where such variations are controlled (see details below). Specifically, we only

2.2 Transformer modeling of RNA sequence

In this work we adapt BERT [9] model to RNA sequences. BERT is a bi-directional transformer80 based model, which learns contextual relations of tokens in a text [43]. The BERT model can 81 be pre-trained on large unlabeled datasets of tokenized text using masked token prediction. Here 82 we considered different tokenizing strategies of RNA sequences which are composed of 4 types 83 of ribonucleotide bases (' A ',' C ',' G ',' U '). We settled on overlapping k-mers of length 6 such that 84 the sequence 'AUUGGCU' is represented by a string containing two tokens, AUUGGC and UUGGCU . 85 During pre-training all k-mers that include a specific nucleotide are masked as in for example the 86 DNABERT model [22]. However, we found the DNABERT architecture to be unstable and opted to 87

pre-train a lighter BERT model with only six layers as describe below. In addition to all possible 88 6-mer combinations of ribonucleotide bases, we include 5 special tokens to represent classification 89 ( [CLS] ), padding ( [PAD] ), separation ( [SEP] ), mask ( [MASK] ) and unknown ( [UNK] ). Finally, we 90 extend the vocabulary with additional tokens to capture additional features and information such as 91 the tissue type, species and length tokens. 92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

2.3 Notation

We measure splicing across c ∈ [1 , . . . , C ] conditions for events e ∈ [1 , . . . , E ] . Each AS event e has a sequence S e comprised of 4 different regions, each centered around the respective splice site S e = { S 1 e , S 2 e , S 3 e , S 4 e , } . Similarly each event has a set of features associated with it such as exon length, conservation etc. denoted F e . Splicing quantification for event e in condition c is denoted Ψ e,c ∈ [0 , 1] and differential splicing as ∆Ψ e,c,c ′ ∈ [ -1 , 1] accordingly. However we frequently drop the event e or condition c index for brevity.

3 Related work

The first splicing code model used boosted decision trees, learned from over 1000 putative regulatory featured derived from the literature [1]. While that first model had only ∼ 3700 exon skipping AS events to learn from, subsequent models took advantage of more samples from RNA-Seq data that had became available to train Bayesian and deep learning models [48, 47, 4, 6, 50]. The best performance on tissue specific splicing prediction was achived in [21] using a similar set of pre-defined regulatory features that were first condensed using an AutoEncoder, then combined in a MLP. Subsequent works aimed to learn a code directly from the genomic sequence using a variety of architectures. MT-Splice for example used a CNN based architecture with 64 length-9 filters while the more recent Pangolin [50] employed a ResNet architecture originally introduced in the SpliceAI model for detecting cryptic splice site [20]. Both MT-Splice and Pangolin focused on predicting mutations that affect splicing outcome and reported moderate accuracy for tissue-specific splicing prediction.

The RNA splicing design task is new and possibly the only directly related work is Deep Exploration Networks (DEN) by [29]. DEN involves a VAE which generates genomic sequence, the generated sequence is then evaluated by a prediction model for the desired task ( e.g. , splicing outcome Ψ ) which is combined, via a hyper parameter λ , with a second target function that penalizes generated sequences too close to previously generated ones. While similar in spirit, DEN is quite different than the work presented here. First, DEN models the genomic sequence as one long position weight matrix (PWM) that is later collapsed into a specific sequence. The VAE itself is based on a feed forward network and the prediction models are either a CNN or a linear model of k-mer counts as in [34]. The splicing task in that work is also different, involving alternative 5' splice site selection with two relatively short regions downstream of each 5' splice site. Finally, the data used for training and testing the DEN for the above task is distinctly different, based on a large pool of approximately 13,000 synthetic sequences tested in cell lines.

4 Data

To pretrain the basic BERT RNA model described above, we extract 1.5 million sequences around splice sites from the GENCODE human pre-mRNA transcripts database. Each sequence was cut to be 400 bases long and centered around the splice site. These sequences are then converted into 6-mers tokens and fed as input to the BERT model.

Similar to previous work, we focus on predicting the inclusion levels of cassette exons. To evaluate performance we use two main datasets. The first is from the mouse genome project (MGP)[23] and

involves six mouse tissues (Heart, Spleen, Thymus, Lung, Liver and Hippocampus) with 4-6 replicates each. We also used the same train/test data split used in [21] so that the results can be compared

directly to their model.

The second dataset is GTEx[8] from which we select six representative tissues/conditions: Heart (Atrial Appendage), Brain (Cerebellum), Lung, Liver, Spleen, and EBV

transformed lymphocytes. Note that some conditions are shared between the datasets. This ensures that our model sees sequences from different species but similar tissues. For all tissues and tissue

pairs in these datasets, we processed the RNA-Seq using MAJIQ (see Section 2.1) to detect cassette

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

events with high-confidence quantification for their Ψ e,c , ∆Ψ e, c, c ′ . In total, we collect E=11346 and E=18278 events from the MGP and GTEx datasets.

Test set filtering. The high number of events in our data is partially due to the fact the cassette exons extracted from MAJIQ's splice graphs may be overlapping ( e.g. , different splice sites used to define the skipped exon). This may be useful for training on diverse exon/intron definitions but care must be taken to avoid information leakage to the test data. This is especially important for large models that can easily memorize genomic sequences [36]. We handle this issue in two ways. First, we fully hide two chromosomes (8, 14 for GTEx and 4, 11 for MGP) for testing, and discard test exons that are too similar to training exons. Sequence similarity was assessed using BLAT [24] with filters for percent identify, difference in length, and the estimated similarity p-value. We consider two filter settings. First, we denote a set of 'Permissive' filters as used in [21], These settings included maxLenDiff=5 , minPval=0.0001 and minIdentity=95 . Because we are using significantly more complex models, we introduce a second set of filters we denote 'Strict' with maxLenDiff=100 , minPval=0.001 and minIdentity=80 . This accounts for short exons with high similarity but that diverge enough relative to their short length to not achieve a significant p-value.

Test data for mutations and knockdown analysis. To evaluate the capability of TrASPr and BOS to predict or suggest mutations, we curated two other sets of experimental data. The first one is the RBP Knockdown (KD) experiments from ENCODE [19]. ENCODE data involves two types of cell lines (K562, HepG2) in which various RBP were knocked down, followed by RNA-Seq experiments to measure the KD effect on the transcriptome. Since the ENCODE RNA-Seq data has been shown to exhibit strong batch effects we first performed batch correction using MOCCASIN [39]. Here, we focused on three well studied RBPs (TIA1, PTBP1, QKI) for which there is relatively better sequence motif definitions ( i.e. , which sequences these RBP are likely to bind) and better experimental binding assays (eCLIP) which indicate regions where these RBPs were found to bind the RNA sequences. To assess whether the splicing code is learning direct regulation by these RBPs we searched for occurrences of these RBPs sequence motifs. Then we filtered those motif locations to be in AS events which had those in the intronic regions proximal to the alternative exon. We furthered filtered those for AS events that had eCLP binding peaks for those RBPs and that their inclusion level was indeed affected upon the RBP KD experiment (| ∆Ψ | > 0 . 15) ). This set of AS events served as putative targets of the above RBPs. We then 'removed' the effect of these RBPs on the set of AS targets by randomly mutating the identified binding motifs. We repeated this process 5 times with different random mutations and the prediction results where then averaged and compared to the wild type (WT) sequence prediction. These in-silico predictions of RBPs effects where then compared to those observed in the actual KD experiment. Finally, we also included experiments from a mini-gene reporter assay where the effect of mutating several regions upstream of exon 16 of the mouse Daam1 gene where tested [1].

5 Methods

Our method involves three main components depicted in Fig. 1: An elaborate data processing pipeline discussed above, a transformer based splicing prediction model (TrASPr), and a Bayesian Optimization algorithm (BOS) to design RNA with desired properties. We now turn to describe the two latter modeling components in order.

5.1 TrASPr

5.1.1 Pre-training RNA splice site BERT model

The foundation model for TrASPr is a 6 layer BERT model which is pretrained on human RNA splice sites (Fig. 1b). Following the pretraining step, as in [22], TrASPrtakes an RNA sequence converted to 6-mer tokens as input, but instead of using the BERT default max length, we feed the model with 400 bases long sequences where the splice site (either 5' or 3' splice site, as shown in the cartoon) is in the center.

For pre-training, we follow BERT in randomly choosing 15% of tokens, but additionally mask the surrounding k tokens for each one to account for our overlapping 6-mer tokenization. We used standard masked autoencoding training, calculating the loss from the original 15% of tokens that

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

a)

b)

c)

d)

Transcriptome annotations

RNA-seq experiments

GFF3

BAM

Wn-2

W'n-1

W'n

ClassificationLayer

MAJIQBUILD+QUANT

Pre-training

Model

Transformer

X6

Define LSVs +filtering/coverage

SourceLSV

Target LSV

PSI

En-2

En-1

Er

Fit Template

Embedding

K-mers input

t...ACTCAG TCAGCG [MASK] AGCGGA...

With mask

...GACTCAGGTCGAT...

PSI

5'Splice Site

OR

...AACATAGAATTAC...

3'Splice Site

皆小→

(3) := Oracle(3)

S

Transformer

PSI, dPSI'+,dPSI-

Decoder

MLPS

Update Data: D ← DU (3,4)

Proposal

Bayesian

Representation

Features

RepresentationZ

Optimization

Fine tuning

Model

Transformer

Transformer

Transformer

Transformer

Transformer

Encoder1

Encoder2

Encoder3

Encoder4

EncoderΦ

Embedding

K-mers

input

K1

(-.ACTCAG.)

Initial s

(- ACCCAG.-)

Sequence S=

s

S2

S3

F3

( GATCAG.中)

Variations3

200 bps

were masked. We pretrain for 110k steps with a batch size of 40. The learning rate was set to 4e-4 and we used a linear scheduler with 10k warm-up steps.

5.1.2 The TrASPr model and fine-tuning

Here, we describe finetuning our TrASPr model from the pretrained model described above, as depicted in Fig 1c. For any given AS event e , the input to TrASPr is a sequence composed of four sequences S e = { S i e } 4 i =1 such that each S i e covers the exonic and intronic regions surrounding one of the four splice sites involved in the exon skipping AS event e . Each S i e is fed through a matching T i F = { F }

The latent space representation from each transformer T , captured by their respective CLS tokens, are concatenated together along with the feature set F e and fed into a 2 hidden layer MLP with width 3080 and 768.

pre-trained transformer , which also accepts additional event features e e,i (see below). i

Event features. The additional feature set F e includes the exon and intron length information as binned tokens, as well as the tissue type. We additionally include conservation values generated based on PhastCons score[37] for each k-mer in the sequence. Exons generally have significantly higher conservation values, as these reflect selection pressure due to non splicing related function (coding for proteins). We therefore used the mean of all conservation scores to fill the exon regions but kept the original scores for the introns.

Supervision. We follow [21] and define targets based on measuring both splicing outcomes and changes in splicing outcome for an event e in two different conditions c, c ′ . Specifically, the target variables included:

Here E [Ψ e,c ] , E [∆Ψ c,c ′ ] represent the posterior expected values for PSI and dPSI as estimated by MAJIQ from the RNA-Seq experiments [42]. The T ∆Ψ+ c,c ′ target captures events with increased

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

inclusion level between tissue c and c' while T ∆Ψ -c,c ′ captures events with increased exclusion, forcing the model to focus its attention on those. To avoid gradient issue, we use random small number between 0.001 and 0.002 as ϵ . For all of those target variables we use the cross-entropy loss function which performed better than regression. In the fine-tuning step, we train the model with 2e-5 learning rate and batch size of 32 for 10 epochs.

5.2 Sequence design for splicing outcomes.

Beyond supervised learning, we also demonstrate that TrASPr can be leveraged to solve sequence design problems. Given a sequence S e = ( s 1 , ..., s n ) , TrASPr measures the probability that the splice site in the center of S e is included in some tissue c , Ψ c ( S e ) . This value can directly be used as the basis for optimization problems, where we seek to design new sequences ˜ S e that differ from S e only slightly, but exhibit altered splicing outcomes. Formally, we define these optimization problems:

̸

Here, lev ( ˜ S e , S e ) denotes the Levenshtein distance between ˜ S e and S e . Solving the minimization problem is equivalent to finding a small perturbation (up to edit distance τ ) of S e that reduces inclusion in the target tissue c by as much as possible. The maximization problem corresponds to increasing inclusion. In practice, we add additional constraints that ∀ c ′ = c , Ψ c ′ ( ˜ S e ) cannot be reduced below 0 . 05 . This additional constraint prevents an optimization routine from destroying splicing to such an extent that all inclusion levels are driven to zero.

To solve these optimization problems, we adapt recent work in latent space Bayesian optimization (LSBO) for black-box optimization problems over structured and discrete inputs [30, 41, 14, 31, 46, 35, 15, 17, 18]. LSBO solves structured optimization problems using two primary components: (1) a deep autoencoder (VAE) model, and (2) a Bayesian optimization routine.

Variational autoencoders for LSBO. In LSBO, we train a DAE that assists in reducing the discrete optimization problem over sequences S to a continuous optimization problem over the latent space of the V AE, Z ⊂ R d . Leveraging the same data used to train TrASPr, we train a 6 layer Transformer encoder Φ : S → P ( Z ) and 6 layer Transformer decoder Γ : Z → P ( S ) [43]. The encoder Φ( S e ) maps sequences S e onto a distribution over real-valued, continuous latent vectors z . The decoder Γ( z ) (probabilistically) reverses this process. The parameters of Φ and Γ are trained so that roughly we have Γ(Φ( S e )) ≈ S e . Because we only care ultimately about the output sequence ˜ S e , here we abuse notation and denote the most probable sequence output from the decoder as Γ( z ) . For optimization, the advantage the VAE provides is the ability to optimize over latent vectors z rather than directly over sequences S e . This is because, for any z proposed by an optimization algorithm, we can evaluate Ψ c (Γ( z )) . We therefore search for a ˜ z such that ˜ S e := Γ(˜ z ) is a high quality solution to the optimization problem.

Bayesian optimization. With the optimization problems in Equation 1 reduced to continuous problems over ˜ z ∈ Z , we can now apply standard continuous black-box optimization algorithms. Bayesian optimization [13] is among the most well studied of these approaches in the machine learning literature. In iteration n of Bayesian optimization, we have a dataset D n = { ( z i , y i ) } n i =1 for which y i = Ψ c (Γ( z i )) is the known objective value. We train a surrogate model of the objective function using this data-most commonly a Gaussian process [33]-and use this surrogate to inform a policy-commonly called an acquisition function -that determines what latent vectors z n +1 to consider next. In this paper, we use LOL-BO [30] as our base, off-the-shelf LS-BO algorithm. To accommodate the constraints in Equation 1, we modify LOL-BO to utilize SCBO [11] rather than TuRBO [10] as the underlying optimization routine. As with the objective, the Levenshtein constraint is evaluated on decoded latent vectors: lev Z ( z , z ′ ) = lev (Γ( z ) , Γ( z ′ )) .

6 Results

In this section, we compare TrASPr with state-of-art methods on predicting condition specific splicing changes, assess its ability to predict the effect of changes in trans (RBP KD) or cis (mutations in a mini-gene reporter assay) using in-silico , then assess the ability of our proposed generative algorithm BOS to propose sensible sequences for a user defined splicing outcome.

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

0.0

0.2

0.4

0.6

0.8

1.0

TrASPr predicted PSI

0.0

0.2

0.4

0.6

0.8

1.0

PSI Lable

0.0

0.2

0.4

0.6

0.8

1.0

Pangolin predicted usage

0.0

0.2

0.4

0.6

0.8

1.0

PSI Lable

6.1 Predicting exon inclusion levels across tissues

We first evaluate TrASPr on the task of predicting Ψ using human GTEx data, comparing to Pangolin. Pangolin uses the SpliceAI model architecture [20] and was originally trained on data from four species, each with four tissues. Pangolin model is unable to define specific splicing events such as cassette exons. Instead it uses a 10Kb sequence window and predicts a 'splice usage' value for the position in the center, constructing a separate model for each tissue. To make Pangolin comparable, we feed the 3' and 5' splice site of each alternative exon e , then calculate the average of these two. Performance was evaluated on shared tissues and test chromosomes as used in [50]. Our model achieved significantly higher Pearson correlation for PSI prediction (0.81 vs 0.17 see Fig 2), even though the training set is smaller due to only using overlapping tissues. Taking a closer look both models work well on most of low PSI cases. However, Pangolin performance suffered on some high inclusion cases, assigning low inclusion values. This result might be because of condition specific regulation, because the relevant sequence context is outside the 10kb fixed window used by Pangolin, or because other splicing signals in that window 'confused' the model with respect to quantifying the inclusion of the cassette exon. We note that as mentioned by the authors in [50], predictions for tissue specific splicing changes were not very accurate and we therefore not include them here.

Next we turned to assess tissue specific differential splicing predictions. We compared TrASPr against a previous model that used the same target function but employed manually curated features parsed through an AutoEncodeer and several layers of MLP (denoted 'AE+MLP feature model') [21]. This curated feature set was only available for the MGP dataset and so we assessed performance on this data using the same train and test set definitions as by the authors. Fig 1 and Table 1 show TrASPr significantly outperformed the AE+MLP model in identifying both differentially included and differentially excluded events, especially in terms of AUPRC (every pair of tissues is a point in the scatter plot with blue crosses and brown minuses each denoting evaluation on a set of differentially included or excluded events respectively). However, when we applied a more stringent filtering criteria on the test set TrASPr performance degraded while, surprisingly, AE+MLP performance improved. The degraded performance of TrASPr may be due to the fact the model was able to relate mouse and human AS events that are somewhat similar, but the fact performance for AE+MLP model improved may point to some specific characteristics of the stricter dataset that may have made it easier to predict using the pre-defined feature set.

AUPRC

0.0

0.2

0.4

0.6

0.8

1.0

AE+MLP feature model results

0.0

0.2

0.4

0.6

0.8

1.0

Our model results

dPSI+

dPSI-

AE+MLP

feature Model

TrASPr

Filter

Default

AUPRC

0.4861 0.4438

0.6079 0.6038

Spearman

0.5503

0.6867

AUROC

0.8712 0.8502

0.8895 0.8892

Filter

Strict

AUPRC

0.5388 0.4874

0.5579 0.5176

Spearman

0.5962

0.5917

AUROC

0.8909 0.8766

0.8740 0.8695

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

0.0

0.2

0.4

0.6

0.8

1.0

0.1

|dPSI|

0.0

0.2

0.4

0.6

0.8

1.0

0.89

Proportion

0.0

0.2

0.4

0.6

0.8

1.0

Ground truth

0.0

0.2

0.4

0.6

0.8

1.0

Prediction

KD

> 0.15

KD

< -0.15

Ground truth direction

0

10

20

30

40

Count

Correct

No Change

Wrong

6.2 Predicting the effect of RBP KD and mutations

We next turned to assess TrASPr ability to predict the effect of RBP KD and mutations. For this we first retrained the model using ENCODE data described in Section 4. First we assessed whether TrASPr is able to accurately predict exon inclusion in those new conditions. As shown in Figure 4a, TrASPr was able to predict Ψ within 10% accuracy in almost 90% of the test cases, indicating the model was able to learn inclusion levels in those cell lines. Next, we focused on the set of putative RBP cassette exons targets shown in Figure 4b, where brown and purple represent events whose inclusion levels went up or down upon KD respectively. We find the WT Ψ predictions for these correlated well with the experimental results (pearson's 0.65), and therefore continued with mutating the specific sequence motifs suspected to be the binding sites for the three RBPs of interest.

Before we could evaluate predictions for in-silico mutations we first needed to assess the significance of any given prediction. To achieve this, we randomly mutated sequences in the same set of exons, selected from the same distribution of distances as the original motifs (the distance can greatly affect the null distribution), but made sure non of these randomly chosen regions hit any of the 'real' motifs. We then used the 95 percentile of effects observed in this set as our threshold to call changes. The results of the in-silico mutagenesis experiment are summarized in Figure 4c. The left stacked bar shows cases whose PSI increased after RBP KD and the right bar shows decrease PSI cases. The correct(blue) and wrong(red) indicates if the predicted direction is the same as the label and no change(grey) means predicted dPSI was below the 95% cutoff described above. Overall, TrASPr performed well on most of the positive direction cases but predicted around half of negative direction cases as no change. The correlation coefficient for the dPSI effects was 0.34 with an associated p-value of 0.0192. The fraction of correctly called changes was over 50% with a p-value of 0.0001 (TNOM based test).

Finally, we assess TrASPr predictions for mutations introduced in a mini-gene reporter assay around a neural specific exon 16 in the mouse Daam1 gene. Similar to the ENCODE RBP analysis, we find TrASPr correctly predicts the effect of mutations in 7 out of 9 the cases (p-value 0.0012), as shown in Fig 5a. Here too, we find the model correctly predicts increased inclusion but the two mutations decreasing inclusion of exon 16 were not predicted correctly. We note these cases both involved region 11 (marked in red) which the model failed to capture.

6.3 Assessing BOS sequence generation

We used TrASPr as an Oracle for our BOS algorithm to generate AS event sequences with edit distances from an original sequence of no more than τ = 30 . First, we asked BOS to increase the inclusion levels of lowly included cassette exons from Figure 4b. From the generated 214 sequences with increased inclusion (dPSI>0.2), our BOS algorithm significantly increased PSI(dPSI>0.5) for 46 of them. Most of the mutations were introduced around the relatively weak splice sites surrounding these AS events, which made biological sense. Scanning for the known motifs we found BOS also generated 15 cases where the known RBP regulatory motifs (TIA1, PTBP1 or QKI) were mutated to increase inclusion. When assessing BOS on the daam1 exon 16 we again found many of the mutations increased inclusion by affecting the splice sites as expected (Figure 5b). However, zooming

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

2

Histogram of mutation index

Histogram of mutation index

20.0

20.0

6

Correct

17.5

5

Wrong

No Change

17.5

15.0

15.0

4

Percentage

12.5

Percentage

12.5

Count

10.0

10.0

7.5

7.5

5.0

5.0

1

2.5

2.5

0

△kD > 0.15

Ground truth direction

k < -0.15|k|≤ 0.15

0.0

0

200

400

600

27bps

800

27bps

1000

1200

1400

1600

0.0

-200

-175

5

-150

6

-125

8

-100

9 1011

75

-50

12

25

0

27bps

in on the upstream intron we found BOS frequently mutated the validated regulatory regions avoiding the region of small/little effect (green) and the area that caused decreased inclusion (red).

To assess the efficiency of BOS, we compared its efficay on reducing Ψ for a given sequence with a baseline method which randomly mutated 3, 6, 15 and 30-mers in different regions. We then calcualted how many of these mutations actually changed the PSI by at least 0.2 based on the TrASPr oracle. In the end, the best setting(30-mers) successfully generated 177 out of 4392 sequences(4.03%). BOS generated 12,066 successful sequences(dPSI>0.2) with only ∼ 40k trials(>25%), significantly outperforming the baseline. Overall, these preliminary results indicate that BOS is able to efficiently capture regulatory elements in a given sequence, including both splice site signals as well as deep intronic elements, then capitalize on those to generate sequences matching a given splicing target function.

7 Discussion

In this study, we offer two main contributions. First, we propose a new tissue specific splicing code model, TrASPr. TrASPr leverages recent advances in LLMs utilizing Transformer based architecture. The architecture of TrASPr allows it on one hand to benefit from the Transformer attention mechanism while at the same time, by utilizing several Transformers each focused on a specific region, keep the model's attention on areas most relevant for splicing regulation without resorting to extremely large models. We demonstrated TrASPr was able to significantly improve performance in both PSI and dPSI predictions on several datasets compared to previous state of the art. These included CNN based models as well as models utilizing expert derived regulatory features that were fed into a DL model.

The second contribution in this study is in formulating the design of RNA sequences with specific splicing characteristics as a Bayesian Optimization problem. We then proposed the BOS algorithm, which uses TrASPr as an oracle, to solve this design problem with biologically plausible mutations. The RNA design task can be leveraged for synthetic biology studies and for therapeutic design ( e.g. , which sequence to target with ASO therapy or with prime editing). We showed BOS can effectively propose sequences that exhibit the desired splicing changes, mutating both core splicing signals and intronic regulatory elements.

It is important to keep in mind that the labels used for assessing the prediction tasks presented here are inherently noisy and limited in number. For example, RNA-Seq quantifcations are noisy measurement, as are the RBP binding assays (eCLIP). The RBP regulatory motifs are crude as well. This means many targets might be missed while the changes upon RBP KD can be due to indirect affects ( e.g. , another RBP affected by the KD) or other sequence motifs. Thus, the work presented here should be viewed more as a proof-of-concept outlining exciting directions for future developments rather than a finished product. Specifically, combining the models we propose with high-throughput mutagensis experiments appears as an exciting direction to explore.

References

Yoseph Barash, John A Calarco, Weijun Gao, Qun Pan, Xinchen Wang, Ofer Shai, Benjamin J Blencowe, and Brendan J Frey. Deciphering the splicing code. Nature , 465(7294):53-59, 2010.

Susan M Berget, Claire Moore, and Phillip A Sharp. Spliced segments at the 5' terminus of 367 adenovirus 2 late mrna. Proceedings of the National Academy of Sciences , 74(8):3171-3175, 368 1977. 369

Nicholas Bogard, Johannes Linder, Alexander B Rosenberg, and Georg Seelig. A deep neural 370 network for predicting and engineering alternative polyadenylation. Cell , 178(1):91-106, 2019. 371

372

373

374

Hannes Bretschneider, Shreshth Gandhi, Amit G. Deshwar, Khalid Zuberi, and Brendan J. Frey. COSSMO: predicting competitive alternative splice site selection using deep learning. Bioinformatics , 34(13):i429-i437, July 2018.

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

Sebastian M. Castillo-Hair and Georg Seelig. Machine Learning for Designing Next-Generation mRNA Therapeutics. Accounts of Chemical Research , 55(1):24-34, January 2022. Publisher: American Chemical Society.

Jun Cheng, Muhammed Hasan Çelik, Anshul Kundaje, and Julien Gagneur. Mtsplice predicts effects of genetic variants on tissue-specific splicing. Genome biology , 22:1-19, 2021.

Louise T Chow, Richard E Gelinas, Thomas R Broker, and Richard J Roberts. An amazing sequence arrangement at the 5' ends of adenovirus 2 messenger rna. Cell , 12(1):1-8, 1977.

GTEx Consortium. The gtex consortium atlas of genetic regulatory effects across human tissues. Science , 369(6509):1318-1330, 2020.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner, and Matthias Poloczek. Scalable global optimization via local Bayesian optimization. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

David Eriksson and Matthias Poloczek. Scalable constrained Bayesian optimization. In 391 International Conference on Artificial Intelligence and Statistics , pages 730-738. PMLR, 2021. 392

Xiang-Dong Fu and Manuel Ares Jr. Context-dependent control of alternative splicing by 393 rna-binding proteins. Nature Reviews Genetics , 15(10):689-701, 2014. 394

Roman Garnett. Bayesian Optimization . Cambridge University Press, 2023. 395

Vladimir Gligorijevi´ c, Daniel Berenberg, Stephen Ra, Andrew Watkins, Simon Kelow, 396 Kyunghyun Cho, and Richard Bonneau. Function-guided protein design by deep manifold 397 sampling. bioRxiv , December 2021. 398

399

400

401

402

Rafael Gómez-Bombarelli, Jennifer N. Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sánchez-Lengeling, Dennis Sheberla, Jorge Aguilera-Iparraguirre, Timothy D. Hirzel, Ryan P. Adams, and Alán Aspuru-Guzik. Automatic chemical design using a data-driven continuous representation of molecules. ACS Central Science , 4(2):268-276, January 2018.

Mar González-Porta, Adam Frankish, Johan Rung, Jennifer Harrow, and Alvis Brazma. Tran403 scriptome analysis of human tissues and cell lines reveals one dominant transcript per gene. 404 Genome biology , 14(7):1-11, 2013. 405

Ryan-Rhys Griffiths and José Miguel Hernández-Lobato. Constrained bayesian optimization for 406 automatic chemical design using variational autoencoders. Chemical Science , 11(2):577-586, 407 2020. 408

Antoine Grosnit, Rasul Tutunov, Alexandre Max Maraval, Ryan-Rhys Griffiths, Alexander I. 409 Cowen-Rivers, Lin Yang, Lin Zhu, Wenlong Lyu, Zhitang Chen, Jun Wang, Jan Peters, and 410 Haitham Bou-Ammar. High-dimensional bayesian optimisation with variational autoencoders 411 and deep metric learning, 2021. 412

Benjamin C Hitz, Jin-Wook Lee, Otto Jolanki, Meenakshi S Kagda, Keenan Graham, Paul Sud, 413 Idan Gabdank, J Seth Strattan, Cricket A Sloan, Timothy Dreszer, et al. The encode uniform 414 analysis pipelines. bioRxiv , pages 2023-04, 2023. 415

416

417

418

419

Kishore Jaganathan, Sofia Kyriazopoulou Panagiotopoulou, Jeremy F McRae, Siavash Fazel Darbandi, David Knowles, Yang I Li, Jack A Kosmicki, Juan Arbelaez, Wenwu Cui, Grace B Schwartz, et al. Predicting splicing from primary sequence with deep learning. Cell , 176(3):535548, 2019.

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

Anupama Jha, Matthew R Gazzara, and Yoseph Barash. Integrative deep models for alternative splicing. Bioinformatics , 33(14):i274-i282, 2017.

Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana V Davuluri. Dnabert: pre-trained bidirectional encoder representations from transformers model for dna-language in genome. Bioinformatics , 37(15):2112-2120, 2021.

Thomas M Keane, Leo Goodstadt, Petr Danecek, Michael A White, Kim Wong, Binnaz Yalcin, Andreas Heger, Avigail Agam, Guy Slater, Martin Goodson, et al. Mouse genomic variation and its effect on phenotypes and gene regulation. Nature , 477(7364):289-294, 2011.

WJames Kent. Blat-the blast-like alignment tool. Genome research , 12(4):656-664, 2002.

Donghoon Lee, Jing Zhang, Jason Liu, and Mark Gerstein. Epigenome-based splicing prediction using a recurrent neural network. PLoS Computational Biology , 16(6):e1008006, 2020.

Kathrin Leppek, Gun Woo Byeon, Wipapat Kladwang, Hannah K. Wayment-Steele, Craig H. Kerr, Adele F. Xu, Do Soon Kim, Ved V. Topkar, Christian Choe, Daphna Rothschild, Gerald C. Tiu, Roger Wellington-Oguri, Kotaro Fujii, Eesha Sharma, Andrew M. Watkins, John J. Nicol, Jonathan Romano, Bojan Tunguz, Fernando Diaz, Hui Cai, Pengbo Guo, Jiewei Wu, Fanyu Meng, Shuai Shi, Eterna Participants, Philip R. Dormitzer, Alicia Solórzano, Maria Barna, and Rhiju Das. Combinatorial optimization of mRNA structure, stability, and translation for RNA-based therapeutics. Nature Communications , 13(1):1536, March 2022.

Yang I Li, David A Knowles, Jack Humphrey, Alvaro N Barbeira, Scott P Dickinson, Hae Kyung Im, and Jonathan K Pritchard. Annotation-free quantification of rna splicing using leafcutter. Nature genetics , 50(1):151-158, 2018.

Donny D Licatalosi and Robert B Darnell. Resolving rna complexity to decipher regulatory rules governing biological networks. Nature Reviews. Genetics , 11(1):75, 2010.

Johannes Linder, Nicholas Bogard, Alexander B Rosenberg, and Georg Seelig. Deep exploration networks for rapid engineering of functional dna sequences. BioRxiv , page 864363, 2019.

Natalie Maus, Haydn T Jones, Juston S Moore, Matt J Kusner, John Bradshaw, and Jacob R Gardner. Local latent space bayesian optimization over structured inputs. arXiv preprint arXiv:2201.11872 , 2022.

Henry Moss, David Leslie, Daniel Beck, Javier González, and Paul Rayson. Boss: Bayesian optimization over string spaces. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 15476-15486. Curran Associates, Inc., 2020.

Qun Pan, Ofer Shai, Leo J Lee, Brendan J Frey, and Benjamin J Blencowe. Deep surveying of alternative splicing complexity in the human transcriptome by high-throughput sequencing. Nature genetics , 40(12):1413-1415, 2008.

Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer School on Machine Learning , pages 63-71. Springer, 2003.

Alexander B. Rosenberg, Rupali P. Patwardhan, Jay Shendure, and Georg Seelig. Learning the Sequence Determinants of Alternative Splicing from Millions of Random Sequences. Cell , 163(3):698-711, October 2015.

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

Benjamin Sanchez-Lengeling and Alán Aspuru-Guzik. Inverse molecular design using machine learning: Generative models for matter engineering. Science , 361(6400):360-365, July 2018.

Jacob Schreiber, Ritambhara Singh, Jeffrey Bilmes, and William Stafford Noble. A pitfall for machine learning methods aiming to predict across cell types. Genome Biology , 21(1):282, November 2020.

Adam Siepel, Gill Bejerano, Jakob S Pedersen, Angie S Hinrichs, Minmei Hou, Kate Rosenbloom, Hiram Clawson, John Spieth, LaDeana W Hillier, Stephen Richards, et al. Evolutionarily conserved elements in vertebrate, insect, worm, and yeast genomes. Genome research , 15(8):1034-1050, 2005.

Ravi K Singh and Thomas A Cooper. Pre-mrna splicing in disease and therapeutics. Trends in molecular medicine , 18(8):472-482, 2012.

Barry Slaff, Caleb M Radens, Paul Jewell, Anupama Jha, Nicholas F Lahens, Gregory R Grant, Andrei Thomas-Tikhonenko, Kristen W Lynch, and Yoseph Barash. Moccasin: a method for correcting for known and unknown confounders in rna splicing analysis. Nature communications , 12(1):3353, 2021.

Christopher WJ Smith and Juan Valcárcel. Alternative pre-mrna splicing: the logic of combinatorial control. Trends in biochemical sciences , 25(8):381-388, 2000.

Samuel Stanton, Wesley Maddox, Nate Gruver, Phillip Maffettone, Emily Delaney, Peyton Greenside, and Andrew Gordon Wilson. Accelerating bayesian optimization for biological sequence design with denoising autoencoders, 2022.

Jorge Vaquero-Garcia, Alejandro Barrera, Matthew R Gazzara, Juan Gonzalez-Vallinas, Nicholas F Lahens, John B Hogenesch, Kristen W Lynch, and Yoseph Barash. A new view of transcriptome complexity and regulation through the lens of local splicing variations. elife , 5:e11752, 2016.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Eric T Wang, Rickard Sandberg, Shujun Luo, Irina Khrebtukova, Lu Zhang, Christine Mayr, Stephen F Kingsmore, Gary P Schroth, and Christopher B Burge. Alternative isoform regulation in human tissue transcriptomes. Nature , 456(7221):470-476, 2008.

Zefeng Wang and Christopher B Burge. Splicing regulation: from a parts list of regulatory elements to an integrated splicing code. Rna , 14(5):802-813, 2008.

Robin Winter, Floriane Montanari, Andreas Steffen, Hans Briem, Frank Noé, and Djork-Arné Clevert. Efficient multi-objective molecular optimization in a continuous latent space. Chemical Science , 10(34):8016-8024, 2019.

Hui Y. Xiong, Babak Alipanahi, Leo J. Lee, Hannes Bretschneider, Daniele Merico, Ryan K. C. Yuen, Yimin Hua, Serge Gueroussov, Hamed S. Najafabadi, Timothy R. Hughes, Quaid Morris, Yoseph Barash, Adrian R. Krainer, Nebojsa Jojic, Stephen W. Scherer, Benjamin J. Blencowe, and Brendan J. Frey. The human splicing code reveals new insights into the genetic determinants of disease. Science , 347(6218):1254806, January 2015.

H.Y. Xiong, Y. Barash, and B.J. Frey. Bayesian prediction of tissue-regulated splicing using RNA sequence and cellular context. Bioinformatics , 27(18):2554-2562, 2011.

Yungang Xu, Yongcui Wang, Jiesi Luo, Weiling Zhao, and Xiaobo Zhou. Deep learning of the splicing (epi) genetic code reveals a novel candidate mechanism linking histone modifications to esc fate decision. Nucleic acids research , 45(21):12100-12112, 2017.

Tony Zeng and Yang I Li. Predicting rna splicing from dna sequence using pangolin. Genome 505 Biology , 23(1):1-18, 2022. 506

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a tissue specific transformer based splicing prediction model, TrASPr along with a Bayesian Optimization algorithm, BOS, capable of designing RNA with desired properties. The authors start by demonstrating the performance of TrASPr on RNA splicing data from both mouse and human tissues. Next, the authors assess the model’s ability to detect condition specific regulatory elements using ENCODE data involving three RBP Knockdown (KD) in two human cell lines, and data for tissue-specific regulatory elements from a mini-gene reporter assay. At last, TrASPr is used as an Oracle for BOS to generate AS event sequences with desired properties and an evaluation of BOS performance is presented.\n\nSTRENGTHS:\n[-] Originality; This paper utilizes recent advances in large language models (LLMs) to define a splicing prediction model. While this is not the first attempt to use LLMs for nucleotide encoding, the authors improved existing models and incorporated existing prior knowledge through dedicated features. Furthermore, the authors use the model as an oracle for a generative model for RNA splicing design. \n\n[-] Quality; The paper is well-written and presented. The framework seems well thought through; combining both expert prior knowledge regarding the problems tackled and state-of-the-art computational models. \n\n[-] Clarity; The paper is self-inclusive, presenting the reader with all necessary information from the background regarding the biological problem, its complexity, and motivation regarding its importance. Following that, all framework parts are clearly presented.\n\n[-] Significance; This work provides promising results towards utilizing LLMs for predicting splicing events and further using such these to train an RNA design model. While this work is only the initial step towards obtaining a robust, reliable framework that solves this task it is of great significance as it advances the field.\n\nWEAKNESSES:\n[-] Reproducibility code; the authors claim for reproducibility however no code was provided. Providing the code could improve the understanding and evaluation of the presented framework.\n\n[-] Structure; the paper is generally well structured in providing all necessary information however the organization could be improved. For example, the background section already contains model implementation details and prior attempts are split between the introduction and related work. In line with the latter, it would be beneficial if the authors could provide a more elaborate description of previous work, specifically for the prediction task. \n\n[-] Evaluation; the author’s explanation regarding the degradation in performance over the “Strict” test set is not convincing, and following the rationale behind the necessity of the “strict” test set raises some concerns. It would be insightful if the authors could look more into this point, potentially testing on alternative data to obtain a better understanding of this. \n\n[-] minor (these are provided to improve the manuscript’s readability);\n\n[--] incomplete/unreadable sentences; a few sentences in the text are incomplete or unclear (e.g. line 78, lines 256-259)\n\n[--] Space after TrASPr is omitted in many places in the manuscript (probably as a result of latex macro).\n\n[--] Figure 1; Figure 1a is never referenced (and its components are hence not explained), the relationship between 1b-1c could be depicted better (in line with many transformer visualization models).\n\nQUESTIONS:\n1. Could the authors provide a code base for evaluation?\n2. Could the authors extend the analysis on differences in performance over the MGP dataset or provide an additional example?\n2. Can the authors suggest an alternative baseline, similar to DEN but which can be evaluated in the same setting, to compare the performance of the design class?\n\nLIMITATIONS:\nYes, limitations have been addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors develop a new framework to predict alternative splicing of RNA. They then deploy it with adaptations and Bayesian Optimization to design new sequences.\n\nSTRENGTHS:\nI find the validation using the RBP KD experiment interesting. It is great that the knowledge of the biological system can be used to inform your computational experiments.\n\n“We repeated this process 5 times with different random mutations and the prediction results where then averaged and compared to the wild type (WT) sequence prediction.“ - Good to do lots of permutations!\n\nGood to try to remove batch effects with ENCODE data, but going to be hard.\n\nLevenshtein distance between designed sequences is good!\n\nFigure 2 comparison to Pangolin is pretty good and convincing\n\nWEAKNESSES:\nTable 1 results of rAUPRC and AUROC are confusing. Can the “feature” and “Model” terms be a bit better defined?\n\nFigure 4b is a bit confusing, and I feel like we need a bit more context. Should things be above or below the line? Can we have a legend for the figure as well?\n\nBO part of paper barely discussed, required changes to core algorithm, and not validated–I would remove. Moreover, the baseline algorithm to compare BO (random mutation) is not a sufficient baseline. What about evolutionary strategies? What about Gibbs sampling?\n\nQUESTIONS:\n“...∼100bp long reads that are mapped back to the genome using 70 dedicated tools (e.g., STAR).”- Citation please?\n\nSection 2.2 What are the “...classification 90 ([CLS]), padding ([PAD]), separation ([SEP]), mask ([MASK]) and unknown ([UNK])....” characters, or what do they mean? Perhaps they are discussed more in detail in the paper, but their relative importance and disambiguation should be described when they are introduced\n\nTo what extent is there synteny between mouse and human chromosomes on the held out set? For human chromosomes 8 and 14, and mouse 4 and 11, it seems like there will be some overlap in homologous sequences in the training and test sets. How many sequences are then removed during the BLAT step? Are the fitlers strict enough? Should BLAT just be done on exonic regions, because they are more conserved?\n\nLine 280 - Should be Fig 3 and Table 1\n\nSection 6.2 - “... To achieve this, we randomly mutated sequences in the same set of exons, selected from the same distribution of distances as the original motifs…” What does distance refer to here? Linear genome distance? Levenstein distance?\n\n“We then ’removed’ the effect of these RBPs on the set of AS targets by randomly mutating the identified binding motifs.” What is random here? Did you preserve GC content? Could you just permute the bases?\n\nWhat is figure 5b trying to show? Is the y axis the number of times a mutation shows up during optimization? Also, is this the figure you are referring to in line 322, where you say figure 4b)?\n\nLIMITATIONS:\nIt is unclear how much the BO section is needed. With a sufficient predictor, do evolutionary strategies work?\n\n“Specifically, we only…” Background section is not complete…sort of just trails off.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new machine learning framework called TrASPr, which is a transformer-based architecture with pretrained RNA language models that is tailored for the prediction and design task of RNA alternative splicing. The authors demonstrate that TrASPr accurately predicts tissue-specific `percent spliced in’ (PSI) scores, and the trained model can facilitate RNA design for specific RNA splicing outcomes.\n\nSTRENGTHS:\n- Overall, the authors introduce and explain the problem of alternative splicing, its significance, and the dataset they used to study this problem quite well for a general reader at a machine learning conference.\n- The evaluation setting is generally rigorous, as the authors carefully curated the test set to avoid any information leakage.\n\nWEAKNESSES:\n- Some additional work and its relationship to this research should be discussed, such as \"RNA Alternative Splicing Prediction with Discrete Compositional Energy Network,\" which also focuses on the prediction of PSI scores in a tissue-specific setting.\n- When evaluating the effect of RBP KD and mutations, the authors first identify a set of RBP motif matching sites that might affect alternative splicing and then check if their model can accurately predict those sites through in-silico mutations. However, this measurement only assesses the model's ability to recover positive samples. The authors should also evaluate and present examples of in-silico mutations on non-regulating motif matches and demonstrate their models' predictions. This is important to show that the model is genuinely learning context-dependent sequence features and not just memorizing motif matches.\n- While the formulation of the RNA design problem and the utilization of language models for RNA sequences can be considered novel in the field of RNA splicing, similar techniques have been introduced and used in protein sequence design and protein language models before; it would be nice to discuss some of those (e.g., for a review, see https://doi.org/10.1016/j.cels.2021.05.017), perhaps in Related work.\n\nQUESTIONS:\nQuestions: from line 83-87, there are some empirical results, such as “we considered different tokenizing strategies of RNA sequences which are composed of 4 types of ribonucleotide bases (‘A’,‘C’,‘G’,‘U’). We settled on overlapping k-mers of length 6,” and “However, we found the DNABERT architecture to be unstable and opted to pre-train a lighter BERT model with only six layers as describe below.” Are there results or analysis supporting these decisions and claims?\n\nSome suggestions:\n- It would be helpful to include a subfigure when introducing the concept of alternative splicing, splicing junction, etc. The authors make good efforts in the main text for getting ML audience to be familiar with the concepts, it would be easier if there are figures or illustrations involved.\n- Some typos: e.g. line 9 “on Oracle” -> “an Oracle”, line 278 “Autoencodeer” -> “Autoencoder”\n\nLIMITATIONS:\nThe authors discussed the limitations of the noisy labels obtained from biological experiments and concluded that this work \"should be viewed more as a proof-of-concept outlining exciting directions for future developments rather than a finished product.\" It would be helpful if the authors could comment on how many datasets exist and are expected to be generated, and whether these limitations would be addressed with more data or more careful model design.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles two tasks in alternative splicing of pre-mRNA, where multiple unique mRNAs are produced by including different segments. First, the authors proposed a transformer-based splicing prediction model, TrASPr. A 6-layer transformer model is pre-trained with 1.5M pre-mRNA sequences centered in splice sites. TrASPr utilizes multiple pre-trained transformer encoders for the splicing prediction in a tissue-specific manner. Second, the authors used TrASPr as an Oracle to train and evaluate splicing sequence design based on the Bayesian Optimization algorithm. Through the experiment results, they argue that TrASPr significantly outperforms state-of-the-art models and BOS can generate sequences with desired characteristics.\n\nSTRENGTHS:\n-\tThe authors tackle important problems in RNA biology. In particular, they argue that the sequence design for RNA splicing is a novel task and it can benefit therapeutics for correcting splicing defects and synthetic biology.\n-\tTo tackle the problems, they adopt a couple of machine learning methods that have shown successes in other domains: pre-training and fine-tuning of language models for biological sequences, and latent space Bayesian optimization (LSBO) over structured and discrete inputs.\n\nWEAKNESSES:\nMajor comments:\n- [Originality] While the methods are novel for their first use for RNA alternative splicing, they still seem like mostly direct applications of widely known machine learning methods. For example, pre-training and fine-tuning of language models seem trivial even for the biological sequences. As referenced in the paper, DNABERT proposed a pre-trained language model for DNAs. There are also plenty of previous works that use pre-trained language models for various protein biology tasks.\n- [Quality] While the paper includes a couple of experiment results, I do not think they are sufficient to verify the effectiveness of the proposed methods. It lacks strong baseline models for comparison and ablation studies for thoroughly understanding the proposed methods.\n- [Significance] The paper does not bring significant and novel ideas that would be valuable to the broader NeurIPS community. \n- [Clarity] I don’t think this is the best version of the paper, considering the broad NeurIPS community is not familiar with computational biology. The explanations are not detailed enough to easily understand the problem and significance of the experiment results. \n\nMinor comments:\n- Typo in L10: on Oracle -> an Oracle\n- Sec 2.1: Incomplete. It suddenly ends with “Specifically, we only.”\n- Sec 7: The authors mostly use the Discussion section for summarizing their contributions rather than discussing notable observations, limitations, and future directions. (except for the last paragraph where they discuss the inherent limitation of experiment data)\n\nQUESTIONS:\n-\tHow well is the pre-training conducted? Can you provide the training curve and evaluation (e.g. perplexity) of the pre-trained model? How much does it differ from the DNABERT model?\n-\tHow do you include conservation values for each k-mer for the Transformer model? Since the conservation values are not considered in the pre-training, the inputs for the Tranformer encoder will be different from the pre-training and possibly incur out-of-distribution problems.\n-\tDo the Pangolion and TrASPr share the same training data? If not, is it possible to compare the TrASPr with the Pangolion trained with the same training data? It would more clearly show the effect of the proposed methods.\n-\tHow important is the pre-training? Is it possible to use the pre-trained DNABERT instead? How does the model perform with a randomly-initialized Transformer model? How important is the 6-mer tokenization compared to 1-mer tokenization?\n-\tHow important is the Transformer architecture? How does the model perform with other model types?\n-\tHow important are the event features?\n-\tIt seems TrASPr is used as an Oracle for both training and evaluation of the sequence design. Wouldn’t it produce over-optimistic results?\n-\tHow important is the LSBO? Can you provide comparisons with other baseline methods?\n-\tThe explanations of the biological problem and interpretations of the experiment results should be easier and more intuitive to understand. In addition, more detailed and friendly backgrounds should be included as supplementary.\n\nLIMITATIONS:\nThe authors adequately addressed that the experiment data are inherently noisy and limited in number, such that this work should be viewed more as a proof-of-concept rather than a finished product. \n\n\n---Post-Rebuttal Comments---\nOverall, I am inclined to believe that incorporating the authors' responses would indeed enhance the manuscript's quality. Consequently, I have adjusted my rating from 3 to 4. Upon reviewing the comments from other reviewers and the authors' clarifications, it seems I'm not the only one who has struggled to understand the authors' contributions and has concerns about the presentation, especially regarding the BO for sequence design. This suggests that substantial revisions are needed beyond the inclusion of additional experimental results. Although the authors' responses have been comprehensive, I could not give a higher score as I respectfully believe resubmission after revision is more appropriate for this manuscript.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-24"}

comments: SUMMARY:
The authors propose two approaches to deal with the problems of alternative splicing (AS).  A transformer architecture-based tissue-specific splicing code model, TrASPr, and a Bayesian Optimization algorithm were performed on the latent variable space of VAE to address the design of RNA sequences with specific splicing characteristics.  The architecture is not so novel, but applying LLM to the AS is worth noting.

STRENGTHS:
The proposed methods are exciting and computationally novel.  Using BERT with masking to pre-train the model was a nice touch.  Using VAE with Bayesian Optimization is interesting.

WEAKNESSES:
The proposed method is a nice modeling experiment exploring using LLM on a novel application. However, it lacks reliability as a tool to deal with biomedical problems that can be used for biological research.   The evaluation the author provides shows that the method fails.  The authors mention that TrASPr significantly outperforms AE-MLP in a particular situation but also point out that the performance degrades based on the filtering criteria.  There also seems to be a performance difference in BOS sequence generation based on the edit distance, a hyperparameter that lay users would not know how to tune on their particular problem.

Minor comment:
Line 78 is incomplete.

QUESTIONS:
Which version of MAJIQ did you use?

LIMITATIONS:
The proposed method has a potential to be a hypothesis generation method.  However, it lacks the biological soundness and it is unclear how to pinpoint the problem when they arise.  Authors are very casual about the evaluation and the problems that arise when the hyper parameters are chosen inappropriately.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-25