Probabilistic Transformer: Modelling Ambiguities and Distributions for RNA Folding  and Molecule Design
Authors: Jörg Franke
Frederic Runge
Frank Hutter
Venue: NeurIPS 2022 Accept (2022)
Keywords: Transformer
RNA folding
Molecular Design
Probabilistic Sequence Modelling
CVAE
Variational Inference
Hierarchical Distribution
ELBO
GECO

Abstract: Our world is ambiguous and this is reflected in the data we use to train our algorithms. This is particularly true when we try to model natural processes where collected data is affected by noisy measurements and differences in measurement techniques. Sometimes, the process itself is ambiguous, such as in the case of RNA folding, where the same nucleotide sequence can fold into different structures. This suggests that a predictive model should have similar probabilistic characteristics to match the data it models. Therefore, we propose a hierarchical latent distribution to enhance one of the most successful deep learning models, the Transformer, to accommodate ambiguities and data distributions. We show the benefits of our approach (1) on a synthetic task that captures the ability to learn a hidden data distribution, (2) with state-of-the-art results in RNA folding that reveal advantages on highly ambiguous data, and (3) demonstrating its generative capabilities on property-based molecule design by implicitly learning the underlying distributions and outperforming existing work.

Probabilistic Transformer: Modelling Ambiguities and Distributions for RNA Folding and Molecule Design
Jörg K.H. Franke', Frederic Runge', Frank Hutter
1 Department of Computer Science, University of Freiburg, Germany Bosch Center for Artificial Intelligence, Renningen, Germany {frankej. funget,fhh0ce uni-freiburg.ce
Abstract
Our world ambiguous and this reflected in the data we use train our algorithms. This is particularly true when we try to model natural processes where collected data is affected by noisy measurements and differences measurement techniques. Sometimes, the process itself is ambiguous, such as in the case of RNA folding, where the same nucleotide sequence can fold into different structures. This suggests that predictive model should have similar probabilistic characteristics match the data models. Therefore, we propose a hierarchical latent distribution to enhance one of the most successful deep learning models, the Transformer, to accommodate ambiguitics and data distributions. We show the benefits of our approach (1) synthetic task that captures the ability to learn hidden data distribution, (2) with state-of-ther -f-ee-arr-t--tttt results in RNA folding that reveal advantages highly ambiguous data, and (3) demonstrating its generative capabilities property-basee based molecule design by implicitly learning the underlying distributions and outperforming existing work.
Introduction
Transformer models [1] are the architecture of choice for many applications. Next to wide range of NLP applications, such as language modelling [2, 3, 4] and machine translation [1, 5], they are also very effective in other disciplines, such as computer vision [6, 7], biology [8, 9], and chemistry [10, 11]. An additional challenging application for which transformers are promising is RNA folding, where the goal is to model a secondary structure (represented in dot-bracket notation [12]) based on a given sequence of nucleotides. RNA data highly ambiguous since is collected with different techniques. resolutions, protocols, and even contains natural ambiguities since there exist multiple structures for the same RNA sequence in the cell [13, 14] and the same structure can be caused by multiple sequences. Similar to RNA structures, molecules can be represented sequences by using the simple molecular line entry system (SMILES) [15] and transformers arise as the architecture of choice in the flourishing field of molecule design [11, 16).
Although Transformers have outstanding performance, the deterministic core of the architecture could harm performance in real-world applications like RNA folding. If collected data contains noisy labels or ambiguous samples, a vanilla Transformer model can only express uncertainties in the softmax output but not in the latent space. When sampling, sequential interdependencies can only be modelled in a decoder setting but not in an encoder-only setting. We address these limitations by proposing Probabilistic Transformer' (Prob Pransrorsformrr) that models hierarchical latent distribution and performs sampling in the latent space. The ProbTranfformer can represent ambiguities in these
Source code available at ggthuub.cm//attoo///oroooooooeot
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

distributions and refine the sampled latent vectors within the computational graph. This is in line with recent findings in cognitive science that suggest that the human brain both represents probability distributions and performs probabilistic inference [17, 18). Our approach is based on the idea of combining the transformer architecture with conditional variational auto-encoder (cVAE) [19], but in hierarchical fashion similar to the hierarchical probabilistic U-Net [20]. Therefore, we introduce a new probabilistic layer and incorporate it after the attention and feed-forward layer (Section 3). In this way, we preserve the global receptive field through the attention mechanism and remain independent of other enhancements in the transformer ecosystem [21], I. To train the latent distributions, we make use of the generalized evidence lower bound (ELBO) with constrained optimization (GECO) [22] and introduce an annealing technique to adapt a hyperparameter online.
We see our contributions in three aspects:
The introduction of the ProbTransformer, novel hierarchical probabilistic architecture enhance-
ment to the Transformer ecosystem. Our training procedure using GECO, the analysis of the sensitivity of its hyperparameter K, and the introduction of the online adaption technique kappa annealing which could be beneficial for variational training with ELBO in general.
A comprehensive empirical analysis that verifies the ProbTransformer's 's capability to learn and recover data distributions on a novel synthetic sequential distribution task, assesses its capability of handling data ambiguities in practice by achieving state-of te--hhe-art art performance in RNA folding, and demonstrates its generative character by outperforming existing work in Molecule Design.
We first clarify notation and recap the cVAE [19] (Section 2), and then introduce the PronTransformrr (Section 3). We then present our experiments on the sequential distribution task (Section 4.1), RNA folding (Section 4.2), Molecule Design (Section 4.3), as well as our ablation study (Section 4.4), discuss related work (Section 5) and conclude (Section 6).
2 Background
Transformer Notation The Transformer [1] is self-attention based requence-to-sequeee model introduced as an encoder-decoder architecture. However, both encoder-only and decoder-only versions are very successful by themselves [2, 23], and in our work, we focus on either of these two versions. An encoder or decoder has N blocks and each block consists of a multi-head (masked) attention followed by a feed-forward layer. Around both of these layers there are residual connections [24] followed by layer normalization [25]. We use the parameterization with moddet dimensions in the residuals and attention, H heads per attention, Djs latent dimensions in the feed- word-forward layer, and N for the blocks in the Transformer. In addition, we define (= , Ts) as the input
sequence of length S, (z Us) as the target sequence, and (fr,...... .9s)) as the predicted sequence.
Conditional Variational Auto-Encoder (cVAE) The cVAE [19] is deep conditional generative model and an extension of the Variational Auto-Encoder [26, 27). During inference, the cVAE aims input More specifically, given input
to generate a distribution for an output y conditional x, a latent variable z is drawn from a (conditional) prior model pa (z]x)) and used as an additional input to the generation model (y |x, 2) in order to generate the prediction y. The cVAE trained by minimizing the negative evidence lower bound (ELBO), LELBO:
CELBO y; P, +/) Egport lo x, ))ot)) + (g
pa (z]x))
(I)
where qu (z)x, y) describes the posterior model (called "recognition network" in [19]). All models are neural networks, and the prior and posterior models each output mean and variance of a Gaussian
distribution which represents the distribution of the latent z. During training, post qu (z)xx y) is sampled from this posterior model and used as input to the generation model (y|xx 2). whose output ý is compared to the ground true with cross-entropy loss. This objective training time can be viewed as a reconstruction task due to the target sequence input to the posterior, which is an easier task than prediction. The Kullback-Leibler (KI.) divergence term (xI .) aims to align the (conditional) prior model pa (z|x) and the posterior model (,,(2)x, y). The two losses are added and then used to train the prior, posterior, and generation models jointly, employing the reparameterization trick [28].
2

---

3 Probabilistic Transformer
Building on concepts of the vVA,, we enhance the Transformer model to a probabilistic Transformer (ranforanformmrr by adding probabilistic feed-forward layers (prob layer) to M E {1..... .N of the existing blocks. We introduce our approach for an encoder-only model simplify notation. also applies to the decoder-only model but not directly to an encoder-decoder model because our training setup requires the alignment of the source and target sequences in the posterior input, see Section 3.2. Whether all or only a selection of blocks are enhanced with prob layers is design decision and examined empirically in Section 4.4. In a block, we place the prob layer after the attention and feed-forward layer. The prob layer parameterizes multivariate Gaussian distribution ((, Σ) with diagonal covariance Σ = I where is the identity matrix, used to sample a latent vector z RRD: for each position in the sequence. We denote by X and Y the input and target sequences of length S, by mm.............................. -
Zm the position- wise sampled sequence of latent distributions in block m, by Z=(Z1..... ZM) all sequences of latent distributions of all blocks, and by Zem (Z1 mm- all sequences of latent distributions of all blocks before m. At inference time, we sample Zm and add it to the computation graph of the current block, similar to sampling from the (conditional) prior in the cVA.. However, in contrast to the cVEE, our sampling is conditioned hierarchically; the latent realization s at block m and sequence position depends all previously sampled latent vectors of any position in previous Transformer blocks due to the Transformer architecture:
Zyn ~ pl om Zem: X).
(2)
A sequential relation between the positional distributions is achieved by the attention mechanism: while the samples Z,,s are drawn independently for each position {I...., the following attention operations relate them and turn the position-wise samples into a joint distribution over sequences refined in higher blocks.
As result of the hierarchical composition of the ProbTransformer and the sequential relation, prior model po Zm Zm: X) and the generation model p, (Y]X; Z) effectively become single predictive model P& X), which differs from the modeling strategy of the e eVAE. This model approximately marginalizes over latent variables Z: (Y Jz Po X) PY Z) dZ, and we can sample from it by hierarchically sampling p X)) decomposed as
Pa = p (M M, (Zo
(3)
followed by sampling from Po (Y|X, Z). At inference time, we can sample different predictions from the predictive model Y PY X)..
However, we can also use the mean of the respective (Gaussian) distri- butions instead of sampling from them. We denote this as mean inference in contrast to sample inference. In the following, we introduce the prob layer architecture detail before explaining our training setup and the learning objective.
Output
Linear
Distribution
μ
a
3.1 Position-wise Probabilistic Feed-Forward Network
Linear
Similar to the feed-forward layer. the prob layer is applied to the latent representation of each position X independently. Figure I provides a visual description. It consists of linear layer that transforms from the model dimension Dmodet to the distribution dimension D. of the probabilistic latent space, followed by an activation function. Further, two linear layers with D. x D, weight matrices generate the latent representation of a conditional Gaussian distribution with mean Mm,s R and log variance2 log σ E DRD
Activation
{I...., S} of the input sequence
Linear
Input
Figure 1: Probabilistic
Feed-Forward layer.
We use log o to avoid negative variance values and for numerical stability. We obtain the variance oi. by 2 logo
3

---

for the prob layer in block m at position s:
Mm.s Linear (Act(Linearaa (Xm,,)))
(4)
log o 2 m, Linear a m (Act(L.ineay in mi (Xm.e)
(5)
Zm N (mmm p m. 1) = p(zm, Zem: X)
(6)
Ym,a mi (Zm,e)
(7)
As mentioned before, during sample inference, we can sample from the distribution (Equation 6) and during mean inference we use - An additional linear layer is used to compute the layer's output ym.= E -Rodeee (Equation 7). Similar to the attention and ferd-forward layer, we employ residual connection [24] followed by layer normalization [25].
3.2 Training Setup and Learning Objective
We optimize the ELBO as the standard practice for conditional training [19]. This requires variational posterior 2 [Z] XX Y) that depends on both the input sequence and target sequence Y. We model this posterior with separate ProbTransformer with the same architecture as the predictive model except for additional input embedding for the target sequence Y and no output generation layers, since we are only interested in the latent sampling Z and not in the actual model output. During training, we first run the posterior model and sample the latent Z Q In a second step, we run the predictive model but use the latent realization Zpost instead of sampling an own Zm in each prob layer m. Figure illustrates this training setup.
Target
CE-Loss
Prediction
Predictive Encoder
Posterior Encoder
Probabilistic Forward
Probabilistic Forward
Sample
Posterior Distribution
Distribution
Distribution
a
μ
KL-Loss
Feed Forward
Feed Forward
Self-Attention
Self-Attention
Prior
Distribution
Source
Source
Target
Figure 2: Training setup of the ProbTransformer The predictive and posterior encoder trained jointly.
The negative ELBO loss LELBO (Equation 1) composed of reconstruction loss Lree and KI. divergence De between the prior distribution Ps conditioned on the latent zoost and the posterior distribution Q The reconstruction loss is the cross-entropy between the predicted sequence Ý and the true sequence Y while using the latent post from the posterior model:
Cree Ezrort log ( X, Z)oet))
(8)
and DKL is the sum of KL divergences between the hierarchically decomposed distributions and Qs at each prob layer m for all positions
DKL XΣ> DRL. (q (2p06) Zpost X, Y) (Zm,, Zpostt X))
With the default LELBO loss, we discovered instability in the training and performance issues, even with weighting factor 3 [29]. Therefore, we follow [20] to avoid convergence issues with the classic

---

ELBO objective and make use of the Generalized ELBO with Constrained Optimization (GECO) objective [22], optimizing the Lagrangian
LGECO X(Cree - k) + DKL.
(10)
The Lagrange multiplier À balances the terms and is initialized to 1, added to the learnable parameter space and updated as function of the exponential moving average of the reconstruction loss [20]. In the beginning of the training, there is high pressure on the reconstruction loss due to an increasing X.
Once the desired reconstruction loss Lree is reached, the pressure moves to the KL-term. Please find more information about the training dynamics in Appendix A.
set to constant in the original work of [22], but this is problematic since it is sensitive hyperparameter due to the training dynamics. Specifically, if x is chosen too small, possible failure modes include that Free never reaches it (with the pressure staying only the reconstruction loss), that K will be reached by over-fitting, or that the posterior distribution collapses which leads to high Dxi (Qw Φ P₂) and destabilizes the training. On the other hand, if x is chosen large, the model can underfit or the reconstruction loss drops below leading to a negative loss value and harming the training success significantly. To address this issue we introduce kappa annealing and adjust during training. Let Le K 2
o Cree(Xx; Y₂)= be the mean constrained difference for K
training samples in one epoch of training or a defined number of update steps. In our experiments, we find that initializing x slightly larger than the optimal CE loss value and updating every epoch with
K +
Le
Le < and S
K =
(11)
otherwise
leads to more stable training behaviour, improved final performance, and reduces the need for expensive tuning of k.
4
Experiments
We demonstrate the benefits of the ProbTransforme in three experiments: (1) using a novel synthetic sequential distribution task, we show the advantages of distributions in the latent space over the standard softmax output distribution of a vanilla Transformer. (2) We show the benefits of the ProbTransformer in dealing with ambiguous data in RNA folding. (3) We use the ProbTransformer to
improve property-based Molecule Design by sampling from the latent space instead of softmax- output. In an additional ablation study, we provide insights on the impact of kappa annealing and hierarchical prob layers.
4.1 Synthetic Sequential Distribution Task
In real-world generation tasks, the true distribution is not available and is only implicitly accessible by a dataset. To provide insights on the quality of distribution learning, we created a synthetic sequential distribution task and compare the ProbTransformer two sampling methods of a vanilla Transformer.
Data We design the task to map a sequence of tokens from a source vocabulary x E VI to sequence of target tokens from a target vocabulary y E V with the same length. The tokens in the source sequence are used to build "phrases" TP. Each phrase exists of 7 tokens sampled with replacement (similar the combination of words to phrases in sentence). For each source token in each phrase, we randomly generate a unique distribution p(y|æ, IP) over the target tokens depending on the current phrase. Further, we design the distribution sparsely so that no more than k tokens from the target vocabulary have non-zen nonz probability. The training data consists of source and target sequence pairs with input sequences sampled with replacement from all phrases and target sequences sampled from the corresponding distributions.
Setup We use an encoder-only ProbTranm Toansssffrrmrr model and enhance each block with prob layer. We configure the task and the model to run on one GPU within few hours. Please find more information about the task and the configuration in Appendix B. For the inference of the vanilla Transformer use MC dropout (using dropout during inference time, based on [30]) and sampling from the softmax distribution. For the inference of the ProbTransformer we sample from the predictive model and use the token with the highest output value. We generate 50 realizations per sample to create a predictive distribution of the target vocabulary.
5

---

Results We measure the generative perfor- mance of model or sampling method with two metrics: The Validity describes the percentage of predicted tokens whose probability in the true distribution is not zero. Second, we measure the KL-divergence between the sampled distribution and the true distribution since sampling should reproduce the true target token distribution. As shown in Table 1, the ProbTransformer outper-
Table The mean measures of five random seeds in the synthetic task.
Model
Validity
KL-div.
ProbTransforme
0.99
0.52
dropout
0.93
12.71
Transformer
softmax
0.73
7.84
forms the MC dropout and softmax sampling, demonstrating its strong performance in probabilistic sequence modelling. Please note that this task does not consider interdependence sampling, where the sampling of one token in a sequence depends on the realization of others, while the next task on RNA folding does.
4.2 RNA Folding
An RNA's structure influences its function drastically [31] and the functional importance of RNA arguably is on par with that proteins [32]. Since cellular RNAs typically have extensive secondary structures but limited tertiary structures [33], RNA folding is typically modeled as function : + r where T and I' denote sequence and secondary structure alphabets, respectively. However, this is simplified view the RNA folding process, which ignores the fact that RNAs alter their structures dynamically, resulting in an ensemble of structures occurring with different probabilities [13, 14]. Further, real-world applications often require structure analysis of very similar sequences [34], sometimes folding into the same secondary structure. These ambiguities can hardly be captured with common approaches and ambiguous data is often removed for a better training result [35, 36, 37]. We address these issues with probabilistic modelling using our ProbTransformer and by keeping ambiguous training data, while implicitly measuring overfitting by explicitly removing similarity to the test data.
Data We collect large data pool of publicly available datasets from recent publications [38, 39, 37, 40, 41] and split the predefined validation and test sets, VLO and TSO [37], from the pool. We derived separate testset, TSsameStruc, from 149 samples of TSO that share the same structure with least one other scquence in TSO and uniformly sampled a disjoint set of 20 sequences with more than one annotated secondary structure from the remaining pool produce an ambiguous testset, 7SsameSeq. Samples without pairs and with a sequence similarity greater than 80% to the test and validation sets were removed from the training pool. However, in contrast to previous work [37, 39, 42], we kept all remaining samples for training to capture ambiguities and the influence of small sequence changes on the structures. The final data consists of 52007 training and 1299 validation samples, and 1304 samples in the testset TSO and 46 samples in the testset T'SaamSeq. We observe 48092, 1304, and 20 unique sequences with up different structures for a single sequence, and 27 17179, 1204, and 46 unique structures with up to 582 different sequences for single structure in the training, TSO, and TSsameSeq sets, respectively. This indicates that many sequences map to the same structure and different structures to the same sequence. We refer to Appendix C.1 for more information about the data.
Setup We use a 6-block encoder-only architecture with D Dmndel of 512 for the vanilla Transformer and ProbTransfommer. We train them for 200 epochs 1M training steps. We compare them against the state-of-the-art deep learning algorithms, SPOT-RNA [37], MXFold2 [39], and a commonly- used dynamic programming approach, RNAfold [43] on the testset TSO and its subset TSsameStruc using mean inference. To analyse the capabilities of the Trobfrosffommer to reconstruct structure distributions, we also infer the model 5, 10, 20, 50, and 100 times using sample inference on TSsameSeq, and compare against three commonly-used algorithms specialized on the prediction of structure ensembles via sampling from the Boltzmann distribution based on experimentally derived thermodynamic parameters: RNAsubopt [43], RNAshapes [44], and RNAstructure [45]. In contrast to previous work [37, 41, 39, 42), we do not use ensembling nor limit the set of accepted base pairs or secondary structures for post-processing, but independently train a CNN head on the output of the ProbTransformer that maps the string representation to an adjacency matrix representing the structure for evaluations on TSO and TSsameStruc. Although we are aware of the problems with exact evaluations via F1 Score due to the dynamic nature of RNA structures [46], we report F1 Score for reasons of comparability to previous work; however, we further report the number of solved structures

---

(the ultimate goal of the task), and the average Hamming distance to achieve better measure of distance. For more details on the setup, CNN head, and metrics, we refer Appendix C.2.
Structure
ProbTransformr
SPOT-RNA
MXFold2
Figure 3: Example predictions of deep learning-basee approaches for challenging RNAs from TSO. We show (top) Group II catalytic intron (RF02001) ) and (bottom) M-box riboswitch (RF00380).
Table 2: Structure fidelity of different RNA folding approaches TSO and TSsameStrue. For the ProbTransformer and the vanilla Transformer we show mean results of three random seeds.
TSO
TSsameStruc
Model
Score
Hamming
Solved
Score
Hamming
Solved
ProbTransformer
62.5
27.4
0.118
93.2
3.2
0.550
Transformer
50.5
35.3
0.084
89.5
4.6
0.481
SPOT-RNA
59.7
39.6
0.005
78.0
14.6
0.020
MXFold2
55.0
42.1
0.014
74.6
17.1
0.067
RNAFold
49.2
48.0
0.008
59.2
25.5
0.020
Results Table 2 summarizes the results of all approaches on TSO and TSsameStruc. Both attention- based approaches generally achieve strong results, but we observe that the ProbTransformer outper- forms the vanilla Transformer across all measures both sets. The ProbTransformer achieves the best performance in terms of Fl Score, solves more than eight times the structures compared to the next best approach (MXFold2) on both sets and achieves four times lower Hamming distance TSsameStruc compared to SPOT-RNA, indicating that it has learned to handle ambiguous sequences. Our model is further capable of accurately reconstructing challenging structures, exemplarily shown for two structures that contain long-range interactions in Figure A functional important class of base pairs pseudoknots [47, 48), non-nested base pairs present in around 40% of RNAs [41] that are overrepresented in functional important regions [49, 48] and known to assist folding into 3D structures [50]. While RNAFold and MXFold2 cannot predict this kind of base pairs due the underlying nearest neighbour model, Figure 3 as well as further results shown in Appendix C.3.2 suggests that the PronTransformer can predict pseudoknots more accurately than SPOT-RNA if these are contained in the structures (Figure 9) and further rarely predicts pseudoknots if the structure is nested. However, detailed analysis on the quality of pseudoknot predictions would be out of the scope of this work. When inferring the model multiple times on TSsameSeq. the FrobTransfomma is the only model our evaluation that could reconstruct more than one structure for given sequence (for two sequences this was already achieved when inferring the model only five times). The average Hamming distances of the best predictions on TSsameSeq for every approach are summarized in Table 15 in Appendix C.3.2. The TrobTransformer improves the Hamming distance by up to 44%, indicating that the ProbTransformer reconstructs the overall structure ensemble very well. Additional results and example predictions for RNA folding are reported in Appendix C.3.
7

---

Table Multerproperty (TPSA+lOgP+SAS) conditional training on GuacaMol dataset (mean on five
different seeds).
TPSA MAD/SD
logP MAD/SD
SAS MAD/SD
Model
Validity
Unique
Novelty
ProbTransformer
0.981
0.821
1.0
2.47/2.04 3.79/4.80
0.22/0.18 0.27/0.35
0.16/0.14 0.18/0.26
MolGPT
0.973
0.969
1.0
4.3 Molecule Design
In the field of generative chemistry [51], deep generative models are employed explore the chemical space [52, 53, 54]. However, biological applications typically require that the designed molecules have certain desired properties. For example, penetrate the blood-brain barrier in order to interact with receptors of the nervous system, a molecule typically requires a certain permeability score [55. 56]. Conditional generation then refers the problem of exploring the chemical space conditioned on molecule properties; a well-suited task to evaluate the Prob Transforre in a decoder-only setting against the saae-of-the-an transformer decoder model, MolGPT [11].
Data We use the GuacaMol [57] training data and the evaluation protocol provided by [11]. For more information about the data we refer Appendix D.I.
Setup We employ the same architecture as [11], enhanced with probabilistic layers, and train our model for 20 epochs or 300k training steps. During training, the model implicitly learns the properties of the training data by conditioning generation on molecular properties together with the input SMILES. At inference, novel molecules with multiple desired property values generated by providing the model with a start token alongside with the desired property values while predicting the next token until either a maximum length is reached the model produces end-token. We condition the generation of molecules on three properties: the synthetic accessibility score (SAS), the partition coefficient (logP), and the topological polar surface area (TPSA), using the same domains of values described in [11]. J. For each combination of property values, the model generates a total of 10000 molecules. Results are reported in terms of the mean average deviation (MAD) and the standard deviation (SD) relative to the range of the desired property values, as well as the validity of the generated compounds, their uniqueness in terms of the internal diversity of valid predictions, and their novelty compared to the training data. For more information about the measures and the general setup, we refer Appendix D.2.
The results for the conditional generation of molecules are summarized in Table 3. We observe a high validity score and novelty of 1.0, indicating that Prob Tronfrrrsfrrmer has learned the underlying SMILES grammar very well and does not suffer from overfitting to the train data. For the main task of conditional generation, the ProbTransformer clearly outperforms MolGPT across all measures, which highlights its ability to control multiple molecular properties during generation. The largest improvement can be observed for the SD and MAD scores of TPSA, important measure for drug delivery in the body, improving the standard deviation (SD) by 57.5% and the mean average deviation (MAD) score by nearly 35%. We do not observe improvements in uniqueness compared to MolGPT but note that nearly perfect uniqueness could, e.g., be achieved by adding carbons posteriori [58]. Additional results and prediction examples are shown in Appendix D.3.
Results
4.4 Ablation Studies
We perform two ablation studies to provide more evidence for the usefulness of hierarchical proba- bilistic construction and kappa annealing. For this ablation, we use the RNA Folding task described in Section 4.2 with the same architecture and training steps. We provide more details in E.
Hierarchical probabilistic design Due to the possibility of having a prob layer in any block, we tested different setups from one prob layer (middle) up all blocks enhanced with prob layer. In Figure 4, we measure the performance improvement in Hamming distance compared to a vanilla Transformer on the TSO and TSsameStruct sets. Performance saturates at 4 prob layers for the TSO set but keeps improving on the TSsameStruct set. This is in line with our hypothesis that especially ambiguous samples profit more from the hierarchical architecture
8

---

1.4
0.20
Improvenent
Kappa
1.2
0.10
0.05
0.02
1.0-
2
25
50
75
100
Number of prob-layer
Epochs
Figure 4: (Left) Performance improvement by number of prob layer: Dot (blue) on TSO and triangle (red) TSsameStruc. (Right) Kappa annealing with different initialization over 100 training epochs.
Kappa Annealing We assess the effect of kappa annealing when initializing к with dif- ferent values. this ablation, we compare constant x to kappa annealing the Hamming distance, see Table 4. these results, kappa annealing always improves the performance and substantially stabilizes final performance across range of initialization values. Figure shows the adaption of a over a training run, demon- strating that it smoothly converges smaller value. Initializing with such a small value would yield much worse performance.
Table 4: Hamming distance with and w/o kappa annealing on different initialization.
Kappa
w/o
annealing
0.02
29.4
29.2
0.05
28.0
28.0
0.1
29.4
27.9
0.2
32.2
29.2
Related Work
Related Transformer Models A series works enhanced the encoder-dccoder Transformer archi- tecture with distribution in the latent space between encoder and decoder. During inference, the model samples from these distributions. There is work in the field of music representation [59] which does not use a posterior model. Work in neural machine translation [60, 61] has separate posterior encoder and enhances the ELBO objective. This is similar to the work of Lin et al. [62] which introduces variational decoder layer for dialogue modelling and has separate posterior encoder. Other work text generation [63, 64, 65) uses encoder the predictive and posterior model. In contrast to our work, neither of the methods mentioned above model hierarchical distributions or use the GECO objective. The work of Pei et [66] introduces hierarchical stochastic multi-head attention mechanism aiming uncertainty estimation. In contrast, mechanism by adding a new layer. The work of Liu et al. [64] also introduces a 3 scheduling in 3-VAB ELBO objective [29] similar to our kappa annealing but focused on the KL loss instead of the reconstruction objective.
preserve the original attention
Related work in RNA folding Until recently, RNA folding was dominated by dynamic program- ming (DP) algorithms using either thermodynamic, statistical, or probabilistic scoring functions [67]. Learning-baa based approaches to the problem benefit from making few assumptions on the folding process and allowing previously unrecognized base pairing constraints [42], which recently led to state-of-the-an performance using deep learning [37, 42]. We discuss these state-of-the-an deep learning approaches in detail Appendix C.4 and refer to a recent review [68].
Related Work in Molecule design While a plethora of deep learning- based models have been proposed for de novo generation of molecules in the past five years [69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79], only some methods yet approached the challenging task of generating molecules with (multiple) predefined property values (conditional generation). These include conditional RNNs [80], vAES [81], conditional adversarially regularized autoencoders [82], and more recently also Transformers [1]]. Since [81] considers inorganic molecules and conditional generation is only performed in case study in [82], and because of the usage of different evaluation protocols in [80] and [11] [],, a correct evaluation and fair comparison of these approaches is out of the scope of this work. We, therefore, compare the Tranffosfformr to the most similar and recent work in the field,

---

MolGPT, which, to our knowledge, is the current state-of-the-an in the field. For more details on the different methods, we refer multiple excellent reviews [83, 84, 51, 53).
Discussion and Conclusion
We propose novel probabilistic layer to enhance the transformer architecture with hierarchical latent distributions while keeping the global receptive field via the attention mechanism. The ProbTransformet samples interdependent sequences one forward path. This sampling happens in the latent space, and the Prob Transsssformmr can refine or interpret a sampled latent representation in further layer. Compared to sampling from the softmax output distribution, this approach yields greater flexibility. also is compatible with other enhancements to the Transformer model since only adds
layer but keeps everything else unchanged.
We showed the benefits of our approach several experiments. To our knowledge, the ProbTrans- former is the first learned RNA folding model that can provide multiple correct structure proposals for given RNA sequence, which opens the doors to novel research paths RNA structure prediction that are in line with experimental evidence for RNA structural dynamics from, c.g., NMR studies, such as fraying [85], bulge migration [86]. and fluctuating base pairs [86]. On the challenging multi- objective optimization task [54] of designing molecules with desired properties, superior control over molecule properties during the generation in a decoder-only setting compared to state-of- state-of-the-ant ofttte-arttt vanilla Transformer architecture. We want point out that molecular research inevitably bears the risk of misuse [87], but we strongly distance ourselves from any such attempts.
demonstrate
However, our approach also has limitations. The additional layer and the posterior model increase the computational and memory needs up to a factor of two. Our approach is also limited to encoder-only or decoder-only setups since it requires a target of the same length as the input sequence. We have not applied our model natural language processing yet, c.g. as a language model, which could improve predictive performance and save compute inference time due to the probabilistic encoder. Another future application could be text-to-speech or automated speech recognition since natural speech contains many ambiguities. In general, our approach and especially the conditional variational training using the GECO objective and kappa annealing, could be used in future probabilistic models. Kappa annealing reduces the need for exhaustive hyperparameter optimization and increases performance. Therefore, we expect that the positive impact of our work is not limited to RNA folding and molecule design but also generative models in general.
Acknowledgments and Disclosure of Funding
Jörg Franke, Frederic Runge, and Frank Hutter acknowledge funding by European Research Council (ERC) Consolidator Grant "Deep Learning 2.0" (grant no. 101045765). Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the ERC. Neither the European Union nor the ERC can be held responsible for them.
Funded by the European Union
A part of this work was supported by the German Federal Ministry of Education and Research (BMBF, grant RenormalizedFlows 011S19077C) and by the German Research Foundation (DFG, grant no. 41 417962828). Furthermore, the authors acknowledge support by the state of Baden- Württember through bwHPC and the DFG (grant no. INST 39/963-1 FUGG). We also thank Tom Kaminski, Fabio Ferreira, Mahmoud Safari and Arber Zela for useful comments on the manuscript.
10

---

References
[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[2] Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810004805, 2018.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877- 1901, 2020.
[4] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint 2rXiv22204.02311, 2022.
[5] Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F Wong, and Lidia S Chao. Learning deep transformer models for machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1810-1822, 2019.
[6] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 882 1-831. PMLR, 2021.
[7] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey, ACM Computing Surveys (CSUR), 2021.
[8] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma. al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15), 2021.
[9] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool Russ Bates, Augustin Židek, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-589, 2021.
[10] Philippe Schwaller, Teodoro Laino, Théophile Gaudin, Peter Bolgar, Christopher A Hunter, Costas Bekas, and Alpha Lee. Molecular transformer: model for uncertainty-caliirated chemical reaction prediction. ACS central science, 5(9):1572-1583, 2019.
[11] Viraj Bagal, Rishal Aggarwal, PK Vinod, and U Deva Priyakumar. Molgpt: Molecular generation using a transformer-decode model. Journal of Chemical Information and Modeling. 2021.
[12] Ivo L Hofacker, Walter Fontana, Peter F Stadler, L. Sebastian Bonhoeffer, Manfred Tacker, and Peter Schuster. Fast folding and comparison of ma secondary structures. Monatshefte für Chemie/Chemical Monthly, 125(2):167 1994.
[13] Elizabeth A Dethoff, Jeetender Chugh, Anthony M Mustoe, and Hashim M Al-Hashimi. Functional complexity and regulation through ma dynamics. Nature, 482(7385):322-335) 2012.
[14] Laura R Ganser, Megan I. Kelly, Daniel Herschlag, and Hashim M Al-Hashimi. The roles of structural dynamics the cellular functions of mas. Nature reviews Molecular cell biology, 20(8):474 2019.
[15] David Weininger. Smiles, chemical language and information system. 1. introduction to methodology and encoding rules. Journal of chemical information and computer sciences, 28(1):31-36, 1988.
11

---

[16] Orion Dollar, Nisarg Joshi, David AC Beck, and Jim Pfaendtner. models for de novo molecular design. Chemical Science, 12(24):8362-837) -872. 2021.
generative
[17] Alexandre Pouget, Jeffrey M Beck, Wei Ji Ma, and Peter Latham. Probabilistic brains: knowns and unknowns. Nature neuroscience, (6(9):1170-1177) 2013.
[18] Marcus Lindskog, Pär Nyström, and Gustaf Gredehäck. Can the brain build probability distributions? Frontiers in Psychology. 12, 2021.
[19] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
[20] Simon AA Kohl, Bernardino Komera-Paredeer Klaus H Maier-Hein, Danilo Jimenez Rezende, SM Ali Eslami, Pushmeet Kohli, Andrew Zisserman, and Olaf Ronneberger. A hierarchical probabilistic u-net for modeling multi-scale ambiguities. In Medical Imaging Workshop at NewrlPS, 2019.
[21] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Comput. Surv., apr 2022. Just Accepted.
[22] Danilo Jimenez Rezende and Fabio Viola. Taming vaes. arXiv preprint 1rXI::111000059, 2018.
[23] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. OpenAl Blog, 2018
[24] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778. IEEE, 2016.
[25] Jimmy Lei Ba. Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint / 06450,6 2016.
[26] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In In 2nd International Conference on Learning Representations (ICLR), 2013.
[27] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning, pages 1278-1286. PMLR, 2014.
[28) Diederik P Kingma, Max Welling, et al. An introduction to variational autoencoders. Founda- tions and Trends in Machine Learning, 1244)3307-392, 2019.
[29] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International Conference on Machine Learning. PMIR, 2017.
[30] Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050-1059. PMLR, 2016.
[31] Minakshi Gandhi, Maiwen Soudron-Herget and Sven Diederichs. Rna motifs and combinato- rial prediction interactions, stability and localization of noncoding rnas. Nature structural & molecular biology, 15(12):1070-1077 2018
[32] Sarah Djebali, Carrie A Davis, Angelika Merkel, Alex Dobin, Timo Lassmann, Ali Mortazavi, Andrea Tanzer, Julien Lagarde, Wei Lin, Felix Schlesinger, et al. Landscape of transcription in 2012.
human cells. Nature, 489(7414):101-108
[33] Suzanne G Rzuczek, Lesley A Colgan, Yoshio Nakai, Michael D Cameron, Denis Furling, Ryohei Yasuda, and Matthew D Disney. Precise molelee recognition of toxic cug ma repeat expansion. Nature chemical biology, 13(2):188, 2017.
12

---

[34] Milad Miladi, Martin Raden, Sven Diederichs, and Rolf Backofen. Mutarna: analysis and visu- alization of nnutationiinduce changes in ma structure. Nucleic acids research, 48(W1):W25$) W291, 2020.
[35] Yuedong Yang, Xiaomei Li, Huiying Zhao, Jian Zhan, Jihua Wang, and Yaoqi Zhou. Genome- scale characterization of ma tertiary structures and their functional impact by ma solvent accessibility prediction. Rna, 23(1):14-2) 22. 2017.
[36] Ivantha Guruge, Ghazalch Taherzadeh, Jian Zhan, Yaoqi Zhou, and Yuedong Yang, B-factor profile prediction for ma flexibility using support vector machines. Journal of Computational Chemistry, 39(8):407-41) 4, 2018.
[37] Jaswinder Singh, Jack Hanson, Kuldip Paliwal, and Yaoqi Zhou. Rna secondary structure prediction using an ensemble of two-dimensional deep neural networks and transfer learning. Nature communications, 10(1):1-13, 2019.
[38] Mirela Andronescu, Vera Bereg, Holger H Hoos, and Anne Condon. Rna strand: the ma secondary structure and statistical analysis database. BMC bioinformatics, 9(1):1-10, 2008.
[39] Kengo Sato, Manato Akiyama, and Yasubumi Sakakibara. Rna secondary structure prediction using deep learning with thermodynamic integration. Nature communications, 12(1):1-9, 2021.
[40] Jaswinder Singh, Kuldip Paliwal, Tongchuan Zhang, Jaspreet Singh, Thomas Litfin, and Yaoqi Zhou. Improved raa secondary structure and tertiary base-pairing prediction using evolutionary profile, mutational coupling and two-dimensional transfer learning. Bioinformatics, 37, 2021.
[41] Xinshi Chen, Yu Li, Ramzan Umarov, Xin Gao. and Le Song. Rna secondary structure predic- tion by learning unrolled algorithms. In International Conference on Learning Representations 2019.
[42] Laiyi Fu, Yingxin Cao, Jie Wu, Qinke Peng, Qing Nie, and Xiaohui Xie. Ufold: fast and accurate ra secondary structure prediction with deep learning. Nucleic acids research, 50(3):e1-e14, 2022.
[43] Ronny Lorenz, Stephan H. Bernhart, Christian Höner zu Siederdissen, Hakim Tafer, Christoph Flamm, Peter F. Stadler, and Ivo L. Hofacker. Viennarna package 2.0. Algorithms for Molecular Biology, 6(1):26, Nov 2011.
[44] Stefan Janssen, Christian Schudoma, Gerhard Steger, and Robert Giegerich. Lost in folding space? comparing four variants of the thermodynamic model for rma secondary structure prediction. BMC bioinformatics, 12(1))1-19, 2011.
[45] Ye Ding and Charles E Lawrence. A statistical sampling algorithm for ma secondary structure prediction. Nucleic acids research, 31(24):7280-730) 2003.
[46] David H Mathews. How benchmark rna secondary structure prediction accuracy. Methods, 162:60-67, 2019.
[47] Edwin Ten Dam, Kees Pleij, and David Draper. Structural and functional aspects of ma pseudoknots. Biochemistry, ( 1992.
[48] David W Staple and Samuel E Butcher. Pseudoknots: Rua struetures with diverse functions. PLoS biology, 3(6)e2113, 2005.
[49] Christine Hajdin, Stanislav Bellaousov, Wayne Huggins, Christopher W Leonard, David H Mathews, and Kevin M Weeks. Accurate shape-direeted ma secondary structure modeling, including pseudoknots. Proceedings of the National Academy of Sciences, 110(14):548--555 2013.
[50] P Fechter, J Rudinger-Thirion C Florentz, and R Giege. Novel features in the trna-like world of plant viral raas. Cellular and Molecular Life Sciences CMLS, 58(1 54:557-161,, 2001.
[51] Joshua Meyers, Benedek Fabian, and Nathan Brown, De novo molecular design and generative models. Drug Discovery Today, 26(11): 2021.
13

---

[52] Christopher Lipinski and Andrew Hopkins. Navigating chemical space for biology and medicine. Nature, 432(7019):855-86)) 2004.
[53] Ola Engkvist, Josep Arús-Pous Esben Jannik Bjerrum, and Hongming Chen. Chapter 13 molecular de novo design through deep generative models. In Artificial Intelligence in Drug Discovery, pages 272-300. The Royal Society of Chemistry, 2021.
[54] Connor W Coley. Defining and exploring chemical spaces. Trends in Chemistry, 3(2):133-145, 2021.
[55] Han van Waterbeemd, Gian Camenisch, Gerd Folkers, Jacques R Chretien, and Oleg A Raevsky. Estimation of blood-brain barrier crossing of drugs using molecular size and shape, and h-bonding descriptors. Journal of drug targeting, 6(2):151-165, 1998.
[56] David E Clark. Rapid calculation of polar molecular surface area and its application to the prediction of transport phenomena. 1. prediction of intestinal absorption. Journal of pharmaceutical sciences, 88(8):807-81) 1999.
[57] Nathan Brown, Marco Fiscato, Marwin HS Segler, and Alain Vaucher. Guacamol: bench- marking models for de novo molecular design. Journal of chemical information and modeling, 1996))009--1108, 2019.
[58] Philipp Renz, Dries Van Rompaey, Jörg Kurt Wegner, Sepp Hochreiter, and Günter Klam- bauer. On failure modes in molecule generation and optimization. Drug Discovery Today: Technologies, 32:55-63 63, 2019.
[59] Junyan Jiang, Gus G. Xia, Dave B. Carlton, Chris N. Anderson, and Ryan H. Miyakawa. Trans- former vae: A hierarchical model for structure-aware and interpretable music representation learning. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 516-520, 2020.
[60] Arya McCarthy, Xian Li, Jiatao Gu, and Ning Dong. Improved variational neural machine translation by promoting mutual information. arXiv preprint arXiv: 2019.
[61] Arya D. McCarthy, Xian Li, Jiatao Gu, and Ning Dong. Addressing posterior collapse with mutual information for improved variational neural machine translation. Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pages 8512-8525, Online, July 2020. Association for Computational Linguisties.
[62] Zhaojiang Lin, Genta Indra Winata, Peng Xu, Zihan Liu, and Pascale Fung. Variational transformers for diverse response generation. arXiv preprint arXiv:2003. 12738, 2020.
[63] Tianming Wang and Xiaojun Wan. T-cvae: Transformer-basee conditioned variational autoen- coder for story completion. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 5233-5239 International Joint Conferences on Artificial Intelligence Organization, 2019.
[64] Danyang Liu and Gongshen Liu. A transformer-based variational autoencoder for sentence generation. In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1-7, 2019.
[65] Le Fang, Tao Zeng, Chaochun Liu, Liefeng Bo, Wen Dong, and Changyou Chen. Transformer based conditional variational autoencoder for controllable story generation. arXiv preprint arXiv:2101. 100002200,.0028,, 2021.
[66] Jiahuan Pei, Cheng Wang, and György Szarvas. Transformer uncertainty estimation with hierarchical stochastic attention. In Proceedings of the Thirty-Sixth AAAI Conference on Artificial Intelligence, 2022.
[67] Elena Rivas. The four ingredients of single-sequence ma secondary structure prediction. a unifying perspective. RNA biology, (07))1185-111 2013.
[68] Qi Zhao, Zheng Zhao, Xiaoya Fan, Zhengwei Yuan, Qian Mao, and Yudong Yao. Review of machine learning methods for ma secondary structure prediction. PLoS computational biology. 17(8) 2021.
14

---

[69] Marcus Olivecrona, Thomas Blaschke, Ola Engkvist, and Hongming Chen. Molecular novo design through deep reinforcement learning. Journal of cheminformatics, 9(1):1-14, 2017.
[70] Gabriel Lima Guimaraes, Benjamin Sanchez- Janccez--engeling. Carlos Outeiral, Pedro Luis Cunha Farias, and Alán Aspuru-Guza Guzik. Dejective-reinforee generative adversarial networks (organ) for sequence generation models. arXiv preprint 1705.
2017.
[71] Benjamin Sanchez-Lengeling Carlos Outeiral, Gabriel L. Guimaraes, and Alan Aspuru-Guzi Optimizing distributions over molecular space. an objective-reinforoed generative adversarial network for inverse-design chemistry (organic). ChemRxiv, 2017.
[72] Artur Kadurin, Sergey Nikolenko, Kuzma Khrabrov, Alex Aliper, and Alex Zhavoronkov. drugan: advanced generative adversarial autoencoder model for de novo generation of new molccules with desired molecular properties in silico. Molecular pharmaceutics, 14(9):3098- 3104. 2017.
[73] Marwin HS Segler, Thierry Kogej, Christian Tyrchan, and Mark P Waller. Generating focused molecule libraries for drug discovery with recurrent neural networks. ACS central science. 4(1):120-131, 2018.
[74] Mariya Popova, Olexandr Isayev, and Alexander Tropsha. Deep reinforcement learning for de novo drug design. Science advances, 4(7)eeapp788,, 2018. 2018.
[75] Rafael Benjamín Sánchez-Lengeling Dennis Sheberla, Jorge sguilera-Iparraguire, Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using data-driven continuous representation of molecules. ACS central science, 42)):2682276 2018.
Jennifer N Wei, David Duvenaud, José Miguel Hernández-Loben
[76] Qi Liu, Miltiadis Allamanis, Marc Brockschmidt, and Alexander Gaunt. Constrained graph variational autoencoders for molecule design. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianche and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[77] Evgeny Putin, Arip Asadulaev, Quentin Vanhaelen, Yan Ivanenkov, Anastasia V Aladinskaya, Alex Aliper, and Alex Zhavoronkov. Adversarial threshold neural computer for molecular de novo design. Molecular pharmaceutics, 1 15(10):4386 4397, 2018.
[78] Oleksii Prykhodko, Simon Viet Johansson, Panagiotis-Christo Kotsias, Josep Arús-Pous, Esben Jannik Bjerrum, Ola Engkvist, and Hongming Chen. A de novo molecular generation method using latent vector based generative adversarial network. Journal of Cheminfornatics, 11(1): 13, 2019.
[79] Daniil Polykovskiy, Alexander Zhebrak, Benjamin Sanchez-Lengeling, Sergey Golovanov, Oktai Tatanov, Stanislav Belyaev, Rauf Kurbanov, Aleksey Artamonov, Vladimir Aladinskiy, Mark Veselov, et al. Molecular sets (moses): a benchmarking platform for molecular generation models. Frontiers in pharmacology, 11:1931, 2020.
[80] Panagiotis-Chrre Christos Kotsias, Josep Arús-Pous, Hongming Chen, Ola Engkvist, Christian Tyrchan, and Esben Jannik Bjerrum. Direct steering of de novo molecular generation with descriptor conditional recurrent neural networks. Nature Machine Intelligence. 2(5):254-265, 2020.
[81] Yashaswi Pathak, Karandeep Singh Juneja, Girish Varma, Masahiro Ehara, and U Deva Priyakumar. Deep learning enabled inorganic material generator. Physical Chemistry Chemical Physics, 22(46):26935-2694) 269443,,3326 2020.
[82] Seung Hwan Hong, Seongok Ryu, Jaechang Lim, and Woo Youn Kim. Molecular generative model based on an adversarially regularized autoencoder. Journal of chemical information and modeling, 20(1):29-36, 2019.
[83] Benjamin Lenchez-Lengeling and Alán Aspuru-Guzik. Inverse molecular design using machine learning: Generative models for matter engineering. Science, 361(6400):300-36, 2018.
15

---

[84] Daniel C Elton, Zois Boukouvalas, Mark D Fuge, and Peter W Chung. Deep learning for molecular design a review of the state of the art. Molecular Systems Design & Engineering. 4(4):828 849, 2019.
[85] Daniele Andreatta, Sobhan Sen, J Luis Pérez Lustres, Sergey A Kovalenko, Nikolaus P Ernsting, Catherine J Murphy, Robert S Coleman, and Mark A Berg. Ultrafast dynamics in dna:"fraying" at the end of the helix. Journal of the American Chemical Society, 128(21):6885. 6892. 2006.
[86] Sarah A Woodson and Donald M Crothers. Proton nuclear magnetic resonance studies bulge-containing dna oligonucleotides from a mutational hot-spot sequence. Biochemistry, 26(3):904 912, 1987.
[87] Fabio Urbina, Filippa Lentzos, Cédric Invernizzi, and Sean Ekins. Dual use of artificial- intelligence-poeeree drug discovery. Nature Machine Intelligence, 4(3):189-111, 2022.
[88] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. H. Wallach, H. Larochelle, Beygelzimer, F. Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035, Curran Associates, Inc., 2019.
[89] Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Courapeaa,, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk. Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis Oliphant. Array programming with NumPy. Nature, 585(7825)
Kevin
September 2020.
[90] Wes McKinney. Data Structures for Statistical Computing in Python. In Stéfan van der Walt and Jarrod Millman, editors, Proceedings of the 9th Python Science Conference, pages 56 61, 2010.
[91] J. D. Hunter. Matplotlib: A 2d graphics environment. Computing in Science & Engineering, 9(3):90-99) 2007.
[92] Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Networks, 107:3-11, 2018.
[93] John R. Prensner, Matthew K. Iyer, O. Alejandro Balbin, Saravana M. Dhanasekaran, Qi Cao, J. Chad Brenner, Bharathi Laxman, Irfan A. Asangani, Catherine S. Grasso, Hal D. Kominsky, Xuhong Cao. Xiaojun Jing, Xiaoju Wang, Javed Siddiqui, John T. Wei, Daniel Robinson, Hari K. Iyer, Nallasivam Palanisamy, Christopher A. Maher, and Arul M. Chinnaiyan. Tran- scriptome sequencing across a prostate cancer cohort identifies peat- an unannotated linerna implicated in disease progression. Nature Biotechnology. 29(8)7742-749 2011.
[94] Bingqing Cao, Tao Wang, Qiumin Qu, Tao Kang, and Qian Yang. Long noncoding ma snhgl promotes neuroinflammation in parkinson's 's disease via regulating mir-7/nlr33 pathway. Neuroscience, 388: 188:118- - 127, 2018.
[95] Michael S Fernandopulle, Jennifer Lippincott-SShwarte and Michael Ward. Rna trans- port and local translation in neurodevelopmental and neurodegenerative disease. Nature neuroscience, 24(5):622-632, 2021.
[96] Seyedch Hoda Alavizadeh, Maham Doagooyan, Fatemeh Zahedipour, Shima Yahoo Torghabe, Bahare Baharich, Firooze Soleymani, and Fatemeh Gheybi. Antisense technology potential strategy for the treatment of coronaviruses infection: With focus on covid-19, 19. IET nanobiotechmology, 2022.
[97] Lijuan Yin, Fei Zhao, Hong Sun, Zhen Wang, Yu Huang, Weijun Zhu, Fengwen Xu, Shan Mei, Xiaoman Liu, Di Zhang, et al. Crispr-cas la inhibits hiv-1 infection. Molecular Therapy- Nucleic Acids, 21:147-155: 2020.
16

---

[98] Elie Dolgin et al. The tangled history of mrna vaccines. Nature, 38 2021.
[99] ENCODE Project Consortium et al. Identification and analysis of functional elements in 1 1% of the human genome by the encode pilot project. nature, 447(7146))799, 2007.
[100] ENCODE Project Consortium et al. An integrated encyclopedia of dna elements in the human genome, Nature, 489(7414):57, 2012.
[101] Ignacio Tinoco Jr and Carlos Bustamante. How ma folds. Journal of molecular biology, 293(2))271-281 1999.
[102] Jörg Fallmann, Sebastian Will, Jan Engelhardt, Bjöm Grüning, Rolf Backofen, and Peter F Stadler. Recent advances in ma folding. Journal of biotechnology, 261:97-104, 2017.
[103] Padideh Danaee, Mason Rouches, Michelle Wiley, Dezhong Deng, Liang Huang, and David Hendrix. bprna: large-scale automated annotation and analysis of ma secondary structure. Nucleic acids research, 46(1 1)538-155394 2018.
[104] Zhen Tan, Yinghan Fu, Gaurav Sharma, and David H Mathews. Turbofold ii: Rna structural alignment and secondary structure prediction informed by multiple homologs. Nucleic acids research, 45(20):1 1570-11581, 2017.
[105] Michael F Sloma and David H Mathews. Exact calculation of loop formation probability identifies folding motifs in ma secondary structures. RNA, 22(12)) 1818, 2016.
[106] Elena Rivas, Raymond Lang, and Sean R Eddy. range of complex probabilistic models for ma secondary structure prediction that includes the nearest-neighhor model and more. RNA, 18(2):193-22 212, 2012.
[107] Limin Fu, Beifang Niu, Zhengwei Zhu, Sitao Wu, and Weizhong Li. Cd-hit: accelerated for clustering the next-generation sequencing data. Bioinformatics, (5123):3550-313 2012.
[108] Ravi Barnwal, Fan Yang, and Gabriele Varani. Applications of nmr to structure determination of rnas large and small. Archives of biochemistry and biophysics, 628:42-2355 2017.
[109] Kévin Darty, Alain Denise, and Yann Ponty. Varna: Interactive drawing and editing of the ma secondary structure. Bioinformatics, 25(15):1974, 2009.
[110] Jimmy Ka Ho Chiu and Yi-Ping Phoebe Chen. Efficient conversion of ma pseudoknots to knot-free structures using graphical model. IEEE Transactions on Biomedical Engineering. 6265):126,11 2014.
[111] Kaiming He, Xiangyu Zhang, Shanqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pottern Recognition (CVPR), pages 770-778, 2016.
[112] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 133: 1997.
[113] Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE transac- tions on Signal Processing, 45(11))2673-263 1997.
[114] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234-241. Springer, 2015.
[115] Nathan Brown. In Silico Medicinal Chemistry. Theoretical and Computational Chemistry Series. The Royal Society of Chemistry, 2016.
[116] David E Clark. What has polar surface area ever done for drug discovery? Future medicinal chemistry, 3(4):469 484, 2011.
[117] Greg Landrum al. Rdkit: A software suite for cheminformatics, computational chemistry, and predictive modeling, 2013.
[118] Masakazu Kondo. Developing generative model utilizing self-attention networks: Application to materials/drug discovery. Molecular Informatics, 101100:2100102, 2021.
17

---

Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We describe our new approach in detail and provide a experimental evaluation with fette-of-the-ann results.
(b) Did you describe the limitations of your work? [Yes] We mention them in Section 6.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We not aware of negative impacts of RNA folding or foundation in probabilistic modelling but in molecule design and mention this in Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We read it and our paper conform.
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as a URL)? [Yes] We include the source code and scripts to build the data in the supplemental material. We also open source our code upon acceptance as mentioned in Section 4 (b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] Details about hyperparameter and data can be found in Appendices B,C, and D.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] We report all experimental results including standard deviation on multi-seed runs in the Appendix.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Details about compute resources can be found in Appendices B.2,C.2, and D)..
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) your work uses existing assets, did you cite the creators? [Yes] We cite used open source libraries in the Appendix B.2.
(b) Did you mention the license of the assets? [No] We use only open source software.
(c) Did you include any new assets either in the supplemental material or as a URL? [No] We do include any new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you' 're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] Our data does not contain personally identifiable
information or offensive content.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] We do not use crowdsourcing nor conducted research with human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] We do not use crowdsourcing nor conducted research with human subjects.
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A] We do not use crowdsourcing nor conducted research with human subjects.
18


Reviews:
[{"review_id": "yLmSUDk6BON", "reviewer": "Reviewer_Ex3E", "summary": "This paper introduced ProbTransformer, which is a transformer based generative model. By adding a probabilistic architecture to the original transformer, ProbTransformer was able to model the hidden distribution of the data and provided probabilistic predictions. In addition, its contributions also include the online adaptation technique, kappa annealing, to gain better stability and performance. This paper is driven by the challenge of modeling the ambiguities of certain tasks, like the RNA folding problem. Deterministic models may harm the performance on those problems when one sample can have ambiguous outputs. The manuscript empirically demonstrates that the addition of their probabilistic layer improves not only the ability to model the latent data distribution but the stability of the learning process combined with the kappa annealing.\n", "questions": "see above", "limitations": "Yes", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n==========\n\nThe paper addresses an important question of modeling uncertainty in structure modeling. They show their modeling approach is applicable to different problems/domains.\n\nThe flow of this paper is clear. It introduces the motivation first and then its solution to the problem fits the motivation nicely. In addition, the experimental design supports their claims by showing their model outperforming the state-of-art models in different downstrain tasks. Specifically:\n\nThe authors nicely demonstrate improvements in the quality of distribution learning by using the ProbTransformer in the synthetic sequential dataset. \n\nTo enhance the claim of distribution learning, the paper further shows that it can succeed in both RNA folding and molecule design problems, which use the encoder-only model and the decoder-only model to solve respectively. In both tasks, the authors provide some examples to illustrate their good performance while their evaluation metrics show that their model outperforms the other models in general. \n\nUsing ablation studies, the authors demonstrate the effect and also the importance of the hierarchical probabilistic and kappa annealing design.\n\n\nWeaknesses:\n==============\nThe paper did not illustrate the model design very clearly. This is not a minor comment but an important one given what the authors are trying to achieve/claim here. The paper’s annotations and formulas give the audience a basic understanding of what the authors aim to do. However, it is hard to follow how the model is constructed. They should have either in the main text or supplementary a “walk through” example to clearly illustrate the model’s working. It is vague about the definition of encoder-only and decoder only models and how they are built differently in the downstream tasks. \n\nThe authors mention that the encoder-decoder can not be built for now because of the same length issue, but it needs to be clarified why the model can’t make the same length for the output of the encoder with the input of the decoder. This relates to the previous comment as well.\n\nThe authors state RNA structure can be expressed by a dot-bracket notation, which means source and target can have the same length. However, do-bracket only captures nested dependencies and is not able to capture structures such as pseudo-knots. This is a key issue with the model which the authors sweep under the carpet. In contrast, other models are not limited by this (some none nested structures are shown in the predicted structures they included). Some discussion of these limitations compared to other methods is warranted, and evaluation would have been nice though possibly out of scope. \n\nPrevious work also showed that RNA structure prediction methods greatly depend on how many canonical nested base pairing occurs - the fewer the ratio, the worse the model.  Again, at least mentioning/discussion of limitations is important.\n\nTo justify the model in the content of RNA the authors seem to mix a bit claims about mutations and molecules that shift between different structures in specific regions. It’s not clear to these reviewers the data actually support claims there. The authors should clarify/clean the text around these statements and make sure claims and results are well connected.\n\nFrom the description in the main text it seems that the authors took a different approach compared to the authors of MXFold2 and SPOT RNA when defining train/test because they wanted to retain ambiguous structures. It is not clear if the comparisons done in the paper allowed these models to be trained in the same way. If not, there is the doubt that improvements are due at least to some extent to differences in training.\n\nMinor comments:\n=============\nLine 21: “Vice-versa” is unclear in this context.\nLine 169: insights in the \nLine 174-175: “token…is a unique distribution” \nLack of labels for different kappa values in Figure 4.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "eUobSjPER0f", "reviewer": "Reviewer_J3YR", "summary": "The authors present a probabilistic Transformer dubbed ProbTransfomer that models hierarchical latent distributions by integrating the transformer structure with a conditional variational auto-encoder. The article reveals that the incorporation of generalized ELBO with limited optimization is advantageous for variational training and that the proposed model excels at managing data ambiguities, including RNA folding and molecule design.\n", "questions": "1.\tFor the RNA folding problem, the authors may like to do a more comprehensive comparison of the ProbTransformer and other tools. For instance, independently evaluating the prediction of canonical and non-canonical base pairs.\n2.\tConsidering adopting the ProbTransfomer model for predicting disordered protein areas. In contrast to the structural sections predicted by Alphafold2, disordered regions do not have fixed conformations and can therefore be seen as various target sequence conformational ensembles.\n3.\tAppendix line 754, the word “BRpRNA” should be changed to “BpRNA”.\n", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.\tThe article is well organized and well written.\n2.\tStrong incentive exists where modeling unstructured molecules such as RNA is natural.\n3.\tCombining the Transformer with the cVAE, the authors propose a probabilistic layer for handling data ambiguity. ProbTransfomer's application to RNA folding and molecular design is a key development.\nWeaknesses: \n1.\tThe innovation is marginal as main part of proposed method is simply combination of existing works (Transformer (Ashish et al., 2017), cVAE (Kihyuk et al., 2015), ELBO (Simon et al., 2019), and GECO (Danilo et al., 2018).) Some recent advances(e.g. Grphformer) are absence in related work and discussions.\n2. The authors introduce the ProbTransfomer model in an effort to solve the challenge of RNA folding. However, the likelihood of alternative RNA structures arising are mostly dictated by biological circumstances, which the authors do not examine. In addition, authors should consider RNA abundance when building datasets.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dWAuM-dO3q-", "reviewer": "Reviewer_GgjC", "summary": "The paper introduces a novel probabilistic layer into the transformer architecture that incorporates cVAE, which allows multiple samples and predictions that as an ensemble may correspond to some distribution of interest. It also introduces a training strategy (\"kappa annealing\") to effectively train models with such layers while reducing need for hypers tuning.", "questions": "See above.", "limitations": "Adequately addressed. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "While the elements of the main ideas are present, they are presented in a mathematically unclear manner with an unconvincing synthetic experiment (section 4.1) or experiments (section 4.2 and 4.3) whose significance is not well described for readers who are not familiar with the exact metrics used.\n\nExamples of imprecise language and notations: \n- Abstract: \"Therefore, we propose a hierarchical latent distribution to enhance one of the most successful deep learning models, the Transformer, to accommodate ambiguities and data distributions.\" What do the authors mean by \"ambiguities and data distributions\" mathematically and how do the chosen experiments address these?\n- L76-75: \"This objective at training time can be viewed as a reconstruction task, which is an easier task than prediction.\" Why is this easier?\n- L99-101: \"As a result of this hierar-chical composition of the ProbTransformer, the cVAE’s (conditional) prior model pθ (zm |Z<m , X )and generation model pρ(Y |X, Z) cannot be disentangled cleanly anymore in it and effectivelybecome a single model pΦ(Y |X).\" What precisely do you mean by \"disentangled cleanly anymore\" and how does this relate to the training strategy?\n- Eq4-6: what is i?\n- Notations in Section 3.2 is under-defined and the material could be better presented for more effective communication. For example, show how the loss terms can be derived in a more principled manner from an optimization objective and variational approximation; what are p_m and q_m in Eq 9? A read could possibly infer from the context but it is on the authors to make this explicit.\n\nSuggestions for experiments\n- 4.1. I suggest to show some examples in the main text and how the tasks, objectives, and metrics are appropriate for quantifying the model's capability for \"ambiguities and data distributions\"; otherwise, readers can only guess what the authors mean. This synthetic example feels contrived and is not intuitive and convincing of how the proposed addition to the transformer architecture provides a meaningful advantage.\n\n4.2. and 4.3. Similar to above, I suggest to show some examples in the main text and how the tasks, objectives, and metrics are appropriate for quantifying the model's capability for \"ambiguities and data distributions\"; otherwise, readers can only guess what the authors mean.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept