Steering Protein Family Design through Profile Bayesian Flow
Authors: Jingjing Gong
Yu Pei
Siyu Long
Yuxuan Song
Zhe Zhang
Wenhao Huang
Ziyao Cao
Shuyi Zhang
Hao Zhou
Wei-Ying Ma
Venue: ICLR 2025 Oral (2025)
Keywords: protein family generation
homologous protein generation
protein design
bayesian flow
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Protein family design emerges as a promising alternative by combining the advantages of de novo protein design and mutation-based directed evolution.In this paper, we propose ProfileBFN, the Profile Bayesian Flow Networks, for specifically generative modeling of protein families. ProfileBFN extends the discrete Bayesian Flow Network from an MSA profile perspective, which can be trained on single protein sequences by regarding it as a degenerate profile, thereby achieving efficient protein family design by avoiding large-scale MSA data construction and training. Empirical results show that ProfileBFN has a profound understanding of proteins. When generating diverse and novel family proteins, it can accurately capture the structural characteristics of the family. The enzyme produced by this method is more likely than the previous approach to have the corresponding function, offering better odds of generating diverse proteins with the desired functionality.

Published as a conference paper at ICLR 2025
STEERING PROTEIN FAMILY DESIGN THROUGH PROFILE BAYESIAN FLOW
Jingjing Gongl, Yu Pei
Siyu Long' Yuxuan Song
Zhe Zhang' Wei- Ying Ma²
Wenhao Huang'
Ziyao Cao' Shuyi Zhang²
Hao Zhou'
Institute of Al Industry Research (AIR), Tsinghua University
2 School of Pharmaceutical Sciences, Tsinghua University
jigonggj, yupei. wp, yx 6, 1 longl I onglongguy @ agma 1 com
z (zhouha,, mawelying Cair tsinguua. edu
ABSTRACT
Protein family design emerges as a promising alternative by combining the ad- vantages of de novo protein design and mutation-base directed evolution. In this paper, we propose ProfileBNN, the Profile Bayesian Flow Networks, for specif- ically generative modeling of protein families. ProfileBFN extends the discrete Bayesian Flow Network from an MSA profile perspective, which can be trained single protein sequences by regarding as degenerate profile, thereby achiev- ing efficient protein family design by avoiding large-scale MSA data construction and training. Empirical results show that ProfileBFN has profound understand- ing of proteins. When generating diverse and novel family proteins, it can accu- rately capture the structural characteristics of the family. The enzyme produced by this method is more likely than the previous approach to have the correspond- ing function, offering better odds of generating diverse proteins with the desired functionality.
INTRODUCTION
Protein design stands as crucial problem with far-reaching implications. In particular, it holds the potential to significantly accelerate progress in numerous areas such as precision medicine and
synthetic biology (Kosorok & Laber 2019; Johnson et al. 202T Benner & Sismour 2005). Re-
cently, artificial intelligence (Al) has brought new possibilities and breakthroughs to protein design (Jumper al.] 202T Abramson et al. 2024, Lin et al., 2023 Hayes et al., 2024). Al-powered techniques are increasingly being employed to accelerate the process and enhance the accuracy of
protein design. The ability to design proteins with specific functions using AI not only scientific pursuit but also practical necessity for addressing various challenges in these fields.
Protein design often involves a combination of de novo design and mutation-based directed evolu- tion. De novo design generates proteins almost from scratch, offering novel protein sequences that expand the diversity of protein libraries (Watson et al. 2023 Dahiyat & Mayo 1997). Although it may have lower success rate in wet lab experiments. it valuable for creating starting points that can be further optimized. Directed evolution (Arnlld 1998: Packer & Liu, 2015) is effective in developing proteins with enhanced functions in vitro. However, the scope of exploration within the vast protein sequence space remains limited due to constraints in both the throughput of library creation and the subsequent screening or selection processes (Wang et al. 2021 Bloom & Amnold 2009).
In this context, protein family design emerges as an approach that combines the strengths of both methods. By generating protein candidates based on multiple existing functional proteins, it ex- plores protein space more broadly than initation-baseed methods alone while utilizing established functional information. This generative process allows for the creation of diverse libraries without being limited to sequences closely related to single wild type. Similar methods, such as Prot- Mamba (Sgarbossa et al. 2024), PoET (Truong Jr & Bepler 2023 and EvoDiff (Alamdari et al.)
"Equal Contribution. Correspondence Hao Zhou @zhouhao aithuhhan@ai..tsnnnuuu..u.

---

Published as a conference paper at ICLR 2025
2023). also aim to balance innovation with reliability in protein design. Overall, protein family design fits within the library creation and optimization pipeline, providing a powerful tool for gen- erating diverse protein candidates that can be further refined through directed evolution.
Recently, single protein sequence modeling has dominated the area due to the analogy to the task of the language model. Hence, there is also rising interest in transferring the techniques from language modeling to protein modeling (Truong Jr & Bepler 2023 Madani et al.) 2023] 2020
Nijkamp al. 2023] Jumper et al. 2021). In contrast, we believe that directly applying the natural language modeling paradigm could be sub-optimal for the protein sequence distribution with very complex global spatial correlation and constraint. In this paper, we consider integrating the evolutionary information from the MSA (Multiple Sequence Alignment) motivated by previous literature (Rao et al. 2021 Alamdari et al., 2023). However, MSA lies a specific data type, i.e. set of sequences, and could vary and hold large length and depth which could bring in practical barriers for efficiently processing the information with a scaled model.
To address the above concern and bring fresh perspective to the protein family generative modeling, we propose the Profile Bayesian Flow Networks (ProfileBEN), which achieves effective yet efficient Protein Family Design by: (i) proposing to use MSA profile (the distribution of MSA) instead of MSA for probabilistic generative modeling, which avoids the heavily direct training of MSA data (ii) ProfileBFN extends the conventional discrete Bayesian Flow Network (BFN) from MSA profile perspective. We formally re-derive the new Bayesian flow and loss terms, tailoring it from the perspective of protein family modeling. (iii) ProfileBFN could escape the heavy construction of large-scale MSA data by training on single protein sequences. Thanks to the mathematical nature of the ProfileBNN, we could generalize the one-hot representation of single sequences as a degenerative profile, which enables the ProfileBNN to be flexible for both single sequence and multiple sequence profiles as imputs.
We evaluate ProfileBFN on a multitude of benchmarks and find that ProfileBFN has the following impressive advantages: (i) ProfileBFN ensures structural conservation while providing the most diverse and novel family protein generation results. For characterizing family structural features, sequences generated by ProfileBNN even surpass the MSA search relied upon by AlphaFold2. (ii) In the evaluation of generating functional enzyme proteins, compared previous advanced methods, ProfileBEN more likely than the previous approach to have the corresponding function, offering better odds of generating diverse proteins with the desired functionality. (iii) In the aspect of protein representation, FrofileBFN outperforms all PLMs under the same parameter scale, demonstrating its profound understanding of proteins.
2
PRELIMINARIES
2.1 REPRESENTING PROTEIN FAMILY AS MSA PROFILES
Multiple Sequence Alignments (MSAs) (Edar & Batzoglou 2006) are commonly used to capture the evolutionary relationship between protein sequences within a family, have been widely used in various aspects of protein modeling, including protein sequence analysis (Gromiha 2010), structure prediction, function prediction, and protein design.
In the context of this paper, a MSA is set of homologous protein sequences that are aligned to each other. Formally speaking. given a set of n protein sequences, the MSA a matrix X E {0. ... K mxmm where is the length of the aligned protein sequences, and X is the j-th amino acid in the i-th aligned protein sequence.
The MSA profile (P()) C K where A represents the space of k-dimensional simplex, P is calculated as follows:
p(t) 1 "
Σ n
1 1x,; j=1
MSS commonly used to capture evolutionary relationship between protein sequences within a family. 2This is analogous to directly calculating the Schrödinger equation and making estimations using density functional theory.
2

---

Published as a conference paper at ICLR 2025
Where K is the alphabet size of amino acids, Pk is the frequency of amino acid & position in the MSA, and 1 (-=) is the Kronecker delta function.
2.2 BAYESIAN FLOW NETWORKS
Bayesian Flow Networks (BFNs) Graves al. 2023) introduce new type of generative model from transmission perspective. In a simpler language, sender leaks it's information through the noisy process of Z, ~ w)] An observer then receives the leaked information and updates its belief about the variable x through Bayesian update and obtain belief about x: p(1/imm). In the context bits-back coding transmission scheme, the total number nats required to transmit with 71:n serving intermediate latents be expressed as llg:(( log px The process also incorporates log (Z1.n |x) nats returned to the sender, thus yielding the expected marginal nats necessary transmit data from p(x), which corresponds the negative Variational Lower Bound (VLB), as:
Pots (ns)) p(zim) - log p(x(21m)) + log ( [x;W] = Ep(x) [ (eana//////x::)lpp((33> (pp((==))) - (o (x:w) log p [x|[)]]]] = VLB
(2)
As p(zm:) can be decomposed regr-segresyyvely with a neural network Ps. where Φ is the governing parameter of the neural network, the loss is
-VLB(Ø) = Ep(())
Σ DEL
Φ)) Bg log Pep (x/Z1:n)
(3)
The VVLB(Ø) is the expected marginal nats required to transfer data sample from p(x). The loss can be derived into simpler form:
((x) 5 exll
(4)
The Bayesian flow required to train the network is:
e Ou
0 x;
E
(5)
Σm ex (0o)
Where the governing parameter of the belief of the variable x. C, is the covariance matrix the multivariate Gaussian distribution. 8(- 0) is dirac delta function that is zero everywhere except at O. For detailed easy to understand derivation refer to Appendix A.I
3 METHOD
To generate a protein that belongs to a specific protein family, it's crucial to leverage the information embedded within that family. As introduced in Section 2. profile serves effective summary of a protein family's 's multiple sequence alignment (MSA). Utilizing profiles allows us to harness the collective information of the entire protein family without incurring additional computational costs compared to singlc-sequence models. However, constructing training set of MSA profiles is computationally expensive (Liu al. 2009 Nag Karlormal 2016).
We introduce our proposed ProfileBFN model, which unifies single-sequence one-hot encoding as special case of profile. This innovative approach enables us to train on single protein sequences while sampling with protein family profiles. Consequently, we can bypass the need to construct an MSA profile training set, offering a more efficient and practical solution. Henceforth, we define profile as a list of PMFs and for simplicity, refer to PMF as profile.
3.1 THE PROPOSED PROFLEBFN
In the original discrete BEN (Graves et al. 2023), the emitted sample x can be viewed as being drawn from a degenerate profile where each component has all its probability mass concentrated on
3

---

Published as a conference paper at ICLR 2025
single category. In this work, extend the discrete BFN to accommodate the input of generalized profiles. This generalization allows for seamless integration with the processing of protein family profiles.
To enable new capabilities, is necessary to derive a new Bayesian flow and a corresponding loss term. The main intuition behind this is to sample from generalized profile, pass it through noisy channel, and then have the parameterized network make predictions based on the received evidence. The Bayesian flow for profile modelling is below, and the derivation and proof can be found in Appendix A.2
Theorem 3.1. Given discrete noisy channel q(ze)p:wh) K w;p(z) where p. Ex Px. 5//1 1, 2 0 is certain profile, with w (i-1))/n ((T) B(t) fo (T) 2
0), ((T) > 0, VT, and B(1) bounded, when + t00, the continuous time discrete Bayesian flow
is:
(0
ex
E Where the accumulated information about the profile p. E R-KK Cij = K1;=j 1, the covariance matrix of the multivariate Gaussian distribution. 6(- 0) is Dirac delta function that is zero everywhere except at O.
PF p0(ppp:
8 B(0))
(6)
k= ex (0o)k)
Where p E A is profile which can also be viewed as Probability Mass Function (PMF) with possible categories, this is the different part compared to vanilla discrete Bayesian flow (Eq.
Additionally, we derive the new loss function as below.
Theorem 3.2. Given a discrete noisy channel q p) 1 +wp(z), p(z) - +wpo(z),w >>>>>>0000 0,
where p. Σ. Ps 1, vp 2 is a certain profile, with nw² = 33 bounded,
lim nD (g(z)p))lx(z) 2 3K 1IPo pl1
For more general case where w()) changes through time, with B(1) So w² (T))dT 12 1212 and 3(1) bounded, the limit of the KL divergence is:
1 81) - 2|1
lim n nDDkL((22lp:t))lld +00
(1)
(8)
There is only little change by substituting ex to p with respect to Eq.
@(m)) represents a neural network, where ( is the ith accumu- lated information about the profile. The primary purpose of the network is to model the interdepen- dency between independently accumulated information about the profiles.
From Eq. 15 P = f (e())..... .....
3.2 TRAINING WITH PROFILE AS INPUT
10 - is the profile, where is the length of the protein sequence, and K is the alphabet size of amino acids. p(ù) is the probability mass function of the i-th position in the MSA profile, indicating the frequency of each amino acid at the i-th position in the MSA.
As introduced in Section 2.1
Unified Profile Representation In the special case where the MSA contains only a single se- quence, the profile at each position p()) becomes a one-hot vector. This scenario simplifies to deter- mining the precise amino acid at each position without ambiguity. However, for typical MSAs with multiple sequences, p(i) provides a richer representation reflecting the variability and conservation of amino acids across the alignment.
ProfileBEN for Protein Generative Modeling From Theorem 3.2 jective function for the training of protein family profile:
is easy to arrive at the ob-
((P)
8(OK])P
-
(9)
2
4

---

Published as a conference paper at ICLR 2025
The p( is the network part, where it takes independently accumulated information about the pro- files 0f as input and tries correlate and guess the true profile. The accumulated information about the profile of can be computed through the Bayesian flow: 0! pr (r p t) During training is sampled uniformly from (0, 1).
Training Strategy We faced a similar representation generation quality trade-off as described in ESM3 (Hayes al. (024) and DPLM (Wang et al.] 2024). Intuitively a smaller & would result in learning with lower quality input, whereas larger would make the objective trivial. During training, for 90% of the time, sample t independently for each amino acid position, and for the remaining 10% of the time, the entire profile is trained with the same t. Additionally, as in DPLM (Wang et al., 2024), our backbone is first trained with masked language modeling objective.
prreseetttion-gggeeaaaioonn
3.3
FAMILY PROTEIN GENERATION
Given a protein family profile (P(1) 1m initial time step lo, then for in [O.... MI, 1j tively:
Jisl C CAK- K-
we first compute its Bayesian flow up to some + to do the following calculation itera- (1-to)í
N
0 ( pr
4z)
(10)
Pa:0+1) (0
(1)
mm)
(11)
Where the initial (P() @= is set to {P())_ Finally we take the arg max sampling over (P(0 (1(+ Im to get the generated family protein sequence, the ith amino acid can be decoded
as follows: a() arg max; (P()) (x(N+)))
The initial time to plays a critical role in the sampling process, setting to too small would lead to severe loss of information from the conditioned sequence family, while setting to too large may limit the exploration of possible proteins, For individual protein sequences, we set to to 0.3. However, profiles typically exhibit greater variance, necessitating larger initial time step. In our experiments, we set the initial time step ty to 0.6 when sampling from a family profile.
EXPERIMENTS
In this section, we validate the advantages of ProfileBEN in family protein generation and protein representation learning through extensive experiments. In the following paragraphs, we present the outstanding performance results of ProfileBFN in family protein generation and protein represen- tation learning tasks, and provide an in-depth analysis of these results. Finally, we analyze the sampling process of ProfileBFN, revealing its efficiency and the biological meaning inherent in this process.
comprehensive overview of the training and evaluation configurations, including the metries used, is provided in Appendix D
4.1 MAIN RESULTS
ProfileBFN Leads in Family Protein Generation We collected 61 primary sequences released by CAMEO starting from May 4, 2024, and searched for their homologous sequences using the same procedure described in AlphaFold2 (Jumper al., (021I. The models, whether provided with primary sequence or a set of homologous sequences, generate 1,000 sequences each for comparison. Refer Appendix D.2. for more detailed information on the experimental settings and evaluation metrics.
Table presents a comparison of the performance of different models in generating family proteins. Based on the results presented in the table, we provide the following analysis:
From structural perspective, Sequences belonging to the same family should share evolutionary information similar to that of the reference family. To evaluate this, we con- ducted non-parameterized contact prediction the generated protein sets using the Potts
5

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of sequence and structural metries mnnn-aarmmetric cluster-level) datasets collected from CAMEO. The results indicate that ProfileBFN outperforms in family protein gener- ation.
Sequence
Structure
Model
Div. +
Nov. 1
LR P@L t
LR P@L/21 t
LR P@L/5 f
Searched MSA
0.186
0.270
0.395
ESM-2 (150M)
0.565
0.691
0.086
0.116
0.167
ESM-2 (650M)
0.619
0.556
0.100
0.146
0.223
PoFT-Single (201M)
0.853
0.200
0.025
0.028
0.031
PoET-MSA (201M)
0.651
0.243
0.036
0.042
0.051
EvoDiff-MSA (100M)
0.225
0.668
0.061
0.089
0.168
DPLM (150M)
0.369
0.463
0.093
0.147
0,284
DPLM (650M)
0.445
0.411
0.102
0.159
0.303
ProfileBFN-Singg Single (150M)
0.368
0.646
0.126
0.197
0.321
ProfileBFN-Singe (650M)
0.421
0,581
0.162
0.262
0.422
ProfileBFN- FN-ileBFN-Profile (150M)
0.283
0.650
0.128
0.210
0.384
ProfileBNN- Profile (650M)
0.293
0.173
0.280
0.474
model implemented in CCMPred. the LR P@I., LR P@L/2, LR P@ L/5 the precision L, L/2, and L/5, respectively. This approach was chosen because parameterized mod- els such as ESMFold or AlphaFold are prone to hallucination issues, as demonstrated in Appendix D.2.2
ProfileBFN shows a considerable advantage, with its performance metrics even surpassing those of MSA obtained through search methods (see the first and last rows). This find- ing suggests that the sequences generated by ProfileBFN effectively capture the structural characteristics of the family, example illustrating this is provided in Figure [
From the perspective of sequence analysis, we expect the generated sequences to exhibit adequate diversity and novelty. To measure diversity, we use the mean identity value among the generated sequences, denoted as Div. Novelty is assessed by calculating the maximum identity between the generated sequences and natural sequences, with novelty defined as 1 max(identity) and denoted Nov.
ProfileBFN excels in terms of diversity and novelty. These results indicate that Pro- fileBFN can generate diverse and novel sequences without suffering from severe mode collapse and ensures the production of varied outputs.
Compared diffusion competitor, DPLM, ProfileBEN consistently outperforms across all metrics with significantly better results at different model sizes (rows 7, 8 vs 9, 10). This demonstrates the superiority of BFN in handling discrete variables over diffusion models.
Comparing the performance of ProfileBFN models in different sizes, larger models gener- ally capture family structure characteristics more effectively. However, they show a slight decline in performance regarding diversity and novelty. This is primarily due to the antago- nistic relationship between structural conservativeness and sequence diversity and novelty.
Regarding the input types for ProfileBEN, utilizing a profile derived from multiple sequence alignment (MSA) as input offers superior structural performance compared single quence, while also enhancing diversity and novelty. This is because the profile or MSA contains richer structural information and ferent sites. As a result, the model can more effectively capture structural features while ensuring diversity and novelty by modifying the more flexible sites.
accurately reflects conservation across dif-
Following PoET (Truong Jr & Bepler 2023). we also use the structure prediction model ESMFold to evaluate the performance different models (see 5) Based on the results shown the figure, the sequences generated by ProfileBEN exhibit higher pLDDT and Max TM-score values, indicating that ProfileBFN still holds an advantage in capturing structural conservation at the instance- metrics. In terms of novelty, ProfileBEN ranks in the middle, but still offers sufficient number of novel options. contrast, while EvoDiff excels diversity, does not effectively capture the structural conserved features of the family. Overall, ProfileBEN still delivers the best performance in

---

Published as a conference paper at ICLR 2025
Long Range P@L: 51.5
Long Range P@L: 39.7
175
175
150
100
100
75
50
25
0
25
50
75
100
125
150
175
125
Figure Example of contact map obtained using ProfileBFN (left) and Searched MSA (right). The family sequences generated by ProfileBFN achieve more accurate predictions than the Searched MSA.
family protein generation. We provide three cases generated by ProfileBFN in Figure 2 important to note that the parameterized instance-level metrics have significant flaws, we provide further discussion in the Appendix D.2.2
However, it
6ET6
2ANX
2ANV
PLDDT: 90.5
PLDDT: 90.5
pLDDT: 88.5
TM-score: 0.967
TM-score: 0.950
TM-score: 0.946
Seq-ldentity: 0.068
Seq-identity: 0.123
Seq-Identity: 0.124
Reference
Generated
Figure 2: Three structurally conserved but sequence-novel lysozymes generated by ProfileBFN.
ProfileBEN Generates Functional Proteins We utilize the enzyme function prediction model, CLEAN (Yu et al. 2023). classify and evaluate enzymes generated by multiple models. Specif- ically, we focus on three representative categories of catalytic enzymes, each extensively validated experimentally. Models in consideration generate new enzymes based on reference sequences from each category. Subsequently, we use CLEAN to predict the EC numbers of these generated en- zymes, thereby assessing their catalytic activity. Refer to Appendix E. for detailed information on the experimental settings and evaluation metrics.
From the results the table 2 we measure Accuracy Uniqueness, more extensive results are shown in Table 6 likely to possess the corresponding functions. From functional perspective, ProfileBEN provides the best capability for generating family proteins.
observe that the enzymes generated by ProfileBFN are considered more
From Table PoET achieves the highest accuracy among all models. However, it suffers from mode collapse, leading to relatively low performance when evaluated using the combined metric of Accuracy x Uniqueness. In contrast, both ProfileBFN and EvoDiff generate a variety of results without observing mode collapse.

---

Published as a conference paper at ICLR 2025
Performance on enzyme tasks. We report the Accuracy Uniqueness metrie, complemen- tary results can be found in T The results show that the enzymes generated by ProfileBFN are likely to be considered as having corresponding functions.
Table 2:
Model
P40925
Q7X7H9 1
Q15165 1
PoET-MSA
3.00%
33.3%
0.05%
SvoDiff-MSA
27.93%
88.69%
1.39%
ProfileBFN-Pron Profile (650M)
95.19% %
98.98% %
42.67 %
ProfileBEN Understands Proteins Deeply To evaluate ProfileBFN's ability to represent proteins, we assess its performance on several protein prediction tasks (Wang al.] 2024 Su et al. 2023 Dallago et al. 202I), including protein function prediction (thermostability and metal ion binding), localization prediction (DeepLoc), annotation prediction (EC and GO), and protein-prove protiin inter- action prediction (HumanPPI). Following DPLM (Wang et al. 2024). we conduct full-parameter
supervised fine-ture seeettnnig on each dataset.
We use accuracy (ACC%) the primary evaluation metric for most representation learning tasks. For thermostability, we compute Spearman's correlation (Spearman's p) (Zar 22005), and for EC and GO annotation tasks, detailed metric description.
the maximum Fi-score (Fmax). Refer to Appendix D.2.3 for
Table 3 shows the performance of different models across various prediction tasks. Based on the results in the table, ProfileBFN outperforms its discrete diffusion competitor, DPLM, across all task metrics (see the last four rows) [ The improvement in performance is attributed to the smoother data denoising process of BFN compared to discrete diffusion, as well as the removal of the adverse impact of unnatural MASK tokens on protein data. Specifically, BFN takes into account changes in the probability distribution of amino acid types at different positions, enabling the model to learn more detailed information about amino acid co- variation (e.g., the probability of amino acid types at two positions increasing or decreasing simultaneously). In contrast, discrete diffusion only considers changes in amino acid types (i.e., both positions undergo type switch), resulting in a coarser granularity of model learning. Moreover, the BFN framework eliminates the need to introduce artificial MASK tokens, avoiding inconsistencies between upstream training and downstream tasks. The benefits of removing the MASK token have also been reported in the field of natural language processing Yang 2019).
Moreover, ProfileBEN demonstrates comparable performance to SaPort (Su et al. 2023). which ex- plicitly utilizes protein structure information. This indicates that ProfileBFN has also developed a profound understanding of protein structure through learning from large volume of protein quences. However, it should also be noted that for tasks directly related to structural information, such HumanPPI, SaProt still maintains leading position, suggesting the necessity of integrating structural information into ProfileBFN in future work.
4.2 SAMPLING PROCESS ANALYSIS
In this section, we analyze the sampling process of ProfileBFN, including sampling efficiency and the biological meaning implied in the sampling process.
ProfileBEN Achieves Higher Sampling Efficiency Figure shows a comparison of sampling times for different models when generating a protein of varying lengths. As observed from the figure, across different model sizes and protein lengths, ProfileBNN consistently demonstrates higher sampling efficiency compared to our main competitor, DPLM. Moreover, this advantage becomes more pronounced as the protein length increases. Although both DPLM and ProfileBFN utilize similar ESM-2 network backbone, the need for resampling trick (where each sampling step requires the model to infer twice) during DPLM's sampling process leads to significant difference in their sampling efficiency. Compared ESM-2, ProfileBFN incurs a slight loss in efficiency.
According to the results provided by Wang et al. 2024). the performance of ProfileBFN and DPLM is comparable, with each having its own strengths and weaknesses. However, based on our replication experi- ments, ProfileBNN consistently outperforms DPLM. This may be attributed to the unstable training process of DPL.M
8

---

Published as a conference paper at ICLR 2025
Table 3: Performance various protein prediction tasks. ProfileBEN shows strong understanding of proteins. *; protein structure is provided. t: results are quoted from SaProt (Su et al. 2023). Q: results are quoted from DPLM (Wang et al. 2024). o: results are reproduced by us using the official code and data. Our model is compared with the o version of the baseline models, if multiple versions exist.
Thermostability
HumanPPI
Metal lon Binding
BC
GO
Deepl.oc
Model
MF
BP
CC
Subcellular
Binary
Spearnan's
ACC(S))
ACC(S)
Finax
Finax
Finax
Finax
ACC(%)
ACC(%)
SaPrott 7
0.724
86.41
75.75
0.884
0.678
0.356
0.414
85.57
93.55
MIF-ST* f
0.694
75.54
75.08
0.803
0.627
0.239
0.248
78.96
91.76
ESM-1 (1B) f
0.708
82.22
73.57
0.859
0.661
0.320
0.392
80.33
92.83
ESM (650M)
0.680
76.67
71.56
0.877
0.668
0.345
0.411
82.09
91.96
AR-LM (650M) 0
0.638
61.16
0.691
0.566
0.258
0.287
68.53
88.31
DPLM (650M) V
0.695
86.41
75.15
0.875
0.680
0.357
0.409
84.56
93.09
DPLM (650M) o
0.698
77.77
70.52
0.881
0.659
0.330
0.388
85,98
93.17
ProfileBNN (650M)
82.22
74.58
0.887
0.342
93.58
DPLM (150M)*
0.687
80.98
72.17
0.822
0.662
0.328
0.379
82.41
92.63
ProtieBEN (150M)
0.701
78.88
77.74
0.874
0.672
0.341
0.394
82.73
93.52
However, this gap narrows as the model size and protein length decrease. Notably, EvoDiff, which has the fewest model parameters, exhibits the lowest sampling efficiency. This is because the model requires MSA as an input for family design. When designing proteins of the same length, the actual input size for the model is larger, leading higher computational complexity.
Sampling Process Reflects Protein Conserva-
tion The sampling process of ProfileBNN is essentially transition from high entropy state to low entropy state. In this paragraph, explore the relationship between this pro- cess and the conservation of different protein sites. Specifically, we sum the entropy at each time step during the sampling process of Pro- fileBFN and compare with the results of the site conservation analysis using MSA. Figure presents an example of the extent of variability in the ProfileBFN sampling process alongside conserved protein sites analyzed through MSA (lysozyme Q37875). The figure shows high consistency between the variation intensity at different sites during the sampling process and the conserved protein sites identified by MSA analysis. This indicates that ProfileBFN suc- cessfully captures the variability and conserva- tion of different sites on the protein, and during the sampling process, reflects this by controlling the extent of amino acid variation at these sites.
DPLM_160W
DPLM_650m
ProfileBFN 11000
ProfileBFX 660ma
50
ESM2.150m
sampple timel()
B5M2 650ml
Evodiff 100m
50
10
100
200
300
100
500
600
soquenee length
Figure 3: Sampling efficiency comparison. Pro- fileBFN has higher sampling efficiency com- pared to its competitors.
RELATED WORK
De novo protein design methods constructs entirely new protein sequences that do not based on homologs. It perform self-supervised learning from large protein databases (Consortium) 2015 Mirdita et al., 2017) Suzek al. 2007), aiming to model the evolutionary constraints across various families (Koonin et al. 2004 Meier et al. 2021 Lin et al. 2022). It demonstrates its advantages in scenarios designing proteins for entirely new properties, especially in cases where there is limited
homologous information (Madani al. 2020 Nijkamp et al.) 2023 Meng al. 2023). However, it performs poorly in tasks involving the design of new proteins within large protein families.
Mutation-bases directed evolution approach mimics the process of protein evolution. By train- ing on evolutionary-seat protein sequences, capture key sites in protein evolution and model

---

Published as a conference paper at ICLR 2025
Conserved
Variable
ProfileBEN
MSA
Figure 4: ProfileBFN's sampling process implies the conservation of proteins.
protein evolution process (Alamdari et al. 2023 Wang al. 2024 Watson al., 2023). It can design proteins that can be verified by wet experiments, but limited by the way of evolution from wild-type, they cannot generate diverse protein sequences for reaching the optimal proteins.
Protein family design is a protein design process that models homologous protein sequences additional signals. These models be further categorized into autoregressive and non- autoregressive models. For example, PoET (Truong Jr & Bepler] 2023), MSAGPT (Chen et al.) 2024). and ProtMamba (Sgarbossa et al.) 2024) are autoregressive models that take sequentially concatenated sequences as input and generate new proteins autoregressively. In contrast, EvoDiff- MSA (Alamdari al., (023) uses TSSansfosfrrmer (Rao et al. 202T) as its MSA module, takes an MSA matrix as input, and generates new proteins in non-autoregressive manner.
CONCLUSION
In this paper, we have made significant contributions to the field of protein sequence generation with several key advancements. We extended the Discrete BFN to design the ProfileBFN model, which effectively utilizes protein family profile information for generating family- protein sequences. Through formal derivation, we introduced new Bayesian flow and loss component, making the ProfileBEN versatile and applicable any data with profile characteristics.
Our ProfileBFN model can accommodate both single-sequence and multiple-sequence profiles. This flexibility allows to train single-sequence data while generating sequences using multi- sequence profiles, thus avoiding the costly process of constructing profile training datasets.
Our model demonstrated exceptional performance in both representation and generation tasks. The generated sequences showed biologically meaningful variations in the amino acid positions, which crucial for practical applications in protein engineering and functional analysis.
Overall, our proposed ProfileBFN have exhibited robustness, efficiency, and biological relevance, offering promising tool for protein sequence generation and functional studies.
ETHICS STATEMENT
In conducting our research on the Profile Bayesian Flow Networks (ProfileBNN) for generative modeling of protein families, we have adhered to the highest ethical standards and address potential concerns as follows:
1. Data Use and Privacy Our research did not involve human subjects or private data. All protein sequence data used in our experiments were obtained from publicly available databases, which are free for academic and scientific research use. No identifiable personal data were used or generated.
10

---

Published as a conference paper at ICLR 2025
2. Potentially Harmful Insights and Applications The development of protein design tech- nologies, including our proposed ProfileBNN, has the potential for beneficial applications in fields such as medicine, bioengineering, and environmental science.
3. Bias and Fairness We have taken steps to ensure that model and methodologies do not inadvertently introduce bias in the generated protein sequences. The ProfileBFNmodel is designed be applicable to wide variety of protein families without favoring any partic- ular family or type. We emphasize the importance of continued evaluation and validation maintain fairness and accuracy in diverse biological applications.
4. Environmental Impact To minimize our environmental footprint, we optimized computa- tional resources by training on single-sequence data, thereby avoiding the need for large- scale MSA data construction and reducing computational power consumption. This ap- proach also contributes to the sustainability of scientific research practices.
5. Research Integrity We uphold the principles of scientific integrity and transparency in our research. All methods and results have been meticulously documented. We encour- age reproducibility by providing detailed descriptions of our algorithms and experiments, facilitating validation by other researchers.
In conclusion, while the potential applications of ProfileBFN offer significant advancements in pro- tein design, we remain committed to conducting our research ethically and responsibly, with careful consideration of potential implications and societal impacts.
ACKNOWLEDGEMENTS
This work is supported by the Natural Science Foundation of China (Grant No. 62376133), Wuxi Research Institute of Applied Technologies, Tsinghua University under Grant 20242001122 and sponsored by Beijing Nova Program (20240484682).
REFERENCES
Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew J Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pp. 1-3, 2024.
Jared Adolf-Bryfogle. Oleks Kalyuzhniy, Michael Kubitz, Brian D Weitzner, Xiaozhen Hu, Yumiko Adachi, William R Schief, and Roland L Dunbrack Jr. Rosettaantibodyyssip (rabd): A general framework for computational antibody design. PLoS computational biology, 14(4)e10061 12, 2018.
Sarah Alamdari, Nitya Thakkar, Rianne van den Berg, Alex X Lu, Nicolo Fusi, Ava P Amini, and Kevin K Yang. Protein generation with evolutionary diffusion: sequence is all you need. BioRxiv, pp. 2023-09, 09, 2023.
Ismail Alkhouri, Sumit Jha, Andre Beckus, George Atia, Susmit Jha, Rickard Ewetz, and Alvaro Velasquez. Exploring the predictive capabilities of alphafold using adversarial protein sequences. IEEE Transactions on Artificial Intelligence, 2024.
Frances H Arnold. Design by directed evolution. Accounts of chemical research, 3253)1125-131,
1998.
Steven A Benner and A Michael Sismour. Synthetic biology. Nature reviews genetics, 6(7):533-543, 2005.
Jesse D Bloom and Frances H Arnol. In the light of directed evolution: pathways of adaptive protein evolution. Proceedings of the National Academy of Sciences, 106(supplement 1):9995- 10000, 2009.
Bo Chen, Zhilei Bei, Xingyi Cheng, Pan Li, Jie Tang, and Le Song. Msagpt: Neural prompting protein structure prediction via msa generative pre-training. arXiv preprint 4rXiv:2406.05347, 2024.
11

---

Published as a conference paper at ICLR 2025
UniProt Consortium.
Uniprot: a hub for protein information. Nucleic acids research, 43(D1):
D204 2212,,221,,22 2015.
Bassil I Dahiyat and Stephen L Mayo. De novo protein design: fully automated sequence selection. Science, 278(5335):82 87, 1997.
Christian Dallago, Jody Mou, Kadina E Johnston, Bruce Wittmann, Nicholas Bhattacharya, Samuel Goldman, Ali Madani, and Kevin K Yang. Flip: Benchmark tasks in fitness landscape inference for proteins. bioRxiv, pp. 2021-11, 2021.
Robert C Edgar and Serafim Batzoglou. Multiple sequence alignment. Current opinion in structural biology, 16(3):368-33 -03,,3332222 2006.
Alex Graves, Rupesh Kumar Srivastava, Timothy Atkinson, and Faustino Gomez. Bayesian flow networks. arXiv preprint arXiv:2308.070:33 2023.
M Michael Gromiha. Protein sequence analysis. Protein bioinformatics: from sequence to function. Elsevier Inc., New Delhi, India, pp. 29 62- 2010.
Tomas Hayes, Roshan Rao, Halil Akin, Nicholas J Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q Tran, Jonathan Deaton, Marius Wiggert, et al. Simulating 500 million years of evolution with a language model. bioRxiv, pp. 6024-07, 2024.
Steven Henikoff and Jorja G Henikoff. Amino acid substitution matrices from protein blocks. Pro- ceedings of the National Academy of Sciences, 99(22):10915-1099 1992.
Kevin B Johnson, Wei-Qi Wei, Dilhan Weeraratne, Mark Frisse, Karl Misulis, Kyu Rhee, Juan Zhao, and Jane I. Snowdon. Precision medicine, and the future of personalized health care. Clinical and translational science, 14(1):86-93, 93,88699,222 2021.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger,
Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. nature, 596(7873):583-588)) 2021.
Eugene V Koonin, Natalie D Fedorova, John D Jackson, Aviva R Jacobs, Dmitri M Krylov, Kira Makarova, Raja Mazumder, Sergei L Mekhedov, Anastasia N Nikolskaya, B Sridhar Rao, et al. A comprehensive evolutionary classification of proteins encoded in complete eukaryotic genomes. Genome biology, 5:1-28, 2004.
Michael Kosorok and Eric Laber. Precision medicine. Annual review of statistics and its application, 6(1):263-29) 226.,26666222 2019.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Allan dos Santos Costa, Maryam ZarandZ, Tom Sercu, Sal Candido, et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hic, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 37966637) 2023.
Yongchao Liu, Bertil Schmidt, and Douglas L Maskell. Msa-cuda: multiple sequence alignment graphics processing units with cuda. In 2009 20th IEEE International Conference on Application- specific Systems, Architectures and Processors, pp. 121-128. IEEE, 2009.
I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv: 1.05100, 2017.
Shitong Luo, Yufeng Su, Xingang Peng. Sheng Wang, Jian Peng, and Jianzhu Ma. Antigen-specific antibody design and optimization with diffusion- based generative models for protein structures. Advances in Neural Information Processing Systems, 35:9754-9767, 2022.
Stuart A MacGowan, Fábio Madeira, Thiago Britto-Borges, and Geoffrey J Barton. A unified analy- sis of evolutionary and population constraint in protein domains highlights structural features and pathogenic sites. Communications Biology, 7(1) (14):447 2024.
12

---

Published as a conference paper at ICLR 2025
Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R Eguchi, Po-Ssu Huang, and Richard Socher. Progen: Language modeling for protein generation. arXiv preprint wrXiv:2004.03499) 2020.
Ali Madani, Ben Krause, Eric R Greene, Subu Subramanian, Benjamin Mohr. James M Holton, Jose Luis Olmos, Caiming Xiong, Zachary Z Sun, Richard Socher, et al. Large language models generate functional protein sequences across diverse families. Nature Biotechnology, 1099) 1106, 2023.
Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alexander Rives. Language models enable zero-shot prediction of the effects of mutations protein function. bioRxiv, 2021. doi: 10. 10.1101/2021.001111111 450444 URL httt:::/www.boorsiv..rrrr.
10.1101 2021. 07. 09. 450648v1
Qiaozhen Meng, Fei Guo, and Jijun Tang. Improved structure-related prediction for insufficient ho- mologous proteins using msa enhancement and pre-trained language model. Briefings in Bioin- formatics, 24(4):bbad217, 2023.
Milot Mirdita, Lars Von Den Driesch, Clovis Galiez, Maria J Martin, Johannes Söding, and Martin Steinegger. Uniclust databases of clustered and deeply annotated protein sequences and align- ments. Nucleic acids research, 45(D1):DI70-DI16 2017.
Akash Nag and Sunil Karforma. heuristic approach to high-speed multiple sequence alignment for phylogenetic tree construction. IPASJ International Journal of Computer Science, 4(4):10-15, 2016.
Erik Nijkamp, Jeffrey A Ruffolo, Eli N Weinstein, Nikhil Naik, and Ali Madani. Progen2: exploring the boundaries of protein language models. Cell systems, 14(1 (1:66-97, 2023.
Tobias H Olsen, lain H Moal, and Charlotte M Deane. Ablang: antibody language model for completing antibody sequences. Bioinformatics Advances, 2(1):baac046, 2022.
Michael S Packer and David R Liu. Methods for the directed evolution of proteins. Nature Reviews Genetics, 16(7):379-39) 2015.
Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. Msa transformer. International Conference on Machine Learning, pp- 8844 6844-8856. PMLR, 2021.
Michael Remmert, Andreas Biegert, Andreas Hauser, and Johannes Söding. Hhblits: lightning-fast iterative protein sequence searching by hmm-hmmn alignment. Nature methods, 9(2):173-775 2012.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, (18(1)::2001223111 2021.
Xavier Robin, Juergen Haas, Rafal Gumienny, Anna Smolinski, Gerardo Tauriello, and Torsten Schwede. Continuous automated model evaluation (cameo)- on the future of fully automated evaluation of structure prediction methods. Proteins: Structure, Function, and Bioin- formatics, 89(12):1977-1996, 2021.
Jeffrey Ruffolo, Jeffrey J Gray, and Jeremias Sulam. Deciphering antibody affinity maturation with language models and weakly supervised learning. arXiv preprint 2rXiv:21 2021.
Stefan Seemayer, Markus Gruber, and Johannes Söding. Cempred and precise prediction of protein residue-residue contacts from correlated mutations. Bioinformatics, 30(21):3 30(21):3128-3130 2014.
Damiano Sgarbossa, Cyril Malbranke, and Anne-Florence Bitbol. Protmamba: a homology-awar aware but alignment-free protein state space model. bioRxiv, pp. 2024-05. 2024.
13

---

Published as a conference paper at ICLR 2025
Zhenqiao Song, Yunlong Zhao, Wenxian Shi, Wengong Jin, Yang Yang, and Lei Li. Generative zyme design guided by functionally important sites and mall--oleecule substrates. arXiv preprint 20Xx::24055088255. 2024.
Martin Steinegger and Johannes Söding. Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature biotechnology, 35(1 15(116)026-1028 2017.
Jin Su, Chenchen Han, Yuyang Zhou, Junjie Shan, Xibin Zhou, and Fajie Yuan. Saprot: Protein language modeling with structure-aware vocabulary. bioRxiv, pp. 2023-10, 2023.
Baris Suzek, Hongzhan Huang, Peter McGarvey, Raja Mazumder, and Cathy H Wu. Uniref: comprehensive and non-redundant uniprot reference clusters. Bioinformatics, 23(10):1282-1288,
2007.
Timothy Truong Jr and Tristan Bepler. Poet: A generative model of protein families as sequences- of-sequences. Advances in Neural Information Processing Systems, 16447,39-77415, 2023.
Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Xinyou Wang, Zaixiang Zheng, Fei Ye, Dongyu Xue, Shujian Huang, and Quanquan Gu. Diffusion language models are versatile protein learners. arXiv preprint 18857 2024.
Yajie Wang, Pu Xue, Mingfeng Cao, Tianhao Yu, Stephan Lane, and Huimin Zhao, Directed evolution: methodologies and applications. Chemical reviews, 121(20):12344-222) 2021.
Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eise- nach, Woody Ahern, Andrew Borst, Robert J Ragotte, Lukas F Milles, et al. De novo design of protein structure and function with rfdiffusion. Nature, 62097976)11099-1100, 2023.
Ruidong Wu, Fan Ding, Rui Wang, Rui Shen, Xiwen Zhang, Shitong Luo, Chenpeng Su, Zuofan Wu, Qi Xie, Bonnie Berger, et al. High-resolution de novo structure prediction from primary sequence. BioRxiv, pp. 2022-07, 2022.
Kevin K Yang, Niccolò Zanichelli, and Hugh Yeh. Masked inverse folding with sequence transfer for protein representation learning. Protein Engineering, Design and Selection, 36:gzad015, 2023.
Zhilin Yang. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv
preprint arXiv:1906.082.77 2019.
Tianhao Yu, Haiyang Cui, Jianan Canal Li, Yunan Luo. Guangde Jiang, and Huimin Zhao. Enzyme function prediction using contrastive learning. Science, 379(6639): 13583991:338833366 1363, 2023.
Jerrold H Zar. Spearman rank correlation. Encyclopedia of biostatistics, 7, 2005.
Jun Zhang, Sirui Liu, Mengyun Chen, Haotian Chu, Min Wang. Zidong Wang, Jialiang Yu, Ningxi Ni, Fan Yu, Dechin Chen, et al. Unsupervisedly prompting alphafold2 for accurate few-shot protein structure prediction. Journal of Chemical Theory and Computation, 19622)8440-8471, 2023a.
Le Zhang, Jiayang Chen, Tao Shen, Yu Li, and Siqi Sun. Enhancing the protein tertiary structure prediction by multiple sequence alignment generation. arXiv preprint 27X.:2230.00/824 2023b.
A PROFILE BFN DERIVATION
A.1 THE ESSENCE OF BAYESIAN FLOW NETWORKS
For easy understanding, the reader can treat the variables as discrete variables. Without loss of gen- erality, the formulation can be easily extended to continuous variables by swapping the summation with integration. This section reviews the essence of Bayesian Flow Networks (BFN) et al.) 2023) in more simple language, there is defined noisy channel x leaks it's information z; ~ [[=]x: w). An observer then receives the leaked information and updates its belief about the variable x through Bayesian update and obtain belief about p(x 71:n).
q( ()),x;)), through which variable
14

---

Published as a conference paper at ICLR 2025
In bits-back coding scheme, the total nats required to transfer with Zi:n intermediate latent - llg p(Z1:)) - log p(([Zim) with - log ((Zin//))) nats put back, so the expected marginal nats required to transfer data from p(x) is:
By(()) By(zun (kw) |- log p((mm) - log p(x log (21mm)((mx))
((71m (x; Betan (n log p(x 71mm)
= Ep(s)
(elaailssw) log
-VBB
p(zin)
B(x)
Bo log p(
(12)
With the conditional distribution of the noisy channel g(Z1m (x) and series of observed variables Ziom following bayesian update rule, the udpated belief of the variable is:
((21m((x)((x)
(x|21m)) Ex (Z (x)((x)
(13)
There could be sparsity problem or curse of dimensionality problem when the variable x is high- dimensional. Thus m-dimensional dently with bayesian update rule. To model the interdependence between variables, an neural net- work is introduced to rectify the posterior distribution p(m ((),, ((m),, where Q is the governing parameter of the posterior distribution of the i-th component and determined by Zim.
treated as independent variables, and updated indepen-
P = fø( Iimm ((1).
((m)))
(14)
(0 ((m)
(15)
Without knowing variables { -.. )n are correlated variables, p(Zi.) in Eq. 12 is then fac- torized autoregressively as p(Zı:n) p(z1) IT p(z, (1:-1) 1), and further parameterized combining
the output distribution pΦ from the neural network in Eq. 14
p(Zim) = p(z1) II p(ze/zia-1)
Σ q(z (x)o(
I ((21x)po(((/zss11
px (2: 11
(16)
Plug Eq.
16 and Eq.
14 into Eq.
12 the -VLB(Ø)) is then:
-V) = Ep(x)
DKL. (a(2 (w:w)lllxzllziieee. (2 Dutt - Be(eem (aw) log Po (x|2
(17)
The -VLB(Φ) the expected marginal nats required to transfer data from p(x), with the transmis- sion system parameterized by Φb The objective is to minimize the transmission cost, and the model trained by minimizing the VLB())).
A.2 PROFILE BAYESIAN FLOW NETWORKS
In the original BFN paper, the continuous variable is regarded as the latent variable that is used for data transmission, treats each components of the categorical variable as binary variable from which y is derived with central limit theorem.
As it's not so straightforward to treat the continuous variable as the latent variable, and kind of "wrong" to treat the categorical variable as binary variables for derivation, we will derive the discrete Bayesian flow from a different perspective, where the latent variables z, are still the discrete evidences that leaked from the data.
We arrive at the Theorem 3.1 that describes the continuous time discrete Bayesian flow with proper derivation and proof.
15

---

Published as a conference paper at ICLR 2025
Derivation and Proof of Theorem 3.7 The noisy channel based on profile is actually a general- ization the original BFN's noisy channel, where the profile is one-hot vector. Exchanging the one-hot vector with profile
be seen hierarchical sampling process: first sample one-hot vector according to the profile, then do the same as the original BFN. Still, we consider the transmis- sion of noisy samples {7, ) i=1 sequential update of the belief of the variable in the profile, and finally push + +00. Since sequential Bayesian update is equivalent to batch Bayesian update, Vx:
4(7m x)p((x 21:n-1)
p(x|Za=n)
Ex (Zn ())((=) Z1:n-1)
Define mx(xx= = p(x(Z1x)), we have the following recursive form
((z))))T=-1()) E. ((
(x(x)
Where ((<x)x;0)) = -س: + +w,12,=x is the one-hot noisy channel (the second hierarchy), wa omitted R
for brevity. After observing new evidence z; the posterior distribution is:
1-w + w/871x) TT: 1xx)
(x)
(mا + w/ (x
w; Ti- 1x-1
I-wi (
where 8.. is the Kronecker delta function.
We then analyze how the observed evidence will affect the distribution in the log space, the aceu- mulated log probability of the distribution is:
In(x,(x)) (n(==1(())) In
Wi
+C
K
In
1-ن
+ w:)
+ C 4= x.
K
In
2 x x, W + C
where C= In -I- (1 + (1m;-1((74) is constant that is irrelevant
Notice that when observing an evidence there will be an extra "energy" on the index matching the evidence by
1 Wi + wi
Wi K
Kw 11 - wi
In
K
Following Graves al. (2023 this term is defined as:
In & der In 11 Kw
Wi
Below we assume all w 's are equal and simply denote (wi, Ee) as (w.
Now analyze the situation of having observed m(m < n) evidences. Assume there ex evidences observed for x such that E, c = m, then the built up log probability for after observing evidences is
In( c. In + (x) C
(18)
16

---

Published as a conference paper at ICLR 2025
The Cx's are the counts of the evidences observed, which follow multinomial distribution M(m. 1-w + wp), so the expectation, variance and covariance of the counts are: K
w
B[cx]
+ wpx
K
1 -
w
(1- [-- -
+ WPs )
Varlcx]
+ wpx)
K
K
1 - w
(+ (==- + wpx (x xx)
Covlex, Cx]]
+ wwx
K
K
Define yx (cx m ) E, the corresponding terms are:
Ely = mwpp I
w
(1- (1-(>~-)) + twpp)) Inn
Varly,] = m
wPx)
K
11-6
+ wp.) (1 (1-W + upx In E(x xx)
Covlyx;Xx1 = -
K
K
Note that n + to0 + w + O, the first order Taylor expansion of In is:
Kw
Inf
1 1- + O
According to the definition and assumption that all wi 's are equal, mw² = ((()) so the expectation and covariance matrix y are E = K m p and 3 m ) with Egj = -11=,- As + t00, n
is replaced by B(t) with continuous time t.
We need to control the expected energy built up for each category to be bounded, thus ((1) need to be bounded.
As the latent variable for data transmission different from the original BFN paper, the KI. term in should be rederived fit the new setting. We propose Theorem 3.2 that describes the KL term in the continuous time discrete Bayesian flow with proper derivation and proof.
Derivation and Proof of Theorem 3.2
lim 2-+0000
(DKL(atzlp)ll(tz)
lim
Σ
logg((()) n Σ p((pp) logp(z)
(19)
Rearrange the inner right term in Eq.
19
n Σ
( 1 11
Σ (gzll)) log
+ PP(())
K
Σ -ng(z (z) p) log K
Σ na(zlp) log(1 (Kpp(z) 1)w)
(20)
Apply second order Taylor expansion on the right term in Eq. 20
Σ
nq(z)p) log(1 + (Kpo(z) - 1)w)
"n Eng(zlp) (KPo(2)) - 1)w マ (Kpo(2) - 1)2w² + o(w))
Σ mg(z)p)(Kpo(z) -1) 2 Σ ng(zlp)((po((z) 1) o()
(21)
17

---

Published as a conference paper at ICLR 2025
The first term in Eq. 21 can be expanded as:
Σ (K(Z)p)((Po(z) 1)w
(1 w- twplz)) (Kpo(z) - 1)
Σn n
K
Σ
w (Kpo(z) 1)w + nw² ((())( -1)
K
=0 + 3
E (())( -) B E((z)(KPolz - 1)
(22)
The second term in Eq. 21 can be expanded as:
Σ
(a(2lp) w)²
Σ3
1 --
+ wp(z)) (Kpo(z) 1)²
K
Σ.
3
w (Kpolz) - 1)² + Σ (p(x)(Kpo(()-1) 1)²
K
Σ 1- w
(K2 P (z) +1 2K po)l))) + Bw [(zz)(Kpo(2) 1)2
+ 3K p 1 + ((1)
(23)
Plug Eq.
22 and Eq.
23 into Eq. 21 and Eq. 21 into Eq. 20 and Eq. 20 becomes:
n
g(z)x) Po
log + 3 + (z) -1)
1
(-3 + 3K + o(1)
2
1
= log K + 3K Σ p(z)pa(z)
3 2
|K] |p
(24)
Similarly, plug the ps with p. into Eq. 24 then the first term in Eq. 19 can be transformed to:
Σ
1 kk|lp| 3 (1)
1
log K
(25)
2
Since 3 = nw? is bounded as - +o0, the o(1) term is negligible. Plug Eq. 24 and Eq. 25 into
Eq. 19
we have:
lim n (Paa((((2)))1l 3 11kK p||
(26)
+0++000
For (t) that satisfies ((t) lo (7) 0, ((1 const, the limit of the KL divergence can be easily derived by the same method with the following substitution:
(tt)att
(27)
1
n At nw (t),,
(28)
(29)
lim lim n-++o At-0
(30)
18

---

Published as a conference paper at ICLR 2025
and the resulted KL divergence is:
lim (Dkale(t/pprt)lllezt))) +=+
(31)
1 8()))/ p||²
lim (kule(2lp:t))llizzt)) At At
(32)
seems although starting from a different perspective from the original BFN paper, the derived KL. term arrived at the same form the original BFN paper.
As the reconstruction term in the right of Eq. 17 will trivially approach when 3(1) is sufficiently large, the training loss for some p at some step t is then:
1 [(x) - = (1)Klpp pl?
(33)
B ALGORITHMS
Algorithm Training Loss Procedure
Require: 31 E I. vocabulary size K E E+, a neural network (0 ... 00m)) f), where Φb is
the parameter of the neural network.
Input: profiles (P() M
CAK-1 where m is the sequence length
(O0 1) Be 481 yt ~ N(K3,,(0) B.C)
0 softmax(y
1==1 = (0 .... (m)) () (p(i) m
C(P) Ei=1 3 (P - 1
Return L(P)
Algorithm Family Protein Generation Procedure
Require: 31 E R. vocabulary size K Zt, initial time to, sampling steps N, neural network (0()), O(()) 1), where & is the parameter of the neural network.
Input: profiles {p( Jiz1 C CAK-I of certain protein family, where m is the sequence length.
for j = to do
t + (1-ta)i + to
3, < tB,
(() N(K3,p(() 3.C)
g(0) = softmax(y(1
(P(() pm = (
0 (m) t)
end for
(1))
arg max (P(1))
Return [mm M=1
C DATASETS
C. EVALUATION DATASETS
Three datasets used to evaluate the performance of our model of protein family generation: dataset from CAMEO, enzyme families, and phage lysozyme families. The dataset collected from CAMEO, which contains 61 proteins with Homo-oligomer Assessment detailed in Table 4 was
19

---

Published as a conference paper at ICLR 2025
Table 4: Detailed information of each protein for CAMEO dataset.
ID
PDB
Chain
Leuga
Tile
XBLS
145
Casital Sincture of Sunil 26
2
SFSQ
Guinea pig sislic acid exeeasee
SPOR
Rahrit cullic acid eitrase
SPSL
116
Homen Menstheii bound nearraliing Orystal dinciunc planseltaasleee ltars SyL0CT0422500 comples with (IDP waate
domons atribuall
SHJP
KISO
Cyyttal descance of exccallkppeseeu claw beli-lastimmae, CESS.)
SIXT
Truscabalania in Comples Clateliio lesthaaii
XIDH
Crysul structure
CRISPR AcrIF25
XIGO
Crystal stracure Dalnccoccus racieduuuss copobyphoshhuaee Chystal stuctue Huml from Plasmodium falciparum
11
Alatine decurhoryyase
complex
SJRB
Crystd Souciare polynerase RCD-1 Neurosport beleenzyme
prospilles
14
XIYX
the
15
3805
Pseudouriline mouophsspate alvcosllaee Auabicopsis thaliana sulfait
16
38400
GarM FAD
17
SOV9
Crystal structure of
black poplar mustroom
18
SUNK
Stracture of N-terminal didomain d1-42 the Thrombospendin CAM.N//CPCCCooolllcc
demust-contaiinng 7A
19
BOYD
Trkll tnasmmmbrrane domain NMR srncture
20
S027
114
Pi donsin AKT-litr kinas Tequrosoma cntri
21
8PTH
118
Seratunc of ml complex with nuncledies
22
ROLD
603
Stuccuue of haman PADS Phoqphmiiiic musnt ViGE:44eF upo
23
SQUI
627
Crytal siuclue of the prcumnsucaa Substeze-lisding protcin ALD in open conformation
24
SULH
633
Orystal simettree of the paromococcal Subatate binding protein ABC is domain soapped dimer redacisse from Methamealleeeeem janneachi
25
RQPM
100
Strecton of metrlmmm--ttttheeeeeeeee Sinchim of SpNos IH domair: a basteral NADIH mnidee..
26
$QQS
A
222
27
ROVE
B
100
Deimeneess srriss TRO125 C-glacayl deglymooaae (CGD) will type cryatal crynprotectol with plyand
28
ROZI
136
Cystal emaciae of human twn purt deneiis peltssiam ion duand TREX-2 (K2P10. is comples with a unnilaally (N&58;
29
ROZ2
134
Crotal sirccure of human two pore dansin potasium knn chame! TREK-2 (K2P10) 1) in sougles with an isbibitery sandilly (N161)
50
80Z3
137
Crytul strucuue of human two pare domain potassim icn chance TREK-2 (K2P10 in congles with an uetiutory sasbody (N167)
31
SRIR
A
673
Trankiziolaee from Steptoeeceus preamonias is comples with thiamis pyrophsphate Transkesoiuee from Supphlocooees aueeus In complex with thimii perophosphare Hallocinated de nowo TIM barre with three helical extetsions HalluTiME I prgl: from plasid PCP10
32
3R35
A
677
33
KRHO
275
34
KS45
145
13
SSUC
100
2 NHU domain
36
The complex of TOL.1 cotodimaii bound LAT-1 Lectis domais
37
HSUF
complex TOL.1 cctodomain bound
Lectis
Procein Phosphatase complex
spezific Phosphatise targeting peptide (PtaTTAP) version Strocture of Sasti (ype tresdoes Crystal stricture Tyrp 36,0101
39
from Suppelacocess suress MW2
Crytal structure
besamenc hazelnst allergen
Cryttal
allengen
Switdgrass Clulcon
AVPP
Suggham Clulkon:
45
SWID
CRYSTAL STRUCTURE OF DPS-LIKE PROTHIN PAUSS0 FROM PSELIDOMONAS AERUGINOSA (DIMERIC FORM) Stastuni X-ny singttuee GAF-PHY domains of Sylt-Cphl
SWAY
somerication Anhiāqpis
ICS)
47
SW26
429
48
KWST
488
Orytaal eineturee USU comples UPS
40
SWEX
468
Croaal simcture of Narrtyl sige amidttrrmfeeee from Legineelaa preumsphill
50
SWOD
100
Coystal sinciune GB97 plandetinatte from Fanobactrram johnsenia cumples with classe
51
SWOP
100
Crysta drustune of Andiikpis daliam UDP glacee 4-spincx ax: 2 (AIL/GE2) complaue with UDP, willdype
52
SWTB
187
Cryld simclize Mc complex truncatzd by diymutypsis
53
SWU7
306
Stracture of cis Gerany lalaassyyy Diphesplute Syuilase from Suepaam clavaligonus
54
8X35
B
34
Cryuul stucure human WDR3 in comples with PTEN
55
XXJE
B
153
Cryttl stricuue of the Yge protein from Compsfobucee Jejuni
56
XXIG
183
Crssul smoture of the YgeY protece from Vihnio parahaemolyticas
57
SYSP
256
Crysaa struturee of bactrnal actrainng silferradeereee Sgd02
58
RYXK
201
X-ny struture of Cosiidiidaa difficlee indolysin Ec diacoaniailas Somsin.
59
9BIR
562
Fanctional inplistiin of the hemotrmers mulidommin vacudas sonting receptur 1 from Anhidoppis hudiuna
50
9BCZ
A
644
Chickes phooppuidyynnsiiill 4,5 hisphoopuue phosph.dissensse 2ot. I (PLCZ)) in comples with calcium and phosphocylzed treeniiee Crysal stucture of Sachhmmmyes cerevzine pll zine sensiove protein I (PNS1)
M
4P63
A
572
Table 5: Detailed information of enzyme data.
ID
EC
Length
family
P40925
1.1.1.37
334
malate dehydrogenase
Q7X7H9
2.7.1.71
287
shikimate kinase
Q15165
3.1.1.2
354
arylesterase
introduced for our model to design protein sequence families separately and based on Multiple Se- quence Alignments (MSAs), forming results by evaluation of CCMPRED (Seemayer et al. 2014).
All targets were filtered from the CAMEO submitted target list, and those discovered before May 2024 were excluded to avoid potential data leakage. Three enzyme families were used to validate our model's ability to generate MSAs with correct functional annotations following (Song et al.) 2024). with detailed information provided in Appendix E.I] forming result by a scoring model CLEAN Yuu et al. 2023). Additionally, lysozyme families were generated and folded into structures using ESM- Fold, tollowing the PoET method (Truong Jr & Bepler 2023) paper, thereby complementing our structural results. All detailed information about evaluation benchmarks are provided in D.2
20

---

Published as a conference paper at ICLR 2025
D EXPERIMENTAL DETAILS
D.1 TRAINING CONFIGURATION
Training Dataset In line with ESM-2, we use protein sequence data from the UniRef database (Suzek et al. 2007) (as of March 2024) to train ProfileBFN. Our training data selection strategy also aligns with ESM-2, starting with an even selection of cluster groups from UniRef50 results, followed by random sequence selection within these clusters based on UniRef90 clustering. In total, the training involves 190 million protein sequences. Notably, although ProfileBNN utilizes MSA profiles as inputs, it does not require the construction of additional profile data, but merely uses existing sequence data for training, which greatly simplifies the implementation.
Training Hyperparameters We use the same Transformer (Vaswani, 2017) module ESM-2 to implement ProfileBNN. For the ProfileBNN model with 650 million parameters, has 33 layers of 20-head self-attention blocks. The hidden and embedding dimensions are 1280, and the feed- forward hidden size is 5120. Note that, unlike the ESM-2 model, we do not use any form of dropout for regularization, the Bayesian flow itself provides sufficient stochasticity. For the Bayesian flow, (11) implies the uncertainty of the last step in the modeling procedure. Based on our empirical expe- rience and cases in the original BFN paper (Graves al.) 2023), we found it could be approximately set according to the equation (eta(1) K = constant (K is the vocab size). With this principle. we could directly obtain a good setting of 3(1) following the previous empirical parameter in Graves et al. (2023) where K is different. We consider three different candidate schedule functions for (tt), linear, square and exponential, then we enumerate all three settings empirically over the small model (8M) and find linear works best in our task. We use AdamW (Loshchilov] (017) to train our model, setting the learning rate at 0.0001, which linearly decays to a minimum 4e-5. We adaptively set the batch size to approximately 2 million tokens.
D.2 EVALUATION DETAILS
D.2.1 EVALUATION OF FAMILY PROTEIN GENERATION
Settings The evaluation for family protein generation involves multiple proteins as targets for generation, including 61 proteins from CAMEO (Robin et al. 2021), phage lysozyme proteins, and three enzyme proteins. Detailed information on these proteins can be found in the Appendix [C.] When using a profile input, the hyperparameter to is set to 0.6; when using a single sequence as input, it is set to 0.3, For the construction of the profile, we first perform an MSA search in the Uniclust30 database (Mirdita et al. (017) using HHblits (Remmert et al.) 2012) based the natural sequence of the protein. Then, we obtain the profile according to the method described in the section 2.1 For each target protein, we require the model generate 1000 sequences (without removing duplicates) for evaluation.
Since the goal of the family protein generation is to generate a cluster of diverse and novel proteins with similar structures and functions, our evaluation metrics are based on three dimensions: sequence, structure, and function.
Metrics
For sequences, we expect the model to deliver diverse, and novel results. Therefore, we consider the diversity, and novelty of generated sequences as metrics.
Diversity: A model that experiences mode collapse, where the generated outputs lack diversity and can only produce a limited number of different proteins, cannot provide users with a rich set of candidate results. We use the mean value of the identity between generated sequences as a metric to measure diversity, denoted Div.
Novelty:
Similarly, a model that simply replicates the natural sequence is inadequate for supporting real-world design scenarios. A useful model needs to produce results that offer novelty. We measure novelty by calculating the maximum identity between the generated
(1 ((xd-mxx((deettt.])))
sequences and natural sequences, defined as the identity between ith among N generated sequence and jth reference sequence.
where identity,, denotes
N
21

---

Published as a conference paper at ICLR 2025
Proteins belonging to the same family typically exhibit high similarity in their tertiary structures. Therefore, structural evaluation of family protein generation primarily focuses on assessing whether the generated sequences contain the structural information corresponding to the proteins. For this purpose, we use the currently popular yet fragile parameterized instance-level evaluation metrics and more robust non-parametric cluster-level metries for evaluation.
Parameterized instance-level: Due to the promising advancements of protein structure prediction models such as AlphaFold2 (Jumper et al.) 2021) and ESMFold (Lin et al. ) previous work has utilized these models to evaluate the structures of generated family sequences (Truong Jr & Bepler (023). Specifically, following [Truong Jr & Be- pler (2023). we use ESMFold to perform structure prediction for each generated family sequence and report the predicted local distance difference test value (pLDDT) output by ESMFold. Additionally, we compare the predicted structure with the natural reference structure and report the maximum template modeling score, denoted as Max TM-score.
Non-parametric cluster-level: This metric is used to avoid incorrect model comparisons caused by bias in parameterized metrics. The instance-level metrics heavily rely pa- rameterized structure prediction models. However, AAlkhouri et al. (2024) have pointed out that structure prediction models can also produce structures similar natural proteins for adversarial samples based on the BLOSUM matrix (Henikoff & Henikoff 1992). This un- doubtedly undermines the reliability of parameterized metrics. Our experimental analysis shown in Appendix D.2.2 further illustrates that adversarial samples using the BLOSUM matrix merely replicate information contained in existing sequences, without providing new insights into our understanding of the family.
Based on the observations above, we design a more robust non- son-parametric metric based on a cluster of sequences to avoid this issue. Specifically, we require the model to generate cluster of sequences for a given family and explain the amino acid contacts in the reference structure by analyzing the mutations within the cluster using the non- non-parametric CCMpred tool (Seemayer et al.) 2014). Following Lin et al. (2023). we report the precision of the top L (length of the protemn,, L/2, and L/5 predicted long-range contacts (amino acid sequence positions differ by 24 or more) as the corresponding metrics, denoted as LR P@I. I. LR P@ @I22, and LR P@ P.1.5. In addition, Long-range contacts are challenging predict and are crucial for understanding protein structure, function, and valuable features (MacGowan et 2024).
The evaluation metrics for protein function are designed assess whether newly generated pro- tein members of given family still retain similar functions. Strictly speaking, evaluating protein function requires wet lab experiments: however, this process both expensive and time-consuming. Instead, perform dry lab assessments based a protein function classification model and have designed corresponding evaluation metrics. Specifically, we task the model with generating en- zymes, special type of protein, and classify the generated proteins using the widely adopted en- zyme function classification model, CLEAN (Yu et al.) 2023). We then assess whether the generated enzymes are correctly classified to determine if the family function is retained in the designs. The proportion of correctly classified results, after deduplication, is reported as a performance metric.
Baselines We select multiple strong protein design models as baseline models for comparison. Specifically, PoET (Truong Jr & Bepler, 2023) is an autoregressive model that uses known family sequences as prompts and generates new sequences for the family by continuously predicting the next amino acid. The model can generate sequences using either a single sequence or a multiple sequence alignment (MSA) as the prompt. Similar to us. EvoDiff (Alamdari et al. 2023) adopts a non-autoregressive generation approach. It leverages MSA to guide a discrete diffusion generation process, achieving the goal of family design. In aoneeeeteeeeessvve paradigm, we have extended the powerful protein language model ESM-2 (Lin et al. (023) to enable its application from protein understanding scenarios to family design. Specifically, for a given sequence in the family, we first mask 15% of the amino acids (consistent with the strategy used during training) and then iteratively replace the masked tokens with generated amino acids using ESM-2. The model most closely related to our ProfileBFN is DPLM (Wang et al. 2024). It is also trained on sarge-scale protein sequence datasets. However, while DPLM adopts a discrete diffusion framework, we utilize a Bayesian Flow Network capable of handling discrete data more smoothly. The original DPLM paper does not ad-
22

---

Published as a conference paper at ICLR 2025
dress scenarios involving family designs. In this paper, extend it to family designs by equipping with a sampling strategy similar to ProfileBFN.
Max TM scocc to natural solved structure
100
PLODT 1
0.8
ProfileBFN PoET Evodiff
ProfileBFN POET Evodiff
0.8
0.3
0.1
0.5 0.6
nn
0.0
0.
Max Seq ident natural
Max Seq ident natural seq
Figure 5: Sequence novelty and predicted structural conservation of phage lysozymes generated by ProfilBBNN, PoET and EvoDiff. ProfileBFN effectively captures the conserved structural features of families while providing sufficient novelty.
D.2.2 NON-PARAMETRIC: WHY IMPORTANT
We assert that non parammaattric methods, like CCMPRED (Seemayer al. 2014), offer distinct ad- vantages in evaluating generated protein sequences. To validate our hypothesis, we have conducted additional BI.OSUM62-bar hacking experiments, which reveal how structural evaluations, such as ESMFold's PLDDT scores, may not perform optimally in certain respects.
To challenge the efficacy of ESMFold, we employed the BLOSUM62 (Henikoff & Henikoff 1992) matrix to score the sequences after randomly substituting amino acid residues from the ground truth sequences. Subsequently, we selected those modified sequences with high scores and analyzed their predicted structures by ESMFold. With sequence identity threshold set at 0.4, we observed that most of these hacked proteins still exhibited favorable PLDDT and pTM scores; however, their structures, depicted in Figure 6 were erroncous and devoid of biological significance.
Additionally, we discovered that some protein samples generated by PoET faced a similar issue, indicating that ESMFold may not provide comprehensive evaluation. As illustrated in Figure 7 sequences with repetitions and those following simple patterns still received high PLDDT scores from ESMFold. To extent, the PLDDT scores in these cases reflect confidence because struc- tures, such as those resembling a stick, are easily recognizable.
D.2.3 EVALUATION OF PROTEIN REPRESENTATION LEARNING
Settings For the evaluation of protein representation learning, we assess the representations of ProfileBNN various protein prediction tasks (Wang al., 2024 Su al. 2023 Dallago al., 2021). These tasks include protein function prediction (Thermostability and Metal lon Binding), protein localization prediction (DeepLoc), protein annotation prediction (EC and GO), and protein- protein interaction prediction (HumanPPI) Following Wang et al. (2024), we perform full-parametoe supervised fine-tuning on each dataset.
We accuracy (ACC%) the primary evaluation metric for most tasks in represen- tation learning since these tasks are primarily classification problems, Accuracy refers to the per- centage of instances where the model accurately predicts the correct class for specific proteins in general it is computed N (uma)) where y, are the ground truth label and model predicted label, N is the total number of samples. In the context HumanPPI and Metal Ion Binding tasks, protein pairs are classified into categories based on whether they interact. For the DeepLoc task, classifications are made either into 10 classes for subcellular localization or into classes for binary localization.
Metrics
N
23

---

Published as a conference paper at ICLR 2025
GT
Hacked
ID
8SUF
8UAI
Identity
0.40
0.40
PLDDT
75.86
72.77
PTM
0.743
0.769
TM-score
0.809
0.736
LR P@LL5
0.043
0.012
Figure 6: Hacking ESMFold's pLDDT by BLOSUM62 Matrix
Spearman's rank correlation (Spearman's p) (Zar 2005) coefficient is a statistical measure that eval- uates the strength and direction of the association between ranked variables. It quantifies the degree of monotonicity in the relationship, meaning it assesses how well the relationship between the two variables can be described by monotonic function. In essence, it indicates whether an increase in one variable consistently corresponds to an increase or decrease in the other, regardless of whether the relationship is linear.
It is used to assess the relationship between the ground truth values of protein thermostability. as outlined by FLIP (Dallago al. (02T). and the predicted values. Specifically, calculated as follows,
6 de
1
d, = 8;
(34)
n(n2 1)²
the prediction and ground truth are both ranked in descending order where èt and indicates the predicted and ground truth rank.
Maximum F1-score (Fmax) is used for EC and GO annotation tasks. Fmax (Maximum F1 F1-score) is a metric that balances the precision and recall classification model, reflecting the best trade-off between these two factors. In classification tasks, predictions can be categorized into four types: True Positive (TP), False Positive (FP), True Negative (TN), and False Negative (FN). A threshold A E [0, 1] determines whether prediction is considered True False. Given N model predicted scores (s; [0, 1] corresponding labels are (4 E {0, [), the True Positive (TP), False Positive (FP), True Negative (TN), and False Negative (FN) Precision (P), Recall (R), F1 score (F1)
24

---

Published as a conference paper at ICLR 2025
high pLDDT
low PLDDT
PLDDT 72.34
80.63
81.42
Figure 7: Trivial cases of PoET generated repeated sequence with high PLDDT after ESMFold.
and finally Faa are subsequently calculated as follows:
NTP(A), NFP(A) 44122 E(M(ONN)
NA, NFN(A) = Σ - (G)()) E (1)
NrP(A)
P(A) = Nrr(A)+ + NFr(A)
NTr(A)
R(A) = NYP(A) + +N)
2P(A)R(A)
FI(A) P(A) + R(A)'
F max max (FI(A))
(35)
Baselines For evaluating representation learning, we use the following baselines: SaProt (Su et al. 2023 is a protein language model that is trained using sequence and structure tokens. MIF-ST (Yang al. 2023) is pre-training model that utilizes inverse folding structural guidance enhance learning ESM-1b (Rives et al., (02T)) and ESM-2 (Lin et al. (002) are two protein language models trained using masked language modeling. AR-LM (Wang al. 2024) protein language model
25

---

Published as a conference paper at ICLR 2025
trained based on autoregression. In contrast, DPLM (Wang et al. 2024) utilizes non non-auttreerressvve discrete diffusion modeling and is the model most closely related to ProfileBFN.
E COMPLEMENTARY RESULTS
E. ENZYME GENERATION
E.I.1 BACKGROUND
Enzymes are special class of proteins with catalytic functions. They significantly accelerate chem- ical reactions within organisms and play crucial role in sustaining life processes. Based on the differences in the types of chemical reactions catalyzed by various enzyme families, researchers have developed the Enzyme Commission Number (EC Number) system to classify enzymes. In other words, two enzyme proteins sharing the same EC Number are considered to have similar cat- alytic functions. Strictly speaking, determining enzyme's EC Number requires labor-intensive and costly wet-lab experiments. However, with advancements in machine learning, the accuracy of using computational methods predict EC Numbers has improved significantly. Among these methods, CLEAN (Yu et al. (023) is one of the most advanced models for predicting enzyme EC Numbers. It employs contrastive learning strategy bring representations of functionally similar enzymes closer while pushing dissimilar ones apart, achieving classification accuracy validated by wet-lab experiments.
E.1.2 SETTINGS
Following the work of previous researchers, we selected three representative enzyme families for model evaluation following (Song et al.) (024). These families possess distinct characteristics that make them important in biological research. Firstly, P40925, which belongs to the family of malate dehydrogenases, plays essential role in the malate-aspartate shuttle and the tricarboxylic acid (TCA) cycle. It catalyzes the reduction aromatic alpha-keto acids in the presence nicotinamide adenine dinucleotide(NADH)) Secondly, Q7X7H9, which belongs to the family of shikimate ki- nases, catalyzes the specific phosphorylation of the 3-hydroxyl group of shikimate acid. It is key enzyme in the shikimate pathway, responsible for the biosynthesís of the aromatic amino acids phenylalanine, tyrosine, and tryptophan. Finally, Q15165 is capable of hydrolyzing lactones and number of aromatic carboxylic acid esters. possesses antioxidant properties, which are crucial in reducing intracellular and local oxidative stress and are related to the pathogenesis of various diseases. For each enzyme family, we require the model to generate 1,000 protein sequences for evaluation. For ProfileBNN, we convert the known protein sequences within the family into pro- file, which serves as the input for generation. For each generated protein sequence, we use CLEAN to classify its function and verify whether it belongs the given family.
E. BASELINES
We have selected several models specialized in generating protein families for comparison. PoET (Truong Jr & Bepler 2023) is an autoregressive model that uses known family sequences as prompts and generates new sequences for the family by continuously predicting the next amino acid. When generating new enzyme family sequences, known enzyme sequences are converted into prompts and input into POET. PoET treats sequences of protein families sequences- -ofeueeees-ussuuccsss..... utilizing both attention modules to capture within sequence and between between-sequence sequence relationships in a hierarchical manner. EvoDiff (Alamdari et al. (023) employs non-autoregeessive generation approach, utilizing multiple sequence alignments (MSA) to guide discrete diffusion generation process and achieve family-specific generation. For this method, known enzyme sequences within the family are organized into an MSA.
sequences,
within-sequence
E.1.4 METRICS
We used diversed benchmark evaluate the performance of generated Enzyme sequences:
26

---

Published as a conference paper at ICLR 2025
Accuracy is defined as the percentage of sequence candidates classified by the CLEAN model into the correct class of EC numbering. Specifically, we deduplicate the sequences beforchand.
Accuracy:
Uniqueness:
Considering that generative models may output the same sequences in differ- ent iterations, we record the survival rates before and after deduplication Uniqueness.
A U: An aggregate indicator which is defined Accuracy Uniqueness both mea- sure model's ability to generate accurate and unique sequence.
Refer to Appendix.
D.2.1 for Novelty and Diversity metric details.
E.1.5 RESULTS
We present detailed experimental results in Table and then present generated sequences it in TO andd2] [2
Table 6: Additional results complementing Table 2 are provided to showcase our model's perfor-
mance. Notably, our model achieves the highest Accuracy Uniqueness. The MSA Depth indicates the depth of MSA that are used as input of the generation model
Model
P40925
Q7X719
Q15165
MSA Depth
572
443
15
PoET 3.00%
33.3%
0.05%
Accuracy x Uniqueness f
EvoDiff-MS.
27.93%
88.69%
1.39%
ProfilcBFN-rrofle
95.19%
98.98%
42.67%
PoET 98.04%
99.93%
100%
Accuracy
EvoDifI-MSA
27.93%
88.69%
1.39%
ProfileBFN-prffile
95.19%
98.98%
42.67%
PoET 3.06%
33.32%
0.05%
Uniqueness 1
EvoDiff-MSA
100%
100%
100%
ProfileBFN-profil.
100%
100%
100%
PoET 0.036
0.366
0,068
Novelty f
EvoDiff-MSA
0.728
0.596
0,497
ProfilBBFN-profie
0.467
0.582
0.288
PoET
0.499
0.645
0.990
Diversity
EvoDiff-MS: MSA
0.138
0.184
0.143
ProfileBEN Proofie
0.374
0,289
0.594
E.2 IMPROVE STRUCTURE PREDICTION VIA ENHANCING MSA
E.2.1 BACKGROUND
Orphan protein structure prediction is an important scientific challenge, aiming improve the accu- racy of models in predicting the structures of orphan proteins. Specifically, orphan proteins refer to those that lack sequence and structure homology information (Wu et al. (022). Due the absence of homologous data, it difficult to construct high-quality Multiple Sequence Alignments (MSAs) for these proteins (Chen al. 2024). The low quality of MSAs strongly limits the performance of current structure prediction models, such as the AlphaFold series (Wu et al. (022) (Chen et al., 2024). Moreover, orphan proteins are not uncommon the protein space; statistics show that ap- proximately 20% of metagenomic proteins and around 1 11% of proteins from eukaryotic and viral origins are classified orphan proteins (Chen et al. 2024). Therefore, addressing orphan protein structure prediction remains a critical challenge in the post-AlphaFoll era.
F.2.2 BASELINES
The current advanced approach to addressing this issue is to use generative models to enhance low- quality MSAs, transforming them into high-quality MSAs. Based this paradigm, MSAGPT(Cher)
27

---

Published as a conference paper at ICLR 2025
al. 2024 reports the best predictive performance to date. Specifically. MSAGPT employs an autoregressiive model, taking low-quality MSAs as input and sampling additional protein sequences to improve the quality of the MSAs. MSAGPT outperforms several methods that enhance MSAs to boost predictive performance, including EvoDiff (Alamdari et al., 2023). MSA-Aug et al., 20236), and EvoGen(Zhang et al. (023a). Due to its advanced performance, we MSAGPT the main baseline method. Additionally, we treat the performance of AlphaFold2 using non-enhanced MSAs on orphan proteins as a lower bound, referred to AF2-MSA. It is worth noting that while ProfileBNN, like MSAGPT, also enhances MSAs using generative models, it differs from MSAGPT in that its training only requires protein sequence data, which is more easily accessible. In contrast, MSAGPT requires training on MSA datasets and is further optimized based on AlphaFold2 feedback using Reinforcement Learning.
E.2.3 SETTINGS
We follow MSAGPT and evaluate the model using orphan proteins from the CASPI4 and CASPI5 datasets. For each orphan protein, we retrieve its MSA using HHblits(Remmert al. 2012) from the UniClust30 Mutabase(Mirditt, et al. 2017). The obtained MSA has a depth of less than 20, meaning fewer than 20 homologous sequences can retrieved. Generation models are required to generate 64 additional protein sequences based on the retrieved low-quality MSA. These sequences supplement the retrieved MSA, forming higher-quality MSA. This high-quality MSA then used as input for AlphaFold2 to improve its structural prediction performance for orphan proteins. In utilizing the retrieved MSA, ProfileBFN transforms it into a profile to be used as model input, while MSAGPT uses it prompt to guide the model in generation.
E.2.4 METRICS
We compare the performance of different methods by analyzing the differences between the or- phan protein structures predicted by AlphaFold2 and those obtained experimentally. Specifically, we use two golden metrics: TM-score, a widely-used metric for assessing the structural similarity between predicted structures and the ground truth, and LDDT, the Local Distance Difference Test score, which measures how well local interactions in a reference structure are conserved in the pro- tein model being assessed. Additionally, Distance Difference Test), which reflects AlphaFold2's confidence in the local accuracy of each residue. All metrics scaled from 0 to 100.
report predictive metric, PLDDT (predicted Local
E.2.5 RESULTS
Table 7 presents the performance metrics of different methods. Based on this table, we observe the following findings:
Generating additional protein sequences can indeed enhance the quality of MSA, thereby improving the model's performance. This improvement stems from the model's pretraining process, which enables it to gain a profound understanding of protein structures. When applied to orphan proteins, the model effectively transfers this understanding. enriching initially low-quality MSAs with structural information and ultimately yielding high-quality MSAs.
ProfileBFN consistently outperforms MSAGPT across metrics, demonstrating that the MSA supplements provided by ProfileBFN contain more comprehensive protein structure information. This result can be attributed to several factors. First, ProfileBNN leverages pretraining to capture deeper protein structural insights compared to MSAGPT, its non- autoregressive strategy aligns more closely with the natural characteristics of protein data. Second, the structural information obtained by ProfileBFN is transferable orphan proteins, unlike MSAGPT, whose pretraining primarily relies on deeper MSAs, while Pro- fileBFN imposes no specific depth requirement on MSAs.
28

---

Published as a conference paper at ICLR 2025
Table 7: Using ProfileBNN to enhance AF2 performance by adding virtual MSAs, the results show that ProfileBFN is capable of generating more appropriate MSAs for models such as AF2 compared the ground truth searched MSA and MSAGPT, All metrics are scaled from 100.
Model
TMscore t
LDDT 1
pLDDT 1
AF2-MSA
53.20
54.01
62.91
MSAGPT
55.72
53.59
66.38
ProfileBNN
56.84
55.72
67.04
T1033
TM-score 0.385 pLDDT 52.39
TM-score 0.617
TM-score 0.842
pLDDT 53.10
pLDDT 76.63
T1064
TM-score 0.291 pLDDT = 37.61
TM-score 0.349 pLDDT 35.76
TM-score 0,730 pLDDT 61.15
T1113
TM-score = 0.639 PLDDT = 68.66
TM-score 0.783 PLDDT 87.93
TM-score 0.826 pLDDT 89.76
Figure 8: Visualization of improved structure prediction sample compared with AlphaFold2 and MSAGPT. Yellow: Ground truth; Blue: Predictions from MSA generated by natural MSA searched with AF2. Purple:
Predictions based on MSA generated by MSAGPT; Red: Predictions based on MSA generated by ProfileBNN;
E.3 ANTIBODY CDR IN- --PAIITIIG
E.3.1 SETTINGS
We further test our Model's ability in the task of Antibody Complementarity Determining Regions (CDR) in-painting. Antibodies are specific types of proteins utilized by the immune system to recognize and neutralize pathogens and are of immense interest for therapeutics. In the structure of antibodies, the so-called Complemenaay--Deermiingg Regions the main regions for binding
29

---

Published as a conference paper at ICLR 2025
with antigens and determining the specificity of the antibodies. Under this cireumstance, CDRs in antibody sequences are masked at once and later predicted conditioned on the framework. We present two versions of our model for antibody generation: ProfileBFN-singllee sing)e without any information trained about antibodies, and ProfleBFN-Anni(55OM, which tuned with the OAS dataset for 8500 steps.
E.3.2 BASELINES
We include several strong baselines, all of which are trained specifically on antibody data. RAbD (Adolf- Soooof-Bryfogleeee et al. 2018) is a renowned software- based method. DiffAb (Luo et al. 2022) uses diffusion models conduct sequence-stng setuuccetttruuurre co-design, which mainly models the geometric aspect. AntiBERTy (Ruffolo et al. 2021) and AbLang (Olsen et al.] 2022) are two sequence-based language models trained on the entire OAS dataset; the former based on the BERT architecture to encode antibody sequences, while the latter is trained on randomly masked antibody sequences and modeled on a Transformer architecture with special head.
E.3.3 METRICS
We used Amino Acid recovery (AAR) of each CDR region for evaluation, with each antibody sample providing candidates.
E.3.4 DATASETS
We used the OAS unpaired dataset (total 2,428,016,345 antibody sequences) to fine-tune our model and the SAbDab Dataset for testing, following the DiffAb paper. To avoid potential data leakage, we removed sequences similar to our test set with MMSeqs2 (Steinegger & Söding 2017) tools by the identity of 0.95. Both heavy chains and light chains included in the tuning process.
E.3.5 RESULTS
The results showed in Table E.3.5 indicate that ProfileBFN had already reached comparable scores before fine-tuning on the antibody dataset, indicating that it learned general rules of protein language that could be successfully transferred to antibodies which are specific and functional proteins. Once tuned on the antibody dataset for a very small number of steps, could surpass the performance of previous models such AntiBERTy and AbLang, indicating the effectiveness of pre-training processes.
Model
CDR-HI
CDR-H2
CDR-H3
CDR-LI
CDR-L2
CDR-L3
RAbD 0.2285
0.2550
0.2214
0.3427
0.2630
0.2073
DiffAb
0.6575
0.4931
0.2678
0.5667
0.5932
0.4647
AntiBERTy
0.7940
0.5932
0,4133
0.7208
0.3996
0.2758
Ablang 0.7039
0,7981
0.3207
0.5799
0.5513
0.3175
ProsileBFN-singlee 0.6766
0.6188
0.1946
0.5356
0.5873
0.3064
BrofilcBFN-Anii 0.8227
0.7236
0.3343
0.6402
0.6156
0.4716
Table 8: Performance of Antibody CDR in-paint task ProfileBFN compared to baselines. The best result indicated in bold, while the second-best result underlined.
E.4 ADDITIONAL RESULTS
E.4.1 INVESTIGATION ON THE RELATIONSHIP BETWEEN PERFORMANCE AND MSA DEPTH
case, where sampled 50, 100, 500. 1000, and 2000 sequences from the searched homologous sequences, and each generate 1000 sequences for contact prediction, report LR P@L L, LR P@ L/2. LR P@L/5 respectively. Results shown in Fig. reveal that the quality of generated sequences tends to increase with the increasing depth of the MSA. The growth rate drops as the depth increases.
We have conducted the experiment
30

---

Published as a conference paper at ICLR 2025
0.40
LR P@L
LR P@L/2
0.35
LR P@LL/
Contact Precision
0.30
0.25
0.29
0.15
MSA Depth
Figure 9: ProfileBFN-prodd -rofille Generation Result of Contact Prediction of Protein 8YXK with differ- ent MSA depth as input
P40925
ATITL WWWA/TTAARR WEW.LEEDOTTVVLLEE.... LOw.O.
PN www.gnAAAnLL... LA.
ProllatF%-PPafie
20657 TAUT
SAPV WWW.P/WWW....000
leag
INETAABAA
Fole TWW
PAADESITEV
Figure 10: Samples of sequences conditioned on enzyme P40925 family by model ProfileBFN, PoET and EvoDiff;
31

---

Published as a conference paper at ICLR 2025
Q7X7H9
(60024)
PSACE TTV-0RLLLLLL Dw
PoET MSA
LLGS SM0T ELLVVER S44 SAYSSNS
9804
AGGRGAPATT PAP.AA2
Dvedut MIAA
POVSADDINTTT
PELDAY LITTY
Figure 11: Samples of sequences conditioned on enzyme Q7X7H9 family by model ProfileBEN, PoET and EvoDiff;
32

---

Published as a conference paper at ICLR 2025
Q15165
PABALLL TULGLL
220
PACEINL
(60024)
POBLL
PAELLALT N1099559C TL
PoET MSA
PALL T- 11200055S /TV
PAILL FALLLALTI TLVLL 11300555 MA-
YILL WWW.Lg VAS
HRLLLY MAAAAAAAAAALLLLL
YAD
Dvedut MISA
DANAGLL MANOOLLLLLLLLL
YESLSVEP 2VOT27
MOVILLAGULTO TOTTTALLLLEE
Figure 12: Samples of sequences conditioned on enzyme Q15165 family by model ProfileBEN, PoET and EvoDiff;
33


Reviews:
[{"review_id": "p1PUXJpOK2", "reviewer": "Reviewer_xv7N", "summary": "The authors build a generative model of protein sequences based on the Bayesian Flow Network model, which they adapt to categorical data. They show that, conditioned on a protein family, their model better predicts structure and generates more realistic proteins than previous models as measured by activity and structure predictors.", "questions": "What does the sampling process have to do with the generative process described by a BFN? It seems you're \"denoising\" a profile, but your claim is that the generative model itself outputs profiles. Please be as rigorous in your answer as possible.\n\nIn \"Sampling Process Reflects Protein Conservation\" the authors show a correlation between conservation at a site and transition rate. Is this something we expect? Is this something we should be interested in?\n\nIn Table 3 the authors show that the representations of their model can be used to predict downstream tasks (by the way, it is misleading to bold their results when they do not beat the baselines). Conspicuously missing is mutation effect prediction on ProteinGym. Is there a reason your model was not benchmarked on this when arguing for \"understanding\"?\n\nCould you release representative sequences suggested by all models for table 2? Were their diversity or similarity to the conditioned family significantly different? How mush of the protein family did you pass to PoET to do this evaluation -- could you be clearer on how the baselines were run?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "They use a new architecture to model protein data.\nThey have many different sources of validation which have the potential to be convincing.", "weaknesses": "I strongly suggest the authors write their introduction more carefully. 1) de novo design, even if it has a low success rate, can be useful for library design. It is not necessarily to be contrasted with directed evolution as an alternative; in practice, libraries designed with these methods are used to find a hit which is then optimized by directed evolution. 2) The statement \"However, it tends to produce proteins too similar to the original wild type, limiting the exploration of protein space and hindering the discovery of globally optimal proteins with desired functions.\" seems intuitively sound but is not backed up by empirical data in the literature. Given these comments as well, it makes little sense to position their \"protein family design\" as a third way (this paragraph should also compare their method with PoET and EvoDiff); I would appreciate them putting it in context in a library -> optimization pipeline. The related works section has a similar issue.\n\nThe paper understates the methods that previously have modeled protein families. I would like the authors to make it very clear that PoET, and ProtMamba (https://www.biorxiv.org/content/10.1101/2024.05.24.595730v1, which was not cited), did something like this previously; the authors mention some of these works in the experimental section, but they should be clearly mentioned in the introduction and related works section. I appreciate that the authors take an alternative approach to these models (which train on protein families); I would therefore also like an argument for why the authors expect their method to outperform these methods in principle given that one might expect their method to perform worse since they didn't train on protein family data.\n\nThe introduction of their model is too colloquial. It would strengthen the paper for the authors to introduce BFNs by describing the generative process and likelihood normally (\"VLB stands for variational lower bound\" for example). This lack of rigour is manifest in many of the ways the authors use the model (see questions).\n\n\"Consequently, we can bypass the need to construct an MSA profile training set, offering a more efficient and practical solution.\" This solution is more \"efficient and practical\" for the authors not the practitioners or those who wish to use the model; the authors should simply curate a training dataset as other model developers have done many times.\n\nImportant aspects of all of the evaluations are in the appendix, including the definition of nearly every metric. Any detail needed to understand the soundness of the evaluation (how you predict the enzymatic activity) should be in the main text.\n\n\nIn terms of soundness, conditioned on a protein family, one wishes to sample more sequences from that family. \"diversity\" and \"novelty\" are not the goal -- these statistics should be compared to those of the real family.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WHWX0nUMk6", "reviewer": "Reviewer_9Yfz", "summary": "This paper introduces a new method for family-based protein generation, as a cross between mutation-based (where a single sequence is mutated one residue at a time) and de novo methods (where the design is entirely structure- or function-driven and the sequence is inferred from scratch). It does by adopting Bayesian Flow Networks. It operates in sequence space and is thus most analogous to a handful of recent methods on sequence-based diffusion, which is in general an underexplored space for protein design.", "questions": "* In Table 1, structural metrics are described (LR P@L) but how structures are predicted is actually never described. The paper repeatedly refers to CAMEO but CAMEO is not a method, it’s an evaluation set. Are actual structures being predicted using AlphaFold or similar method and then assessed? Or are only contacts being predicted using e.g., Potts models and then assessed? This is never stated in the main text or the appendix as far as I can tell.\n* Similarly in Table 2, percentages are provided for what fractions of enzymes are considered functional, but the method of assessment is never mentioned! How is an enzyme being assessed as being functional? Is this based on a prediction tool? If so which tool?\n* Similarly in Table 3, the focus is no longer on just enzymes but function prediction in general, but again how any of the accuracies are computed is never mentioned.\n\nIf these ambiguities are properly addressed I would increase my score.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The method is novel and explores a region of algorithm space that is generally underexplored for proteins, namely sequence-based design and in particular MSA-based design.\n* Results are competitive if not necessarily state of the art. I consider the paper more about new ideas and approaches as opposed to necessarily achieving the best results.\n* The method essentially finds a niche between quality and novelty of generated structures, sitting in between existing methods on the Pareto frontier.\n* Experiments are well done and the algorithm itself appears clever.", "weaknesses": "* Biggest weakness (see question) is that how the evaluations are precisely done is not well described at all.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tm0zMNUgvi", "reviewer": "Reviewer_BQXP", "summary": "Introduces a novel modification to the pre-existing Bayesian Flow Networks (BFN) learning scheme to create the ProfileBFN. It then applies this new ProfileBFN to the protein generation task. Like AlphaFold (and many other works) they use MSA, however, they use the the distribution statistics of MSA rather than the actual MSA data directly. They show that they technique performs well across most metrics, and produces state of the art results in some.", "questions": "1 - Why do you use sigma for both the Kronecker delta and the Dirac delta functions? This is confusing.\n\n2 - You do not define PMF. What does it stand for?\n\n3 - You do not define phi, it just appears in equation 3 (and in the line above). What does phi represent?\n\n4 - Why do you not cite the paper (Z Lin’s or another) that introduces ESM-2?\n\n5 - How did you choose the hyper parameters in lines 266-267?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "The results are overall quite impressive. Despite being outperformed in some metrics, ProfileBSN still achieves decent results in those metrics while achieving state of the art results in other measurements. \n\nAdditionally, I like the use of Bayesian Flow Networks, which are not entirely in the mainstream techniques in protein modeling. This gives this paper a high degree of novelty\n\nI appreciate the introduction to BFNs provided in Section 2.2. This greatly helped the readability.", "weaknesses": "The mathematics is not presented in the easiest way to understand. It is not so problematic that I reject the submission, but it is not written in an easy manner for the reader to follow. Reuse of variable names (see Questions) is an issue, but the largest issues are:\n\nA - In line 170: dimensions are given for the summation symbol. This does not make sense, as summation is an operator, not a data type. Please explain what this means. I have held off on lowering the score too much for it in hopes there is a good explanation, but I definitely would like to see an answer for this. \n\nB - Explanation for many of the results feels quite rushed, and I had to put a lot of effort into parsing it out.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S5yQn85UJl", "reviewer": "Reviewer_vGzf", "summary": "The authors introduce Profile Bayesian Flow Networks (ProfileBFN), an adaptation of the Bayesian Flow Networks from Graves et al. (2023), designed specifically for aligned protein sequences in multiple sequence alignments (MSAs) and their frequency profiles. They derive a simplified loss function measuring the squared difference between the predicted latent embedding p_phi and the encoded representation e_x, effectively capturing how well the model’s predictions align with encoded information in the data.  Furthermore, the authors present techniques to efficiently sample from the model.\nArguing that training directly on MSA profiles rather than raw MSAs offers key advantages, the authors highlight that MSAs can be large, computationally intensive, and challenging to generate. By relying on sequence profiles instead, their method reduces computation and even enables training on single sequences rather than full MSAs. Furthermore, the approach does not require additional profile data construction, but instead leverages existing sequence profile data for training.\nTheir results show that the generated sequences better capture protein contacts than standard MSA-based methods. Additionally, in-silico refolding of these generated sequences predicts structures that closely resemble native structures. Despite these advances, the generated sequences remain highly similar to native sequences in terms of sequence identity.", "questions": "To better showcase the strength and applicability of ProfileBFN, I’d suggest the following points:\n* The authors claim that their method is suitable for de-novo design, potentially enabling the creation of novel sequences, structures, or even entirely new functions. However, it appears that the sequences generated by ProfileBFN are optimized variants closely resembling native sequences in terms of similarity and identity. Could the authors elaborate on this point and clarify how ProfileBFN supports de-novo design?\n* Could the authors specify which protein families were used to evaluate their method? In particular, is there a dependency between the depth of the MSA and the quality of the generated sequences? Certain regions, such as antibody loops (and loops in general), are challenging to model and generate due to the complexity of viable variations. Have the authors considered testing ProfileBFN in these difficult sequence spaces?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The papers' emphasis is focused on the mathematical derivation and has clear explanations of key theoretical components. The authors benchmark ProfileBFN against multiple SOTA methods, demonstrating that it performs on par with or surpasses existing approaches.", "weaknesses": "While the paper is strong on the mathematical side, it lacks some depth in application.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)