PepTune: De Novo Generation of Therapeutic Peptides with Multi-Objective-Guided Discrete Diffusion
Authors: Sophia Tang
Yinuo Zhang
Pranam Chatterjee
Venue: ICML 2025 poster (2025)
Keywords: Discrete diffusion
multi-objective optimization
therapeutic peptides
Primary area: applications->health_medicine
TL;DR: PepTune generates therapeutic peptides in discrete space using Monte Carlo Tree Guidance for multi-objective optimization.

Abstract: We present **PepTune**, a multi-objective discrete diffusion model for simultaneous generation and optimization of therapeutic peptide SMILES. Built on the Masked Discrete Language Model (MDLM) framework, PepTune ensures valid peptide structures with a novel bond-dependent masking schedule and invalid loss function. To guide the diffusion process, we introduce **Monte Carlo Tree Guidance (MCTG)**, an inference-time multi-objective guidance algorithm that balances exploration and exploitation to iteratively refine Pareto-optimal sequences. MCTG integrates classifier-based rewards with search-tree expansion, overcoming gradient estimation challenges and data sparsity. Using PepTune, we generate diverse, chemically-modified peptides simultaneously optimized for multiple therapeutic properties, including target binding affinity, membrane permeability, solubility, hemolysis, and non-fouling for various disease-relevant targets. In total, our results demonstrate that MCTG for masked discrete diffusion is a powerful and modular approach for multi-objective sequence design in discrete state spaces.

PepTTue:: De Novo Generation of Therapeutic Peptides with Multi-Objectiv--Guddee Discrete Diffusion
Sophia Tang" Yinuo Zhang" *2 Pranam Chatterjee
Abstract
20 wild-type amino acids have limitations, including sus- ceptibility to enzymatic degradation and low membrane permeability (Wang et al., 2022). To overcome these limita- tions, non non-natural amino acids (nAAs) containing diverse chemical modifications to the peptide backbone and side chains have been integrated into peptides to enhance their therapeutic properties. Despite this progress in peptide drug development, searching for the vast space of chemically modified peptides remains a major limitation (Muttenthaler et al., 2021; Vinogradov et al., 2019). This motivates the development of generative deep learning models that can effectively learn the space clinically relevant peptides and sample de novo peptides conditioned with various therapeu- tic properties.
We present PepTune, a multi-objective discrete diffusion model for simultaneous generation and optimization of therapeutic peptide SMILES. Built on the Masked Discrete Language Model (MDLM) framework, PepTune ensures valid peptide structures with a novel bond-dependent masking schedule and invalid loss function. To guide the diffusion process, we introduce Monte Carlo Tree Guidance (MCTG), an inference- time multi-objective guidance algorithm that bal- ances exploration and exploitation to iteratively refine Pareto-optimal sequences. MCTG inte- grates classifier-basee rewards with search- tree expansion, overcoming gradient estimation chal- lenges and data sparsity. Using PepTune, we peptides
Generative structure-based models are considered state-of- the-art for de novo binder design, but they often rely on stable tertiary structures of target proteins (Rettie et al., 2025; Bryant & Elofsson, 2023; Pacesa et al., 2024; Li et al., 2024; Watson et al., 2023), precluding the design of peptide binders to disordered and dynamic targets, which drive a sizable portion of diseases (Uversky et al., 2008). Gener- ative peptide design language models that depend only on the target sequence (Bhat et al., 2025; Chen et al., 2023) have demonstrated robust success on disordered and struc- turally diverse targets, but their use of only 20 wild-type amino acids limits these models from sampling from the space of chemically- modified or eyclic peptides. Further- more, discrete generative models still face significant limita- tions in multi-objective-uuiiee (Austin et al., 2021; Lou et al., 2024; Shi et al., 2024; Sahoo et al., 2024; Gat et al., 2024; Rector- Peng et al., 2025a; Davis et al., 2024; Tang et al., 2025b). Classifier-based and classifier-free free guidance strategies have been explored to steer discrete diffusion objectives toward specific properties (Wang al., 2024; Rector-Brooks et al., 2024; Stark al., 2024; Nisonoff al., 2024), yet these ap- proaches often struggle with conflicting objectives, gradient estimation, and the sparsity of quality data.
generate diverse, themically-modiifee simultaneously optimized for multiple therapeutic properties, including target binding affinity, mem- brane permeability, solubility, hemolysis, and non-fouling for various disease-relevan targets. In total, our results demonstrate that MCTG for masked discrete diffuusion a powerful and mod- ular approach for multi-objective sequence design in discrete state spaces.
1. Introduction
generation and optimization
Peptides possess unique advantages as a therapeutic modal- ity, including their low cytotoxicity and structural flexibility to bind to diverse set of binding motifs without requir- ing stable binding pockets, making them ideal for target- ing structurally diverse protein surfaces (Dang et al., 2017: Wang et al., 2022). However, peptides containing only the
et al., 2024;
'Equal contribution Department of Computer and Information Science, University of Pennsylvania *Center of Computational Bi- ology, Duke-NUS Medical School 'Department of Bioengineering, University of Pennsylvania. Correspondence to: Pranam Chatter- jee <pranam @ seas. upessuuenn.cuu.
In this work, we introduce PepTune, the first multi- objective-guides discrete diffusion model for de novo pep- tide SMILES generation. Our key contributions include:
Proceedings of the 4dnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

De Novo Generation of Therapeutic Peptides with Multi-Objecive--uidee Discrete Diffusion
1. Masked Diffusion Language Model for Peptide SMILES. We introduce the first discrete diffusion model for generating peptide SMILES with non- canonical amino acids and cyclic modifications.
probability of masking a token within a peptide bond in- creases slower rate in earlier times É in the masking process compared to non-peptide bond tokens. To achieve this, we define the discrete-time log- log-linear masking sched- ule (t)) = - log(1 - t) for non-peptide bond tokens and the log-poly geg-ooyyomiil masking schedule o = - - - t*)) for peptide-bon bond tokens. We show in Appendix G.I that the continuous- time probability of remaining unmasked at time in the forward diffusion process is given by the function on (xo) R - IR that takes the vector encoding the token Xo and returns a probability
2. Bond-Dependent Masking Schedule. We derive the NELBO and reverse- prrerse-osttriorr for bond-dependen masking schedule that increases the structural validity of our generated peptide SMILES.
3. Global Sequence Invalid Loss. We introduce novel invalid loss based on our peptide SMILES filter that ef- ficiently propagates penalties from discrete sequence to continuous probability distributions.
1 11-t" tt
Xo = b Xo o
(xo)
(I)
11-6
4. Monte Carlo Tree Guidance. We develop ro- bust framework for classifier- multi-objective guidance for discrete diffusion by iteratively refining Pareto-optimal peptide sequences across therapeutic properties without gradient estimation or re-training.
where b is the vector with ones at indices of peptide bond tokens and zeroes in remaining indices. The tokens corre- sponding to peptide bonds are identified with our BOND- MASK function (Algorithm 7). Since the probability of tran- sitioning to a [MASK] token at time I is given by (xo), there is lower probability t" for t E (0. 1] of masking peptide bond token than the probability of masking a non-peptide bond token, especially in earlier time steps for smaller t (Figure 15A). As t - 1, the probability of re- maining unmasked approaches 0 (ax (Xo) + 0) and the probability of masking for both peptide and non-peptide bond tokens approaches 1, ensuring that the model can learn to reconstruct the full token sequence during the reverse diffusion process.
5. Property Prediction Toolkit. We train a set of clas- sifiers and regressors for binding affinity, membrane permeability, solubility, hemolysis, and non-fouling,
2. Bond-Dependent Masked Discrete Diffusion
2.1. Masked Discrete Diffussio Model
We based our unconditional generator on the Masked Diffu- sion Language Model (MDLM) framework, which learns to reconstruct clean sequences from sequences corrupted with [MASK] tokens (Figure 1) (Sahoo et al., 2024; Shi et al., 2024; Ou et al., 2024; Zheng et al., 2024). The backbone model used to generate the predicted probabilities, denoted xo(2,,) t) y4 x [0, 1] + A of transitioning from a masked state to any token in the vocabulary V is predicted by a backbone RoFormer architecture (See Appendix D.2). RoFormer leverages rotary positional embeddings (RoPE) (Su et al., 2021), which effectively captures the relative inter- token interactions in peptide SMILES, especially for cyclic peptides.
With our bond nond-dependent masking rate ave we define the forward transition matrix as
g(z,) [x)) Cat(zz; az (Xn)xx + (1 (1-ar (x)o)))))mm)
Proposition 2.1 Dependee Reverse Posterior). The reverse posterior defining the probability distribution of the loken 2, at time ( Al given the token 2n at time L with our bond cond-dependente forward masking schedule is defined as
{Zz Zz, Xu)
b + 1,, X p b
1,)
m
== m
(3)
2.2. Bond- Dependent Masking Schedule
Zt
Z + m
Since all peptides follow a distinct SMILES structure con- sisting of un-modified or modified peptide bonds before and after each central carbon atom with an amino acid side chain, we hypothesized that applying bond-dependent masking and unmasking schedules would allow the reverse diffusion pro- cess to learn to unmask the crucial structural components of a peptide SMILES that are common across all peptides before filling in the segments in-between with diverse amino acid side-chains.
When the clean token is a peptide bond token (i.e. Xo = b).
the transition distribution for a masked token z, = m re- duces to q(z, i2, = m, Xo = b) = (1 8" When the clean token is not a peptide bond token (i.e. Xu # b), the transition distribution for a masked token 2, m reduces to q(z, z; = m, xo # b) (1- 4) xo + (+) m. If the token is already unmasked, it remains unmasked at the same token with probability 1.
for Xg +
1- m.
Extending previous work in state-dependee masking (Shi et al., 2024), we devised a masking schedule where the
The derivation is provided in Appendix G.2. To estimate the reverse posterior, we define a parameterized RoFormer
2

---

De Novo Generation of Therapeutic Peptides with Mult--hjjeciv--Cuidee Discrete Diffusion
t
Bond- Dependent Masked Discrete Diffusion
RoFormer Backbone
CCICIO
Zero Meaking Probubiliiyyy
Camy Unimeaking
(iyy k) mi)
iilty sy then ma
Or
Input Peptide SMILES
Reconstructed Peptide SMILES
C NELBO + Civalidd
Continuous- Time State State- Deaee-Deeedenntt NELBO Invalid Loss
Figure PePMDLM. PepMDLM a discrete masked diffusion model for unconditional de novo generation of peptide SMILES representations.
model (z t) 44 x [0, 1] M that takes the partially masked sequence time t and predicts vector of token probabilities over the |Vlddimensional simplex for each po- sition in the sequence. By substituting Xy = xo(2,, t) into the true reverse transition, we get the predicted reverse tran- sition distribution.
probability distribution X (Z,, is exactly the one-hot en- coding vector Xy for each position é in the true sequence, the loss reduces to O, which supports our objective.
Proposition 2.2 (Bond-Depende) NELBO), The bond- dependent continuous-time NELBO decomposes into the sum of the negative log- log-losses (NLLs) for all non non-peptide bond tokens that follow a log-linear masking schedule and the sum of the NLLs for all peptide bond tokens that follow pog-polynomial schedule.
Po (1,, (x))
=") b t 1,1100 (Zt, t)) 2zt
+) b + (zr,0(zr,1 ())
2: m
(4)
2
21 + m
Co0 NELBO
Σ Ex
7
logixo
(2e, ())
(0)
(0=b
For larger w, peptide bonds are masked later timesteps, encouraging earlier unmasking in the reverse diffusion pro- cess. However, setting too large can result in the model over-fitting to the dataset (Shi et al., 2024). Empirically, we found that w = increased peptide validity while maintain- ing diversity across generated samples.
Σ Exx z/
log(x
(6)
(0 xx (2(.))))
(5)
where E {L....,.) L} denotes the position the sequence.
The derivation is provided in Appendix G.3. Since the NLL term is minimized when the predicted probability of the ground truth token is close to 1, we show that applying the log-polynomial masking schedule for an exponent w > 1 scales the diffusion loss NELBO by a factor of w from the log-linear schedule. However, for earlier timesteps as / -> O, both NLL weights increase oc, ensuring high precision in the final unmasking steps (Appendix Figure 15).
2.3. Loss Functions
Bond-Dependent Continuous- Time Diffusion Loss To optimize the parameters 9 of the reverse diffusion model, we maximize the evidence lower bound (ELBO) of the distribu- tion log p(xo), which is the probaabityy distribution of generating the peptide sequences X. present in the dataset. Therefore, we define our loss function as the negative ELBO (NELBO) (Appendix B.2).
Given that peptide bonds form the fundamental backbone structure of a peptide, our bond-dependent masking strategy for peptide bonds acts as peptide bond loss that introduces higher penalty when the token predictions at positions of peptide bonds are inconsistent from the ground truth tokens
Training on samples masked for continuous values of t ~ Uniform (0, 1) yields tighter lower bound compared to dis- crete values of (Kingma et al., 2021). When the predicted
3

---

De Novo Generation of Therapeutic Peptides with Multi-Objectiv--Cuddee Discrete Diffusion
during training. foreing the model to learn the specific struc- ture of peptide SMILES strings in vast space of SMILLES strings that are not valid peptides.
each time t by 8= [
1 2 B........ B
to get a vec-
tor E- (f++) 5) mod of evenly distributed time steps to ensure the model learns to regenerate the clean sample 2 for continuous range of time steps. After applying bond- dependent masking to each training sequence, obtaining the predicted probabilities X (o(2,, () and sampling the discrete sequence Xn from greedy ar sampling, we minimize the total loss function L given by
Invalid Peptide Loss To further discourage the genera- tion from predicting token logits that produce invalid pep- tide SMILES, we incorporate a loss to penalize sampling of invalid peptide SMILES during training by taking the argmax of the predicted logits and assigning penalty based on our peptide validity filter (Appendix 8). Given the clean peptide sequence &o E V/ generated from the argmax tokens with the highest probability from the pre- dicted logits X (Zz, t), we minimize a penalty determined by our validity filter 1[x, is Invalid] which returns when the sequence is a valid peptide SMILES and 1 when the sequence either contains invalid SMILES notation or cannot be decoded into peptide sequence. Since the argmax function is not differentiable, we use the softmax proba- bility of the sampled tokens to scale the penalty score for each token in the loss function.
L NNLLOO + Linvalid
(8)
By increasing batch size and applying dynamic batching (Appendix B.1), we obtain tighter ELBO of the true dis- tribution p(g)( The model used generate the vali- dation results in this manuscript is trained on in-house 8 x A6000 Nvidia GPUs (50G memory) for 1600 GPU hours using the AdamW optimizer with a learning rate of 0.0003 and weight decay of 0.075. After training for 8 epochs with 11 million peptide SMILES (Appendix C.I), we achieved a train loss of 0.832 and a validation loss of 0.880.
Linvalid
x (0T (OTSM(X (2.)) ()- 1 180 is Invalid]
(2)
Sampling To sample from the unconditional PepMDLM model, start with sequence X₂- [MASK]* / of length L of only [MASK] tokens. We first compute the diffu- sion time steps & E { 2 7.... 1} where T is the number of denoising steps (T = 128). From the predicted token probabilities X (z t) generated by feeding z, through the trained RoFormer backbone, we compute the reverse transi- tion token distribution pe pz(z, (2)) following Equation (4) and perform Gumbel- mma sampling to get the next token Zg-
exp(=(.1)
1 (Rx is Invalid]
(6)
j=1 exp(x!g)) (z))
K (0)
where A == arg max,(x;
(2) (x, (zr, 1)) is the token with the high- est predicted probability at position of the sequence.
Proposition 2.3 (Gradient Flow of Invalid Loss). By dif- ferentiating the invalid loss with respect to the probability vector x (ze, l) for position& the gradient with respect to the predicted probability of the sampled token j = k and all other tokens the vocabulary j f k is given by
2g ~ arg max (log (e
(z. (z)) + G)
(9)
= log(- logg--oog( + c) + c) is the i.idd. sampled Gumbel noise applied to the jth token probability, Wj Uniform(0, 1), and e = Le - 10. Then, we return the newly sampled tokens only when z, = m, while keeping all un- masked tokens unchanged. After T timesteps, we obtain a fully unmasked sequence x.
(M( (6) (1- --MM((
(1-
j=k
V Limvalid =
(7)
-SM(M (M( SMM()
j
The derivation is provided in Appendix G.4. Minimizing this objective function updates the parameters to lower the predicted probabilities for tokens that result in invalid pep- tide SMILES and increase the probabilities of the remaining tokens proportional to their original distribution, such that the relative probability distribution of all other tokens j of k is maintained.
3. Multi-Objective Guided Discrete Diffusion
In this section, we describe the concept of Pareto dominance and non-dominance for multiple objectives and introduce Monte Carlo Tree Guidance (MCTG), a novel algorithm that reformulates the Monte Carlo Tree Search (MCTS) framework (Coulom, 2007) for caltti-bjeetiie-guiccee dis- crete diffusion.
Training To train the MDLM to accurately approximate the true reverse transition distribution g(z,/zr, Iz,, Xo) of train- ing sample Xa for all continuous timesteps t 1 + 0, we train a parameterized model that takes the partially masked sequence z; and returns a probability distribution (2, t) that approximates the clean one-hot vector Xa (Al- gorithm 1). For each dynamic training batch B, we ran- domly sample B values E Uniform(O, 1) and oef-set
Pareto Optimization When optimizing sequences for multiple objectives (e.g., affinity to multiple protein targets, membrane permeability, solubility, etc.). there is likely no single best sequence that achieves the highest score across all objectives. Optimizing one objective often leads to sacri- ficing performance on another objective.
4

---

De Novo Generation of Therapeutic Peptides with Multi-Objectvv--Gudde Discrete Diffusion
Iteration
Selection
Rollout Back- propagation
Rewards
WHA! 11 Niaal
Upay a/
Wie) Wisd F(
Updatts
Expansion
Objective Classifer
Figure 2. PepTune. PepTune a multi-objective discrete diffusion model with Monte Carlo Tree Guidance (MCTG). The full algorithm
detailed in Algorithm 3.
Therefore, we focus on finding a set Pareto optimal se- quences that minimize the trade-offs between objectives to achieve overall optimal performance across all objectives. Formally, Pareto-optimal sequences (or non-dominated quences) cannot be further optimized in any single objective without sacrificing performance in another objective.
Since infinitely many trade-offs can exist between the K objectives, there can be an infinite number of Pareto-optimal sequences. Therefore, multi-objective optimization aims approximate finite set of Pareto-optimal sequences with reasonable number of iterations.
Let sxx) = [s1 (x)...., ... sK (x)] E R be a vector of scores that measures the performance of a sequence x in K differ- ent objectives, with higher scores indicating better perfor- mance. A sequence x* is said to dominate another sequence (denoted as s(x") > s(x) if and only if it satisfies the following property. For all objectives & E {1...., K}. the score for the kth objective for x* is greater than or equal to the score for the kth objective for x, and for at least one objective k', the score for x" is strictly greater than the score for x.
Notation Let z, denote the partially unmasked sequence at time l. also corresponds to node in the MCTS tree with set of M children nodes denoted as children (z)) = [2,,1,...,4 22,ar) Each child node is itself a partially un- masked sequence at time 8 derived from sampling the MDLM reverse posterior po [z.))))) The children nodes at each iteration of MCTS are rolled out into of clean sequences denoted {x,,1..... Xx, for each of which we compute a score vector s(x..)) R and rewards vector r(x,,)) E RK where K is the number of objectives guiding the MCTS search.
Pareto-optimal sequence x is sequence where there does not exist another sequence x* in the current Pareto-optimal set P that dominates it. Since there are trade-offs between objectives, this does not mean that x is dominant over all other sequences.
Let P = (xx) the set of [P'| Pareto non-dominated sequences indexed n E [I.....//P*] IP"1), which is updated at each iteration. At each node zr, store cumulative rewards vector W(z,) and counter for the number of times the node has been visited across all iterations Nvisa (2:). Finally, we denote the total number of search iterations with Nner
E s.t. s(x") > s(x)
(10)
non dominnted
We define the Pareto front as the set of non-dominated se- quences x and their d-dimensional objective score vectors.
Initialization We initialize sequence Z = [MASK] of length L of [MASK] tokens as the root node of the MCTS tree corresponding to time t(T) and an empty set P that will maintain clean sequences with Pareto-optimal score
p' (xx)s())) | 1xx" p s.t. s(x') > s(x)) (11)
5

---

De Novo Generation of Therapeutic Peptides with Multi-Ohjectv-- Discrete Diffusion
vectors. We initialize a of scoring functions s VL R that takes clean sequence x,, E V generated from the partially masked sequence ž,, and outputs a vector of real values s(x,,) E R that measures its performance in each of the K objectives. We also set the hyperparameters. including the number of iterations Nuer and the number of children M.
greater than that of 2,,i in at least one of the K objectives and equal scores across all the remaining objectives.
P selectt =(Zo,i Pas.s E (chllree(zz)
s.t. U(ze, Zs.3) > U(zz,Ze.))
(12)
If the selected node is non-leaf node, the loop repeats with the selected node n,,; as the new parent node. If a fully unmasked node with t = 0 is reached, we restart the selection process from the root node. Once a leaf node is reached, the loop ends and the next step executes.
At each iteration, four steps are performed to update the set of Pareto optimal solutions: traversing the tree by se- lecting a Pareto-optimal unmasking step until reaching unexpanded leaf node (selection), expanding the leaf node into M distinct partially unmasked sequences (expansion), fully unmasking each child node into a clean sequence and computing multi-objective score and reward vector (rollout), and finally back-propagating the total rewards to the pre- decessor nodes to guide the selection process at the next iteration (backpropagation).
Expansion At the iteration at time t, we sample M se- quences from the reverse posterior pe (z(zllz) defined in Equation (4) to get a set of partially masked sequences which form the set of children nodes of z₁: children(zz) (x) = [Ze.1....,,z. Zz,ar }. All the children nodes are added to the tree.
Selection At each iteration, we traverse the tree start- ing at the root node (fully masked sequence) Zi(T) and selecting a child node based on the selection score vector U(z,, Zs.5) that balances child nodes that generate high re- ward sequences from previous iterations and unexplored unmasking actions that could lead to a larger pool of diverse sequences.
To ensure that the expansion step results in M distinct un- masking steps, we experimented with two different batched unmasking techniques from the single partially masked se- quence at parent node. For the first method, we repeated the array corresponding to the parent node tokens over M dimensions and added independently sampled Gumbel noise values Gi,j. where denotes the sequence in the batch and j denotes the token index.
Ulz 20.i)= = W(zss)) + c po 1 + (visa(Zo..) (z.s)
Nvisi (2t)
log po. (g., (2)) = log pe (Ze, (2)) + G2 G = --llog log (- log (ui. + e) +c)
(13)
(14)
The first term is the cumulative reward vector W ()),;) nor- malized by the number of times the node was previously visited. This guides the selection process towards the un- masking step that has resulted in fully unmasked sequences with optimal properties without biasing towards highly vis- ited nodes. The second term is a scalar added element-wise to the normalized rewards. The scalar probability of the unmasking step based on the unconditional reverse poste- rior (o(2,.5 (2:) guides the selection towards the unmasking step with the highest probability to generate a valid peptide based on the pre-trained MDLM. When the number of times the parent node has been explored is high and the number
where "4 ~ Uniform(O, 1) and Po.: denotes the ith per- turbed reverse transition distribution after applying Gum- bel noise independently sampled for each sequence i in the batch, where i E {1..... M). Then, we sample M distinct child sequences from each of the distributions Z,,i ~ po. (Z,,i/z)).
The second method involves taking the softmax (denoted as SM) across the top & probabilities after applying Gumbel noise and drawing random samples from the re- re-normalized softmax distribution over only the top k most probable to- kens.
of visits to a child node is low, the v (M,)
1+ (N,,,) term en- courages exploration of the unexplored unmasking scheme given that Po (2s,3 (24) is sufficiently high. However, as the number of visits to a child node increases, the impact of the second term decreases and the cumulative rewards dominate the selection score vector. cs scalar hyperparameter that determines the degree of exploration compared to exploiting high-reward nodes, which is selected to be 0.1.
:o. (Z. (29 SM
SM( topk log Po (2.9) + (( (15)
After empirical experimentation, we found that the first method results in higher diversity across sequences whereas the second method prevents unlikely tokens. Since the re- ward generated by a sequence ultimately determines whether it is selected in subsequent iterations, we chose the first method to allow for greater exploration during the expan- sion step.
Then, we select uniformly at random from the pool of chil- dren nodes ž,, E Po whose selection score vectors are non-dominated, such that there does not exist another child node 2æ.j where the selection score vector has a score strictly
Rollout From each child node generated at time s, we completely unmask the sequence by greedily sampling the
6

---

De Novo Generation of Therapeutic Peptides with Multi-Ohjeetive-CCuddee Discrete Diffusion
a argmax tokens from the predicted reverse transition distri- bution po.: (2,. (2,) for all remaining time steps s 0 and s' = T get set of clean sequences XX..1....,,X..11 of SMILES tokens. We feed each clean sequence x as in- put to the scoring functions for all of the K objectives to gen- crate the score vector s( s(x,,)) = [s1(xs.)...... ,..)).......... (xx,)) E R Then, we use the score vector to compute a vector of rewards r(x,,)) (xs..).. ,'KK TK. (xx,)) E RK To gen- erate the property scores given input peptide SMILES, we train regression models for target-binding affinity and cell membrane permeability and binary classification mod- els for solubility, hemolysis, and non-fouling specifically on peptide SMILES data (Appendix E).
2 parent(Ze..) to Ze 2 we apply the following
update
M
W(z)) t- W(2)) +
Σ r(x.,))
(18)
(z N (2t) +1
(19)
These updated scores are used to guide the selection process in the next iteration, such that the unmasking paths that result in the highest reward sequences have a greater chance of being selected and explored further.
The output after Nue iterations is the set P of Pareto-optimal sequences across the K objectives. Our strategy simultaneously guides the unmasking process to- wards optimality across multiple objectives directly in the discrete state space while exploring the diverse space of peptide sequences using the trained unconditional MDLM generator. Furthermore, we generate set of Pareto-optimal sequences from a single run through the MCTSS algo- rithm which are non-dominated from the total of Nuerr M total sequences sampled across all iterations.
Output
The reward of a child node sequence for the kth objective is the fraction of the sequences x, in the current set of Pareto- optimal sequences P where the child node has a higher classifier score in that objective. Specifically, the reward for the ith child node 2,,i and the resulting unmasked sequence x for the kth objective is given by
P Σ 1 (xx,)) 2 (xx)
(x =
(16)
where 1 is an indicator function that returns 1 the score for the ktth objective of the ith child node is greater than or equal to the score of the nth sequence in the Pareto- optimal set P In parallel to computing the reward, we add all non-dominated children sequences to the set of Pareto optimal sequences O and remove all dominated sequences (Algorithm 6).
4. Therapeutic Property Classifiers
While several classifiers exist for predicting properties of small-molecule SMILES sequences and amino-acid repre- sentations of peptides, there exists a gap in high-quality property models trained specifically on peptide SMILES data. To fill this gap, we train regression models for target- binding affinity and cell membrane permeability (Appendix Table 8; Figure 14) and binary classification models for sol- ubility, hemolysis, and non-fouling specifically peptide SMILES data. Our prediction models achieve significantly enhanced performance in peptide property prediction com- pared to the state-of tfate-of-thh-art PeptideBERT (Guntuboina et al., 2023) baseline (Table 2).
P pr -P*U (L0((12))) (1 s(x,,))) I VX E s(x,,) 2 s(X) p p' 1/x1| Ex,,; E children(z.) s.t. s(x,,)) > s(x)
In Appendix I.1, we show a proof-of-concepp for a time- dependent multi-objective guidance strategy where the up- date to the Pareto-optimal set P* depends on the rewards for only a subset of the K objectives that varies depending the current iteration, enabling the prioritization of properties with larger influence on peptide structure and function in earlier iterations and fine-tuning on additional properties in later iterations.
Table 2. Benchmarks of solubility, hemolysis, and non-fouling prediction against PeptideBERT (Guntuboina et al., 2023). We leveraged PeptideCLM embedding representations of the SMILES tokens and trained XGBoost models for binary classification.
Solubility
Hemolysis
Non-fouling
Back- pacckppoppggtion At each child node Zs.i. the reward vector r(x,,)) is used initialize the cumulative reward vector W (z..., and the number of visits Nvisits (Za,)) is initialized to 1.
Metrie
Ours
Peplid-BERT
Ours
PeptideBRR
Ours
PeprideBERT
0.660
0.597
0.846
0.483
0.768
0.699
Aceuracy
0.661
0.651
0.823
0.766
0.873
5. Experiments
W(z.,) f- r(Xe.),, Nvisi +
(17)
PepMDLM generates diverse coamicallymmddifiee and cyclic peptides. Our optimized unconditional MDLM (PepMDLM) shows increased uniqueness and diversity with lower SNN compared to the autoregressive generator of
Then, we backtrack through the predecessor nodes of z up to the root node Zurr), adding the child reward vector to the cumulative reward vector and incrementing the number of visits for each node in the path. For all nodes from
7

---

De Novo Generation of Therapeutic Peptides with Multi-Objeciv--Cuudee Discrete Diffusion
Table Evaluation metrics for generative quality peptide SMILES sequences of max token length set 200.
Model
Validity (1)
Uniqueness (1)
Diversity (1)
SNN (1)
Randomness (f)
KI.-Divergence (1)
Data
1.000
1,000
0.885
1.000
4.55
0 (Reference)
PepMDLM
0.450
1,000
0,705
0,513
4.11
0,174
PepTune
1.000
1.000
0.677
0.486
4.12
0.173
macrocyclic peptides, HELM-GPT (Xu et al., 2024), demon- strating our capability to comprehensively search the sub- space of valid peptide SMILES (Table 7). Furthermore, our unconditional generator PepMDLM generates valid peptides with higher average nAA frequency than experimentally- validated peptide SMILES for membrane permeability and binding affinity (Figure 12), demonstrating our unique abil- ity to design de novo peptides with cyclic and nAA modifi- cations and expanding the search space of therapeutic pep- tides well beyond any generative model trained on canonical amino acid representations.
Semagbrtide
Swer
GLP-1R Binder
Linaglatde
Debling Bura
Binder
Figure 3. Comparison of docked PepTlune-generated peptides to existing GLP-1R agonists. (A, B) Docking images of semaglu- tide (score: -5.7 kcal/mol) and liraglutide (score: -5.1 keal/mol) binding GLP-IR. (C) Full view of the positive control GLP-IR agonists and the PepTune-generated generated binders on GLP-IR. (D, E) Docking images of binder (score: -7.4 kcal/mol) and (score: -7.0 kcal/mol) were generated using PepTune conditioned on pre- dicted affinity to GLP-IR, solubility, hemolysis, and non- non-fouling. Shared polar contacts between binder 1 and either controls are highlighted in pink, shared polar contacts between binder 2 and either controls highlighted in green, and the shared contacts across both binders are highlighted in purple.
Even though the multi-objective selection process favors high-reward unmasking steps, we show that the resulting pool of PepTuunegenerraee peptides retains similar unique- ness and diversity scores to the peptides generated by Pep- MDLM and the training dataset (Table 1). In addition, the fraction of valid peptides consistently reaches 100% after only 20 iterations of the MCTS search algorithm, demon- strating the effectiveness of backpropagating the classifier- based rewards.
Pep'Tune enables multi-objective generation of thera- peutic peptide binders. Given the sigmificant develop- ment of glucagon-like peptide-1 (GLP-1R) peptide ago- nists for the treatment of type-2 diabetes and obesity (Al- faris et al., 2024), compared GLP-1R binding affinity- conditioned peptides generated using PepTTnee with recent blockbuster GLP-1R 1RP agonists: semaglutide and liraglutide. Both semaglutide and liraglutide are over 30 amino acids in length and act by mimicking the binding of natural GLP-1 by binding to the activation pocket of GLP-1R with high preci- sion (Figure 3) (Mahapatra al., 2022; Nauck & D' AlAsssio, 2022).
kcal/mol for the two best candidates. Our peptides show superior docking affinity to GLP-1R while interacting at overlapping binding motifs to semaglutide and liraglutide derived from the natural hormone ligand, GLP-1 (Figure 3). These results suggest that our PepTune-derived peptides can serve potent agonists or antagonists of GLP-JR signaling.
PepTTune generates optimized dual-target-binding pep- tides. Multi-target drug discovery is of significant interest in various fields, including cancer therapeutics and drug delivery for neurological disorders, given their ability to perform multiple different functions such as binding to bi- ological barriers like the blood-brain barrier, penetrating target cells, and inhibiting protein-protein interactions (Tang et al., 2025a; Li et al., 2023b; Chan et al., 2016).
Shorter agonists or antagonists to GLP-IR would serve sev- eral benefits to the treatment of insulin-related disorders, including reduced cost and complexity of synthesis, lower immunogenicity, and faster tissue penetration. Therefore, we sought to generate shorter-chain peptides that are capable of binding to GLP-IR with comparable affinity to the exist- ing agonists. We first generated a pool of peptide binders conditioned on binding affinity with the GLP-IR sequence, solubility, hemolysis, and non-fouling. After selecting the peptides with the highest predicted binding affinity scores from the Pareto non-dominated set, we performed docking and determined docking scores of -7,4 kcal/mol and -7.0
To evaluate PepTune's capabilities in multi-target guid- ance, we generate bi-specific peptide binders to TíR and glutamate-aspartate transporter (GLAST) protein abundant on the surface of astrocytes, a type of glial cell in the brain. Successfully generating these peptides can facili- tate BBB BBB-crossing via TiR binding and uptake in astrocytes via GLAST binding for intravenous delivery of therapeutics
8

---

De Novo Generation of Therapeutic Peptides with Multi-Objecive-Cuidee Discrete Diffusion
Table Property metrics for Pepiuume-generated dual-target binders to TYR and GLAST. The predicted binding affinity scores by
our trained classifier are placed in brackets beside the docking score. Larger scores indicate stronger binding for our classifier.
Binder ID
TYR Docking Score (kea Vmol) (U)
GLAST Docking Score (kcal/ool ())
Solubility (1)
Hemolysis (1)
Non-fouling (1)
Binder
-8.9 (7.775)
0.975
0.743
0.118
Binder
-8.0 (7.599)
-7.9 (6.751)
0.938
0.835
0.309
Binder
-8.3 (7.537)
-8.2 (6.662)
0.972
0,914
0.214
Binder
(7.748)
-7.5 (6.946)
0.959
0,902
0.290
Binder
-10.5 (8.77 (8:714)
-8.5 (7 398)
0.811
0.748
0,202
Binder
-8.4 (8.197)
-7.5 (7.076)
0:971
0.855
0.165
Binder
-9.3 (8.321)
-9.2 (7.19))
0.881
0.860
0.212
Binding TIR
Binding GLAST
Eecting Sooee- -3.5
Boiking Juare
Basl Blnder
Figure 4. Pep/tune-generated peptides to TfR and GLAST. Full protein binding location and close-up binding position for (A) dual binder 1, (B) dual binder 6, and (C) dual binder 8 with TfR (left) and GLAST (right). Polar contacts within 3.5 Á are highlighted.
for a multitude of neurological disorders where astrocytes involved, including Alexander disease (Li et al., 2018 8), Alzheimer's disease (Habib et al., 2020), Parkinson's 's dis- ease (Yun et al., 2018), Huntington's 's disease (Khakh et al., 2017), multiple sclerosis (Wheeler et al., 2020), and several psychiatric disorders (Martin-Fermandeez et al., 2017).
6. Discussion
We introduce Pep'Tune, a multi-objective-guidee discrete diffusion model for de novo generation of peptide SMILES containing non-natural amino acids and cyclic modifications. We propose a done-dependent masking schedule and invalid loss to ensure peptide validity. PepTTune leverages Monte Carlo Tree Guidance (MCTG), novel multi-objective guidance framework for discrete diffusion, to identify pep- tide sequences optimized across multiple therapeutic proper- ties. Unlike previous discrete guidance methods (Nisonoff et al., 2024; Gruver al., 2023), MCTG operates strictly in the discrete state space and can be integrated at inference time with additional training. By balancing exploration through batched unmasking with Gumbel noise and reward- based exploitation of optimal unmasking paths, MCTG en- ables the generation of a diverse set of Pareto-optimal se- quenees across an arbitrary number of objectives. Unlike recent binder design methods (Rettie et al., 2025; Pacesa et al., 2024; Li et al., 2024; Watson et al., 2023), PepTTune requires 3D target structures, enabling the design of pep- tides for conformationally diverse proteins- such as fusion oncoproteins ( (Vincoff et al., 2025) and post-translationally modified isoforms (Peng et al., 2025b) while optimizing for properties beyond local geometric interactions.
We generated 100 peptide binders conditioned on five prop- erties: predicted binding affinity to TiR, binding affinity to GLAST, solubility, hemolysis, and non-fouling. Remark- ably, all property scores improved over iterations, with final solubility, hemolysis, and non-fouling scores surpassing binders conditioned solely on TfR binding affinity (Ap- pendix Figure 10). This highlights that our multi nulti-target guidance strategy avoids significant property trade-offs.
To validate binding to both TIR and GLAST, we selected seven binders and docked them against each target. All seven achieved docking scores < -7.5 kcal/mol, with the top binder scoring - 10.5 kcal/mol for TIR and -9.2 kcal/mol for GLAST (Table 3). These top binders exhibited diverse secondary structures (Figure 4), positive solubil- ity, and low hemolysis probabilities (Table 3). Candidates' binding positions and polar interactions varied, showing PepTune can discover diverse, high-affinity peptides with- out relying specific motifs.
9

---

De Novo Generation of Therapeutic Peptides with Multi-Objectv--Cuudee Discrete Diffusion
Impact Statement
References
We propose robust framework for multi-objective guidance of diffusion that could be applied to various generative tasks requiring simultaneous optimization of multiple constraints. Our work focuses designing therapeutically-viane tides, potentially advancing treatments for many diseases by enabling the discovery of peptides optimized for bind- ing, solubility, and other critical properties. However, like any powerful generative framework, there is a potential for misuse, such as generating peptides with harmful biological properties. We encourage careful oversight and ethical ap- plication of Pep'Tune to ensure its use aligns with positive socictal outcomes.
Alfaris, N., Waldrop, S., Johnson, V., Boaventura, B., Kendrick, K., and Stanford, F. C. Gip-1 single, dual, and triple receptor agonists for treating type 2 diabetes and obesity: a narrative review. eClinicalediccine 75: 102782, 2024.
pep-
Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and Berg, R. v. d. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 2021.
Bhat, S., Palepu, K., Hong, L., Mao, J., Ye, T., Iyer, R., Zhao, L., Chen, T., Vincoff, S., and Watson, R. e. De novo design of peptide binders to conformationally di- verse targets with contrastive language modeling. Science Advances, 1 14), 2025.
Declarations
Acknowledgments We thank Alexander Tong for review- ing the theoretical formulations of PepTune. We also thank Sophia Vincoff and Lauren Hong for their assistance with figure generation.
Bi, Y., Liu, L.., Lu, Y., Sun, T., Shen, C., Chen, X., Chen, Q. An, S., He, X., and Ruan, C. a. T7 peptide- functionalized peg-plga micelles loaded with carmustine for targeting therapy of glioma. ACS Applied Materials &amp; Interfaces, 8(41))27465-27473 2016.
Author Contributions S.T. devised and developed Pep- Tune architecture and theoretical formulations, and trained and benchmarked generation, prediction, and sampling mod- els. Y.Z. advised on model design and theoretical frame- work, trained classifier models, and performed molecular docking. S.T. drafted the manuscript and S.T. and Y.Z. de- signed the figures. P.C. conceived, designed, supervised, and directed the study, and reviewed and finalized the manuscript.
Brenner, M., Johnson, A. B., Boespflug-
O., Ro- driguez, D., Goldman, J. E., and Messing, A. Mutations in gfap, encoding glial fibrillary acidic protein, are asso- ciated with alexander disease. Nature Genetics, 27(1): 1 117-120, 2001.
Bryant, P. and Elofsson, A. Peptide binder design with inverse folding and protein structure prediction. Commu- nications Chemistry, 6(1), 2023.
Cai, L., Yang, C., Jia, W., Liu, Y., Xie, R., Lei. T., Yang, Z., He, X., Tong, R., and Gao, H. Endo/lysosome-escapahh delivery depot for improving bbb transcytosis and neu- ron targeted therapy of alzheimer's 's disease. Advanced Functional Materials, 30(27), 2020.
Data and Materials Availability Our pep- tide filtering, analysis, and visualization tool, SMILES2PEPTIDE, is freely available on Hugging-
Face: https: bttps://huggingaace cttpceces//ppcc////cccsssssccssc ShatterrjeL.ab/MMIEEEEEEPEEP
The Pep- Tune codebase is freely accessible to the academic community via a non-commercial license at https:
Chan, L. Y., Craik, D. J., and Daly, N. L. Dual-targeting anti-ange angiogenic eyclic peptides potential drug leads for cancer therapy. Scientific Reports, 6(1), 2016.
Chen, T. and Guestrin, C. Xgboost: A scalable tree boost- ing system. In International Conference on Knowledge Discovery and Data Mining, pp. 785-794, 2016.
Funding Statement This research was supported by NIH grant R35GM155282 as well as gift from the EndAxD Foundation to the lab of P.C.
Chen, T., Dumas, M., Watson, R., Vincoff, S., Peng, C., Zhao, L., Hong, L., Pertsemlidis, S., Shaepers- Cheu, M., and Wang. conditioned generation of therapeutic peptide binders via span masked language modeling. arXiv preprint 2023.
e. a. Pepmlm: Target sequence-
Competing Interests P.C. is co-founder of Gameto, Inc and UbiquiTx, Inc. and advises companies involved in pep- tide therapeutics development. P.C., S.T., and Y.Z. have and are currently filing patent applications related to this work. P.C.'s interests are reviewed and managed by Duke Univer- sity in accordance with their conffict-of-interes policies.
Chen, T., Zhang, Y. and Chatterjee, P. mop- pit:de novogeneration of motif-specific binders
10

---

De Novo Generation of Therapeutic Peptides with Multi-Objectve-CCuieee Discrete Diffusion
with protein language models. bioRxiv preprint bioRxiv: 10.1 10.11100000002240.311111 6060988 2024.
meta-analysis of gfap gene variants in alexander disease. Scientific Reports, 14(1), 2024.
Coulom, R. Efficient Selectivity and Backup Operators in Monte Tree Search, pp. 72-83. Springer Berlin Heidelberg, 2007. ISBN 9783540755388.
Gruver, N., Stanton, S., Frey, N. C., Rudner, T. G. J., Hotzel,
I., Lafrance-Vanasse, J., Rajpal, A., Cho, K., and Wilson, A. G. Protein design with guided discrete diffusion. Ad- vances in Neural Information Processing Systems, 2023.
Dang, C. V., Reddy, E. P., Shokat, K. M., and Soucek, L. Drugging the "undruggable" cancer targets. Nature Reviews Cancer, 17(8):502-508, 2017.
Guntuboina, C., Das, A., Mollaci, P., Kim, S., and Barati Fa- rimani, A. Peptidebert: language model based on transformers for peptide property prediction. The Jour- nal of Physical Chemistry Letters, 14(46):1047-104444 2023.
Davis, O., Kessler, S., Petrache, M., Ceylan, I. I., Bronstein, M., and Bose, A. J. Fisher flow matching for genera- tive modeling over discrete data. Advances in Neurul Information Processing Systems, 2024.
Habib, N., McCabe, C., Medina, S., Varshavsky, M., Kits- berg, D., Dvir-Szternfeld, R., Green, G., Dionne, D., Nguyen, L., and Marshall, J. L. a. Discase-associated astrocytes in alzheimer's s disease and aging. Nature Neu- roscience, 23(6):701-706, 2020. ISSN 1546-1726.
di Clemente, N., Racine, C., and Rey, R. A. Anti-müllerian hormone and polycystic ovary syndrome in women and its male equivalent. Biomedicines, 10(10):2506, 2022.
Duffy, J., Verniere, M., Devocelle, M., Bernard, E., Shields, D. C., and Chubb, A. Cyclops: Generat- ing virtual libraries of cyclized and constrained peptides including nonnatural amino acids. Journal of Chemical Information and Modeling, (144)8829 836, 2011.
Hart, K. N., Stocker, W. A., Nagykery, N. G., Walton, K. L., Harrison, C. A., Donahoe, K., Pépin, D., and Thomp- son, T. B. Structure of amh bound amhr2 provides insight into unique signaling pair in the tgf-3 family, Proceedings of the National Academy of Sciences, 118 (26), 2021.
Eberhardt, J., Autodock vina 1.2. New docking methods, expanded force field, and python bindings. Journal of chemical information and modeling, 61(8):3891-3888. 2021.
D., Tillack, A. F,, and Forli, S.
Hol, E. M. and Pekny, M. Glial fibrillary acidic protein (gfap) and the astrocyte intermediate filament system in diseases of the central nervous system. Current Opinion in Cell Biology, 32:121-130, 2015.
Eng, L. F. Glial fibrillary acidic protein (gfap): the major protein of glial intermediate filaments in differentiated astrocytes. Journal of Neuroimmunology. 8:203-214, 1985.
Imbeaud, S., Belville, C., Messika-Zeitoun L., Rey, R., di Clemente, N., Josso, N.. and Picard, J.-Y. A 27 base- pair deletion of the anti-müllerian type ii receptor gene is the most common cause of the persistent müllerran duct syndrome. Human molecular genetics, 5(9):1269-127) 1996.
Feller, A. L. and Wilke, C. O. Peptide-aware chemical language model successfully predicts membrane diffusion of cyclic peptides. bioRxiv, 2024.
Gat, I., Remez, T., Shaul, N., Kreuk, F,, Chen, R. T. Q., Synnaeve, G., Adi, Y., and Lipman, Y. Discrete flow matching, Advances in Neural Information Processing Systems, 2024.
Jefferies, W. A., Brandon, M. R., Hunt, S. V., Williams, A. F., Gatter, K. C.,, and Mason, D. Y. Transferrin receptor on endothelium of brain capillaries. Nature, 3 312(5990): 162-163, 1984.
Gfeller, D., Michielin, O., and Zoete, Swisssidechain: molecular and structural database of non-natural sidechains. Nucleic Acids Research, 41(D1):D327-D332 2012.
Johnsen, K. B., Burkhart, A., Melander, F., Kempen, P. J., Vejlebo, J. B., Siupka, P., Nielsen, M. S., Andresen, T. L., and Moos, T. Targeting transferrin receptors at the blood- brain barrier improves the uptake of immunoliposomes and subsequent cargo transport into the brain parenchyma. Scientific Reports, 7(1), 2017.
Gosk, S., Vermehren, C., Storm, G., and Moos, Target- ing transfernnnri receptor antibody (ox26) and ox26- conjugated liposomes brain capillary endothelial cells using in situ perfusion. Journal of Cerebral Blood Flow &amp; Metabolism, 24(11):1193-1200 2004.
Khakh, B. S., Beaumont, V., Cachope, R., Munoz-Sanjura I., Goldman, S. A., and Grantyn, R. Unravelling and exploiting astrocyte dysfunction in huntington's 's disease. Trends Neurosciences, 40(7):422-433 2017.
Grossi, A., Rosamilia, F., Carestiato, S., Salsano, E., Cec- cherini, I., and Bachetti, T. A systematic review and

---

De Novo Generation of Therapeutic Peptides with Multi-Objeciv--Cuudee Discrete Diffusion
Kim, G., Kim, M., Lee, Y., Byun, J. W., Hwang. D. W., and Lee, M. Systemic delivery of micromm-221 antisense oligonucleotides to the brain using 17-peptide decorated exosomes. Journal of Controlled Release, 317:273-281, 2020.
and He, S. Smprot:
A reliable repository with comprehen- sive annotation of small proteins identified from ribosome profiling, Genomics, Proteomics &amp; Bioinformatics. 19(4):002-6) 610, 2021.
Liang, M., Gao, C.,, Wang. Y., Gong, W., Fu, S., Cui, L.,
Kingma, D. P., Salimans, T., Poole, B., and Ho, J. Varia- tional diffuusion models. Advances in Neural Information Processing Systems, 2021.
Zhou, Z.,, Chu, X., Zhang, Y., and Liu, Q. Enhanced blood-brain barrier penetration and glioma therapy me- diated by t7 peptide-modified low-density lipoprotein particles. Drug Delivery, 25(1): 1663, 2018.
Krenn, M., Häse, F., Nigam, A., Friederich, P., and Aspuru- Guzik, A. Self-referencing embedded strings (selfies): A 100% robust molecular string representation. Machine Learning: Science and Technology, 144):045024, 2020.
Lin, J. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory, 37(1):145-))) 151, 1991.
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., and Shmueli, Y. e. a. Evolutionary tein structure with language model. Science, 379(6637); 1123-1130, 2023.
Kuang, Y., An, S., Guo, Y., Huang, S., Shao, K., Liu, Y., Li, J., Ma, H., and Jiang, C. T7 reptide-functional nanoparticles utilizing mna interference for glioma dual targeting. International Journal of Pharmaceutics, 454 (1):11-20, 2013.
prediction of atomic-level pro-
Lou, A., Meng, C., and Ermon, S. Discrete diffusion mod- eling by estimating the ratios of the data distribution. International Conference on Machine Learning, 2024.
Lazaros, L.. Fotaki, A., Pamporaki, C., Hatzi, E., Kitsou, C., Zikopouus,,, A., Virgiliou, C., Kosmas, I,, Bouba, I,, Stefos, T., et al. The ovarian response to standard go- nadotropin stimulation is influenced by amhrii genotypes. Gynecological Endocrinology, 32(8):641-645 2016.
Mahapatra, M. K., Karuppasamy, M., and Sahoo, B. M. Semaglutide, a glucagon like peptide-1 receptor agonist with cardiovascular benefits for management of type 2 diabetes. Reviews Endocrine and Metabolic Disorders, 23(3):521-539, 2022.
Lee, J. H., Engler, A., Collawn, J. F., and Moore, B. A. Receptor mediated uptake of peptides that bind the human transferrin receptor. European Journal of Biochemistry, 268(7):2004-200 2012, 2001.
Fernande, M., Jamison, S., Robin, L. M., Zhao, Z., Martin, E. D., Aguilar, J., Benneyworth, M. A., Marsi- cano, G., and Araque, A. Synapse-specific astrocyte gat- ing of amygdala-related behavior. Nature Neuroscience, 20(1 I): 1540-18888 2017.
Li, J.-N., Yang, G., Zhao, P.-C., Wei, X.-X., and Shi. J.-Y. Cpromg: controllable protein-orientee molecule genera- tion with desired binding affinity and drug-like properties. Bioinformatics, 39:1326-i336. 2023a.
Mendez, D., Gaulton, A., Bento, A. P., Chambers, J., De Veij. M., Félix, E., Magariños, M. P., Mosquera, J. F, Mutowo, P., Nowotka, M., and Wordillo-Marañioe M. e. a. Chembl: towards direct deposition of bioassay data. Nucleic Acids Research, 47(D1):D930-DD D9400, 2018.
Li, L., Tian, E., Chen, X., Chao, J., Klein, J., Qu, Q., Sun, G., Sun, G., Huang, Y., Warden, C. D., and Ye, P. e. Gfap mutations in astrocytes impair oligadendrocyte pro- genitor proliferation and myelination in an hipse model of alexander discase. Cell Stem Cell, 23(2))239-25) 2018.
Morris, G. M., Huey, R., Lindstrom, W., Sanner, M. F., Belew, R. K., Goodsell, D. S., and Olson, J. Autodock4 and autodocktools4: Automated docking with selective receptor flexibility. Journal of computational chemistry, 80(16):2785-2799 2009.
Li, Q., Vlachos, E. N., and Bryant, P. Design of linear and cyclic peptide binders of different lengths from protein sequence information. bioRxiv preprint bioRxiv: 10.1110//2224.0220.999993 2024.
Muttenthaler, M., King, G. F., Adams, D. J., and Alewood, P. F. Trends in peptide drug discovery. Nature Reviews Drug Discovery, 20(4):309-32) 2021.
Li, X., Pu, X., Wang, X., Wang. J., Liao, X., Huang, Z., and Yin, G. A dual-targeting peptide for glioblastoma screened by phage display peptide library biopanning combined with affinity-adaptabiie analysis. Interna- tional Journal of Pharmaceutics, 644: 2023b.
Nauck, M. A. and D'Alessio, D. A. Tirzepatide, a dual gip/glp-1 receptor co-agonist for the treatment of type 2 diabetes with unmatched effectiveness regrading gly- caemic control and body weight reduction. Cardiovascu- lar Diabetology, 21(1), 2022.
Li, Y., Zhou, H., Chen, X., Zheng, Y., Kang, Q., Hao, D.,
Zhang, L.,, Song, T., Luo, H., Hao, Y., Chen, R., Zhang, P.,
12

---

De Novo Generation of Therapeutic Peptides with Multi-Objectvv--Cudee Discrete Diffusion
Nisonoff, H., Xiong, J., Allenspach, S., and Listgarten, Unlocking guidance for discrete state-space diffusion and flow models. arXiv preprint arXiv:2406.01577, 2024.
Sahoo, S. S., Arriola, M., Schiff, Y., Gokaslan, A., Marro- quin, E., Chiu, J. T.,, Rush, A., and Kuleshov, V. Simple and effective masked diffusion language models. Ad- vances in Neural Information Processing Systems, 2024.
Ou, J., Nie, S., Xue, K., Zhu, F., Sun, J., Li, Z.,, and Li,
C. Your absorbing discrete diffusion secretly models the conditional distributions of clean data, arXiv preprint arXiv:2406.0m 03736, 2024.
Schrödinger, LLC. The PyMOL molecular graphies system, version 1.8, November 2015.
Shi, J., Han, K. Wang, Z., Doucet, A., and Titsias, M. K. Simplified and generalized masked diffusion for discrete data. Advances in Neural Information Processing Sys- tems, 2024.
Pacesa, M., Nickel, I.., Schellhaas, C., Schmidt, J., and et al., Bindcraft: one-shot design of functional protein binders. bioRxiv, 2024.
Paratcha, G., Ledda, F., and Ibáñez, F. The neural cell ad- hesion molecule neam is an alternative signaling receptor for gdnf family ligands. Cell, 113(7)867-877) 2003.
Singh, S., Pal, N., Shubham, S., Sarma, D. K., Verma, V., Marotta, F,, and Kumar, M. Polycystic ovary syndrome: Etiology, current management, and future therapeutics. Journal of Clinical Medicine, 12(4):1454, 2023.
Peng. F. Z., Bezemek, Z. Patel, S., Rector-Browd Brooks, J., Yao, S., Tong, A., and Chatterjee, P. Path planning for masked diffusion model sampling. arXiv preprint arXiv:2502.03.3 035440 2025a.
Sosunov, A. A., McKhann, G. M., and Goldman, J. E. The origin of rosenthal fibers and their contributions to astro- cyte pathology in alexander disease. Acta Neuropatho- logica Communications, 5(1), 2017.
Peng. F. Z., Wang, C., Chen, T., Schussheim, B.. Vincoff, S., and Chatterjee, P. Ptm-mamba: a ptm-aware protein language model with bidirectional gated mamba blocks. Nature Methods, 22(5):945-949, 2025b. ISSN 1548- 7105.
Stark, H., Jing, B., Wang, C., Corso, G., Berger, B., Barzilay, R., and Jaakkola, T. Dirichlet flow matching with appli- cations to dna sequence design. International Conference on Machine Learming, 2024.
Polykovskiy. D., Zhebrak, A., Sanchez-Lengeling B., Golo- vanov, S., Tatanov, O., Belyaev, S., Kurbanov, R., Arta- monov, A., Aladinskiy, V., and Veselov, M. e. a. Molecu- lar sets (moses): benchmarking platform for molecular generation models. Frontiers Pharmacology, 11, 2020.
Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint 1rXiv:2104.09866 2021.
Tang, S., Han, E. L.,, and Mitchell, M. J. Peptide- functionalized nanoparticles for brain-targeted therapeu- tics. Drug Delivery and Translational Research, 2025a. ISSN 2190-3948.
RDKit, online. RDKit: Open-source cheminformatics. http: http://wwwww..tttt// org.
Rector-Brooks, J., Hasan, M., Peng. Z., Quinn, Z., Liu, C.,, Mittal, S., Dziri, N., Bronstein, M., Bengio, Y., Chat- terjee, P., Tong, A., and Bose, A. J. Steering masked discrete diffusion models via discrete denoising posterior prediction, arXiv preprint orXiv:2410.0813.1 2024.
Tang, S., Zhang, Y., Tong. A., and Chatterjee, P. Gumbel- softmax flow matching with through guidance for controllable biological sequence generation. arXiv preprint arXiv:2503.17366 2025b.
Rettie, S. A., Campbell, K. V., Bera, A. K., Kang, A., Ko- zlov, S., Bueso, Y. F., De La Cruz, J., Ahlrichs, M., Cheng, S., and Gerben, S. R. e. a. Cyclic peptide structure pre- diction and design using alphafold2. Nature Communica- tions, 16(1), 2025. ISSN 2041-1723.
Uversky, V. N., Oldfield, C. J., and Dunker, A. K. Intrin- sically disordered proteins in human diseases: Introduc- ing the d2concept. Annual Review of Biophysics, 37(1): 215-246, 2008
Rigon, C., Andrisani, A., Forzan, M., D' Antona, D., Bru- son, A., Cosmi, E., Ambrosini, G., Tiboni, G. M., and Clementi, M. Association study of amh and amhrii poly- morphisms with unexplained infertility. Fertility and sterility, 9444):1244-1288 2010.
Vignac, C., Krawezuk. I., Siraudin, A., Wang, B., Cevher, V., and Frossard, P. Digress: Discrete denoising diffusion for graph generation. International Conference on Learning Representations, 2022.
Vincoff, S., Goel, S., Kholina, K., Pulugurta, R., Vure, P., and Chatterjee, P. Fuson-plm: a fusion oncoprotein- specific language model via adjusted rate masking. Na- ture Communications, 16(1), 2025. ISSN 2041-1723.
Rogers, D. and Hahn, M. Extended conneeeidtoneetiiitty finger- prints. Journal of Chemical Information and Modeling, 50(5):742-75,, 2010.
13

---

De Novo Generation of Therapeutic Peptides with Mult--Objeciv---uidee Discrete Diffusion
Vinogradov, A. A., Yin, Y., and Suga, H. Macrocyclic pep- tides as drug candidates: Recent progress and remaining challenges. Journal of the American Chemical Society, 141(10):4167-4181, 2019.
Yu, M., Su, D., Yang, Y., Qin, L.,, Hu, C., Liu, R., Zhou, Y., Yang, C., Yang. X., Wang, G., and Gao, H. D-17 peptide-modified pegylated bilirubin nanoparticles loaded with cediranib and paclitaxel for antiangiogenesis and chemotherapy of glioma. ACS Applied Materials &amp; Interfaces, (1(1):176-188)) 2018.
Vukojevic, V., Mastrandreas, P., Arnold, A., Peter, F., Ko- lassa, I.-T., Wilker, S., Elbert, T., de Quervain, D. J.-F.,
Papassotiropoulos, A., and Stetak, A. Evolutionary con- served role of neural cell adhesion molecule-1 in memory. Translational psychiatry, 10(1):217,200) 2020.
Yun, S. P., Kam, T.-I,, Panicker, N., Kim, S., Oh, Y., Park, J.-S., Kwon, S.-H., Park, Y. J., Karuppagounder, S. S., and Park, H. e. a. Block of al astrocyte conversion by microglia is neuroprotective in models of parkinson's disease. Nature Medicine, 24(7):931-938, 2018.
Wang, L., Wang, N., Zhang, W., Cheng, X., Yan, Z., Shao, G., Wang, X., Wang. R.. and Fu, C. Therapeutic pep- tides: current applications and future directions. Signal Transduction and Targeted Therapy, 7(1), 2022.
Zhang, D., Duque-Jimenez, J., Facchinetti, F., Brixi, G., Rhee, K., Feng, W. W., Jänne, P. A., and Zhou, X. Trans- ferrin receptor targeting chimeras for membrane protein degradation. Nature, 2024.
Wang, S., Witek, J., Landrum, G. A., and Riniker, S. Improv- ing conformer generation for small rings and macrocycles based on distance geometry and experimental torsional- angle preferences. Journal of chemical information and modeling, 60(4):2044-205 2020.
Zhang, R., Wu, H., Xiu, Y., Li, K., Chen, N., Wang, Y., Wang, Y., Gao, X., and Zhou, F. Pepland: a large-scale pre-trained peptide representation model for a compre- hensive landscape of both canonical and non-canonical amino acids. arXiv preprint arXiv:231 04444, 2023.
Wang, X., Zheng, 7., Ye, F., Xue, D., Huang, S., and Gu, Q- Diffusion language models are versatile protein learners. International Conference on Machine Learning, 2024.
Zhang, T. Li, H., Xi, H., Stanton, R. V., and Rotstein, S. H. Helm: A hierarchical notation language for com- plex biomolecule structure representation. Journal of Chemical Information and Modeling, 52(10):279-2288, 2012.
Wang, Z., Zhao, Y.. Jiang, Y., Lv, W., Wu, L., Wang. B., Lv,
L., Xu, Q., and Xin, H. Enhanced anti-ischemic stroke of z1006 by (7-conjugated pegylated liposomes drug delivery system. Scientific Reports, 5(1), 2015.
Zhao, Y., Jiang. Y., Lv, W., Wang, Z., Lv, L.., Wang, B., Liu, X., Liu, Y., Hu, Q., Sun, W., Xu, Q., Xin, H., and Gu, Dual targeted nanocarrier for brain ischemic stroke treatment. Journal of Controlled Release, 233:64-71, 2016.
Watson, J. L., Juergens, D., Bennett, N. R., Trippe, B. L.., Yim, J., Eisenach, H. E., Ahem, W., Borst, A. J., Ragotte, R. J., and Milles, L. F. a. De novo design of pro- tein structure and function with rídiffusion. Nature, 620 (7976):1089-1100, 2023.
Zheng, K., Chen, Y., Mao, H., Liu, M.-Y., Zhu, J., and Zhang, Q. Masked diffusion models are secretly time- agnostic masked models and exploit inaccurate categori- cal sampling. arXiv preprint (4l::2409002908 2024.
Weininger, D. Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules. Journal of Chemical Information and Computer Sciences, 28(1):31-33 1988.
Wheeler, M. A., Clark, I. C., Tjon, E. C., Li, Z., Zandee, S. E. J., Couturier, C. P., Watson, B. R., Scalisi, G., Alkwai, S., and Rothhammer, V. a. Mafg-driven astrocytes promote inflammation. Nature, 578(7796):593-599 2020.
Xu, X., Xu, C., He, W., Wei, L., Li, H., Zhou, J., Zhang, R., Wang, Y., Xiong, Y., and Gao, X. Helm-gpt: gpt: de novo macrocyclic peptide design using generative pre-trained transformer. Bioinformatics, 40(6), 2024.
Yang, Q., Zhao, J., Chen, D., and Wang. Y. E3 ubiqui- tin ligases: styles, structures and functions. Molecular Biomedicine, 2(1), 2021.
14

---

De Novo Generation of Therapeutic Peptides with Mult--Objectv--Cuddee Discrete Diffusion
Overview of Appendix
In Appendix A, we discuss additional case studies demonstrating PepTune's ability to generate peptides with high binding affinity and enhanced therapeutic properties to several therapeutic targets, including receptors on the blood-brain barrier (A.1), intracellular proteins with enhanced cell permeability (A.2), targets without existing peptide binders (A.3), and dual-targeting for target-protein degradation (A.4).
Appendix B provides background on continuous-time discrete diffusion (B.1), the NELBO loss objective (B.2), and guidance for diffusion models (B.3). Appendix C provides details on data curation and tokenization. Appendix D provides additional implementation details and generation results of our unconditional cond-dependent masked discrete diffusion model, PePMDLM. Appendix E provides details on the model architecture and training of our property prediction models for binding affinity 1F.1), membrane permeability (E.2), solubility, hemolysis, and non-fouling (E.3). Appendix F contains details on our evaluation methods, including our SMILES2PEPTIDE filter (F.1).
In Appendix G, we provide the theoretical basis for Section 2, including formal proofs for Proposition 2.2 (G.2), Proposition 2.1 (G.3), and Proposition 2.3 (G.4). Appendix H discusses the choices of hyperparameters. Appendix provides results of additional experiments, including one that integrates time-dependence into the MCTG algorithm (1.1) and ablation studies investigaaing the impact of oond-dependent masking and the invalid loss on generation quality (I.2). Finally, we provide pseudo code for all of our algorithms in Appendix J.
A. Additional Case Studies
With our trained property classifiers, we conduct experiments for diverse, therapeutically relevant protein targets to evaluate our multi-objectivv MCTS guidance strategy. To demonstrate generalizability, we include targets with known peptide binders such as TfR, and proteins with no known binders, including GFAP, NCAM1, and AMHR2. We also design bi-specific binders for GFAP and RBXI as a case study for Alexander disease therapeutics. These targets include both receptor proteins involved with active transport pathways as well as intracellular targets where cell membrane permeability is crucial to achieving therapeutic effects. For each target, we condition the generation on the binding affinity score given the target protein sequence, along with solubility, hemolysis, non-fouling, and cell membrane permeability for intracellular targets. For external testing and validation, we use Autodock Vina (Eberhardt et al., 2021) to compute in silico binding affinities of our generated binders (Appendix F.3).
A.1. Targeting Receptors on the Blood-Brain Brain Barrier
The Transferrin receptor (TfR) is a receptor protein abundant the selectively permeable blood-brain barrier (BBB) that is responsible for transporting iron- iron-binding transferrin (Tf) proteins into the brain parenchyma (Johnsen et al., 2017). Given its selective expression on brain endothelial cells and glioma cells and its ability to recycle back to the luminal surface after facilitating the internalization of cargo through the BBB (Jefferies et al., 1984), TyR has been extensively studied as a target for the intravenous delivery of various therapeutics and therapeutic nanocarriers through the BBB (Gosk et al., 2004; Zhang et al., 2024).
To generate relevant binders for TfR, we condition PepTune binding affinity with the TfR sequence, in addition to solubility, hemolysis, and non-fouling, At each iteration, we measured the mean of the property scores across all rolled-out sequences from the selected node to evaluate the effectiveness of the optimization strategy. We show that all properties, except solubility, exhibited an upward trend over iterations, with the average score of the binding affinity classifier exhibiting a significant increase in score to over 9.0 (Figure 5B). After plotting the distribution of 100 peptides generated from a single run of PepTuune with the minimum number of sequences set to 100, we confirm that our multi-objectivee MCTS algorithm higher predicted binding affinity than the unconditionally generated peptides (PepMDLM) and
shifted the distribution the data used to train the binding regression model (Figure 5A). Despite being conditioned on four distinct properties, PepTunee is capable of generating higher-affinity binders than the unconditional model, supporting the effectiveness of our multi-objective guidance strategy.
Encouraged by these results, we sampled the Pareto-optimal sequences from the generated peptides and used Vina docking to compute their optimized docking score. Notably, we observed that all of the generated binders that were selected for docking produced affinity scores below -6.0 kcal/mol, with our per-rerfonnnng binder achieving a -8.4 kcal/mol binding affinity (Figure 5C). From the docking scores, we took the two binders with the best docking scores and visualized their binding conformation with TiR, showing that they bind to distinct motifs on the protein surface (Figure 5B, F, G).
15

---

De Novo Generation of Therapeutic Peptides with Mulli-Obeectve--uiiee Discrete Diffusion
Density Plot of TIR Binding Affinity
PANAOLLA
TI Binder VINA Cooking Booe 40 Honolpso 0 857 (N Fouerg 3.0
TEE Ginder 2
Rivding cocking Score 88 Mamobas 0.943 (N Foud 9.4225
TER Rinder 2
TER d Binding: 9.375 (ubo 8.705 Hemolyeke 6.942 Kon-Rouling 0.064
Binding 8:330 Sanabiity 8.745 Hemelyckc 6.926) Rine-Roling 0. 109
Figure 5. PepTuune-generated peptide binders TfR. (A) Density plot depicting the frequency of predicted binding affinity scores from our trained regression model for the sequences in the data used to train the regression model, the generated peptides from our unconditional PepMDLM model, and our PepTune model conditioned on TıR binding affinity. solubility. hemolysis. and non-fouling, (B) Plots depicting the mean scores for each property over the number of iterations or traversals of the MCTS algorithm for 128 iterations and a maximum token length 200. The shaded region represents the standard deviation. (C) Two-dimensional visualization of generated binders with token length 100, their corresponding docking scores (L) computed using Vina docking, and predicted classifier scores (t) from the trained classifiers. (D) Visualizations of generated binders with token length 200, their docking scores, and predicted classifier scores.
To further confirm binding affinity TiR, we compared our peptides the well-established 7-amino acid peptide T7 (sequence: HAIYPRH) that selectively binds to alternative site compared to endogenous Tf on TYR (Lee et al., 2001).
T7 has been extensively explored for targeted delivery of nanoparticles to the brain (Kuang et al., 2013; Kim et al., 2020;
Bi et al., 2016; Cai et al., 2020; Wang et al., 2015; Zhao et al., 2016; Liang et al., 2018), and has demonstrated 7.89-fold
enhanced brain penetration in in vivo mice models (Yu et al., 2018). After docking T7 with TiR, we obtained a docking score of -8.4 kcal/mol. Notably, our peptides optimized on all four therapeutic properties, including TfR binding affinity show competitive docking scores T7 (Figure 6A, C, E), suggesting that PepTune is capable of generating promising candidates for in vivo targeting and delivery across the BBB. Furthermore, after annotating polar contacts within 3.5 Á we determine that both of the generated peptides with the best binding affinity scores have shared residue contacts when binding with TfR as T7 (Figure 6B, D, F), indicating that our generated peptides have similar binding properties to T7, enabling it to bind strongly to a shared binding site. Furthermore, our generated binders have diverse structural features, such as cycles in binder and side-chain modifications in binder 2. Since T7 is known to bind to an alternative site than endogenous Tf (Lee et al., 2001), we show that PenTune can generate viable candidates for non-competitive binding to TíR for BBB-targeting applications.
A.2. Targeting Intracellular Proteins
Glial fibrillary acidic protein (GFAP) is an intracellular protein differentially expressed in astrocytes, a family of glial cells in the brain (Hol & Pekny, 2015). Dysregulation of GFAP expression has been found to cause Rosenthal fibers, astrocytic cytoplasmic inclusions that are responsible for Alexander discase, fatal neurodegenerative discase affecting infants (Brenner et al., 2001; Grossi et al., 2024). Discovering potent binders that inhibit degrade GFAP proteins can have significant therapeutic implications. However, no established peptide binders exist to GFAP, which motivates their de novo design. In addition to achieving high binding affinity with GFAP, we posit that an optimal peptide binder must also cross the astrocyte cell membrane into the cytosol to access GFAP. Therefore, we condition the generation of GFAP binders on five properties: binding affinity to GFAP, solubility, hemolysis, non-fouling, and cell membrane permeability using our permeability regression model, demonstrating optimization across all of these properties (Figure 7). To confirm GFAP
16

---

De Novo Generation of Therapeutic Peptides with Multi-Objecive-Guidee Discrete Diffusion
Peplide (НATYPRH)
rax
TIR 2inder VINA Docking Score--84
Figure 6. Comparison of PepTune-generated peptides and established 17-peptide to TIR. Two-dimensional chemical structure of (A)
HepTune-generaae binder 1, (C) established peptide, and (E) aapTune-generatee TYR binder and their Vina docking scores to TiR (L), Zoomed in visualization of the docked binding positions of (A) binder 1, (B) T7, and (C) binder 2 with TIR. Polar contacts within 3.5 Á are annotated, and shared contacts between T7 and binder (purple) and between and binder (bluc) are highlighted. (C) Overlay of peptide binders on full TIR protein
engagement, our docking peptides demonstrate strong affinities below -7 kcal/mol, motivating downstream experimental validation in astrocyte cultures (Figure 7B and D).
A.3. Targeting Proteins Without Existing Binders
To test the ability of our model to generate binders to challenging extracellular targets without existing binders, we evaluate PepTune-generated generated peptides for NCAMI and AMHR2, two therapeutically relevant receptor proteins. Neural cell adhesion molecule (NCAMI) a transmembrane protein expressed on the surface of neurons and glial cells (Paratcha et al., 2003). Beyond its roles in neuronal migration and synaptogenesis, NCAMI is also crucial for memory formation, highlighting its significance in brain development (Vukojevic et al., 2020). As NCAM1 is extracellular protein, generated library of peptides with repTTune-optimizer NCAM1 binding affinity, solubility, hemolysis, and non-fouling (Figure 8F, G). All properties exhibited an upward trend across optimization iterations.
We selected two binders with the highest Vina docking scores for visualization (Figure 8A-E). Notably, in silico docking analysis revealed that binder exhibits markedly high affinity binding (-8.6 kcal/mol) while binder 2 wraps around the NCAM1 structure via numerous polar contacts, suggesting extensive and specific interactions (Figure 8B and D).
17

---

De Novo Generation of Therapeutic Peptides with Multi-Objectiv--Cuiee Discrete Diffusion
Dansity Mambrans Permuability
Hesi 2000
GIAP
Solubility 8.822 Hamo
Docking Score: -8.5
Denoty Bloding Afferty
Data
6.972 Permeebllty 6.001
Solbilly: 0.906 Foulin 0.876
Docking Score:
Seadin mnyloae
Figure Pep'tune-gemerated peptide binders to intraceltular protein GFAP. (A, C) Two-dimensional structures of GFAP binder I and 2 with predicted property scores, including cell membrane permeability. (B, D) GFAP binders 1 and 2 docked GFAP with scores of -8.5 kcal/mol and -7.1 kcal/mol, respectively. (E) Full GFAP protein structure with docked binders and 2. (F) The distribution of peptides compared 100 peptides generated using unconditional PepMDLLM .M model, and 100 peptides generated with PepTune conditioned on both cell membrane permcability and affinity shifted towards higher permeability with predicted binding affinity scores to GFAP for the PepTune-generated peptides is shifted to higher scores with mean of 8.053 compared set of experimentally-tesse peptides and unconditional PcpMDLM --gnerrted peptides.
PAMPA membrane permeability scores from 34,853 superimennallly-valddaad
GFAP. The permeability
of -6.295. (G) Simultancously. the distribution of
B
Denotyy Piot NCAM1 Binding Affinity
VINA Docking Score
nddngg 7.697| Soubilliyy 0.976
Prac Graling Aiaiiyy Mileet
Hemelyass 0.822 Non Fouling 0.102
D
VINA Docking Score -70
anding 7.289 SSmbiilit 0 934
Hemotysic: 98911 Non-Fouling 0.218
Figure repTuune-eeeerteed peptide binders NCAM1. Two-dimensional structures binder and binder 2 genered with PepTune. Docking positions of (B) binder and (C) binder 2 on NCAMI with annotated polar contacts within 3.5 Å(G) Full NCAM1 protein structure with docked peptide binders and (H) (Top) Density plot of NCAMI binding affinity scores for PepTune (mean: 6.708). PepMDLM (mean: 5.298), and peptides from control set of experimentally- tested peptide SMILES (mean: 5.360). (Bottom) Plots depicting the average predicted score for NCAM1 binding affinity, solubility, hemolysis, and non-fouling over iterations MCTS.
Anti-Mülleria hormone type-2 receptor (AMHR2) is transmembrane receptor involved in differentiation. Mutations in the AMHR2 gene are leading cause of Persistent Müllerian duct syndrome (PMDS) in males, resulting in the retention of female gonads alongside male reproductive structures (Imbcaud et al., 1996). In females, polymorphisms of AMHR2 have been associated with infertility (Rigon et al., 2010; Lazaros et al., 2016). Most interestingly, antagonism of AMHR2 with therapeutic peptides can potentially serve as specific therapy for polycystic ovarian syndrome (PCOS), which affects an estimated 4% 10% of women globally (Singh et al., 2023), as AMHR2 signaling has been implicated in follicular
18

---

De Novo Generation of Therapeutic Peptides with Multi-Objectve--uuieee Discrete Diffusion
Denuity Plut of AMHI2 Binding Aflinty
AMRR2 Binder1 VINA bozking Scor: 8.8 Sinding,75 7.829 Solub Hemoolyie Fodlagg
G Norley
Nen Fhaling Crose Paeredery
AMER2 Binder 7.561 (Saiiiltty 918
Homotys 0.361| Non-Foulngg 0.107
MCC
Figure 9. PepTuune-generated peptides to AMHR2. Two-dimensional structures of (A) binder and (B) binder generated with PepTune. Docking positions of (A) binder and (B) binder 2 on NCAMI with annotated polar contacts. (G) Full AMHI2 protein structure with docked peptide binders and 2. (H) (Top) Density plot of AMHR2 binding affinity scores for PepTTnee (mean: 8.212). PepMDLM (mean: peptide SMILES (mean: 6.740). (Bottom) Plots depicting the average
6.832), and peptides from control set of experimentally-teee predicted score for AMHR2 binding affinity, solubility, hemolysis, and non-fouling over iterations of MCTS.
Table 4. rep'lune-generatee dual-target binders to GFAP and RBX1. The predicted binding affinity scores by our trained classifier are placed in brackets beside the docking score. Larger scores indicate stronger binding for classifier.
Binder ID
GFAP Docking Score (kcal/mol) (1)
E3 Docking Score (kcal/mol) (J)
Solubility (f)
Hemolysis (1)
Non-fouling (t)
Binder
-8.0 (8.384)
-8.4 (7.468)
0.730
0.894
0.111
Binder
-8.3 (7.395)
-9.3 (7.089)
0.972
0.869
0.134
Binder 3
-7.3 (7.925)
-8.7 (7.158)
0.935
0.812
0.143
Binder
-8.8 (7.144)
-8.7 (7.000)
0.897
0.807
0.158
arrest and dysregulated ovarian function (di Clemente et al., 2022).
Following similar computational setups as described previously, we generated in silico binders with high Vina predicted binding affinities (<-6 kcal/mol), despite observing a decrease in the predicted solubility along iterations (Figure 9). However, our observation of reduced solubility upon binder docking can be attributed the presence of hydrophobic patches within the AMHR2 extracellular domain, particularly near the binding site to its ligand AMH (Hart et al., 2021). This phenomenon highlights the importance of balancing solubility and binding affinity in binder development. With further optimization of their therapeutic properties, we hope to demonstrate the potential of these binders for applications in fertility treatment in the future.
The examples above demonstrate the versatility of our method, which can effectively applied to discover peptide binders for single target proteins lacking known ligands, thereby unlocking their therapeutic potential.
A.4. Dual- Targeting of GFAP and an E3 Ubiquitin Ligase for Target Protein Degradation
As another dual-target case study, we used PepTune to generate peptides with high binding affinity to the GFAP protein and an E3 ubiquitin ligase protein RBXI, a protein in the Skpl/Cullin- 1/F-boxx (SCF) E3 ubiquitin ligase complex that recruits E2 to catalyze ubiquitination and subsequent degradation (Yang et al., 2021). A peptide generated for this task would be capable of binding to GFAP proteins overexpressed in Alexander disease and mediate their proteasomal degradation, which could alleviate the production of disease-causing Rosenthal fibers in astrocytes (Sosunov et al., 2017). After conditioning PepTune generation on binding affinity to GFAP, binding affinity to RBX1, solubility, hemolysis, and non-fouling (Table 4), we selected three non non-dominated binders with predicted affinities greater than 7.0 for docking experiments. For these Pareto- careto-optimal peptides, we indeed observed strong binding affinities for both GFAP and RBXI post-docking, indicating their unique potential for multi-targer interaction (Figure 11). GFAP an intermediate filament protein (Eng, 1985) and
19

---

De Novo Generation of Therapeutic Peptides with Mult-Objectiv--Cuidee Discrete Diffusion
Average Binding Affinity to GLAST Over Iterations
Average Binding Affinity Scores TIR Over Iterations
Average Solubility Score Over Iterations
Average Binding Affinit Score
Average Binding
Average
Iteration of MCTS
Iteration of MCTS
Iteration of MCTS
D
Average Hemolysis
Iterations
E
Average Fouling Iterations
Average Non- FouliFouiigg Score
plal AMAr
Average Hemplyii
Iteration MCTS
Iteration MCTS
Figure 10. Property Scores Over Iteration for Dual-Targe Conditioning on TfR and GLAST. (A) Plot of average predicted binding affinity score to GLAST over iterations. (B) Plot average predicted binding affinity score to TIR over iterations. (C. D, E) Plot of average predicted solubility, hemolysis, and non-fouling scores over iterations.
thus forms unique rod-like structure with a head domain and a tail domain. The docking positions of all three candidates were along the rod domain, binding in the gap between adjacent rods in the filament. Contrarily, docked candidates to RBX1 consistently bound close to its interaction site of Cullin, rather than at the Skp2 F-box adaptor site (Figure 11), indicating that further motif conditioning, as done with recent peptide design language models (Chen et al., 2024), would benefit PepTune's ' clinical potential.
B. Extended Background
B.1. Continuous-Timm Discrete Diffusson
Discrete diffusion models (Austin al., 2021) are subset of diffusion models where the forward corruption and reverse denoising processes operate in the discrete latent space via categorical probability distributions for discrete variables.
We denote token in a sequence from the dataset one-hot vector x( {0, 1 The discrete-time forward diffusion process involves applying categorical noise to Xo at varying degrees based on noise schedule a (tt) for total of T time steps ranging from no noise at 0 to maximum noise at & = 1. To clearly distinguish each step, we denote the with transition in the forward pass as the transition from (n) to t(n2. where snn) m and t(n) = 7. The marginal noise that transforms the sequence (z(n) at time s(n) to a progressively noisier sequence Ze(n) at the next time step t(n) = s(n) + +
is given by VI x [V] marginal transition matrix Q = (t) + (1 - (tt)) where the (i, entry denotes the probability of transitioning from token to token j at each position in the sequence.
Therefore, the marginal categorical distribution of Z(()) in the discrete-time forward-pass diffusion process can derived as
((2e()) Be(n)) = Cat (((()) (o(n))
Cat(Ze(()) o (((n))))()) Zz(n) + +(1 (((()))))
(20)
where (t(n)) the marginal probability of a single position in the sequence remaining the same token during the transition s(n) + t(n) and (1 (t(n))) is the marginal probability of transitioning different token based on the token probability distribution & E A IV. For simplicity, we denote (t))) = (n))
20

---

De Novo Generation of Therapeutic Peptides with Mult-Objectiv--Cuide Discrete Diffusion
Binding to GFAP
Binding RBX1 E3
Booking Blnder
Dooking
Dodidng
Dasd Bidder
king Stowe
Docking Sconc 8.8
Daal Beder4
Docking Scone &,7
Figure PepTTune-gent generrrde peptides with dual GFAP and RBXI affinity. Full protein binding location and close-up binding position for (A) dual binder 3, (B) dual binder 2, and (C) dual binder 4 with GFAP (left) and RBXI (right). Polar contacts within 3.5 are annotated, shared polar contacts between binders are highlighted.
The cumulative transition from time 0 time (()) denoted the product of the marginals Q -
Im Qujs which is the product of marginal transitions (n) t(n) for ranging from 0 to applied the clean input sequence Xg-
f Q:
[ I+ 1 II n==
[[a-o())) 1x
a(m))
(21)
==0
For the continuous-time forward pass diffusion process where T + oo and + O, we can take the limit as -> oo to derive an expression for the continuous-tin transition probability, dy.
lim II
(n)) lim exp
log II (1 -(n)) n=0
T-000
lim exp T-x00
log(1 a(n))
lim exp T0000
(n)
(log(1 x) se x for small x))
exp
(n))d
nnm
((t)
exp
(s)ds
(22)
We have shown that the continuous-time forward transition probability from 0 ((l) is az exp exp(-a(t)) where ott) _[()) (s)ds. Letting t(t), define the continuous-time
cumulative transition matrix Q, at the limit where T + oc and the continuous time distribution g(z,/x,) as
Q = a.I + a=
(23)
q(z (x)) Cat(zz: Qx
Cat(zr axxo (1 (T)
(24)
21

---

De Novo Generation of Therapeutic Peptides with Multi-Objectvv--Cuddee Discrete Diffusion
It follows that the marginal transition Qait is the cumulative transition Q₁ divided by all previous transition probabilities, denoted as Q. -o Qe/r- 50 ata a,"
Following Austin et al. (Austin et al., 2021) and substituting the marginal and cumulative probability distributions, we derive the true reverse transition from t + conditioned on a clean sequence Xo as
g(z, Xx), (2))))))))))))))
g(z,|Z(,xo)
(Zz Xx)
Q:
Q Xu
Cat
Zsi
Z Q2 Xo
latts Ze + (1 (1]s)]1 2 [a,x0 + (1 - ag)=]
Cat
Zsi
(25)
az Xo + (1 0: )z π
where the numerator is the element-wise product of -ilddmmenil vectors representing the marginal probability distribution of sampling 2e given 2, and the cumulative probability distribution for z, from the original clean sequence Xo- The denominator a scalar probability of the specific token z, being drawn from the noisy probability distribution at time t.
B.2. Continuous-Tine Negative Evidence Lower Bound (NELBO)
The objective of denoising diffusion probabilistic models (DDPMs) (Hol & Pekny, 2015) is to iteratively sample slightly less noisy intermediate sequences z, until obtaining a clean sequence x that has a high probability of being drawn from the data distribution p(xo). To train model that accurately samples from p(xp), we maximize the Evidence Lower Bound (ELBO) of log (xx) which measures how accurately the model parameterized by 0 generates true samples Xy given corrupted sequence zr by iterative sampling from the reverse posterior po |o.).(z)). The ELBO is maximized when pe (Xo) = 1 and log(po(Xxo) 0 for all sequences X6 in the dataset, which supports the objective of accurately generating sequences from the data distribution. To convert this into loss minimization objective, we define the loss function the negative ELBO (NELBO). First, we compute log (Xa) by integrating over the joint probability of all possible paths of intermediate states from the noisy state 27 at = to the clean sample To at = 0, denoted by (x Since our goal is to reverse the forward masking of the clean sample zo from all time steps, we introduce an encoder term q (xo) denoting the combined distribution of obtaining any noisy sequence between times
T from the clean sequence XD.
logpo (x0) = log / pe po(zoT)
log /
pa (Z(:T))
((Z):((x))
dzi:T
=
Po (P((z))))
log
(ig(
(26)
9(21:T (x))
where Zgrr includes Xo-
Using Jenson's inequality, we move the logarithm inside the expectation and reverse the sign to get the NELBO for log pa (xo)). We split the terms associated with the forward noising process g(Z1.r (xo) and the reverse denoising model pe(Zo (Zo:T) into reconstruction term, the prior term, and the intermediate diffusion term.
log pe (Z(:T)
LNLLB Eggtarr (xa)
(Z11. (xo)
(x (1 (e [- po (z(z.(m) ((()))
(x))
log
a(Z((T) ( IITh1 q ((Re(m)) (2a(m))
Pe po(Z)(ri)
(Ze(n) 2((x))
log po (Xo (x(1)) - llog
log
(Z((T) Za(r))
4 ((2(n)) Ze(n))
po(2)(T))
log (Zz(m) (z(2))
log pe (Xo (x(1))
-log g(Z((T) (x(T))
g(2((()) (s(n))
22

---

De Novo Generation of Therapeutic Peptides with Multi-Ohjectve- Discrete Diffusion
Pe (a(2)(2))
B (xo)
log (o(olxx)))))) (xolze(1))
+ E (x
log
((Z((T) (x(T))
po (o(()())) ((((n))
R (
log
((Zz(n)) (((n))
18 (ante Macry y (x) log po (o(
E (x xx)
log pe (x0 k(()))
((z((2) (()(r))(as(r))
reconstruction loss
prior loss
(Z( Ze(n))
Bu(xstn) Ra (xx)
log
(27)
ql ((Ze(n) (e((n))
diffusion
Now, we can take the limit for each of the loss terms as T + oo to derive the continuous-time MDLM objective.
Reconstruction Loss Crecoust The reconstruction loss evaluates the final step of the reverse diffusion process that denoises the sequence time ((1) the clean sequence time 0. Since ((0) 7. the distribution that the sequence 2((1) drawn from in the forward pass diffusion is given by
p(2((1) (xo) =
+ az(1) (Xo))
(28)
Since have oz (xx) = 1 1. for Xg b and
for xo * b, write
(4(11) (x0))x + (1 (xo1)(Xo))
(1
T
1
X +
T m
Xo =
(29)
Xo
7m
Xo X
In the limit T oo, both cases converge to Xo. so have 2((1) Cat(Z Xo) and 2y(1) Xo. Since ((((1) (xo) X0 in the forward pass, by parameterizing the reverse posterior to copy-over unmasked tokens, we get po(xo (Z((1)) Xo. Therefore, the reconstruction loss reduces to O.
E Ey(Meccy [xo) log (xo (xolz((1)
log po (xo (xx)
Prior Loss Sprior
The prior loss measures the first reverse transition from the fully masked sequence Zi(r) to a slightly
unmasked sequence (.))-
log p(Z((T)
log p(Ze(T))
Eg(neryy 180)
Eg(onuery (xo) (a(
((Z((T)) (Z,(()))
q (Z)(r) ((()))
KL Divergence
-ggg (xo) KL (oloetrrooeeccp)lllllloo
(30)
Since t(T) 1, we have (xo) = 0. Therefore, we derive
((xTT)xx0 Cat(zur) (x(r)(x0)xx + (1 - (u- m)
= =Cat(zur)) Oxg + 0)m)
= -Cat(zurr): m)
(31)
Since all sequences are completely masked at time T. it follows that the marginal distribution ((Zer)l).(T))
(at(zury):)) m) and the prior distribution po(2)())- = Cat Cat(zury; m), so the KI. divergence reduces 0.
23

---

De Novo Generation of Therapeutic Peptides with Multi-Ohjectve-Guidee Discrete Diffusion
Diffusion Loss LT The diffusion loss measures the consistency of each predicted reverse transition with the forward marginal transition for all T time steps.
T-
Po (Za(n) Ze(())
(n (ain) Ba(++)) (80)
log
(32)
g( g(2((n) Bs(n))
T 1
Po (1)((())) (2e((s)) ((Zz(m)) (x(n))
B Ba ( Poleeenssaacc)
log
KL divergence
KL (((Z)()))stn))lle)) k(cwo)
Since the objective is to accurate predict 2a(n) given ((()), we cannot condition on the term (z(n)- Instead, we can condition ())(())Z(())) on Xo and use Bayes' theorem to derive
(ZZ(m))Za(n)) xo) (((s(n) [[(()): x xx)g(<((m))xx)) q (Z).(n) (x))
Rearranging the terms we get an expression for the true reverse transition o (m)) (e(n)) xo) conditioned on the clean data Xo. By minimizing the KL divergence between the learned reverse transition pa (Z)(m) ((()) and ((Ze(1)) (2(())) Xo). we can rewrite the diffusion loss as
CT
Σ
Eat (sco)
[e(n)) xo) KL ((2))))))<(())
X0))po(()))))) B(()))
(33)
In Appendix G.3, we derive the bond-dependent continuous-time NELBO loss from its general form above.
B.3. Guided Diffusion Models
Guided diffusion aims to sample from the data distribution conditioned some property y. g(xp. 3). such that ((y/xx)
is maximized. Therefore, the marginal reverse transition aims sample z, from distribution q(z, Zze y) conditioned on
the current sequence z, and property y. Using Bayes" theorem, we can decompose the guided conditional distribution as
q3y/22, 2()((z)(z))
q Ze, v)
(34)
((vl2z)
There are two strategies to generate samples from this conditional distribution: classifier-fre and classifier-based guidance.
Classifier-Free Guidance Classifier-free guidance strategies aim to model the conditional distribution q(z, [zg y) by directly training the diffusion model on subset of the unconditional data with property y. such that after training, the model samples from learned distribution po (z 2e, y). However, classifier-free guidance fails at tasks where high-quality annotated data is scarce, including peptide sequences. Furthermore, this strategy cannot scale to multiple objectives, which would require data conditioned on more than one property.
Classifie--Based Guidance Classifier-based guidance trains an unconditional diffusion model po (z, |2,) and classifier model (y(2)) with learned parameters & that generates score measuring the probability that the intermediate sequence 2, has property y. By Bayes' theorem, we can model the conditional distribution as
Pe. Iz,, 3) Po(ylzz,)pozz Iz.)
(35)
po(y)zz)
Since the model parameters implicitly learn the normalized distribution, we can drop the po(y)u)) term. Then, at each iteration, we update the parameters 0, e in the direction of the gradient of log (z, |z,, y) obtained as the sum of the gradients of the unconditional distribution log pa (z.lz.) and classifier probability po(2)z.) with respect the sampled sequence Zg-
v, log (Z. v) Vz, logzo(ulz.) + Vz, log log.e(z.)z.)
(36)
24

---

De Novo Generation of Therapeutic Peptides with Multi-Objectve--uudeee Discrete Diffusion
After joint training with the classifier and unconditional data distribution, we can sample from the learned conditional distribution p0.0(2,/24) y)-
Unlike classifier-fns free guidance, classifier-based guidance does not require training generative model from conditioned dataset. However, the gradient-based strategy for classifier- based guidance is not directly applicable to discrete state spaces due to the lack of a defined gradient. To mimic gradient- -aased updates to each sampling step, Gruver et al. (Gruver et al., 2023) leveraged iterative guidance steps on continuous latent embeddings for each token before decoding back to a discrete sequence at each time step. However, projecting to and from the continuous and discrete spaces results in inconsistencies in the guided diffusion process, where optimized hidden embeddings do not always map to optimal tokens.
Guidance in the Discrete State Space To directly guide the diffusion objective in the discrete space, must compute the optimality of a single discrete reverse transition 2, against all other possible transitions to maximize the conditional probability ply|z,, zz). That is, we need to compute Equation (34) with the denominator expanded to represent all possible transitions from Zz.
Po Po pe (z. (o.))))o(zzzz) (z))
pa.o Zn (2,(Zz,3)
(37)
Ex; Po Po <2' (2,,22)
However, computing Po z for all the possible transitions from state Z, is computationally intractable. Previous work has bypassed this limitation by approximation. For continuous and differentiable classifier functions p(y|x) : 12 - R, we can approximate the denominator using the first-order Taylor expansion given by
log py/z. 21) = log (y)lz.) + (z. z1) V log
(38)
which approximates the likelihood of observing property y at the slightly denoised state 2, = 2 given the known pog-pobabbility of observing y for state z,. This eliminates the need to explicitly sample z, for all possible state transitions and compute log Po (y) z.. z.).
Digress (Vignac et al., 2022) has used this approximation strategy for guided discrete diffusion on categorical graph generation. Furthermore, Nisonoff et al. (Nisonoff et al., 2024) uses the Taylor-approximated conditional distribution log (b (z,) to adjust the unconditional transition rates (z, ze given the unconditional rates Rg (Zr, z.) for predictor- guidance of Continuous-Tiin Time Markov Chains (CTMCs) in the discrete state space.
log Pa (y) 2s. 2:)
Re zolv)= = Re(z,, 2.)
(39)
HogPo(ylze
where Re (Zz, zzly) is the predictor-ane adjuussed rate of transitioning from state z, to state Z,
However, this strategy fails to scale to multiple objective guidance since it would require computing the joint probability over K objectives po (y1, 1/.... ................................ .K (z,, zz) for some K > 1. If all properties are mutually independent, we can factorize the distribution and compute the estimated probability of each objective and take their product (0 U2...... UK [2,, z,) II I z., z:). For the majority of multi-objective tasks, including therapeutic peptide generation, independence across properties is not reasonable assumption, and computing the joint distribution is required. Moreover, for objectives that guide toward contradictory optimal rates or transitions, training model conditioned on these objectives could prevent the model from generating optimal sequences for either objective. Given these limitations, there remains gap for efficient classifier-based conditioning for discrete diffusion that is robust to multi-objective tasks, which we address in this work.
C. Data Curation and Tokenization
C.1. PepMDLM Training Data
To train the unconditional masked diffusion language model generator, we collected 11 million peptide SMILES consisting of 7451 sequences from the CycPeptMPDB database (Li et al., 2023a), 825,632 unique peptides from SmProt (Li et al., 2021), and approximately 10 million modified peptides generated from CycloPs (Duffy et al., 2011; Feller & Wilke, 2024), which consists of 90% canonical amino acids, 10% unnatural amino acids from SwissSidechain (Gfeller et al., 2012), 10% dextro-chiral alpha carbons, 20% N-methylated amine backbone atoms, and 10% PEGylated peptides. All possible cyclization conformations were attempted on the peptides generated with CycloPs. We used SELFIES (Krenn et al., 2020) to check the integrity of the SMILES sequences.
25

---

De Novo Generation of Therapeutic Peptides with Mult--Objectv--Cuidee Discrete Diffusion
We split our data by k-means clustering into 1000 groups of sequences with similar chemical properties based on their Morgan fingerprint (Rogers & Hahn, 2010), which is bit-vector representation of the full peptide sequence where each bit encodes feature relating to the SMILES atom types, connectivity, and bonding environment. The final dataset was 0.8 to 0.2 split based on the clusters, maintaining similar diversities of the SMILES strings, Since the degree of masking is evenly spread between / = 0 to / = within each training batch, grouping similar SMILES in the same batch ensures the model learns to reconstruct a diverse set of peptide SMILES from various degrees of masking.
C.2. Dynamic Batching
We applied dynamic batching to handle variable-length token sequences and increase computational efficiency. Inspired by ESM-2's dynamic batching technique (Lin et al., 2023), input SMILES are sorted by length to maximize the utility of GPU memory, The maximum tokens per GPU was set to 16,000.
C.3. SMILES Tokenization
To enable the novel generation of non-natural amino acids containing cyclizations and diverse backbone and side-chain modifications, we trained our generative diffusion model on Simplified Molecular- Line-Entry System (SMILES) (Weininger, 1988) representations of peptides. We experimented with several tokenization schemes that capture common motifs in the training data enhance the generation of valid peptide SMILES. We find that the SMILES Pair Encoding (SPE) tokenization scheme (Li et al., 2021) with the PeptideCLM (Feller & Wilke, 2024) vocabulary of 581 SMILES tokens and 5 special tokens with an average length of four characters per token, demonstrated superior performance, generating precise but valid peptides (Appendix H).
D. PepMDLM Implementation Details
D.1. Notation
Let X E {0, 1 represent the one-hot vector of token in sequence in the training data and Xg(2,, () vector of predicted token probabilities across the vocabulary V given the current state z, at time In most contexts, X will be used to denote single token, but when discussing the full sequence, x! is used to denote the token at position { in the sequence. Let T denote the total number of time steps in the discrete forward and reverse diffusion processes. In the context of all time steps, we expand
A be the
t(n) E (0, 1] when denoting a single time step in the forward and backward diffusion process with n [ T 1]. Let s(n) t(m) + denote the previous time step in the forward process.
Then, let (((n)) and Z denote the state of specific token at time ((n) and s(n) in the diffusion process, respectively. Let (xo) [R[ + R denote a function that takes the unmasked token Xo and outputs value in [0.1] representing the probability of remaining unmasked at time t in the forward diffusion process. Let b E R denote vector with ones at
indices of peptide bond tokens and zeroes at all remaining indices, and let xo b indicate that x is peptide- token.
D.2. Model Architecture
To predict the token probabilities at each reverse step x (z,, t), we trained RoFormer model (Su et al., 2021) that leverages rotary positional embeddings (RoPE) robust to varying input lengths and dependencies between tokens. The specific hyperparameters of our model are given below.
Table 5. Roformer Architecture Hyperparameters
Hyperparameter
PepTTnee
Input Dimension
581 (vocab size)
Hidden Dimension
768
Intermediate Dimension
3072
Number of Layers
8
Attention Heads
8
Max Positional Embeddings
1035
Hidden and Attention Dropout Probability
0.1
26

---

De Novo Generation of Therapeutic Peptides with Multi-Objectvv--Gudde Discrete Diffusion
Table Training and Validation Loss P. values are taken after convergence at epochs when training PepMDLM on Il million peptide SMILES with bond-dependent masking and invalid loss.
Model
Train Loss (4)
Train PPL ())
Val Loss (4)
Val PPL (L)
PcpMDLM
0.832
2.460
0.880
2.277
D.3. Unconditional Generation Results
Here, we provide additional tables and figures demonstrating the performance of our unconditional PepMDLM model. Table 6 shows the training and validation metrics after 8 epochs of training on 11 million peptide SMILES. Table 7 presents our benchmark results in comparison to the HELM-GPT model (Xu et al., 2024). We demonstrate that PepMDLM generates peptides with higher uniqueness, diversity, and lower similarity to their nearest neighbor (SNN) (details on metrics are provided in Appendix F.2). While peptide unconditional peptide validity lower, peptides generated by HELM- GPT are represented in HELM notation (Zhang et al., 2012), where each token corresponds is trained on SMILES tokens that decompose amino acids into smaller tokens that can be pieced together into valid amino acids during generation. While this enables us to represent greater diversity of non-natural amino acids, even a single token generated in the wrong position can result in an invalid peptide sequence, resulting in a lower validity rate. However, we note that our MCTS guidance strategy increases the validity rate to 100% due to its iterative unmasking process that rewarded on high-scoring and valid peptides.
amino acid. In contrast, PepMDLLM
In Figure 12, we show that the frequency of non-natural amino acids from SepMDLL-geeneraa peptides is similar the membrane permeability and binding datasets containing experimentally- validtated modified peptides that used to train property classifiers (Appendix B.1). In addition, show that 10% of unconditionally generated peptides from PepMDLM contain modifications that result in cyclicization (Figure 12). In total, we demonstrate PepMDLM's unique capability of generating diverse non-natural and cyclic peptides.
(Gfeller et al., 2012) present in
Table Benchmark of PepMDLM unconditional model against HELM-GPT. The best scores are bolded. HELM-GPP was trained on HELM notation, where each token is monomer encoding natural and modified residues. Since there existing peptide SMILES generative models, we chose HELM-GPT as the closest comparison. The validity is assessed differently, as all HELM sequences
correspond to valid peptide, while not all SMILES sequences can be decoded into peptide.
Model
Validity (t)
Uniqueness (1)
Diversity (t)
SNN (L)
HELM-GPT
0.839
0.913
0.595
0.975
PepMDLMM
0.450
1.000
0.705
0.513
27

---

De Novo Generation of Therapeutic Peptides with Mult-Ohjecciv--Cuidee Discrete Diffusion
Non-Natural Amino Acid Frequencies
Frequency Non-Natural Amino Acids Per Peptide
PepMDLM
Permeability Datase:
Binding Dataset
Permeability Data
Binding Data
PepMDLM
Mcan nAAs Per Peptide Cyclic Peptides (%)
2.215
2.150
2.940
0.467
0.027
0.100
Figure 12. PepMDLM generates cyelic and modified peptides. (Ahove) Distribution comparison of non-natural amino acid frequency for 100 unconditionally-genee generrtdd peptide SMILES with the peptide SMILES dataset of experimentally-validdeee peptides for membrane permeability (PAMPA) and binding affinity (Appendix B. (Bottom) peptide frequency non-natural amino acids (nAAs) and percentage of cyclic peptides in PepMDL.M-generade sequences and experimentally validated membrane- permeeble peptides.
E. Therapeutic Property Prediction for Peptide SMILES
While several classifiers exist for predicting properties of small-molecule SMILES sequences and amino-acid representations of peptides, there exists a gap in high-quality property models trained specifically peptide SMILES data. To fill this gap, we train regression models for target-binding affinity and cell membrane permeability and binary classification models for solubility, hemolysis, and non-fouling specifically on peptide SMILES data (Table 2).
E.1. Protein Target-Binding Prediction
To guide the generation of peptides with high binding affinity to given protein target, we trained a transformer-basedd model with cross multi-head attention layers that learn the joint latent space of ESM-2-650M (Lin et al., 2023) embeddings of the protein amino acid sequence and PeptideCLM (Feller & Wilke, 2024) embeddings the peptide SMILES sequence (Figure 13; Table 8).
We trained on 1806 sequences from the PepLand (Zhang et al., 2023) canonical and non-canonical binding datasets containing the protein-target sequence, peptide SMILES sequence, and the experimentally-valdaaaee Ka/ K; K /C50 binding affinity score. Given a peptide SMILES sequence and protein amino acid sequence, the model predicts score that indicates weak binding (< 6.0), medium binding (6.0 7.5), and high binding (2 7.5). After training for 50 epochs, our regression model achieved strong Spearman correlation coefficient of 0.869 on the training data and 0.633 on the held-out validation data.
E.2. Cell Membrane Permeability Prediction
For cell membrane permeability, we trained an XGBoost (Chen & Guestrin, 2016) boosted tree regression model on PeptideCLM (Feller & Wilke, 2024) embeddings which returns the predicted PAMPA lipophilicity score (log P) given a peptide SMILES sequence, where values -6.0 indicate strong permeability and values < 6.0 indicate weak permeability. The XGBoost regression parameters were optimized with 50 trials of OPTUNA search and are provided Table 10.
28

---

De Novo Generation of Therapeutic Peptides with Multi-Objeciv--Cuudee Discrete Diffusion
Table 8. Cross-Attention Model Architecture for Target-Binding Affinity Prediction
Layers
Protein Dimension
Peptide Dimension
Embedding Module
1280
768
Linear Layer
512
512
Layer Norm
512
512
Cross-Attention x3
Multi-Head Attention (h- 8)
512
512
Linear Layer
2048
2048
ReLU
2048
2048
Dropout
2048
2048
Linear Layer
512
512
Shared Prediction Head
Linear Layer
1024
ReLU
1024
Dropout
1024
Regression Head
1
Classification Head
3
Cross Multi-Head Attention x3
4
Peptide Attending Protain
Peptide SMILES
Predicted Affinit
Target Sequance
ochewet)
Sorne
Protein Attending Peptide
Figure 13. Architecture of binding affinity regression model. Embeddings for the target protein sequence are generated with ESM-2 and embeddings for the peptide SMILES are generated using PeptideCLM. Cross multi head attention layers combine the embeddings and predict a binding affinity score.
The dataset contains 34,853 experimentally validated peptide SMILES, consisting of 22,040 SMILES sequences obtained from the ChEMBL database (Mendez et al., 2018) and 7451 sequences from the CycPeptMPDB database (Li et al., 2023a). Data was randomly shuffled and split into 1.8/0.1/0.1 ratio for train, validation, and test. Our model achieved strong Spearman correlation coefficient of 0.998 the training dataset and 0.943 on the dataset (Figure 14, Table 9).
Table 9. Held-out validation performance of binding affinity and membrane permeability regression models trained on peptide SMILES. Spearman rank correlation and MSE were calculated on the 20 percent held-out validation set.
Metric
Binding Affinity
Membrane Permeability
Spearman Rank Correlation
0.633
0.943
MSE
0.566
0,088
29

---

De Novo Generation of Therapeutic Peptides with Multi-OOjeedive-Cuidee Discrete Diffusion
Predicted True Membrane Permeability
Predicted Membrane Permeability
A
Spearman Correlation: 0.943
Spearman Correlation. 0.998
Predicted Permeabilly (logP)
Predi ctod Permiabllity (logP)
True Permeability (logP)
True Permeablityy (ogP)
C
Predicted vs True Binding Affinity
D
Predicted vs True Bindlngg Affinity
Spearman Correlation:
Spearman Correlation:
hro dies
Predicted Binding Affimity (log acale)
Predicted Binding Affinity (log-sca
True Binding Affinity (logsscale)
Truc Binding Affinity (log scale)
Figure 14. Correlation plots for binding affinity and membrane permeability classifiers. Plot of true permeability (logP) on the r-axis and predicted permeability on y-axis for the (A) validation set and (B) training set. Plot true binding affinity (log-scale) on the a-axis and predicted permeability the y-axis for the (C) validation set and (D) training set.
E.3. Solubility and Toxicity Prediction
For solubility, hemolysis, and non-fouling, we collected binary data from the PepLand and PeptideBERT datasets (Zhang et al., 2023; Guntuboina et al., 2023), where indicates the positive class and 0 indicates the negative class. Since increased solubility improves drug-loading, we seek to maximize the probability of the positive class. Since positive hemolysis indicates that the peptide sequence induces destruction of red blood cells, seek minimize the probability of the positive class, maximize the inverse. Positive non-fouling indicates lower off-target binding, non-fouling.
seek to maximize positive
We leveraged PeptideCLM embedding representations of the SMILES tokens and trained XGBoost models for binary classification. The optimal thresholds for the positive class were determined to be 0.500 for solubility, 0.800 for hemolysis (non-hemolysis), and 0.450 for non-fouling. The XGBoost binary classification parameters were optimized with 50 trials of OPTUNA search and are provided in Table 10. We achieved strong Spearman correlations across all three tasks, showing improvements against the state-of oftte-of-theart the-art PeptideBERT (Guntuboina et al., 2023) baseline model (Table 2).
F. Evaluation
F.I. Peptide Validity Filter
Among the sequential representations of peptides, including amino acid sequences, HELM (Zhang et al., 2012), and SMILES (Weininger, 1988), SMILES is the most intricate representation of peptide sequences. Although this enables the representation of non-natural amino acids, diverse side chains, backbone modifications, and cyclic peptides, also means
30

---

De Novo Generation of Therapeutic Peptides with Multi-Objectvv-CCudeee Discrete Diffusion
Table 10. XGooo Hyperparameters for Classification and Regression. Classification hyperparameters were used the solubility. hemolysis, and non-fouling binary classification models (left). Regression hyperparameters were used for the membrane permeability regression model (right).
Classification Hyperparameters
Regression Hyperparameters
Hyperparameter
Value/Range
Hyperparameter
Value/Range
Objective
binary: logistio
Objective
reg: squarederro
Lambda
[1e-8, 10.0] [1e-8, 10.0] 1.01
Lambda
[0.1, 10.0] (log scale)
Alpha
Alpha
[0.0, (log scale)
Colsample by Tree
Gamma
[0,
Subsample
0011, 1.0]
Colsample by Tree
[0.5,
Learning Rate
0.3
Subsample
[.96
Max Depth
2, 30]
Learning Rate
[1e-5,0.1]
Min Child Weight
[1, 20]
Max Depth
[.30]
Tree Method
hist
Min Child Weight
[1,20]
Tree Method
hist
Scale Pos Weight
[0.5, 10.0] (log scale)
that the vast majority of SMILES strings not synthesizable peptides. Therefore, we devised algorithm that determines whether a SMILES string is valid peptide, characterized by peptide bonds and central carbon atoms. The filter first checks if the SMILES sequence is valid molecule using RDKit (RDKit, online).
Then, we use regular expressions to detect bond patterns for peptide bonds, N-methylated peptide bonds, reversed peptide bonds, and ester bonds, along the sequence to split the sequence into several segments with a bond before and after each segment. The filter checks each segment for chemical modifications based on their bond type, including N-methylation (N-Me) and O-linked glycosylation. The remaining segment is matched to the corresponding natural non-natural amino acid side chains (Algorithm 8). Our filter is capable of detecting library of over 200 nAAs from SwissSidechain (Gfeller et al., 2012) and can classify peptide SMILES as cyclic non-cyclic. The tool is freely available on HuggingFace: ht https: huggingface..eeeccccooo/oo/oaaee co/ httpre/ehhhaatceeeeeeaaaaLLLLEBS
F2. Metrics
To evaluate the generation quality of our unconditional MDLM, PepMDLM, and our MCTS-guided MDLM, PepTTune we leverage the Moses metrics, including validity, uniqueness, diversity, similarity to nearest neighbor (SNN), and novelty (Polykovskiy et al., 2020).
Validity defined as the fraction peptide SMILES that pass our SMILES2PEPTIDE filter (Algorithm 8), indicating that it translates to a synthesizable peptide.
Uniqueness defined as the fraction of mutually distinct peptide SMILES.
Diversity is defined one minus the average Tanimoto similarity between the Morgan fingerprints of every pair of generated sequences, which measures the similarity in structure across generated peptides.
f(x:) f(x;) f f(xx)) + [((x)) f(x;) f(x,)
Diversity
(40)
(Ngreenne)
where f(x,) and f(x,) are the 3048-dimensional Morgan fingerprint with radius for a pair of generated sequences x; and Xz.
31

---

De Novo Generation of Therapeutic Peptides with Multi-Objectve-Cuudee Discrete Diffusion
Similarity to Nearest Neighbor (SNN) defined as the maximum Tanimoto similarity between generated sequence x; with sequence in the dataset Xj-
f(xx) f(X,)
SNN
max JEP
(41)
[f(x:)] + - f(x;)
Randomness
is defined as the Shannon Entropy (Lin, 1991) on tokenized sequences given as:
E E= Σ p. loga (p:)
(42)
where is the probability of i-th unique token divided by the total number of tokens L in the sequence.
KL-Divergence is defined as the divergence between the token distribution in the generated peptide SMILES p; and the token distribution in the training data.
T pi logg (t)
if q > 0
KL(PI
= Σ
(43)
iEV p. logg To
if
= 0
where p; the probability of token i the training data, and qi is the probability of token in the generated data.
Due to the limit of memory and CPU time required load all the training dataset of 11 million peptide SMILES, chose to sample subset of 1000 batches randomly (~100kk sequences) for novelty and SNN caleulation. To assess the novelty of generated sequences, we employed Shannon entropy (Lin, 1991) to quantify the SMILES token randomness between 100 PepTune-generated and 100 DepMDLM--ennraadee sequences and the same randomly sampled 1000 subsets from the training set. Then, Kullback-Leibler (KL) divergence was used to evaluate divergence across token distributions from the generated peptides compared to the training data.
F.3. Docking
For valid generated peptide SMILES with non-dominated scores across objectives, we used Autodock Vina (Eberhardt et al., 2021) (v 1.1.2) for in silico docking of the peptide binders to their target proteins (Appendix 11) to confirm binding affinity. Targets were preprocessed with MGITools (Morris et al., 2009) (v 1.5.7) and the conformations of the SMILES were optimized by ETKDG from RDKit (Eberhardt al., 2021; Wang et al., 2020). The final results were visualized in PyMol (Schrödinger, LLC, 2015) (v 3.1), where the residues in the protein targets with polar contacts the peptide binder with distances closer than 3.5 À are annotated.
Table PDB structures target proteins used docking.
Protein
PDB
GFAP
6A9P
TYR
3KAS
GLP-IR
3CST
AMHR2
7L.01
GLAST
5LM4
NCAMI
2HAZ
RBXI
ILDJ
G. Theoretical Details
G.1. Bond-Dependent Masking Schedule
From Equation (22), we define the continuous- time forward masking probability 1- ag at time with a = exp(-o(tt).
where [0, 1] + is the cumulative discrete-time masking schedule. Following Lou et (Lou et al., 2024), we apply
32

---

De Novo Generation of Therapeutic Peptides with Multi-Objecive--uidde Discrete Diffusion
State- Peate-Dependent Masking Schedules
B
Seate-Dependent Weighted Negative-Log Loss
10
1.0
w-1
w=2
0.8
W-4
w=4
0.6
W
0.4
0.2
0.0
0.2
0.4
0.0 0.2
0.4
0.6
0.8
1.0
Time Step t
Time Step
Figure Plots of bond-dependent masking schedules. (A) The probability of remaining unmasked during the continuous-time forward diffusion process over time given different values of w as the exponent of the masking schedule a, = 1 1 We use for non-peptide bond tokens and w = for peptide bond tokens, resulting in slower masking of peptide-bond tokens. (B) The weight of the negative log-loss for different exponents in the pogloolylaiaal masking schedule. The weight of the loss is higher for larger in earlier time steps, which results in higher penalty for inaccurate predictions of peptide bond tokens compared to other tokens.
log-line linea masking schedule (t) log(1 - t) for the forward diffusion process, which has been shown to result in the lowest variance in the NELBO loss (Sahoo et al., 2024). Therefore, the continuous-time probability of remaining unmasked at time is equal to ar = exp (- log(1 t)) = 1 t and the weight that scales the negative log loss (NLL) is given by 7 by our derivation in Appendix G.3.
For peptide-bond tokens, we alter the masking schedule such that peptide-bonds are masked at a slower rate at earlier time steps by defining log-polynn log-polynomial masking schedule (t) = - log(1 t"), for some constant exponent w > Note that when w 1, the log-polynomial schedule reduces the log-linear schedule. Therefore, the probability of remaining unmasked becomes a₁ (- (- log(1 - (t))) = =1- t" - and the weight that scales the negative log loss (NLL) is given by w by our derivation in Appendix G.3.
Since E (0, 1]. the probability that a peptide-bond token remains unmasked at time is equal to a; = - 1", which larger than the log-linear schedule for w 1. Conversely, the probability that a peptide-bond token is masked before t a = t", which is smaller than the log-linear schedule for w > 1. As t + 1, a - for both the log-linear and polynoolll time schedules, which means that both peptide-bond and non- peptide bond tokens will have a high probability of being masked in later times in the forward pass diffusion process.
The NLL of the peptide-bond tokens is weighted more heavily than non-peptide bond tokens for close to 1. As t + 0, the NLL weight approaches 3o for all tokens. This biases the reverse diffusion process to unmask peptide bond tokens earlier since it was trained to minimize the loss associated with each unmasking step. As - 0, the large NILL weight ensures that the final unmasking steps during the reverse diffusion process result in an unmasked sequence that lies within the space of valid peptide SMILES.
G.2. Derivation of Bond- SenddDependent Reverse Posterior
Proposition 2.1 The reverse posterior defining the probability distribution of the token z, at time 5 = - At given the token 2, at time with our dependett forward masking schedule defined as
((
= b t xn) Xo + (( +) b bb m
2,
(2,,) 21, xo)
(44)
Z + m
When the clean token is peptide bond token (i.e. Xo = b), the transition distribution for masked token z, m reduces to g(z, 12, m, X b) - Xo (:") m. When the clean token is not peptide bond token (i.e. xo A b), the transition distribution for masked token 2, = m reduces to q(z, i4 m, Xo * b) (1 :) x + ( m. If the token
33

---

De Novo Generation of Therapeutic Peptides with Multi-Objectvv--Cuieee Discrete Diffusion
already unmasked, it remains unmasked at the same token with probability 1.
Proof. For single token, the bond-dependent forward difffusion process is defined by the probability distribution a(z, xo)
which transforms the clean inputs to sequences with varying degrees of masking based on probability distribution a, (Xo). We define ay (Xo) RM + R as function that takes the clean token encoding and outputs the probability of remaining unmasked at time depending on whether Xy encodes peptide bond token.
q (x) = Cat(zı; (xx(x))) + (1 (x))))
(45)
Then, the marginal forward transition from time s(n) + ((12) is defined as
Cre (xo) (x(()))
Xp + (1-
ar ax(xo) Or (Xo)
((z(()) [s(n)) = Cat
Ze(n)i
m
(46)
In this work, we classify each token into one of two states: peptide-bond tokens and non-peptin non-peptide-bond tokens, which represent amino acid side chains and modifications. We define a function that generates mask with values of 1 indicating tokens containing contained within peptide bond, and 0 otherwise (Algorithm 7). Let b R denote vector with (f)
ones at indices of peptide-bond tokens. For derivation purposes, we let b bxx 1 and x a peptide bond token. Note that is defined differently depending on the context of the token in the full sequence, which handled by the BONDMASK function. Then, have or (Xn) (1 b x²xx) (1 t) b xo (1 (*)) equivalently can write
when a token at position !
1 1 1 b -
or (xo
(47)
Xo 7
By Bayes' rule, the general state-independent form of the true reverse posterior is given by
2g Ze, xo) = a(2, 1z (x,)((z(xx))
q Zz x0)
ac
2 ze +
(1- 01 m Zu [asx] zz + (1 - a.) m Iz]]
(48)
[acx)] Ze + (1- az) m Zi
With bond-dependent masking, the value of are (Xo) and a, (Xo)) depend on the state of xo, so the bond-dependent reverse posterior becomes
[( (2,))
(ee(zz)
2 24 (1 (1- ac(x.) m 2z (x0)xx 22 (1 as, (xo)) m "z
q(2, [2,, xo)
(49)
aa (x + (1 - (xx)) m m ze
When z, = Xy, the true reverse posterior simplifies to
(.(x..) (2.)
z xo + (1 (x,) a. (2,) m xo] Jas (xx())xx)) xo + (1 - (xo)) m T xo]
(Z2//Z Xo) =
(x)( Xo + (1 or (xn)) m xxo]
or((z)) (2.)))
z Xg
[as (xu)]
(50)
as (xx)
When 2, # No. 2,.x0 X 0 so q2, * xol2e = N0, xo) = O. When z, = Xo, we have
a.(xx) (x))
x xo
(xx))
( xoler Xx, Xo)
(xx)
(x((xo) )
(xx(Xx)
as (xo)
cx
34

---

De Novo Generation of Therapeutic Peptides with Multi-Objectve-Cuudee Discrete Diffusion
which means that 2 remains unchanged after unmasking. This supports the carry-over unmasking scheme, which explicitly sets the probability of changing an unmasked token equal -00.
In the forward diffusion process, token either remains unchanged or is masked, so the only other case we need to consider is m. Since the masking schedule differs only when the ground truth token is peptide bond token, or Xo = b, we can consider two cases: first, when Xp = and second, when X * b.
Case 1. Consider the case when X = b or the ground truth token Xo is a peptide-bond token. From our modified masking schedule, we have (b) = 1 tt. Therefore, we can write the probability distribution for unmasking a peptide-bond token
Ae(Ra] (z.
zg m (1-
(.(2a) (z.)
m T m
[a.(b)bb z. + (1 ass (.s(b)) m n.
(ZzlZz m, Xo b)
[ar(b)b m + -1-a8 aaa(bab)))))) m m
(e(Za) az(2,)
zg m + (1 (z.)))
(.(2.)
[a.(bbb z, + (1 (b)) m z,
(51)
(1-a((b))
The probability of transitioning from masked state to peptide-bond token is simplified to
(b) a. (b)
b m + (1- cx(bb) [b.(b] b+ (1 a, (b)) m bb]
(b)
gz. = -blz;- -m, Xo- = b)
(1 04 (b))
1-p
(1 - (w-
I-g
))))
3 11
(1 g²)
t
500
t
8"
(52)
1"
The probability of remaining in masked state is
(m)
m m (1- a (m)
[b. m (1 a, (b()) m m]
(m)
a, (m)
q(z. m/z: zc m, x0 =b)
(1 (b))
(m)
1 (m)
} (1-a,(b)
am(m)
mm(m)
(1 an (b))
(b))
1 ox (m)
1 (1 5w)
1 (1 1n)
8
(53)
tw
which aligns with the constraint that 2, E Xo) in the forward diffusion process.
Case 2: Consider the case when Xo * b or the ground truth token Xo is not a peptide-bond token. From the baseline log-linear masking schedule, we have ai X 1 t. Therefore, we can write the probability distribution for unmasking a peptide-bond token
(z.) (2,)
(x.)))) m m [x=(x))x] m + (1 as (xn)) m T m (z,)
2 m +
g(2, Iz, m. X0 # b)
(xx(Xx) (x)) x m + (1 (x m mmm
(z,) (2,)
z m +
(2.) (xx m + (1 axo))]
(54)
Or (Xo) )x] m (1 are (xx))]
35

---

De Novo Generation of Therapeutic Peptides with Multi-hjecciv--Cuidee Discrete Diffusion
With similar steps Case 1, the probability of transitioning from a masked state to non-peptide-bond token is given by
<<(Xx) ax(xa)
ai (xx))
gzz = Xo Zz = m. Xo + b)
(1 O2.y (x(Xo))
1-1
(1 - (1 - s))
1-8
(1 (1 (1t)) t))
t-8
t
(55)
follows that the probability of remaining masked state in the reverse process
glz. m 2 mo b)
(56)
This demonstrates that the probability of remaining in masked state when Xo b is smaller than when x # b, since taking the exponent of fraction results in smaller value. So we have f for w > 1.
Combining Equations (53) and (56) we get the following distribution for the case when z m and z, = m
q(z = m zz = m. xp)
=
b
X0 +
t
g
5 xo
T
b
+ -1
t
b.
b + 7
Xo
(57)
Similarly, combining (52) and (55) get the following distribution for the case when and 2, o m or equivalently
2, Xg.
g(z, Xo Zz = m, X²)
$
" +
T
xo
1 8) 7)
t
1
8
b
{1) X0
l 1
b
xo
(58)
Now, we can write the true reverse posterior as
(( b+ Xo + (( -) b (1,, m
2, = m
((2, ke, xo)
(59)
24
Therefore, we get the following expression for the parameterized reverse posterior
(( b + ()),,)) xo (o(2),
(( +) b 11, 1))
2,
po (2:) =
(60)
Zt
Z: +
G.3. Derivation of Bond-Dependent NELBO Loss
Proposition 2.2 for all non-peptide bond tokens that follow log-linear masking schedule and the sum of the NLLs for all peptide bond tokens that follow a log-polynomial schedule.
The bond-dependent continuous-time NELBO decomposes into the sum of the negative log-losses (NLLs)
L
NELBO = Eg, [E,q(22(10))
20
Σ
W
log(x) ,x0 (Zt, 1)) - Σ
(t)
(6)
log(xi (0) .x00 (2), 1))
(61)
Exx
(2)
txx 10
zb
36

---

De Novo Generation of Therapeutic Peptides with Multi-Objectve--uudeee Discrete Diffusion
Proof. The diffusion objective in its general form is given by
LNLLBO
Σ
B Raccn (xa)
KL. ((Za(t)) Re(n)) (e(i)) (o))po)(s))) (a((m)))
n=1
B f.3....... fr...... Be(ar
T. KL ( Ze, xo) po(zolzz))
(62)
First, we will derive an expression for the cond-depended diveeeege, which measures the difference between the learned reverse posterior q( (2,) (2,, (o t)) and the true reverse posterior g(2,) 22, xo) conditioned the training distribution Xn-
(L(g(Z, Zz.
(z:))
(Z,,/zz = xo) log gZz, /2: = m, Xo)
Σ =e
po 2 m)
g(2,|2, |2 m.) log g(2, 1r
m, X0)
po 1, 12 m
x, m
{(a = x 2 = m, x0)
=2
Xol2: = m, xo) log
p6 (z, = Xy | m)
(z. = m lz = m. X
+ g((z
m zt
x0) log
po(Zz m
[z m)
T
b +
1 1- 1
Xy log
((
por
b + 1
x0
((
=") b + 1 x2(Zi,t)
st
b + 11) X
+
b
+ t1
Xo log
(63)
fw
b + 11)
xo (z(.1) 1)
In the case where the true token Xy b, we can simplify to
+) x Xn 2) x xo 1)
(L(g(Za/zz, (o) (2)) =
7
8
1
7
8
log
5
+
s
1
log (xf T
Xo (Z),t)
t
s"
log(xo, xo(ze, t))
(64)
1
Substituting t 1. T> we can simplify s to
5"
T
1
w0
1
tT
1
10
]
tT
1
((1 + x)
+ wr o(x
tT
T
wt w
t
(65)
T
T
where o (7x) denotes higher order terms that grow slower than T2-
37

---

De Novo Generation of Therapeutic Peptides with Multi-Objecive--uidee Discrete Diffusion
Now, we can write
1 we
1t
+ 1 (7t))
(L(g(2,) Zz, Xo)lpp
log log x (z 1))
t"
t 0 (+t)
7
log x (Z( t))
W
1
log(xo, xo (z)
tT
T
(66)
In the where the true token Xn A b. simplify
(1 {) x xo
(1- t
(L, xo))oo
log (1 2) x5J xxxx0(e xo (ze, t)
1 8) log (xn) xx (2) t))
t
log(xo. xA (zc, t))
(67)
Similarly, substituting 1 7. we have
1
+) t
Kl.(g(2./2z, x0)lpo((.12)))
log(xo: xo(2),
log(Xo, xo ())
(68)
IT
Now, we can combine the two cases using the indicator functions [ b] that evaluates to when Xo b and 0 otherwise and [xx 전 b] that evaluates to 1 when Xo o b and otherwise. Since this definition of KL divergence is only applicable when = m. we have
KL (L(((2, Re, x (p)(l1((2))21))
1(x0 = b]
w
-
(
(
log log(Xo. xo(21, (x,, t)) 1(x0 * b) tT
log(xu, xo(Zi,
(69)
tT
Substituting this back into the equation for the discrete-time diffusion loss, we get
CNELBO -EuE( 2 J.......... Beg(rr T. -KLL q(zz Ize, (o))(po((zzzzz)
-EEee 4.7 .......) 1E xoo) 1
1(x00 bl
log(xo, x (z t))
IT
T
1[x0 # b) ET
log(xo, (z((z(t)) t)}
wT
-Erer 3. ..... Ey(ze xx0)
1 1|x00 = b]
To
log (z t))
tT
72
T
1(x0 + b)
log(xu: X (Ze, b)}
IT
1
= =E f. .......) Ey moo
1/20 = bl
To
log(xo, xo (z. t))
72
-1/x0 * b) log(Xo,x0 xo t)}
(70)
38

---

De Novo Generation of Therapeutic Peptides with Multi-Ohjectie--Guidee Discrete Diffusion
Finally, taking the limit as T xc, the higher-order term lim To (++)
0 and we get
NELBO
Jim LNFLBO T-000
1(x0 =bj (24 To
-/] T2 ) log(xo, (0 1))
lim BLEl 1..5 f....... B Be(xx)se)) T-+00
1[x00 # blj 1
log(xo, x (o(z,,)) t)]
1 = -bl w log(xu, Xo ())
10-2(00,111111x xo)
- # b) & log(xu, xa (ze, t))
(71)
which is the continuous-time NELBO loss for a single token. Therefore, the loss across a sequence of L tokens denoted as (2)
X
we have
Ee Ej (xo)
Σ Exx
log xo(2,,))
Σ
1 log(xo, xa(zi,
(72)
t
Lxx y
which proves the loss defined in (5).
G.4. Gradient Flow of Invalid Loss
(8) Proposition 2.3 By differentiating the invalid loss with respect to the probability vector X (z,, t) for position E, the gradient with respect to the predicted probability of the sampled token j = k and all other tokens in the vocabulary j o k is given by
SM(= (2) (1-SM(0
j= k
V Linvalid
(73)
(SM(
j j k
Proof. We aim to show that the penalty for invalid token samples through the argmax function on predicted logits can be effectively backpropagated through the model parameters via our softmax scaling strategy. Here, we will denote the predicted probability for the token is = arg max, (x! (zz, t)) with the highest probability as 8, and all remaining token
(e())
probabilities as {..j
(f) for = [1... []]-
First, we define the softmax function as
exp(=!!1)
(0) SM
(M(
(74)
M
exp(x!1))
The partial derivative of the softmax probability z/f for every token j is given by equation
(6) (8) exp(= 0x!
(Σ exp(==²))
a ΣM exp(=!?) (exp(=??))
(10)
(1)
exp(x(()) E.V exp(a!!))
ou!?
(75)
(6) amgg
MM exp(x!13)
(2)
Therefore, we have two cases for the derivative: first, the derivative with respect to T which denotes the predicted (8)
probability for the token that was sampled, and second, the derivative with respect to Tg for j * lo which denotes the predicted probabilities for all remaining tokens.
39

---

De Novo Generation of Therapeutic Peptides with Multi-Objectvv--uuddee Discrete Diffusion
For the first case when j = ki the partial derivative simplifies
a
exp(=)ph) j=1 exp(xxe))
exp(x),e) Σj1 M
exp(x)1) exp(2 -xp((= exp(=/.)
(8) 0xgg.
VI
Σ exp(x (8)
8.k
exp(xex))
Σ 14
= exp( ( - exp(2b1))
Σイ E exp(2)e)))
Σ/= expl=!!))
= sM (1 (1-M((z))))
(76)
For all j + k, the derivative simplifies to
0 - 1-cxp(xl0 (00) exp(xae (1)))
0
exp(x.61)
0x
Σj= exp(x!?)
exp(=
exp(=
Σ MI exp(x
1P] Sesseexpuu )
SM( (SM(
(77)
The parameters are updated such that the predicted probability of sampling the token E with argmax probability ( which resulted in an invalid peptide SMILES sample, is reduced. The gradient update is minimized for predicted probabilities near 0 and 1, suggesting that the loss function pushes the model towards higher confidence predictions from uncertain predictions to minimize invalid sampling.
((E) (E) Ja.k Jg.kk - -n- - (1- SM(2
(78)
where 1) the learning rate.
In contrast, the parameters of the remaining tokens Toj are updated so that the predicted probability of sampling the other tokens increases proportionally to their original softmax probabilities. This prevents extreme changes in the predicted probabilities of the remaining tokens and ensures that the token distribution remains relatively consistent with the previous iteration.
(f)
00.5 (00) < - t.- SM (9) (M
(6) T0,
(79)
Here, we show that our invalid loss effectively updates parameters to reduce the position-specific token probabilities that result in invalid sequence samplings and push the model predictions toward other high- ligh-likelihood tokens.
H. Hyperparameter Selection
In this section, we discuss the choice of hyperparameters for PepTune. Specifically, we discuss the effects of changing the number of expanded children nodes, the number of iterations, and the tokenization scheme.
Number of Children The number of children M the hyperparameter that determines the batch size during the expansion step of MCTS. A small number of expanded nodes would limit the degree of exploration and the number of generated sequences for evaluation each iteration. If the initial iterations resulted in sub-optimal unmasking steps for all children, this could prevent the algorithm from discovering local or global optimum across objectives. Suppose the number of children is too large. In that case, this can result in a lack of diversity if several sequences from the same expansion step are added to the Pareto-optimal set given their sequence similarity, leading to similar property scores. A large M also slows down runtime significantly. To determine a value for M within the two extremes, we evaluated the performance of the MCTS search for M- = 10, 50, 70, 100. Overall, we found that M = 50 yields consistently increasing scores across all properties, which we use for the remainder of the study.
40

---

De Novo Generation of Therapeutic Peptides with Multi-Objectve-Cuudee Discrete Diffusion
Number of Iterations The number of iterations Nuer determines the number of selection, expansion, rollout, and hackpropagation loops executed in single MCTS run. In addition, Naer is the maximum value of the number of unmasking steps that can be executed before the rollout begins, which corresponds to the maximum tree depth. We found that equating the number of diffusion steps T to the number of MCTS iterations Nuxr results in convergence on the property prediction scores as the selection process becomes biased towards a single unmasking scheme. As shown in Figure 5, all property scores converge for Nuer = T 128, which we use for the remainder of the study.
Tokenization Scheme To evaluate the effect of different tokenization methods on the generation quality, we experimented with three different tokenization schemes: SMILES Pair Encoding (SPE) tokenization with the trained vocabulary used by PeptideCLM and Atom Pair Encoding (APE) tokenization for SMILES and SELFIES (Krenn et al., 2020) representations. Overall, we found that the SPE tokenization scheme decreased perplexity and maintained precision while capturing common peptide motifs like bonds and recurring side chains.
Table 12. Effect of Tokenization on Sequence Length, Training, and Validation Loss after Convergence
Tokenization Scheme
Vocab Size
Avg Sequence Length in Data
Train Loss (4)
Val Loss (4)
Val PPL (1)
SMILES SPE Tokenizer
581
84
0.65
0.75
2.12
SMILES APE Tokenizer
605
19
1.33
2.32
10.18
SELFIES APE Tokenizer
605
21
1.56
2.50
12.12
Additional Experiments
L.1. Case Study for Time-Dependent Multi-Objective Guidance
Some properties of peptides require more intense guidance towards specific structural or sub-structural features, while others may only require small changes in the side chain composition or non-natural modifications. To enable the prioritization of properties during guidance, we introduce a time-dependent multi-objective guidance strategy that guides the generation based on only a subset of properties, depending on the current iteration number of the MCTS search. To achieve this, we define --dimensional vector [&1, 12, .., ik] where each ig is the iteration number to begin guidance for the kth objective. Properties where 4 0 are used guide all iterations, whereas properties where is are used to guide only the iterations from ig Nier-
Our time-dependent guidance operates as follows. During the expansion and rollout steps on iteration i, the rolled-out child sequences Xs.i that are non-dominated across the sub-vector of property scores S; = [sk 44 S i Nuee] dependent on the iteration i are added to the Pareto-optimal set P Therefore, x, does not need to be non-dominated in the properties where in > i. Similarly, only the sequences x* p that become dominated when adding x, in the subset of properties represented in (x") are removed from Pt
pr = P* U { (Zz,s(x,)) Vx* E P* sx,x) 2 sx()))
(80)
p"=P" \ fx" Ex, children(zz) s.t. (xx) > (x())}
(81)
Then, during selection, we only consider the cumulative rewards W, = [W is Si Nuerl for the properties where in s.t. i < i S Niter when computing the selection score vector U, to form the Pareto-optimal selection set.
Pretee = Pielect U { (2s
Vx" E Peeeect U.(2;, 2.) Z U,(2, x x")}
(82)
Pio = Pielect \ {z" 3z3 E children(z:) s.t. U,(2,,2,) > U,(Z,, z")}
(83)
Finally, we select the next node z, ~ Plece uniformly at random from the Pareto-optimal selection set.
To test this strategy, we generated 100 peptides conditioned only membrane permeability for the first 50 iterations since we found it as the most challenging property to optimize. Then we conditioned all properties, including membrane permeability, binding affinity to GFAP, solubility, hemolysis, and non-fouling. We show that during the first 50 iterations, all properties except membrane permeability show relatively constant average scores, whereas the permeability score increased (Figure 16). Then, after the 50 iteration mark, GFAP binding affinity and solubility curves increased significantly while the hemolysis and foullng curves increased slightly for the remainder of the iterations (Figure 16). Although all the results

---

De Novo Generation of Therapeutic Peptides with Muli-Ohjectiv--uudee Discrete Diffusion
A
Average Membrane Permeability Scores Over Iterations
Average Binding Affinity Scores GFAP Over Iterations
Average Solubiity Score Over Iterations
Average Membrane Permesbiity
Average Binding Affinity Score
Average Solubility Score
100
D
Average Hemolysis Score Over Iterations
E
Average Non-Foulde non-Fouling Score Over Iterations
Average Hemolysis Score
Average Non Non-Foulin Score
Iteration of MCTS
Iteration of MCTS
Figure 16. Time-Deper Deree-Dependett Multi-Objectie Guidance. (A) Plot of average membrane permeability score for 50 sampled sequences the expansion and rollout step over iterations where MCTS search is conditioned on permeability for iterations. average predicted binding affinity score GFAP over iterations when conditioned starting from epoch 50. (C, D. E) Plot of average predicted solubility, hemolysis, and non-fouling scores over iterations when conditioned starting from epoch 50, Pink dotted lines denote the iteration where the MCTS search began conditioning the property
(B) Plot
in this paper leverage peptides without time-dependent guidance, this serves as proof of concept for future experiments varying the start times across properties to refine certain properties at later time steps, where the generated sequences already constrained to specific predefined substructures.
I.2. Ablation Studies
We conduct ablation study evaluate the effect of our bond-dependent masking schedule and invalid loss the fraction of unconditionally generated SMILES sequences that are classified as valid peptides by our SMILES2PEPTIDE decoder. We demonstrate that both components of the model are critical for the generation of sequences containing the necessary components that define valid peptide (Table 13). Furthermore, our bond-dependent masking schedule and invalid loss can be applied for diverse sequence generation tasks, where preservation of fundamental structural components of the sequence necessary (e.g., sentence structure in natural language, protein motifs, etc.).
Table 13. Ablation study on the effect of bond-dependent masking and invalid loss. We unconditionally sampled 100 sequences from each model and evaluated validity with SMILES2PEPTIDE decoder.
Model
Fraction of Valid Peptides
PcpMDLM
0.40
PepMDLMM + No Bond Dependent Masking
0.16
PcpMDLM No Invalidity Loss
0.21
42

---

De Novo Generation of Therapeutic Peptides with Mult--Objectv--Cuudee Discrete Diffusion
J. Algorithms
Algorithm I outlines the training algorithm for PepMDLM, our bond-dependent masked discrete diffusion model for unconditional peptide SMILES generation. Algorithms 3. and describe PepTune, our MCTS -ggiddd peptide SMILES generator. Algorithms and describe the bond mask function and peptide sequence decoder which can also validity filter.
Algorithm 1 PepMDLM Training
1: Inputs: Batched training examples Xo
2: Output: Trained unconditional MDLM for peptide SMILES generation po(z,|zi)
3: procedure TRAIN
4: Sample Uniform 1)
sample continuous times
5: -bond-dependent masking schedule
6: a1(Xu) (1 (--BONDMAS((x.))
B)OODMMASK(x))(11-- th)
7: mask each sequence in batch at varying degrees
8: Sample ~ Cat(z;; a (x0))xx))))))))))))))())))) + (1 (x(x)))
9: X9 (Z, t) f- RoFormerg (Z,,)
b predict token logits with RoFormer backbone
10:
if 21 * m then
X (o( t) f- Zz
carry-over unmasking
else if 2t m then
13:
Xg(Zi, f xo(Z,, t) com
zero-masking probability
14:
end if
15:
C NELBO
B ExEE
( -exio0 w log(xb) (0) (2)
Xx (2x, t)) - E
(6) (2) & log(x Xx ( t))
16:
x + arg max X1 (Zi,t)
(6)
17:
Linvalid f- [BT EneB
Et (0) SM(x)) (2t, 1)) 1[20 is Invalid)
C t NEBLO + Linvalid
total loss
19:
H' < 0 - VoL
update backbone parameters to minimize loss
20: end procedure
Algorithm 2 Reverse Diffusion Step
1: Inputs: Partially unmasked sequence at time Zz
2: Output: Token probability distribution pa Z. 2,) for all positions in the sequence with the bond-dependent reverse
posterior and SUBS parametrization
3: procedure REVERSEDDIFUSSINNSTPP
4: X (z( t) < RoFormery (z,,t)
5:
86 [
6:
if m then
7:
(z(z,lz,) + ( (00
b+ 1
(0 11)2)) + ((f" -4) b f1, (1,, m -+)bt
8:
(z. = m 1z)) t- 0
zero-masking probability
9:
else 2, # m then
10:
2. + Ze end if
carry-over unmasking
return pe(z, 22. (c:)
13:
end procedure
43

---

De Novo Generation of Therapeutic Peptides with Multi-Ojeectve-Gnne Discrete Diffusion
Algorithm 3 PepTTune: Multi-Ojjective Guided Discrete Diffusion with Monte Carlo Tree Guidance (MCTG)
1: Inputs: Trained MDLM denoiser (2, zz),, score function s(x) 1/ R containing classifiers for K objectives
$1. $2..... S.. number of time steps T, number of iterations Nier
2: Output: Set of Pareto-optimal sequences for the objectives and their K-dimensional classifier score vectors P
{(x), {{}}}
3: procedure SAMPLEPRPTUNE
4: ZT - [MASK]
b initialize fully masked sequence iniitialize Pareto-front
5:
P 00
6:
for I,..,, Niter do
7:
Ze(n); ((n) SELECT(ZT) r+0 0
select expandable leaf node unmasked time t(n)) initialize vector that will store the sum of all rewards from expanded children
8:
9:
children(Z (Z((n)) f BATCHEDREVERSESTEER..
10:
Noolou f- T -
(()(
11:
n n-1
11
T
T
di -- T
for 1....,M M do
14:
Zi(n) n,
15:
for 1 to Nrotlout do
b rollout fully unmasked sequence
16:
Po.i (Z)(())())) (e(n)) -
17:
(((x) + arg max Pe, (Z)())))))))))) ((()))
18:
end for
X f- arg maggmax/o Po. (x, (((1))
get clean sequence compute score vector
s(x,,)) + s(x,,i)
21:
add sequence if non-dominated
22:
r(Ze,)). P *UDDATPPARTTOFFONTTPPPP (EE (Ze,i,
23:
children(z:)) (appreenz))..peeddlz...... rer- rZz,i) end for
add child node b> add child reward total reward for node z
24:
25:
26:
t parent(z,.)) (2,,))
27:
while not None do
t backpropagate scores
28:
Wzz) t W(z) +r
Nvssin (z) t Nussis (z) +
30:
parent(z)
repeal for parent node until root node
31:
end while
32:
end for
33:
return P
b return Pareto -preial sequences
34:
end procedure
44

---

De Novo Generation of Therapeutic Peptides with Multi-Objectvv-Giine Discrete Diffusion
Algorithm 4 Batched Reverse Step
1: Inputs: Partially unmasked sequence at time / (representing the selected node in MCTS search), value of k for top k: sampling (A: O for batched Gumbel-max sampling), total time steps 'I'
2: Output: Set of M slightly unmasked sequences children(z:) { 2x,M) } at time a that become the child nodes of zz
procedure BATCHEDREVERSESTTE
4: children (2.) 0
5: (z) t RoForrem, (Z,,t)
6:
s t +
7:
z; m then
8:
po(2.1zz (x)== - ( pa0
b f-g 1
xo (0 1)) 23 + ((i= -3) b 11, (o(2e,)) m
9:
pa (z. = m zz)) 6- 0
zero-masking probability
10:
else v, + m then
Z, + Ze end if
carry-over unmasking
12:
13:
for i
M 1...M do
D define slightly different distribution for each sample in batch
14:
Ui Uniform (0, 1)
15:
G..j +- -log log(us, + E) + +)
16:
k = then
Pa, (2a,i + loggo (2n) + G.
batched Gumbel-max distributions
18:
Zn,i ~ Po.s (2, (2,, (+)
19:
else 4/4 > 0 then
20:
6o.s (2., (2:) + SM topki log po (2.,(zi) + Gii}
batched top le sampling
21:
end if
22:
Z, po. (Z,,ilze))
23:
children (ap))d (2.i)) end for
24:
25:
return children(z:)
26:
end procedure
45

---

De Novo Generation of Therapeutic Peptides with Multi-Objectvv-Giune Discrete Diffusion
Algorithm 5 Selection
1: Inputs: Masked root node ZT(T)
2: Output: Expandable leaf node Ze
3: procedure SELECT
4: while True do
5: if z, is non-leaf node and + then
6:
Pretect f for Zu in children (z.) do
b initialize Pareto front of select scores
7:
8:
if Zo is non-leaf expandable leaf node then
9:
(Z 20,1)
Non(Zz..)) + c. po(Zz,ilzi) It 1+Nna (Zo.e)
W(z..)
(as
P seeeca -UPDRTEEARETTOFONT Pielects (Zo,s, U(Ze, Ze,t))
11:
end if end for
12:
13:
b set parent node for next iteration as a child node selected uniformly at random from Pareto-optimal set
14:
2, Pielect
SELECT(2,
recursively call select until leaf node reached node is already fully unmasked restart selection process from root node return leaf node for expansion
else then
17:
SELECT(ZT)
18:
else
19:
return 2,
20:
end
21:
end while
end procedure
46

---

De Novo Generation of Therapeutic Peptides with Multi- Salii-Objeciiv-Guidee Discrete Diffusion
Algorithm 6 Update Pareto Front
Inputs: Current Pareto-fromt sequences and score vectors p {(x7, ss)) newly sampled sequence and score vector (Zg; (x,))
2: Output: Reward vector r(z,) and updated Pareto-optimal set p*
3: procedure UPDATEPARETOFRONT
4:
if* is empty then
5: "P"'appendde [g, s(x.)))
6: r(z.) +
set reward vector to ones
7: else
8:
vector of boolean flags indicating which sequences nondominant bool[[P*|]
nondominateFlag 10: toDelete +- 0
9:
11:
r(z.) t 0
D set reward vector to zeroes
12:
for (x) ) in p do
b define vector with where x, is non-dominated in the property - [mx = 1 sx(x), ł s
define vector with where Xa,i dominant the property [d+ 1 sx t $*]
16:
17:
r(z,,i) r(2,) +
update reward vector x dominates x"
18:
if (vn n s.t. n& = 1) A (3d E d s.t. d₂ = then
19:
toDelete.ag "pDelete.append(x)
20:
nondominnttellagi]] + True
else if Vng E n s.t. п₁ then
x is not dominated by
22:
nondominateFllg[i] True
23:
else
x" dominates
24:
nondominateFlagli] False end if
25:
26:
end for
27:
if x, is either dominant or non-dominated by all x" Pareto-optimal set Pt, then add to P Vi nondominateFllgllj] = True then
28:
P append (z.. s(x))
30:
end if for in toDelete do
31:
32:
p* ss)
33:
end for end
34:
b return reward vector and updated Pareto-optimal set
35:
return r(z.), p
end procedure
47

---

De Novo Generation of Therapeutic Peptides with Multi-Ojective-Giinn Discrete Diffusion
Algorithm 7 Bond Mask
Inputs: List of peptide SMILES strings smi les __iist
2: Output: Position-wise bond mask for each sequence with in positions of peptide bonds and O otherwise ma sk
3: procedure BONDMASK
4: bond_paaterng +
[(r "OC (=0) 'ester'),
(r ((-))), 'n m__mehhy'', (=)) 'n_methyl') (I'N[12]C(=O)', 'peptide'), (F'0(=0)N(12)3', 'peptide')]
5:
for warbatch_ddde smiles in enumerate(amilee__1stt do
6: positions + empty_list ()
list to store bond positions set to track used positions identify bonds using patterns
7:
used + empty_set ()
8:
for pattern, bond in bwtdbooonterrnreeeeeeeeeeeeeeeee do
9:
for match in re fin.iterrrrrrteem,, smiles) do
if not any(p E range(match. start (), wammaahccnnd... ()) for in used) then
11:
positions appendd(starri match.staze (), end: match. end ()
type: pend_type.pateentt
match. group () 1)
used. update (rangematch. (aadd (), match. end ()))
end if
14:
end for end for
15:
16:
for pos in positions do
b update the mask for the current SMILES
17:
mask [batch_idk pos [start] :pos [end] ] + 1
end for end for
20: return mask
21: end procedure
48

---

De Novo Generation of Therapeutic Peptides with Multi-Ojeetivv-GGue Discrete Diffusion
Algorithm 8 SMILESZPEPTIDE
1: Inputs: SMILES String
2: Output: Batch of M sequences at time 8.
3: procedure ANALYZER
4: if is correct SMILES format then,
5:
if's s contains peptide bond [NC(=O)] or N-methylated peptide bond [N(C)C(=O)] then,
6:
IS_ "S_PEPTIDE TRUE positions empty_list ()
7:
8:
for bop in bond_ panterss do
9:
for match in in ftn.....nniteerraatettnn smiles) do
10:
+
11:
emp
12:
positions.sort
if [ then,
first segment
segments. content: smiles [0:positions [C] ["] ['starr
bond_after; positions [0] pattern'))
15:
end if
16:
for i in len (positions) -1 do
other segments
17:
current = positions [i]
18:
next_ positions [i+1]
segments. append content: smiles [current ('end') :next_pos [' [start']],
bond_before: current [' pattern']] ],
bond_after: next_pos [' pattern']]]
20:
end for if positions [-1] end'] < len (smiles) then, segments. append content: smiles [positions ['end' ['end') :]
21:
b last segment
22:
bond_after: positions [-1] [ (ppttteenn) )
end if end for end for
24:
25:
26:
for residue due in segments do
27:
residue +- Regex pattern
Empirical Amino Acid Regex Pattern
28:
end for end end
29:
30:
end procedure
49


Reviews:
[{"review_id": "NxW8OX9YSC", "reviewer": "Reviewer_whBR", "summary": "The paper proposes PepTune, a discrete diffusion model that enables multi-objective guidance for generating and optimizing therapeutic peptide SMILES. PepTune adapts a bond-dependent masking schedule and global sequence invalid loss to improve discrete diffusion performance on peptide, and uses an MCTS-based strategy to guide the masked discrete diffusion generation and demonstrates good results for several therapeutic properties.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback.\n\nFirst, we would like to respectfully disagree with the assertion that the guidance strategy is inadequately evaluated. Given that the paper is on **Application-Driven ML Track**, we chose case studies that provide robust evaluation to prove efficacy in peptide-based therapeutic design. This is far more impactful and applicable to society and medicine than classical benchmarks. We ask that the reviewer reconsider their review of the paper with this in mind.\n\n**Addressing Concerns Experimental Designs Or Analyses:**\n\n- We conduct an ablation study as suggested. From each model, we sampled 100 sequences of token length 100 and checked validity with our SMILES2PEPTIDE decoder. We show that both the bond-dependent masking and invalid loss improve the validity of peptides.\n\n| **Model** | **Fraction of Valid Peptides** |\n| --- | --- |\n| PepMDLM | 0.40 |\n| PepMDLM + No Bond Dependent Masking | 0.16 |\n| PepMDLM + No Invalidity Loss | 0.21 |\n- In HELM-GPT, the peptides are represented in HELM notation [1], where each token is a valid, synthesizable amino acid. In contrast, PepMDLM is trained on SMILES tokens that decompose amino acids into smaller tokens that can be pieced together into valid amino acids during generation. While this enables us to represent a greater diversity of non-natural amino acids, even a single token generated in the wrong position can result in an invalid peptide sequence, resulting in a lower validity rate. However, we note that our MCTS guidance strategy increases the validity rate to 100% due to its iterative unmasking process that is rewarded on high-scoring and valid peptides.\n- We selected three target classes to demonstrate PepTune’s robustness: (1) proteins with known binders to benchmark against known binders, (2) proteins without known binders, and (3) dual-target cases. This is a rigorous evaluation for application in therapeutic peptide design.\n    \n    While previous guidance methods for discrete diffusion are limited to optimizing 1-2 objectives, we evaluate guidance across 4-5 non-trivial objectives that are highly relevant to therapeutic peptide design.\n    \n- We appreciate the reviewer’s suggestion to conduct best-of-N comparison. However, we believe that our comparisons between PepTune and PepMDLM are sufficient in demonstrating the efficacy of our guidance strategy for the following reasons:\n    - In Pareto optimization, scalarizing or ranking sequences across conflicting objectives without access to the Pareto frontier leads to suboptimal trade-offs, making it difficult to rank best-of-N.\n    - Despite exploring more sequences, PepTune achieves comparable runtime by reusing high-reward unmasking paths within the MCTS tree. Rather than starting from scratch, PepTune expands only unexplored nodes, reducing redundant computation. Empirically, we find similar runtimes for generating 100 *valid* sequences using PepMDLM (403 sec) and PepTune (347 sec for top 100 sequences with $M=10$ and $N_{\\text{iter}}=128$), while PepTune yields substantially higher property scores (Figures 4–9).\n- We also appreciate the suggestion to conduct a benchmark against ParetoFlow. However, ParetoFlow is designed for continuous flow-matching models, whereas our approach is designed specifically for discrete diffusion. Due to the fundamental differences in model class and data type, a direct comparison would not be meaningful; to our knowledge, PepTune is the first to apply Pareto-aware guidance for discrete diffusion, and we hope this novelty is considered.\n- Given that our method supports robust guidance beyond 2-3 objectives, it is impossible to visualize the high-dimensional Pareto frontier. Instead, our graphs demonstrate that our approach simultaneously raises the scores across 5 objectives (see Figure 9) to a plateau, indicating that no further optimization is possible without sacrificing another objective.\n- Each MCTS run explores $M\\times N_{\\text{iter}}$ sequences. The sampling cost scales linearly with respect to both of these parameters. We compared $M=\\{10, 50, 70, 100\\}$ and found similar guidance curves and set $N_{\\text{iter}}=128$ constant.\n\n**Responses to Questions**\n\n- In this experiment, we optimized 5 properties: binding affinity to GFAP, membrane permeability, solubility, hemolysis, and non-fouling, all of which are not related to the values in Table 1. In Table 1, we aim to show that our MCTS-guidance algorithm does not significantly lower the diversity of sequences and increases validity through our iterative MCTS-guidance algorithm that rewards valid expansion steps.\n\nWe thank you for your suggestions and hope you reconsider raising your score in light of these responses and the broader goals of the Application-Driven ML Track.\n\n[1] Zhang et al. Helm: A hierarchical notation language for complex biomolecule structure representation. http:[//dx.doi.org/10.1021/ci3001925](https://dx.doi.org/10.1021/ci3001925)", "claims_and_evidence": "Most claims in the submission are supported by convincing evidence. However, there are several gaps in the proposed method and experiments to support the novelty and performance of this paper. Please refer to details in \"Methods and Evaluation Criteria\" and \"Experimental Designs or Analyses\" sections.", "methods_and_evaluation_criteria": "There are two major components of the proposed method: 1. PepMDLM that modifies the MDLM method with prior knowledge and inductive bias of the peptide data distribution; 2. MCTS based guided sampling for multi-objective design. The datasets and evaluation criteria for PepMDLM look fine, while for guided sampling, only case studies are applied without a general rigorous evaluation. Also, commonly used criteria for Pareto optimization, eg, the hypervolume metric, should be calculated.", "theoretical_claims": "There are no theoretical claims in this paper.", "experimental_designs_or_analyses": "- For PepMDLM, an ablation study is missing for the importance of each component (i.e., bond-dependent masking schedule, bond-dependent diffusion loss, invalid peptide loss, etc.).\n- In Table 5, the validity of HELM-GPT is much higher than PepMDLM. The author claims that validity is assessed differently. Could the author clarify this more and how this affects the validity comparison with HELM-GPT?\n- The effectiveness of the guided sampling part is majorly demonstrated using case studies. How are these case study settings chosen? It is helpful if the authors could provide some evidence to support that these case studies offer a general and reliable evaluation for the proposed method that is applicable to a broad range of cases without cherry picking.\n- For guided sampling, there is no baseline model other than the pretrained PepMDLM that is compared, for example the simplest best-of-N approach with N setting as a value comparable to the computational cost in the MCTS sampling, as well as guided sampling approaches, eg. [1].\n- For each case study, the paper only shows the distribution of each objective individually. It would be helpful to draw the Pareto frontier of the generated peptides to better demonstrate how the model output balances different objectives and explore the Pareto frontier compared to other baselines.\n- An analysis of the sampling cost of MCTS with respect to different hyperparameter choices and how the model performance scale with the sampling cost is helpful\n\n[1] Paretoflow: Guided flows in multi-objective optimization. ICLR 2025.", "relation_to_broader_scientific_literature": "The paper is related to: machine-learning wise, discrete diffusion models, guided sampling; and biological application wise, de novo peptide design.", "essential_references_not_discussed": "- Multi-objective guided flow models: \n\n  [1] Paretoflow: Guided flows in multi-objective optimization. ICLR 2025.", "other_strengths_and_weaknesses": "The paper has good domain-specific designs in the method for peptide and evaluates the method in many real-world use cases, which have good potential to be applied in real-world peptide design problems. My main concern in this paper is the lack of baselines, benchmarks, and ablation studies in the evaluation of the method apart from the case studies.", "other_comments_or_suggestions": "NA", "questions_for_authors": "- In Table 1, for PepTune, what are the rewards/properties optimized by the MCTS, and how are they related to the values reported in Table 1? They seem to be some general metrics that are irrelevant to the reward and thus cannot measure how well the rewards are optimized by the model.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YPrsn8XY19", "reviewer": "Reviewer_zw2U", "summary": "The paper proposes a new model for novo generation and optimization of peptide. The model operates on SMILES and performs multi-objective-guided discrete diffusion; it can handle non-canonical amino acids and cyclic modifications. PepTune employs a bond-dependent masking schedule. To guide the diffusion, PepTune uses Monte Carlo Tree Search. The model is evaluated on different targets, e.g, generation of peptide binders for the GLP-1 receptor and bispecific binders for TfR and GLAST; PepTune produces peptides with better predicted properties compared to existing therapeutics.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very grateful for the reviewer’s thoughtful review.\n\n**Essential References Not Discussed:**\n\nWe request that the reviewer refer to Appendix C, which provides an extensive analysis of prior discrete diffusion and classifier-based and classifier-free guidance methods. We also discuss at the end of Appendix C.3 the limitations of prior discrete guidance methods and challenges in multi-objective guidance. \n\n**Addressing Weaknessses:**\n\nTo ensure that our model can scale to an arbitrary number of objectives, we leverage a classifier-based guidance approach which doesn’t take the protein sequence explicitly as input to the diffusion model but instead injects the target protein information at each denoising step by computing multi-objective rewards that are back-propagated through the MCTS tree. This means that unmasking steps that produce high-scoring peptides are explored further, limiting the “wasted” peptides. We also note that this exploration method enables us to generate a batch of Pareto-optimal sequences in a single run of the algorithm, which is much faster than generating only one sequence per run like prior models. \n\nIn addition, we show that our binding affinity and property classifiers generalize well on the validation data (Table 6 and Figure 14). We have to accept that only limited protein-peptide binding quantification data is available, and downstream in silico validation like VINA docking can be used to narrow down candidate peptides to be synthesized for experiments. \n\n**Answers to Questions:** \n\n- PepMDLM is trained on SMILES tokens that decompose amino acids into smaller tokens that can be pieced together into valid amino acids during generation. While this enables us to represent a greater diversity of non-natural amino acids, even a single token generated in the wrong position can result in an invalid peptide sequence, resulting in a lower validity rate, which we calculate by decoding the SMILES sequences with regular expression matching. However, we note that our MCTS guidance strategy increases the validity rate to 100% due to its iterative unmasking process that is rewarded on high-scoring and valid peptides.\n- For the experiments, we generate peptides of token length 50 (7 amino acids) up to 200 (30 amino acids). This aligns with existing therapeutic peptides in clinical use, where the average length is about 20 amino acids or less [1].\n- Yes, our MCTS-guidance algorithm is a modular multi-objective guidance framework that can be adapted for any type of discrete data (e.g. protein sequences, promoter and enhancer DNA, RNA, small molecule drugs, natural language, etc.) and any pre-trained classifier in a plug-and-play fashion. Furthermore, our bond-dependent masking schedule can be extended to preserve structural components of other data types (e.g. functional motifs in proteins, punctuation and grammar in natural language, etc.).\n- In our work, we use “*de novo*” to refer to the generation of novel peptide sequences from scratch—i.e., not based on templates or fragments from known binders—and guided by pre-trained classifiers that predict binding affinity, solubility, and immunogenicity of these peptides. The model does not rely on known peptide backbones or motifs but instead generates sequences purely from noise through discrete diffusion sampling.\n\n    Furthermore, the target protein is not restricted to the training set of the classifier. Both the target protein and de novo peptide binder sequence are converted into feature-rich embeddings and passed through a classifier trained to understand binding patterns for accurate prediction. Our classifiers generalize well across unseen sequences, as shown by the high validation Spearman correlation coefficients (Table 6 and Figure 14).  \n    \n\nWe appreciate the reviewer’s thoughtful feedback and hope the clarifications provided demonstrate the uniqueness and adaptability of our approach. Our response highlights how PepTune addresses key limitations in prior work through a scalable, modular guidance framework, efficient multi-objective generation via MCTS, and strong generalization to arbitrary discrete data types and guidance objectives. Given our detailed answers, we hope you can consider raising your score to demonstrate continued support for our work.\n\n[1] Dougherty, Patrick G et al. “Understanding Cell Penetration of Cyclic Peptides.” *Chemical reviews* vol. 119,17 (2019): 10241-10287. doi:10.1021/acs.chemrev.9b00008", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "The paper introduces a sequence-based approach for peptide generation based on smiles and using a masking schedule that depends on the bonding to ensure that e.g., peptide bonds, side chains are preserved during the diffusion process; the model generates sequences that are more likely to be chemically valid and optimize multiple properties. As such few existing models can model non canonical amino acids and generate cyclic peptides.", "essential_references_not_discussed": "the authors do not detail much the related work; it would be helpful to have a separate related work section that is more detailed than the current introduction paragraph. for example, the authors could discuss more how multi property optimization is done for other molecular tasks e.g. for proteins; Accelerating Bayesian Optimization for Biological Sequence Design with Denoising Autoencoders - Stanton et al.", "other_strengths_and_weaknesses": "* The paper presents a new combination of masked diffusion with MCTS for multi-objective optimization in discrete sequence spaces. This  addresses the challenges with guiding discrete diffusion processes without gradient estimation.\n* The bond masking schedule is well suited to the peptide SMILES representation as it helps preserve structural elements of peptides.\n* PepTune successfully optimizes multiple properties simultaneously.\n* The paper includes extensive experiments and case studies, such as the generation of GLP-1R binders and bispecific peptides.\n\nWeakness\n* it seems like the generation process is not conditioned on the target, which if I understood correctly is only involved through the affinity oracles. as such the generation process can be wasteful as the generated molecules are not conditioned to the target of interest. Moreover, trained affinity oracles are known to generalize poorly out-of-distribution. As such, although target conditioned generation is possible, there might be several caveats with it.", "other_comments_or_suggestions": "see questions", "questions_for_authors": "* if model trained on valid peptides, why would it sample invalid peptides ? \n* which size of peptides can PepTune handle ?\n* can PepTune be adapted to other types of molecules, e.g. proteins ?\n* can the authors clarify what they mean with de novo ? it looks like the model would still need to have the targets to be in distribution for the affinity oracles to work; as such the model might not perform correctly under a more challenging train/test split with OOD test targets which is usually qualified as \"de novo generation\"; as such the current terminology can be misleading.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HJyn1rheCF", "reviewer": "Reviewer_evkB", "summary": "This paper presents PepTune, a Masked Discrete Language Model (MDLM)-based generative framework for generating new peptides. Unlike existing approaches that struggle with multi-objective optimization or rely on continuous approximations, PepTune operates natively in a discrete sequence space while optimizing for properties such as binding affinity, membrane permeability, solubility, hemolysis, and non-fouling.\n\nThe key contributions of the paper are:\n\n1. Introduction of Bond-Dependent Masking Schedule during training and inference.\n\n2. Introduction of Invalid Loss Function: A penalty term that discourages the generation of invalid SMILES sequences.\n\n3. Multi-Objective MCTS Guidance: Classifier-based rewards steer sequence generation to ensure Pareto-optimal solutions across conflicting therapeutic properties.\n\nThe paper uses several real case studies to demonstrate the efficacy of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed suggestions.\n\n**Claims and Evidence:**\n\n- We conduct an ablation study as suggested. From each model, we sampled 100 sequences of token length 100 and checked validity with our SMILES2PEPTIDE decoder. We show that both the bond-dependent masking and invalid loss improve the validity of peptides.\n\n| **Model** | **Fraction of Valid Peptides** |\n| --- | --- |\n| PepMDLM | 0.40 |\n| PepMDLM + No Bond Dependent Masking | 0.16 |\n| PepMDLM + No Invalidity Loss | 0.21 |\n- Due to the significant compute time needed to train additional baseline models on our 11 million peptide dataset, we decided it was not feasible to do so in this rebuttal period. Moreover, we note that Gruver et al. evaluate at most two objectives and require performing guidance updates in the latent embedding space based which can result in invalid conversion back to a discrete sequence, and Nisonoff et al. is limited to single-objective guidance. To the best of our knowledge, PepTune is the first framework for classifier-based **multi-objective guidance** for **discrete diffusion**, overcoming previous limitations via reward-based tree expansion and Pareto-aware backpropagation.\n\n**Essential References Not Discussed:**\n\nShi et al. leverage a per-token parameter $w$, which is trained in parallel with the discrete diffusion model. In contrast, we fix $w=3$ for tokens that exist within a peptide bond to ensure the model learns to reconstruct them earlier. This construction presents a practical extension of state-dependent masking to preserve peptide structure, which could be further explored for preserving various other structured data types. In addition, we rederive the state-dependent NELBO loss in a fundamentally different manner than Shi et al. (See Appendix D.3).\n\n**Other Comments or Suggestions:**\n\nWe thank you for the thoughtful suggestions. We have made the notations and figures clearer upon your request in the manuscript for the camera-ready version. \n\n1. We notice the inconsistencies in notation that you refer to, but in Equations 1-4, we simplify the notation so that $\\mathbf{x}_0$ refers to the one-hot vector of the true token at an arbitrary position in the sequence. We have added clarification and removed the $(\\ell)$ superscript in the other equations to maintain consistency. \n2. Yes, $w$ is a scalar greater than one. Please refer to the last sentence of Section 2.1, where the exact value is specified: “Empirically, we found that $w=3$ increased peptide validity while maintaining diversity across generated samples.”\n3. The sentence is fixed to “where $\\mathbf{b}$ is the vector with ones at indices of peptide bond tokens and zeroes in remaining indices.” \n4. We have increased the font size for clarity and fixed Equation 11 to include the $\\log$. \n\n**Answers to Questions:**\n\n1. In equation 3, the first $\\mathbf{x}_0$ is a one-hot vector is used to extract the probability associated with the unmasking the true token by taking an inner product. The second $\\mathbf{x}_0$ indicates that if the current token $\\mathbf{z}_t$ is a mask token, it has a $q(\\mathbf{z}_s=\\mathbf{x}_0|\\mathbf{z}_t=\\mathbf{m}, \\mathbf{x}_0)$ probability of being unmasked to $\\mathbf{x}_0$ and a $q(\\mathbf{z}_s=\\mathbf{m}|\\mathbf{z}_t=\\mathbf{m}, \\mathbf{x}_0)$ probability of remaining masked. For clarity, we have changed the transpose to an inner product:\n    $$\n   q(\\mathbf{z}_s|\\mathbf{z}_t, \\mathbf{x}_0)=\\begin{cases}        \\left\\langle \\left(\\frac{s}{t}-\\frac{s^w}{t^w}\\right)\\mathbf{b}+\\frac{t-s}{t}\\mathbf{1}, \\mathbf{x}_0\\right\\rangle \\mathbf{x}_0+\\left\\langle\\left(\\frac{s^w}{t^w}-\\frac{s}{t}\\right)\\mathbf{b}+\\frac{s}{t}\\mathbf{1}, \\mathbf{x}_0\\right\\rangle\\mathbf{m}&\\mathbf{z}_t=\\mathbf{m}\\\\\\\\ \\mathbf{z}_t&\\mathbf{z}_t\\neq \\mathbf{m}    \\end{cases}\n$$    \n2. $K$ represents the token vocab size. $\\tilde{\\mathbf{x}}\\in \\{0, 1\\}^K$ denotes a $K$-dimensional one-hot vector where the index of the token is indicated with 1.\n3. In HELM-GPT, the peptides are represented in HELM notation [1], where each token is a valid, synthesizable amino acid. In contrast, PepMDLM is trained on SMILES tokens that decompose amino acids into smaller tokens that can be pieced together into valid amino acids during generation. While this enables us to represent a greater diversity of non-natural amino acids, even a single token generated in the wrong position can result in an invalid peptide sequence, resulting in a lower validity rate. However, we note that our MCTS guidance strategy increases the validity rate to 100% due to its iterative unmasking process that is rewarded on high-scoring and valid peptides.\n\nWe appreciate the reviewer’s constructive feedback and hope the new clarifications, ablations, and improved notation address your concerns. Given the novelty, empirical strength, and thorough revisions, we hope you will consider raising your score to demonstrate your support for our work.", "claims_and_evidence": "1. Token-dependent masking and unmasking help in the generation of peptides.\n\nThere is not direct evidence supporting this claim in isolation. Idealy, PepMDLM should have been compared with vanialla MDLM for this.\n\n2. The proposed MCTS-based decoding procedure improves pareto optimality w.r.t multiple objectives.\n\nThis claim is supported using the results in Table 1 PepMDLM vs PepTune. However, it would be good to have comparisons to simple existing baselines for guided discrete diffusion for single objective guidance like Gruver et. al. (2023) and Nisonoff et. al. (2024), and them compare with them even for multi-objective guidance by making appropriate extensions.", "methods_and_evaluation_criteria": "Currently, the paper only presents results for two versions of their own method: PepMDLM and PepTune, where both methods only differ in the inference procedure. Since the paper uses multiple modifications for training as well as inference over vanilla MDLM, it is important to show ablations for each one of them.\n\n1. Comparing PepTune just with PepMDLM is not sufficient. Before going to the case studies, it would be good to have comparisons to simple existing baselines for guided discrete diffusion for single objective guidance like Gruver et. al. (2023) and Nisonoff et. al. (2024).", "theoretical_claims": "I could not check the correctness of the proof for equation (3) because I had some trouble understanding the notation. Please see the questions and the comments section below.", "experimental_designs_or_analyses": "Please see the \"Claims and Evidence\" section above.", "relation_to_broader_scientific_literature": "The paper adds a promising approach for scalable inference time guidance for discrete diffusion models using MCTS. To the best of my knowledge, the use of MCTS-like algorithm for sampling from a discrete diffusion model is novel and has not been explored before.", "essential_references_not_discussed": "Please discuss how the state-dependent training objective used in this paper is different (if it is different) from Shi et. al. (2024).", "other_strengths_and_weaknesses": "One of the major strengths of the paper are the real case studies. However, some sections of the paper are not as clear (see comments and questions). Moreover, the paper can be improved by including more baselines (see Claims & Evidence and Methods & Evaluation sections above).", "other_comments_or_suggestions": "1. Line 058, column 2: $\\mathcal V$ is used without specifying its meaning, and at this point in the paper it is unclear what is meant by \"takes the vector encoding of the token $\\mathbf x_0$\". Isn't bold-faced $\\mathbf x_0$ a sequence? There are a couple of missed words around the equation, which makes it an awkward read: \"where $b$ represents _the with_ ones..\". Also, what is $w$ ? I'm assuming it is a fixed scalar greater than $1$, but this should be mentioned. \n\n2. In equation 3, what is the meaning of $(...)^T x_0 x_0$? The left multiplier is transposed, so it must at least be a vector, but what is $x_0 x_0$? Similary, what is the meaning of $x_0 m$? While I could still understand equation 1, in spite of undefined symbols. I can't follow equation 3 at all. \n\n3. The colors and fonts used in the main figure of the paper (Figure 1) make it extremely difficult to read. I'm unable to read the small fonts even after zooming in quite a bit.\n\n4. I think there should be a $log$ on the left-hand side of equation 11.", "questions_for_authors": "1. In equation 3, what is the meaning of $(...)^T x_0 x_0$? The left multiplier is transposed, so it must at least be a vector, but what is $x_0 x_0$? Similary, what is the meaning of $x_0 m$? \n\n2. What is $K$ in $\\tilde x^{(l)} \\in \\{0,1\\}^K$? \n\n3. Why is the validity percentage for PepTune much lower than HELM-GPT?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gVuWZdCKbt", "reviewer": "Reviewer_hF7T", "summary": "The authors propose a framework based on the Masked Discrete Language Model (MDLM) (PepTune) to generate and optimize therapeutic peptides. They claim that the main contributions of their model are: 1) using MDLM to generate peptide SMILES representations, 2) introducing NELBO and reverse-posterior, 3) using Monte Carlo tree search to guide the generation of SMILES, and 4) training a set of classifiers and regressors for multiple properties.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your insightful review. \n\n**Essential References Not Discussed:**\n\nWe appreciate the reviewer’s suggestion to consider Wang et al. (2024). Although the title may suggest a close relation to our work, the scope, modeling assumptions, and core goals differ significantly. MMCD generates peptides composed of 20 natural amino acids only using a multi-modal approach. While MMCD uses the label “therapeutic peptide generation,” its evaluation is limited to docking scores and embedding-level metrics, without any testing of therapeutic properties such as solubility, hemolysis, or non-fouling. In contrast, our work directly targets real-world therapeutic peptide design by expanding the chemical space to include non-canonical residues and cyclic structures and by incorporating classifier-guided objectives tied to **experimentally actionable properties**. This direction aligns closely with the goals of the **Application-Driven Machine Learning Track**. Given these fundamental differences in scope and design goals, a direct comparison is not appropriate.\n\n**Addressing Weaknesses:** \n\n**Weakness 1:** We would like to push back on the assertion that our paper lacks significant novelty. Although NELBO is used in previous works, we derive a **unique** case of the NELBO given our novel bond-dependent masking schedule (See Appendix D.3). While the MDLM was already introduced by Sahoo et al., we make a novel extension with a bond-dependent masking schedule and derive a reverse posterior that differs from the general case in Sahoo et al. (See Appendix D.1-D.2). In addition, we introduce a novel invalidity loss that penalizes invalid peptide structure during training. \n\nFinally, despite the use of MCTS in prior discrete models, our paper introduces the first application of MCTS to diffusion models, with our method being published in *arXiv* before Yu et al. (2025). In addition, our submission date to ICML was 9 Jan 2025, which was before the publication of Yu et al. (2025) on 11 Feb 2025. \n\n**Weakness 2:** The baseline mentioned (Wang et al.) is designed to generate peptides consisting only of the 20 natural amino acids, while the core innovation of PepTune is to expand the search space to peptides with modified amino acids and cyclicizations which has not been achieved by prior models, which has been shown to enhance various therapeutic properties. These are two fundamentally different problems that are difficult to compare. Furthermore, the motivation behind the key components of our architecture, including our bond-dependent masking schedule and invalid penalty loss, stems from our unique goal of generating valid peptides from SMILES tokens despite the increased granularity.\n\nOur source code is included in the arXiv version of the manuscript; however, given the anonymity of the review process, please refer to the anonymous repo link: [**https://anonymous.4open.science/r/peptune-86CB/README.md**](https://anonymous.4open.science/r/peptune-86CB/README.md).\n\n**Weakness 3:** We believe you may have misinterpreted the comparisons in Table 1. PepMDLM is ***our*** model trained with our bond-dependent masking schedule only without MCTS guidance. We make this comparison to demonstrate that our MCTS-guidance strategy does not significantly lower the diversity of sequences and increases validity through our iterative MCTS-guidance algorithm that rewards valid expansion steps.\n\nGiven that most of your concerns are regarding clarity and not the methodological and experimental soundness of our paper, and we have addressed your doubts on the novelty of our approach, we hope you consider raising your score in response to our answers. We will plan to introduce clarifications into the main text for the camera-ready version.", "claims_and_evidence": "The source code is not provided. The experiments are incomplete, and there is no comparison with other baseline models, which makes the paper unconvincing.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Using MDLM to generate peptide SMILES representations and introducing NELBO and reverse-posterior", "essential_references_not_discussed": "Wang Y, Liu X, Huang F, et al. A multi-modal contrastive diffusion model for therapeutic peptide generation[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(1): 3-11.", "other_strengths_and_weaknesses": "Strengths: \n1) This paper is well presented and well-written.\n2) The appendix provides clear supplementary material for better understanding.\n3) Therapeutic peptide generation is a challenging and novel area.\n\nWeaknesses:\n1) The paper lacks significant novelty. The authors claim to use NELBO as a loss function, but this is not a new approach, as many prior works have already utilized similar loss functions, such as in Bartosh et al. (2023) [1]. MDLM itself was proposed last year in Sahoo et al. (2025) [2]. Furthermore, MCTS has been used for calculating rewards in discrete models since 2017, as demonstrated by Yu et al. (2017) [3], and there are several papers that have applied MCTS to diffusion models, such as Yoon et al. (2025) [4].\n2) The paper lacks essential comparisons with baseline models, such as Wang et al. (2024) [5]. Additionally, the authors did not provide the necessary resource code, which undermines the credibility of the model.\n3) The experimental results are incomplete and do not outperform existing models. The experiments only measure two statistical metrics (validity and uniqueness), and the novelty metric is not provided. Moreover, the results do not surpass those of Data and PepMDLM, indicating that the proposed method does not offer significant improvements over such models.", "other_comments_or_suggestions": "None", "questions_for_authors": "See Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Every reviewer agrees on the acceptance due to its methodological and application contribution.  I also recommend its acceptance. 

This is a suggestion, but it appears that this work did not cite several highly related works, such as guidance in discrete diffusion models or methods [a] that can easily be adapted to a discrete diffusion model [b]. While the focus there is not on multi-objective optimization, it proposed a searchish based method in discrete diffusion models. I believe it should be cited in the final version. 

[a] LI, Xiner, et al. Derivative-free guidance in continuous and discrete diffusion models with soft value-based decoding. arXiv preprint arXiv:2408.08252, 2024.

[b] WU, Luhuan, et al. Practical and asymptotically exact conditional sampling in diffusion models. Advances in Neural Information Processing Systems, 2023, 36: 31372-31403.
Decision: Accept (poster)