Aligning Target-Aware Molecule Diffusion Models with Exact Energy Optimization
Authors: Siyi Gu
Minkai Xu
Alexander S Powers
Weili Nie
Tomas Geffner
Karsten Kreis
Jure Leskovec
Arash Vahdat
Stefano Ermon
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion Models
Reinforcement Learning from Human Feedback
Computational Biology
Primary area: machine_learning_for_other_sciences_and_fields

Abstract: Generating ligand molecules for specific protein targets, known as structure-based drug design, is a fundamental problem in therapeutics development and biological discovery. Recently, target-aware generative models, especially diffusion models, have shown great promise in modeling protein-ligand interactions and generating candidate drugs. However, existing models primarily focus on learning the chemical distribution of all drug candidates, which lacks effective steerability on the chemical quality of model generations. In this paper, we propose a novel and general alignment framework to align pretrained target diffusion models with preferred functional properties, named AliDiff. AliDiff shifts the target-conditioned chemical distribution towards regions with higher binding affinity and structural rationality, specified by user-defined reward functions, via the preference optimization approach. To avoid the overfitting problem in common preference optimization objectives, we further develop an improved Exact Energy Preference Optimization method to yield an exact and efficient alignment of the diffusion models, and provide the closed-form expression for the converged distribution. Empirical studies on the CrossDocked2020 benchmark show that AliDiff can generate molecules with state-of-the-art binding energies with up to -7.07 Avg. Vina Score, while maintaining strong molecular properties. Code is available at https://github.com/MinkaiXu/AliDiff.

Aligning Target-Aware Molecule Diffusion Models with Exact Energy Optimization
Siyi Gu Minkai Xu Alexander Powers', Weili Nie², Tomas Geffner? Karsten Kreis², Jure Leskovec', Arash Vahdat", Ermon
Stanford Univeristy NVIDIA (sgu33, minkai, jure, ermon) stanford edu Ixpowers@stanforddc edu {wnie, tgeffner, kkreis, avahdat]@nvidia. com
Abstract
Generating ligand molecules for specific protein targets, known as structure- based drug design, fundamental problem in therapeutics development and biological discovery. Recently, target-aware generative models, especially diffusion models, have shown great promise in modeling protein-ligand interactions and generating candidate drugs. However, existing models primarily focus on learning the chemical distribution of all drug candidates, which lacks effective steerability the chemical quality of model generations. In this paper, propose a novel and general alignment framework to align pretrained target diffusion models with preferred functional properties, named ALIDIFF. ALIDIFF shifts the target-conditioned chemical distribution towards regions with higher binding affinity and structural rationality, specified by user-defined reward functions, via the preference optimization approach. To avoid the overfitting problem in common preference optimization objectives, Preference Optimization method to yield an exact and efficient alignment of the diffusion models, and provide the closed-form expression for the converged distribution. Empirical studies on the CrossDocked20200 benchmark show that ALIDIFF can generate molecules with state-of-the-art binding energies with up to -7.07 Avg. Vina Score, while maintaining strong molecular properties. Code is available https: pectps://githhe /hu.. com homm//innaiiuuaaa Allifff.
further develop improved Exact Energy
Introduction
Generating ligand molecules with desirable properties and high affinity to specific protein targets, known as structure-based drug design (SBDD), is a fundamental problem in therapeutic design and biological discovery. It necessitates methods that produce realistic and diverse drug-like molecules with stable 3D structures and high binding affinities. In the past few years, numerous deep generative models have been proposed generate molecules in SMILES string representation [Kusner et al., 2017, Segler et al., 2018] or graph representations [Jin et al., 2018, Shi et al., 2020, Guan et al., 2023]. Although these models have shown promise in generating plausible drug-like molecules, they lack sufficient modeling of the 3D potein-ligand interaction with proteins and therefore can hardly be adopted target-aware molecule generation. As result, generating ligands conditioned on protein targets remains an open research problem.
Recently, with rapid progress in structural biology and the increasing scale of structural data [Fran- coeur et al., 2020, Jumper et al., 2021] 12 numerous target-aware generative models have been proposed directly generate molecules within the protein targets in 3D. Initial work proposed to sequentially
'Equal contribution; junior author listed earlier. *Coreesponencee to: Minkai Xu <minkai @ cs.stanford.edu>
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

place atoms within the target via autoregressive models [Luo et al., 2021, Liu et al., 2022, Peng al., 2022]. while later work learns diffusion models to jointly design the whole ligand with state-of- the-art results [Guan et al., 2023, Lin et al., 2022, Schneuing et al., 2023, Huang et al., 2023, Guan et al., 2024]. Following the biological principle to model the coteinn-ligand complex interactions, these methods have shown great promise in generating realistic drugs that can bind toward given targets. However, all existing models solely focus on learning the chemical distribution of candidate molecules and treat all training samples equally, while in practice, only the ligand molecules with strong binding affinity and high synthesizability are preferred for real- real-world therapeutic development. As result, existing learned models generally lack sufficient steerability regarding the relative quality of model generations and cannot generate faithful samples with the desirable properties.
To bridge the gap between existing SBDD mod- els and the necessity for designing ligands with favorable properties, in this paper, we intro- duce a novel and comprehensive alignment framework to align pretrained target-aware dif- fusion models with preferred functional prop- erties, named ALIDIFF. ALIDIFF adjusts the turget-conditione chemical distribution toward regions characterized by lower binding energy and structural rationality, as specified by a user- defined reward function, using preference opti- mization approach. To this end, we derive a uni- fied variational lower bound align the likeli- hoods of both discrete chemical type and contin- uous 3D coordinate features. We further analyze the winning data overfitting problem commonly associated with preference optimization objec- tives, and introduce an improved Exact Energy Preference Optimization (E'PO) method. EọPO analytically ensures precise and efficient aligmment of diffusion models, and we provide a closed- form expression for the converged distribution. Our key contributions can be summarized as follows:
High Reward Molecule
Low Reward Molecuie
Po (m (p)
Po (In' (p)
Protein Structure
Incresse
Decreasse
Likelihood
Likelhood
Figure 1: High-level illustration of ALIDIFF. For a protein target, we can have multiple candidate ligands and rank the preference by certain reward functions, e.g., binding energy. We align the target- aware molecule diffusion model with these prefer- ences by adjusting the conditional likelihoods.
We address the challenge of designing favorable target-aware molecules from the perspective of aligning molecule generative models with desirable properties. We introduce the energy preference optimization framework and derive variational lower bounds to align diffusion models for generating molecules with high binding affinity to binding targets.
We analyze the overfitting issue in the preference optimization objective, and propose an improved exact energy optimization method yield an exact alignment towards target distribution shifted by reward functions.
We conduct comprehensive comparisons and ablation studies on the CrossDocked2020 [Fran- coeur et al., 2020] benchmark to justify the effectiveness of ALIDIFF. Empirical results demonstrate that ALIDIFF can generate molecules with state-of-the-arr binding energies with up to -7.07 Avg. Vina Score, while maintaining strong molecular properties.
2 Related Work
Structure-Based Drug Design. With increasing amount of structural data becoming accessible, generative models have attracted growing attention for cruuctur-based molecule generation. Early research [Skalic al., 2019] proposes to generate SMILES representations from protein contexts by sequence generative models. Inspired by the progress in 3D and geometric modeling, many works proposed solve the problem directly in 3D space. For instance, Ragoza al. [2022] vox- elizes molecules within atomic density grids and generates them through Variational Autoencoder framework. Luo et al. [2021], Peng et al. [2022]. Liu et al. [2022], Powers al. [2023] developed autoregressive models to generate molecules by sequentially placing atoms or chemical groups within the target. Following the autogressive backbone, FLAGG/Zhang et al., 2023) and DrugGPS [Zhang and Liu, 2023] take advantage of chemical priors of molecular fragments to generate ligand molecules piece by piece, leading to more realistic substructures. More recently, diffusion models achieved exceptional results in synthesizing high-quality images and texts, which have also been successfully
2

---

Preference Pair Selection
High Rewward Molecule
8.2
Diffusion
Denoising
-7.3
-42
Milj
noise
E
Motecule
Diftusion
Denoising
Figure 2: Overview of ALIDIFF. This workflow can be summarized as 1) For each protein target (pocket) p the training set, retrieve candidate ligands m; Label the ligands wining sample m" and losing sample m' by desirable properties, e.g. binding energies; 3) Calculate the preference optimization objective Equation (12) and update the molecule diffusion model pg.
used for ligand molecule generation [Guan et al., 2023, Lin et al., 2022, Schneuing et al., 2023, Huang et al., 2023, Guan et al., 2024]. These models generate molecules by progressively denoising atom types and coordinates while maintaining physical symmetries with -qu3)-equi neural networks. While the existing works focus on designing molecules using various deep generative models, they often struggle with generating molecules that exhibit different desirable properties, e.g., strong binding affinity, high synthesizability, and low toxicity. Real-word drug discovery projects almost always seek to optimize constrain these properties (D) Segall, 2012, Bickerton et al., 2012]. In this work, we aim to address the challenge with a novel and general preference optimization framework.
Reinforcement learning from human feedback (RLHF), Recently, significant efforts have been devoted to aligning generative models with human preferences. The use of reinforcement learning to incorporate feedback from humans and Al into finetuning large language models is exemplified by Reinforcement Learning from Human Feedback (RLHF) [Ziegler et al., 2020, Ouyang al., 2022]. Research works have incorporated human feedback to improve performance across various domains, such machine translation [Nguyen et al., 2017], summarization [Stiennon al., 2020), and also diffusion models [Uehara et al., 2024b,a4 Notably, Rafailov et al. [2023] designed a new preference paradigm that enables training language models to satisfy human preferences directly without reinforcement learning. This algorithm was later applied to diffusion models for text-to-image generation tasks [Wallace al., 2023]. Coneurrent work [Zhou et al., 2024] attempts to apply DPO for designing antibodies with rationality and functionality. To the best of our knowledge, we are the first alignment approach for target-aware ligand design, where the conditional distribution is shifted toward desirable properties.
3 Method
In this section we present ALIDIFF, a general framework for aligning target-aware diffusion models with various molecular functionalities. We first provide an overview of the target-aware ligand diffusion model and our Reinforcement Learning from Feedback formulation (section 3.1). Next, we introduce the energy optimization approach for aligning the diffusion model and analyze the potential limitations of the framework (section 3.2). We then further introduce an exact energy optimization method from distribution matching perspective to align the generative model efficiently and exactly (section 3.3). visualization of the framework shown in Figure
3.1 Overview
Notation. We focus on aligning molecule generative models for structure- based drug design, which can be abstracted as generating molecules that bind to a given protein target. Following the

---

convention in the related literature [Luo et al., 2021, Guan et al., 2023], the molecule and target ( (6)
(8) (6) M and P {(xp p Di respectively, where Nar and Np denote the number of atoms of the molecule M and the protein P. x E R and IR
protein are represented as MM- {(x VM
denote the atomic 3D position and chemical type, respectively, with K being the dimension of atom types. For brevity, we denote the molecule as a matrix m = [XM, Varl where XM E IR x3 and VM E 1R x K and denote the protein as a matrix = [Xp, vpl where Xp E R RNxx3 and Vp R. RNpxK The task can then be formulated as modeling the conditional distribution p(mlp))
Preliminaries. Diffusion Models have been previously used to model the joint distribution of atomic types and positions [Guan et al., 2023, Schneuing et al., 2023, Lin et al., 2022]. This approach consists of a forward diffusion process and a reverse generative (denoising) process. Both processes are only defined on the ligand molecules m, with fixed proteins p. In the forward process, small Gaussian and categorical noises are gradually injected on atomic coordinates x and types v as follows:
((ma)m, p) = -N(x; v
C(v:: (1- - - - -
(1)
where N and C stand for the Gaussian and categorical distribution respectively, and Be corresponds to a (fixed or learnable) variance schedule. Note that, in certain recent work process can be learnable with dependence on the conditioning p [Huang et al., 2023]. We omit the subscript M for the ligand molecule without ambiguity here and denote the atom positions and types at time step & as x, and v Using Bayes theorem, the posterior conditioned on ma can be computed in closed form:
q(m: mm mo: p) = -x ju (x(x;; xo), B.1) C(v+-1;; :((v,, vo)):
where ix(x;, xo) &
1-a; Xx, 3. T Br, ar = 3,, a. II. Oce. c" and c* (V,. Vo) = [aov, + (1 ō(ve; Vo) Σt= oi
l-a: Xo yā, (1-a,-1)
(1-0011)/ K] [Ho et al., 2020, Austin et al., 2021]. At timestep T, q converges to the prior with Gaussians on coordinates and uniforms on atom types. The reverse process, also known as the generative process, learns neural network parameterized by 0 to recover data by iterative denoising. The denoising step can be approximated with predicted Gaussians pso and categorical distributions co as follows:
po (m; | |m, p) (x Mo ( & vol. f, p), B.1) C(v--1;oo(x), velit,
(3)
= -N (x &n). B.T) C(v.-1;((v, vo)),
where [&o. vol = en (xx, vel. t. p) are predictions from a denoising network ep. Importantly, the denoising network here is specifically parameterized by equivariant neural networks, resulting in an SE(3)-invariant likelihood po (m p) on the protein-ligand complex [Xu et al., 2022].
Overview. As ligand molecules with desirable properties, e.g., high binding affinity and synthesizabil- ity, are required for real-world therapeutic development, we aim to align the ligand diffusion model with these preferences. Such preferences can be defined as a reward model r(-) M x P + R calcu- lated from various cheminformaties software, e.g., binding affinity, drug-likeness, synthesizability, or their combinations. We fine-tune and align the pre-trained diffusion model with the reinforcement learning framework. Specifically, given a dataset D containing given protein targets, inspired by RLHF [Ouyang et al., 2022], this fine-tuning is achieved by maximizing the reward:
max Ep-D.mmpl Pe
[r (m, p)] 3TD (po (ppomm) (m) Ip) (m)p))
where Pe and Pref are the distributions induced by the fine-tuned and pre-trained models, respectively. In this work, Pa and Pret are the fine-tuned and pre-trained molecule diffusion models, as introduced above. 3 a hyperparameter controlling the KL divergence regularization. Note that, here the reward is a known black-box function, unlike typical RLHF where it is unknown and has to be estimated from preferences. In the following section, we claborate on how the alignment objective is rewritten with diffusion forward and reverse processes defined atomic types and coordinates.
3.2 Energy Preference Optimization
Though the reward function is known, evaluating reward values such as binding affinity is com- putationally expensive and we instead resort to aligning with a labeled offline dataset. We start with a dataset D {(p, m", m²)) where p denotes the protein condition and m" " > m' is a pair of winning and losing ligands with respect to certain specified energy, e.g., binding energy.

---

The optimal solution to the RLHF objective from Equation (4) can be written in closed-form (5(m)p) oc (re((mp)) exp( (mm p)) [Peters and Schaal, 2007]. Following the preference opti- mization algorithm [Rafailov al., 2023], we use the Bradley Terry (BT, [Bradley and Terry, 1952])
model p(m) m p) = (r((m)) p) r(mm, p)) reformulate the RLHF objective as:
Po ( Ip) Pret ( (p)
Po (mmm) (p) Peet (m/) (p) )
CDPO - (p. m')>
log
log
3 log
(5)
Due to the intractability of pe (m)p) for diffusion models, we instead follow recent work on diffusion- based preference optimization [Wallace et al., 2023] to align the whole reverse process and utilize Jensen's 's inequality to optimize its negative evidence lower bound optimization (ELBO):
log Pe (m Pref (.T))
(mm 0:7
LDPO-Diffusion (0) = - -E (p.mg "m
log a
B log
Peef (m::T)
where omit the conditioning the protein target p for compactness. We further approximate the reverse process pe ( mo) with the forward process q(m mo) ) for efficient sampling of m1.7. and obtain the following expression after some derivations [Wallace et al., 2023]:
EpPO-Diftusion (0) (p (p.my
logo - 3T (D (q (m, (m; (m"")) - DKL
(q (m mlPee (m] (m"))
llpp:pllpmmmmmmm |mm)) + Dxx (g(m) |m [llpPer((i (m/-1 (mm
Let [8o: Vol be the predicted atom position and type, which are fed into Equation (3) to obtain posterior distributions. With the joint diffusion processes Equations (1) to (3) both continuous and discrete v features, the above KL divergences can be decomposed and calculated as:
DKL
= 1x0
+ DXL 1.v.4-11 ((((er1
1
DEL
= || xo) - Mxxe, xo) C x| xo - koll +C,
Be
DMM (ale//llaaaa)alllleeelo = Σ ē(ve. vo)a log o(vi, vo) k
o(Vi, vo)x
where (1- and C is a constant. Let &0,9. Vo.o and Ko.ee. Vo,eer be the predictions from the fine-tuned and from the original pretrained model, respectively. Then, we can further obtain the preference optimization loss on x and v, respectively, as follows:
188
(0) - E log of ( ( - ,,, llvv( vo,,)) DKI vo)l,, vor,er)
||x"" x -1x6 x 1|x %o
1)]]
CY_, (0) -E [I[
- DDL vo)lle(v) vo,)) + DKL (É(v), vo)ll v).re))))]
(9)
With Jensen's s inequality and the convexity of log a, we can derive the final objective as (weighted)
sum of atom coordinate and type preference losses LX + LY 1. which turns the sum of the KI. terms outside- log a and serves as an upper bound of Equation (7):
CALDDIF () = - (p.m. m.
mq [A_] + __1] 2 EDPO LDPO.DiITusion
DifTusion (0),
(10)
where the preference is assigned separately to atom types v and coordinates The loss decomposition imposes a fine-grained preference assignment on chemical elements and geometric strcutures and enables to choose weights to balance the training of the two variables [Guan et al., 2023, 2024]. The overall training and sampling algorithms of ALIDIFF are summarized in Appendix B.
3.3 Exact Energy Optimization
Although DPO enjoys the advantage of efficient fine-tuning without fitting reward function, recent theoretical investigations reveal that it is highly vulnerable to overfitting by pushing all the probability mass the winning sample [Azar et al., 2024]. Specifically, the non-linear transformation log

---

of Equation (5) pushes the log Pa (m log (mm'm p) towards infinity, completely removing the likelihood for the losing sample regardless of any regularization in the original RLHF setup Equation (4) [Azar et al., 2024, Tang et al., 2024]. Let us analyze the problem with an example consisting of two ligand molecules m T and m' with their rewards measured as r" and r' (e.g.. calculated from binding energy). The DPO objective in Equation (10) tends to just greedily maximize towards p(m" > m'(p) + 1. However, the optimal preference probability can be calculated by the BT model [Bradley and Terry. 1952] as p(m" > m' pp) = orr -r), r). and our alignment goal is to shift the distribution to align with this 5 instead of greedy maximization. To address the wvr-opptiiiztiinn issue, we introduce an improved objective with regularization on preference maximization, named Exact Energy Preference Optimization (F2) Let L% (0) and Ly (0) denote terms for reverse preference optimization:
(0) = 1 - 1 L = 1 ,
(11)
Our EPPO objective function takes cross-entropy form align the distributions po (m > m (p) towards f(mm th m ml)p). Formally, it is given by:
L EAUDII (0) = - -B mp mb)=D:;-10.T],m/ -.
(12)
(o(r" (r(r--r))) (f
+ (1 (1- -01-0(r3 --
where the second term 1=_ + EYー weighted by 1 - o(r - r²) helps to alleviate the overfitting
on the winning data sample. Notably, for r >> r', we have o("" - = 1, indicating that the regularized objective in Equation (12) will still change back to the original objective in Equation (10), where overfitting on the extremely better data is expected. In principle, with the regularization objective, we have:
Theorem 3.1. The finctiono in Equation (12) optimizes a variational upper bound of the KL- divergence DKI.. (6" (m/p) (mlp))léo(mmp)) where 5a (mip) x (mpp exp(r(m, (m. p)) and bo (m p)) x
Piet Pre((m|p) (p)
(o(m))) (m)p))
The theorem provides an analytical guarantee for the optimal shifted distribution after alignment that avoids over-optimization. Assuming we achieve convergence on the KL divergence, we have that (mr(mmp)) (p) exp(r(m,, x Pief (m/p)) (mlp) which further gives us pe (mip) oc (ppp)) exp( (mm, p)), where a smaller B encourages a sharper shift towards the user-defined reward function. We give the full derivations in Appendix C, and analyze the empirical effect on generation quality in Section 4.
4 Experiment
4.1 Experiment Setup
Dataset. We train and evaluate ALIDIFF using the CrossDocked2020 dataset [Francoeur et al., 2020]. Following the common setup in this field [Luo et al., 2021, Guan et al., 2023]. we refined the initial 22.5 million docked protein binding complexes by selecting docking poses with RMSD lower than IÅ with the ground truth and diversifying proteins with a sequence identity below 30%. To apply ALIDIFF, we further preprocess our data and construct a dataset of the form D = ((p, m m))] where p denotes the protein, m " denotes the preferred molecules, and m' denotes rejected molecules based on the user-defined reward. In our setting, we choose two ligand molecules per pocket site and label the preference by a certain reward, e.g. binding energy for our main benchmark. We provide ablations with more reward functions in Section 4.3. Details of preference pair selection are presented in Appendix E. The final dataset uses a train and test split of 65K and 100.
Baselines. We compare our model with the following baselines: liGAN [Ragoza et al., 2022] is conditional VAE model that utilizes 3D CNN architecture to both encode and generate voxelized representations of atomic densities; AR [Luo et al., 2021], Pocket2Mol [Peng et al., 2022] and GraphBP [Liu et al., 2022] are autoregressive models that learn graph neural networks to generate 3D molecules atom by atom sequentially; TafgetDiff [Guan et al., 2023] and DecompDiff [Guan et al., 2024] are diffusion--asse approaches for generating atomic coordinates and types via a joint denoising process; IPDiff [Huang et al., 2023] is the most recent state-of-the-at diffusion-baut

---

Table 1: Summary of binding affinity and molecular properties of reference molecules and molecules generated by ALIDIFF and baselines. (1) (L) denotes whether larger smaller number preferred. Top 2 results are bolded and underlined, respectively.
Methods
Vina Score (1) Avg.
Vina Min (1) Avg.
Vina Dock (1)
High Affinty(t)
QED(T)
SA(1)
Diversity(f))
Med.
Med.
Avg.
Med.
Avg.
Med.
Avg.
Med.
Avg.
Med.
Avg.
Med.
lICAN*
-6.33
-6.20
0.39
0.57
0.67
GraphBP
-480
4.70
14.2%
6.7%
0.43
0.45
0.49
0.48
0.79
0.78
AR
5.75
-5.64
-6.62
0.70
PocketMMll
-5.14
-4.70
-6.42
-5.82
-7.15
-6.79
48.4%
51.0%
0,56
0.57
0.74
0,75
0.69
0.71
TargetDiff
-5.47
-6.30
-6.83
-7.80
-7.91
39.19
0.48
DecompDift
-5.67
-6.04
-7.04
-7.09
-8.39
-8.43
64.4%
71.0%
0.45
0.43
0.61
0.60
0.68
0.68
IPDdI
-6.42
-7.01
-7.45
-7.48
-8.57
-8.51
69.5%
75.5%
0.52
0.53
0.61
0.59
0.74
0.73
ALIDIFF
-7.07
-7.95
-8,09
-8.17
-8.90
-8.81
73.4%
81.4%
0.50
0.50
0.57
0.56
0.75
0.71
Reference
-6.36
-6.41
-6.71
-6.49
-7.45
-7.26
0.48
0.47
0.73
0.74
Median Vina Energy
-12
target
target
target 20
target 30
target 40
50
target 60
target 70
target
target
TargetDiff (lowest in 9%)
IPDIf (lowest 42%)
Ours (lowes in 49%)
Figure 3: Median Vina energy for different generated molecules (TargetDiff, IPDiff, ALIDIFF) across 100 testing samples, sorted by the median Vina energy of molecules generated from ALIDIFF.
approach that further integrates the interactions between the target protein and the molecular ligand into the generation process.
We evaluate the generated molecules by comparing binding affinity with the target and critical molecular properties. We analyze the generated molecules across 100 test proteins, reporting the mean and median for affinity-based metrics (Vina Score, Vina Min, Vina Dock, and High Affinity) and molecular property metrics dirug-likeness QED [Bickerton al., 2012], synthesizability SA [Ertl and Schuffenhauer, 2009], and diversity). We use AutoDock Vina [Eberhardt et al., 2021] to estimate binding affinity scores, using the common setup described by Luo et al. [2021]. Ragoza et al. [2022]. Specifically, Vina Score estimates binding affinity from the generated 3D structures, Vina Min refines the structure through local minimization before estimation, Vina Dock uses a re-docking procedure to reflect the optimal binding affinity, and High Affinity gauges the percentage of generated molccules that bind better than reference molecules per protein.
Evaluation metries.
4.2 Results
Binding Affinity and Molecular Properties. We compare the performance of our proposed method ALIDIFF against the above baseline methods. Our model is fine-tuned from IPDiff, the ligand generative model. We report the results Table 1, and leave more implementation details in Appendix D. As shown in the results, ALIDIFF significantly outperforms all con-diffusion--asee models in binding-relann related metrics, and also surpasses our base model IPDiff in all binding affinity related metrics by notable margin. In particular, ALIDIFF increases the mading--ellteed metrics Avg. Vina Score, Vina Min, and Vina Dock by 10.1 %, 8.5 8.56%, and 3. 3.9% compared with IPDiff. Our superior performance in binding-related metrics demonstrates the effectiveness of energy preference
Reference
TangetDiff
IPDif
Ours
Reference
TargetDiff
IPDIff
Ours
Wac-750G 37P3R
Wes NE--LLIOEE 2318
-LA-LLLOOER LEPALED
Figure 4: Visualizations of reference molecules and generated ligands for protein pockets (1131, 2e24) generated by TargetDiff, IPDiff, and ALIDIFE. Vina score, QED, and SA are reported below.
7

---

Table 2: Effect of combining multiple reward objectives. Affinity denotes ALIDIFF, whereas
AfA denotes combining both synthetic accessibility and affnnity as reward function.
Choice reward
Score (1)
Vina Min (L)
Vina Doek (L)
High Affinity(t)
QED(T)
SA(f)
Diversity(
Avg.
Med.
Avg.
Med.
Avg.
Med.
Avg.
Med.
Avg.
Med.
Avg.
Med.
Avg.
Med.
Affinity
-7.07
-7.95
-8.09
-8.77
-8.90
-8.11
73.4%
81.4%
0.50
0.50
0.57
0.56
0.73
0.71
AffinitySSA
-6.87
-7.76
-8.00
-8.08
-8.1
-8.22
72.7%
80.8%
0.52
0.55
0.60
0.59
0.74
0.73
Affinit+QED
-7.11
-8.02
8.001
-7.9
8..17
-8.72
73.7%
82.0%
0.52
0.57
0.57
0.73
0.73
Table 3: Comparison of DPO and E.POO with pretrained and supervised fine-tuned models. ALIDIFF with DPO takes energy ranking, and with E.POO uses exact energy for preference optimization.
Methods
Vina Scure (1)
Vina Min (1)
Vina Dock (1)
High Affinity(f)
QED(T)
SA(T)
Diversity(f)
Avg.
Med.
Avg.
Med.
Avg.
Med.
Avg.
Med.
Avg.
Med.
Avg.
Med.
Avg.
Med.
IPDif
6.42
-7.01
-7.45
-7.48
-8.57
-8.51
69.5%
75.5%
0.52
0.53
0.61
0.59
0.74
0.73
IPDITT
6.53
.6.2
-7.27
-7.99
-8.14
-8.09
67.5%%
72.5%
0.48
0.48
0.61
0.59
0.72
0.69
ALIDIFY-DPO
.6.11
7.62
-7.75
.7.79
-8.88
8.55
69.7%
71.1%
0.50
0.51
0.56
0.56
0.74
0.72
ALIDIFF-E'PO
-7.07
-7.95
-8.09
-8.17
-8,90
-8.81
73.4%
81.4%
0.50
0.50
0.57
0.56
0.73
0.71
optimization. Figure 3 shows the median Vina energy of the proposed model, compared with TargetDiff and IPDiff, two diffusion-basee state-of-the-an models in target-aware molecule generation. We observe that ALIDIFF surpasses these baseline models and generates molecules with the highest binding affinity for 49% of the protein targets in the test set. In property-related metrics, we observe only slight decrease in QED, SA, and diversity, compared with IPDiff. Specifically, with approximately 10.1 % improvement on Avg. Vina Score, we observe a minor decrease in Avg. SA (-6.5%), Avg. QED (-3.8%), and diversity(-1.4%) Figure 4 presents examples of ligand molecules generated by ALIDIFF, Diff and IPDiff. The figure shows that our generated molecules maintain reasonable structures and high binding affinity compared with all baselines, indicating their potential as promising candidate ligands. Additional experimental results and visualized examples of these molecules are Appendices E and F.
We also notice a trade-off between binding affinity and property-related metries. While we achieve state-of-the-art performance all binding affinity metries, the performance on QED and SA metrics slightly decreases. This phenomenon has been commonly observed in previous studies where achieving high binding affinity can often sacrifice other molecular metries [Guan et al., 2023, Huang et al., 2023]. This is because the highest affinity can potentially only be achieved by rather specific and unique molecules, which are harder synthesize than simple molecules, and hence these trade-offs are expected. Besides, in real-world drug discovery, binding affinity is typically a more critical metric as molecules with more stable interaction with the pocket site are important, whereas QED and SA work mainly as rough filters [Guan et al., 2023]. For these reasons, we believe the deterioration in molecular properties well compensated by the improvement in binding affinity, especially with such little deterioration in property metrics. In addition, in the following section (Table 2), we further discuss incorporating molecular properties into the reward, which shows slightly lower performance gain on affinity but archives improvements also on molecular properties.
4.3 Ablation Studies
Effect of reward objectives. To further explore the potential of ALIDIFF, we evaluate the effect of combining optimization objectives (r I'fffinity + rSA: Fnffinityy EQED) and investigate whether such a combined reward function can lead to better molecular properties to counter the trade-off we discussed before. As shown in Table the results indicate that finetuning solely with binding affinity apparently achieves better performance in terms of binding affinity metrics. However, ALIDIFF- AffinitySSA generates compounds with better drug-likeness (QED) and synthetic accessibility (SA). Both models exhibit similar performance in terms of structural diversity. This suggests that while ALIDIFF-Affnity is superior for binding affinity, incorporating synthetic accessibility considerations (Affinity + SA) results in compounds that are more drug- drug-like and easier to synthesize, enabling more efficient multi-objective drug development. Moreover, ALIDDFF-Affinity-- binding affinity compared with ALIDIIF--ffinity while the improvement in QED is relatively minimal. Thus, balancing these objectives highlights the potential for overcoming trade-offs in molecular optimization.
achieves better
8

---

014
Affinty
955
Vine -.4
Aeg
Med
Hal
Het
leastte
basine
3350
942
berelte
20
barelte
12
10
Figure 5: Ablation analysis of ALIDIFF under different B. Vina Score, High Affinity, QED, and diversity are reported, where blue lines represent AL.DDIFF-DPO, and orange lines represent ALIDIFF. The dotted lines represent the baseline IPDiff.
Comparison with Supervised Fine- Tine-Tuning, Supervised Fine-Tuning (SFT) serves as an alternative method for generating molecules with user-defined optimization objectives. We seleet the top 50% protein-ligand samples with higher quality in user-defined reward from the training dataset and fine-tune the baseline model with the same training and sampling setting. The results in Table show that SFT did not show improvement over the baseline, and ALIDIFF demonstrates significantly superior results compared to SFT.
Effect of preference optimization methods. As discussed in Section 3.3, the original DPO objective is vulnerable to overfitting and we propose to avoid it with regularization by weighting preference losses with the user-defined rewards. We compare the direct use of energy preference optimization by ranking molecule pairs (ALIDIF--DPO) and exact energy optimization with user-defined reward function (ALIDIFF-EPPO in Table 3. The results show that ALIDIFF-EPPO achieves superior performance over ALIDIFE-DPO in all binding affinity metrics (Vina Score, Vina Min, and Vina Dock) while maintaining competitive scores in QED, SA and diversity. In terms of drug-likeness and structural diversity, ALIDIFF-EPPO performs competitively, indicating that while prioritizes binding affinity, it still maintains favorable drug-like properties and diversity. This further supports our previous hypothesis regarding the trade-off between binding affinity and molecular properties. An additional ablation study on the effect of exact energy optimization is presented in Appendix E.
General applicability to ligand diffusion models. We further justify the general applicability of the proposed ap- proach by finetuning another iefffusion-bbsedd SBDD model, TirgetDiff [Guan et al., 2023], with exact energy optimiza- tion (ALIDIFE-T), As shown in Table 4, ALIDIFF-T sur- passes Tiff on all binding affinity and molecular properties, with 6.2%, 16.6%, 6.9%, 2.8% increase in Avg. Vina Score, QED, SA, and diversity, respectively. The results further justify that our approach is generally applica- ble to diffusion- based SBDD models. Notably, ALIDIFF-T archives even better QED and SA compared with ALIDIFF, which allows users to choose the model based on the spe- cific purpose for molccular properties. Also, we notice the percentage of improvement of binding affinity from Target- Diff to ALIDIFF-T slightly lower than that from IPDiff to ALIDIFF. This can be explained as preference optimization is more effective when the model distribution is more similar to the prefer- ence data distribution, and IPfiff is shown to fit CrossDocked data better than TirgetDiff [Huang et al., 2023].
Table 4: Finetuning TargetDiff with ALIDIFF. ALIDIFF-T denotes fine- tuned model with the same reward objec- tive on TargetDiff.
Metric
Tiff Avg.
ALIDIFF-T
Med.
Avg.
Med.
Vina Score
-5.47
-6.30
-5.81
-6.51
Vina Min
-6.64
-6.83
-6.94
-7.01
Vina Dock
-7.80
-7.91
-7.92
-7.97
QED SA Diversity
0.48
0.48
0.56
0.56
0.58
0.58
0.62
0.60
0.72
0.71
0.74
0.75
Strength of 3. We further evaluate ligand molecules generated by ALIDIFF trained with varying 3 values in Figure 5. Recall that 3 influences the scale of energy preference optimization and regularization with respect to the reference model. The results indicate a clear trade-off between binding affinity and molecular properties with varying 3. Lower 3 values (c.g.. 0.01) significantly enhance binding affinity metrics (Vina Score, Vina Min, Vina Dock), but at the cost of lower drug-likened (QED) and diversity. Conversely, higher 3 values improve QED, suggesting that these configurations generate more drug-like compounds while maintaining consistent synthetic accessibility and diversity. We believe / reaches an equilibrium around 3 = 1. where binding affinity maximized without sacrificing too much loss in molecular properties. This ablation study demonstrates that the parameter 3 can offer a useful tool to train ALIDIIF models with different desired trade-offs between binding affinity and useful molecular properties, which can vary for different drug development use cases.

---

Conclusion
In this paper, we present ALIDIFF, a novel framework to align pretrained target-aware molecule diffusion models with desired functional properties via preference optimization. Our key innovation the Exact Energy Preference Optimization method, which enables efficient and exact aligmment of the diffusion model towards regions of lower binding energy and structural rationality specified by user-defined reward functions. Extensive experiments on the CrossDocked2020 benchmark demonstrate the strong performance of ALIDIFF. By incorporating user-defined reward functions and an improved Exact Energy Preference Optimization method, ALIDIFF successfully achieves state-of-the-ar performance in binding affinity while maintaining competitive molecular properties. In the future, we plan to explore more expressive molecular reward function classes within our framework and extend ALIDIFF to real-world prospective drug design settings by integrating into online drug discovery pipelines.
Acknowledgement
We thank Jiaqi Han for the discussions on this project. We gratefully acknowledge the support of ARO (W91 1992 ONR (N00014-23-1-2159) NVIDIA, and Chan Zuckerberg Biohub. We also gratefully acknowledge the support of NSF under Nos. OAC-1835599 (CINES), CCF-1918940 (Expeditions), 232-7227709 (THBEM), IIS-2403318 (II);; Stanford Data Applications Initiative, Wu AI, Chan Zuckerberg Initiative,
Tsai Neurosciences Institute, Stanford Institute for Amazon, Genentech, GSK, Hitachi, SAP, and UCB. Minkai Xu thanks the generous support of Sequoia Capital Stanford Graduate Fellowship.
References
Jacob Austin, Daniel Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference Artificial Intelligence and Statistics, pages 4457-4455. PM 2024.
Richard Bickerton, Gaia Paolini, Jérémy Besnard, Sorel Muresan, and Andrew L Hopkins. Quantifying the chemical beauty of drugs. Nature chemistry, 4(2):90-98, 2012.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: L. the method of paired comparisons. Biometrika, 39(3/4):324-344 1952.
Huayu Chen, Guande He, Hang Su, and Jun Zhu. Noise contrastive alignment of language models with explicit rewards. arXiv preprint arXiv:2402.053699 2024.
Matthew D Segall. Multi-parameter optimization: identifying high quality compounds with balance of properties. Current pharmaceutical design, 1299--122--3300 2012.
Jerome Eberhardt, Diogo Martins, Andreas F Tillack, and Stefano Forli. Autodock vina 1.2. O: New docking methods, expanded force field, and python bindings. Journal of chemical information and modeling, 61(8):3891-38 3898, 2021.
Peter Ertl and Ansgar Schuffenhauer. Estimation of synthetic accessibility score of drug-like molecules based molecular complexity and fragment contributions. Journal of cheminfor- matics, 1:1-11, 2009.
Paul G Francoeur, Tomohide Masuda, Jocelyn Sunseri, Andrew Jia, Richard lovanisci, Ian Snyder, and David R Koes. Three- convolutional neural networks and cross-docked data set for structure-based drug design. Journal of chemical information and modeling, 60(9))4200-42155 2020.
10

---

Richard A Friesner, Jay L Banks, Robert B Murphy, Thomas A Halgren, Jasna Klicic, Daniel T Mainz, Matthew P Repasky, Eric H Knoll, Mee Shelley, Jason K Perry, et Glide: new approach for rapid, accurate docking and scoring. 1. method and assessment of docking accuracy. Journal of 2004.
medicinal chemistry, 47(7):1739-
Jiaqi Guan, Wesley Wei Qian, Xingang Peng, Yufeng Su, Jian Peng, and Jianzhu Ma. 3d equiv- ariant diffusion for target-aware molecule generation and affinity prediction. arXiv preprini arXiv:2303.03543, 2023.
Jiaqi Guan, Xiangxin Zhou, Yuwei Yang. Yu Bao, Jian Peng. Jianzhu Ma, Qiang Liu, Liang Wang, and Quanquan Gu. Decompdiff: diffusion models with decomposed priors for structure-based drug design. arXiv preprint arXiv:2403.07902, 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-665 68501, 2020.
Zhilin Huang, Ling Yang, Xiangxin Zhou, Zhilong Zhang, Wentao Zhang, Xiawu Zheng, Jie Chen, Yu Wang, CUI Bin, and Wenming Yang. Protein-ligand interaction prior for binding-aware 3d molecule diffusion models. In The Twelfth International Conference on Learning Representations, 2023.
Haozhe Ji, Cheng Lu. Yilin Niu, Pei Ke, Hongning Wang, Jun Zhu, Jie Tang, and Minlie Huang. Towards efficient and exact optimization of language model alignment. arXiv preprint M.Xiv:2402.008566 2024.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In International conference on machine learning, pages 2323-2332.
PMLR, 2018
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger,
Kathryn Yunyasuvunakool,, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873) 2021.
Matt J Kusner, Brooks Paige, and José Miguel Hernández-Lobato. Grammar variational autoencoder. In International conference on machine learning, pages 1945-1954. PMLR, 2017.
Haitao Lin, Yufei Huang, Meng Liu, Xuanjing Li. Shuiwang Ji, and Stan Z. Li. Diffbp: Generative diffusion 3d molecules for target protein binding, 2022.
Christopher A Lipinski, Franco Lombardo, Beryl W Dominy, and Paul J Feeney. Experimental and computational approaches to estimate solubility and permeability in drug discovery and development settings. Advanced drug delivery reviews, 64:4-17, 2012.
Meng Liu, Youzhi Luo, Kanji Uchino, Koji Maruhashi, and Shuiwang Ji. Generating 3d molecules for target protein binding. arXiv preprint arXiv:2204.0944 2022.
Shitong Luo, Jiaqi Guan, Jianzhu Ma, and Jian Peng. A 3d generative model for structure-based drug design. Advances in Neural Information Processing Systems, 34:6229-6233 6239, 2021.
Khanh Nguyen, Hal Daumé III, and Jordan Boyd-Graber. Reinforcement learning for bandit neural machine translation with simulated human feedback. arXiv preprint arXiv:1707.074000 2017.
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback, 2022. URL https://arxiv. org/abx/203..02755, 13, 2022.
Xingang Peng, Shitong Luo, Jiaqi Guan, Qi Xie, Jian Peng, and Jianzhu Ma. Pocket2mol: Efficient molecular sampling based 3d protein pockets. In International Conference Machine Learning, pages 1 17644-17777 17655. PMLR, 2022.
Jan Peters and Stefan Schaal. Reinforcement learning by weight-weighted regression for operational space control. Proceedings of the 24th international conference on Machine learning, pages 745-750, 2007.
11

---

Alexander Powers, Helen Yu, Patricia Suriana, Rohan Koodli, Tianyu Lu, Joseph Paggi, and Ron Dror. Geometric deep learning for structure-based ligand design. ACS Central Science, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305. 2023.
Matthew Ragoza, Tomohide Masuda, and David Ryan Koes. Generating 3d molecules conditional on receptor binding sites with deep generative models. Chemical science, 13(9):7701-2713, 2022.
Arne Schneuing, Yuanqi Du, Charles Harris, Arian Jamasb, Ilia Igashov, Weitao Du, Tom Blundell, Pietro Lió, Carla Gomes, Max Welling, Michael Bronstein, and Bruno Correia. Structure-based drug design with equivariant diffusion models, 2023.
Marwin HS Segler, Thierry Kogej. Christian Tyrchan, and Mark P Waller. Generating focused molecule libraries for drug discovery with recurrent neural networks. ACS central science, 4(1): 120-131, 2018.
Chence Shi, Minkai Xu, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. Graphaf: flow-based autoregressive model for molecular graph generation. In International Conference on Learning Representations, 2020. URL wtw.htttss//peereiiiieeeee
Miha Skalic, Davide Sabbadin, Boris Sattarov, Simone Sciabola, and Gianni De Fabritüs. From target to drug: generative modeling for the multimodal structure-based ligand design. Molecular pharmaceutics, 16(10):4282 4291, 2019.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3000 3021, 2020.
Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2402.05749, 2024.
Masatoshi Uehara, Yulai Zhao, Kevin Black, Ehsan Hajiramezanali, Gabriele Scalia, Nathaniel Lee Diamant, Alex M Tseng, Tommaso Biancalani, and Sergey Levine. Fine-tuning of continuous time diffuuson models as retropy-regularrized control. arXiv preprint 24Xiv:24022.11944 2024a.
Masatoshi Uehara, Yulai Zhao, Kevin Black, Ehsan Hajiramezanali. Gabriele Scalia, Nathaniel Lee Diamant, Alex M Tseng, Sergey Levine, and Tommaso Biancalani. Feedback efficient online fine-tuning of diffusion models. arXiv preprint 162552240..16359, 2024b.
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. arXiv preprint arXiv:231 12908 2023.
Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. International Conference on Learning Representations, 2022. URL
Zaixi Zhang and Qi Liu. Learning subpocket prototypes for generalizable structure-based drug design. In International Conference on Machine Learning, pages 41382-41398. PMIR, 2023.
Zaixi Zhang, Yaosen Min, Shuxin Zheng, and Qi Liu. Molecule generation for target protein binding with structural motifs. In The Eleventh International Conference on Learning Representations, 2023.
Xiangxin Zhou, Dongyu Xue, Ruizhe Chen, Zaixiang Zheng, Liang Wang, and Quanquan Gu. Antigen-specific antibody design via direct energy- based preference optimization. arXiv preprint arXiv:2403. 165766 2024.
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint 1iXiv:1909.00 08593, 2020.
12

---

A Limitations and Future Work
While ALIDIFF exhibits promising performance, there are still potential limitations to our current approach. For example, ALIDIFF takes binding affinity as our reward function which is computed by AutoDock Vina [Eberhardt al., 2021] in this work. However, computing binding energy via software is an approximation and sometimes can be very inaccurate. In the future, we plan to explore experiment-measured energy or ensemble different binding affinity calculation software, egg,, GlideScore [Friesner et al., 2004] In addition, in this work, we focus setting where the preference pairs are off-the-shelf. This is because computing binding affinity is computationally expensive. An important future direction to extend the work toward real-world drug discovery scenarios could be incorporating the online setting but with limited number of query.
offline learning
B
Algorithm
The pseudo-code for ALIIDIF and ALIDIFF-T are provided below. Sampling procedures the same as Guan et al. [2023] and Huang et al. [2023].
Algorithm 1 Training Procedure ALIDIFF
1: Input: Protein-ligand binding dataset (P,M", ,M pre-trained neural network dia, reference
network Øres. learnable neural network vuu and pretrained interaction prior network TTP.
2: while deg and 1/g2 converge do
3: [Po, m mij ~ {P,M" M' where m. = {xx". vor m' = {x/ vol
4: Obtain r", r' for m m', respectively.
5:
t~ (O, T")
6:
Move the complex to make CoM of protein atoms zero
Obtain shifts [s^ ,sh s.M1] and interactions [r,"., . [.p from the and the2 according to
7:
[Huang et al., 2023].
Perturb x xt, to obtain xi" x with shifts s. .s s.M:
8:
9:
~ (N,
10:
x Va,x"" + Ma + vI oie, x = s Mi v1 ate
11:
Perturb v vo to obtain v v
g Gumbel(O, 1)
13:
log ct log( V. a1 log log(ā,v)) (1- kz/K)
14:
Ve" onchot arg max, (g. log c.")) )), v onchot (arg max + log c))
Embed vi, v into h .h M..0 and embed vp into hi
Obtain features [hr.0 h Mio h..0] through prior-conditioning
Predict (*t) Voje) from da ([b; M...0 h 2..11 Me, .fP1))
17:
Predict (x)) vojo) from de [h; Mr.0
h. p.0
I.II Mr ..PI)
18:
Predict (*N rr..r Vojc.er) from over( ([h; M...0
Mi.. h P.0 16 1
19:
Predict (x)) r.ref v Vojcuet from cher [li Mi,0
Mr.0 hb P. I.IM.. GD
20:
Compute loss with (x) (oje), (xb, vo). (R vi according to Equation (12)
re and vil V
22:
Update 0 and 02 by minimizing I.
23: end while
13

---

Algorithm 2 Training Procedure for ALIDIFF-T
1: Input: Senein-ligand binding dataset (P, M M N. pre-trained neural network do, reference
network Piet
2: while (hg not converge do
3:
[p, [ m/] ~ ; ji= where m {xx V0) mb =xx0= vo}
4:
Sample diffusion time 1 ~ U(O,.....
5:
Move the complex make CoM of protein atoms
6:
Perturb x x, to obtain x," x x where e ~ (0,
Vaxx + 01)c, x = Vaxx (1 04)c,
7:
Perturb vo vó to obtain et of: (a))/)K
8:
lage Vā +
9:
loge' = log(asvo + at)/K)
10:
V." = one _aoeeaaaogggaa max(g; + l)gc;"))
11:
V; = one hot(arg max(g. + logdl)))))))) where g (0,
12:
Predict [x" vo" from [x"", v(]] with dg: 8y v (x([") vt'.tt,))
13:
Predict [ v/l from xx) vill with do: [&). [ol (0( t.
14:
Predict [x vo from [x vi with Oref [XD v0' = Orer[[x;" V. t.p)
Predict xo. V rerr from xx, vil with dief [xh. Vo.eer = Ser((x t.l.t.p)
Compute L(0) = (0) + a. according Equation (10)
17:
Update 0 by minimizing L
end while
C Proof
Theorem 3.1. The objective function Equation (12) optimizes a variational upper bound of the KL-
divergence IKLL (6* (mlp))lig(m(p)) where (m p) Pre((mlp) exp(r(m,p) p)) and pe (m|p) x (m(pp)) (m/p)))))
(m)(m)
We prove the theorem with Lemmas C.I and C.2. Lemma C.I justifies the least square objective the variational upper bound for preference optimization, and Lemma C.2 shows that regularized preference optimization corresponds to exact KL divergences between the optimal and parameterized distributions. A version of similar proof can be found in Wallace et al. [2023] and Ji et al. [2024], Chen et al. [2024] respectively, and to be self-contained with Wallace et al. [2023], we introduce an additional term into the diffusion optimization. And compared with Ji al. [2024], Chen et al. [2024], explicitly drop the assumption for drawing infinite samples m for each pocket p.
incorporate these proofs here. Compared
Lemma
C.1. The objective function Equation (12) L (ALDII-EPP((O)
E (p. D. a variational upper bound of:
+
(g) = (p mmmmmm)22 (a(r" -r)) (log log po(m"(() B log Pry (p) (m p)
pa pa(
po (m (m pp)
3 log (m Ip), (mo pp) ))
+ (1- -1- -r') (lo log
Peo (m
(13)
We refer readers to Appendix S2 of Diffusion--POOO [Wallace et al., 2023] for the full proof. The bound is derived from Jensen's inequality and the convexity of the function log σ.
Lemma C.2. The objective function Equation (13) optimizes the KL-divergence DKL. (f* (mp()) ( (mlp)). where p² (mlp) & Pre((mpp) exp(r(m,, p)) and po(ml)) o pa(m)p) (m)
(pp(
14

---

Proof. First of all, we can rewrite the objective Equation (13) in the following form, expanding the sigmoid function:
(p(p.m
8 log
Py (m,/m)) (10g IP?
Ppppo =
log
j=1 er(p.m,)
log
Pa (p)
Phal mny
j=1
( log
er(p.m.)
= Pa
log
(14)
==1 er(p.my)
log pa Fiot (al (pl
Ej=1
2
(p(pm
[mm.m.p) Pur( p?)
E P B
log
e)(p.my)
ΣØ=1
[Po(m,
Purt(m (p)
1- 2 By the definition (mpp) x Pie (mlp)r (m p). we have both sides with Pre((mpp) Then we can substitute this equation and rewrite Pe 0):
po (m pp) Part (ma(m
pa m p
(by dividing
cr(p.m:)
(mm, |p)
(0) (ppDBpa(miaall)
log
Pier mm p)
i=1 j== er(p.m,)
2
Po P)) 8
Pert (p)
2
ex(p.m.)
Po (m,/p)
Ep-D B (pa((maapp))
log
(15)
j=1 (x(p.my))
2
Da(m, P)
im1
==1
(m,;
Since Pre(-1lp is supervised fine-tuned samples {m, ) assume (m, ) takes most of the probability mass and thus (pppp))p)) Rs (m1:221))p)) Then we have the following approximation:
2
Po po(m;lp)
(m(p) (me((m
Prer Prer(m)) (mpp) Po (qmm)) = 2 Σ
Prof (my) pp)
x 2E 2E por (mi p)
=
Σ ME.M
Pref (m|p)
Po (m|p) = 2,
j=1
MEM
Σ
e mp)
2 pm (ml p)
((().m)
Σ
Prer( (ep((ml 27(p))
/=1
meM
Then we can plug the above results into Equation (15) and further simplify Eer
r(p.m:
Do (m, pp) 2p ( pp)
(0)
log
(7(p)
(p.
bo (m p))
(p,,),
Ep-pP par (p)
log
22(p)
Piet (m: p)
22(p)
i=1
Z(D)
(pp...))
Po (mmm/p)
e (pp.m.
(c((..m))
)
E
log
log
2Z(p)
Piet [ p)
prtp.me)
2Z(p)
2Z(p)
i=1
Z(p)
where the second term remains constant C to 8, and thus be omitted when analyzing the optimization for O. Notice the normalized form of f² (m/p) = (m)perr we replace
15

---

Z(p) Prer Prer((mlpp)) mmppee with 8r and further simplify the above equation:
1
e m.).m
Po
m
(0) = BpDD
log
+ C
2 i=1
(())
m
(((D.m)
Po (m
= BDDD
E (mpp)
log
+ C
7(p)
p
Σ Piot m p) (p(mm)
pg m p pr
= BDDD
log
C
(p)
p)
meM
po pp)
EppD
Σ meM
0" (m(pp)
log
+
10 (i((mpp)
= BpDD (f +C].
which completes the proof of Lemma C.2.
D Implementation Details
Data. Following [Guan al., 2023], proteins and ligands are expressed with atom coordinates and a one-hot vector containing the atom types. For proteins, each atom type is represented by one-hot vector covering 20 distinct amino acids. Ligand atoms are encoded using a one-hot vector that discriminates among several elements, specifically H, C, N, O, F, P. S, Cl. Additionally, one-dimensional binary flag incorporated to differentiate whether atoms are part of the protein or the ligand. We further apply two separate single-layer Multi-Layer Perceptrons (MIPs) to transform the input data into 128-dimensional latent spaces, providing a compact and informative representation for subsequent computational stages.
Preference Pair Generation. For each synthetic molecule, we first locate its corresponding protein binding site and compute reward according to user-defined reward function for all synthetic molecules of the corresponding the binding site. We select losing sample with lower reward and construct the preference. The selection process is detailed in Appendix E.
Architecture. We follow the architecture IPDiff (Huang al., 2023], which includes learnable diffusion denoising model Φ91, learnable neural network Φ02 and pretrained interaction prior network IPNET. The architecture of all models used in our method is the same as IPDiff.
Pretraining Details. Following existing work, we adopted the Adam optimizer with a learning rate of 0.001 and parameters 3 values of (0.95, 0.999). The training was conducted with batch size of and a gradient norm clipping value of 8. To balance the losses for atom type and atom position, we applied scaling factor À of 100 to the atom type loss. Additionally, with standard deviation 0. to the protein atom coordinates as a form of data augmentation. Our parameterized diffusion denoising model, IPDiff trained single NVIDIA A6000 GPU and achieved convergence within 200k steps.
introduced Gaussian noise
Training Details. For finetuning, the pre-trained diffusion model is further fine-tuned via the gradient descent method Adam with init learning rate=5e-6, betas=(0.95,00 0.999). We keep other setting the same as pretraining. We use 3 = 5 in Equation (5). We trained our model with one NVIDIA GeForce GTX A A100 GPU, and it could converge within 30k steps.
E
More Experimental Results
Effect of diffusion steps. In fig. 6, we present a comprehensive ablation study examining the impact of diffusion steps on the optimization of molecular properties using our novel ALIDIFF framework. The visualizations at the top of the figure showcase the progressive refinement of molecular structures across increasing diffusion steps (1 = 200 to / = 1000). These images clearly illustrate how our model gradually enhances the molecular fitting within the target binding site, which critical for improving drug efficacy. The plotted data below provides quantitative analysis of QED, SA, Vina
16

---

Reference
T=200
T-400
T-600
T-800
T-1000
2' SALO
0.57
SALO
SAUO 0,41
BALO 0
0170771
BAL 1L
Soare:
Soare
Vino
Sonn::
OuterlAeg! Pillo))
Oureliee Gurn Ouru 10oal
039
0.00
Duts (MMc
Durs Mud!
31 413
Whe
6.43
6462
0.9
Nam of mime 500ps
Num of Cm/ 2top3
Num Cmn Stops
Figure 6: Ablation study on diffusion steps. The top shows a visualization of the generated molecule (4aua) under different time step. The bottom reports QED, SA and Vina Dock reported under different diffusion steps(200, 400, 600, 800 and 1000). Blue lines represent ALIDIFF-DPO and Red lines represent ALIDIFF-E'PO.
Dock across all test targets. Notably, both ALIDIFF (P) and ALIDIFF (R) demonstrate significant improvements in QED and SA scores as the number of diffusion steps increases and exhibit notable decrease Vina Dock. Particularly, ALIDIIFFEEPO model have shown better performance across all three metries, with significant improvement on binding affinity across the diffusion steps.
Lipinski. We further compared Lipin- ski's Rule of Five [Lipinski et al., 2012] across all comparison methods. Lipin- ski's Rule of Five is another measure- ment for assessing dtrug-likened besides QED, and we would like to incorporate this metric validate our performance in generating drug-like molecules. The results of Lipinski's 's scores are reported in Table 5. The results are consistent with our evaluation using QED score, as all diffusion- models are not achieving high drug- We maintain similar drug-likeness as our backbone models targetDiff and IPDiff.
Table Lipinski results for all methods.
Methods
ALIDIE
IPDIf
TargetDirt
AR
Pocket2Mal
Reference
Avg. Lipincki (1)
4.48
4.52
4.51
4.75
4,88
4.27
Table 6: Ablation study results with different choice of m'.
Choice of mm
Vina Score (1)
Vina Min (L) Med.
Vina Dock ()) Avg.
High (ffinity(f) Avg.
QED(T)
SA(1)
Diversity(f)
Avg.
Med.
Avg.
Med.
Med.
Avg.
Med.
Avg.
Med.
Avg.
Med.
worst
-7.07
-7.95
-8.09
-8.17
-8.90
-8.81
73.4%
81.4%
0.50
0.50
0.57
0.56
0.73
0.71
best
6.80
-7.66
.7.83
7.69
8.64
-8.05
70.2%
76.8%
0.50
0.52
0.56
0.55
0.74
0.71
random
-6.96
-7.82
-8.03
-8.00
-8.77
-8.20
72.1%
77.8%
0.50
0.51
0.56
0.55
0.74
0.72
median
.6.66
-7.85
8.01
-7.6
8.80
-8.24
72.5%
78.9%
0.50
0.51
0.57
0.55
0.74
0.72
Choice of m'. Our generated dataset is obtained by directly transforming standard labeled dataset into a pairwise preference dataset. Yet the binding affinity labels are continuous values where sometimes the difference between preferred and dispreferred is minimal. Therefore, the effect of energy preference optimization is highly sensitive to the overall data quality. Table 6 compares the performance of applying different strategies for selecting the dispreferred samples. "worst" indicates that the losing sample has the worst score from the user-defined reward function (lowest binding affinity). "best" suggests that the losing sample has the second-to-higee highhtt binding affiniyy(esiddes the preferred one). "random" and "median" mean that the losing samples are extracted randomly or from the median. Vina Score, Vina Min, Vina Dock, QED, SA, and Diversity are reported as average (Avg.) and median (Med.) values. Overall, the "worst" strategy, selecting the least favorable sample based on optimization objectiveness, consistently achieves the best performance binding affinity metrics (Vina Score, Vina Min, and Vina Dock), while maintaining competitive likeness (QED)
17

---

and synthetic accessibility (SA). The "best" strategy, which may involve selecting the most favorable samples, performs poorly overall, which implies that energy preference optimization works better when there exists a larger discrepancy between r and r'. This allows the model to learn how to favor to m and avoid m' during the finetuning process. The "random" and "median" strategies show intermediate performance, suggesting that a strategic approach to sample selection can significantly impact the efficacy of the resulting models.
More Visualizations
Reference
TargetDiff
IP
Ours
1e8h
Vina: &.9 QED: 0.28 SA: D. 65
Vina: 8.1 QED 0.52 SA: 0.54
Vina: 12.4 QED: 0.45 SA:0.44
Vina: 14.1 QED: 0.52 SA: 0.51
1phk
Vina:-8 0.18 0.57
-3. QED SA: 0.64
-2.44 QED: 0.65
Vina: -14.3 0.65 0.53
2e24
Vina: 4.4 QED: 0.41 SA 0.62
Vina: 533 QED SA:0.61
Vina: -100 QED 0.50 0.50
4tqr
Vina:
0:54
2zen
Vina: QED: 0.65 0.93
Vina: 6.4 QED: 0.58 SA: 0.58
-9.3 QED. SA 0.55
Vina:110.2 QED: 0.67 SA: 0.5
Figure 7: More visualizations of generated ligands for protein pockets generated by TargetDiff, IPDiff, and ALIDIFF.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Our claim is reflected through comprehensive experiments and theoretical proofs.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed limitations of our work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

Justification:
Yes we have provided proof in Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided all the codes needed to reproduce the results presented. Pseudo code is also included in appendix.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification:
We have provided all the codes needed to reproduce the results presented.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. cc/ mublic/guides Sowiidde//Coeeuumissstttee weeiiissssooooliiio oolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //wpps..cppuu/////// ctcccc//p/o wwt.ccccpppccc.eeeeeeeeee.eeeeee wewiic/ggids//ooooo Sttes/CCoeSuuiiiiseeee citbmmisssio onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have provided implementation details in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars are not reported because would be computationally expensive. Since our diffusion model sample 100 samples for each pocket, we believe reporting median and mean will be well reflected of the overall performance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
21

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided computer resources details in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http://eeei.../.eeeee wttppssscccpppccccpeeeeeeecseeee Dpublit/Ehhicstuuddellnnee ines?
Answer: [Yes]
Justification: General machine learning method without specific concern in our mind.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: General machine learning method without specific concern in mind,
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: General machine learning method without specific concern in our mind.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets get credited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have provided the code along with files to run the training process directly in the supplementary.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No crowdsourcing and research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.


Reviews:
[{"review_id": "ZeRfQuhKXq", "reviewer": "Reviewer_5bxb", "summary": "This paper proposes a novel alignment framework, known as ALIDIFF, to align pretrained target diffusion model with preferred functional properties for structure-based drug design. ALIDIFF shifts the target-conditioned chemical distribution towards regions with higher binding affinity and structural rationality, specified by user-defined reward functions, via the preference optimization approach. The experiments show that this method can generate ligands with better binding energy and maintain competitive molecular properties.", "questions": "Please refer to the weakness part.", "limitations": "Limitation is discussed.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- This method is novel in using energy preference optimization framework to align molecule generative models with desirable properties in order to generating molecules with high binding affinity to binding targets.\n- They analyze the overfitting issue in the preference optimization objective, and propose an\nimproved exact energy optimization method to yield an exact alignment towards target distribution shifted by reward functions.", "weaknesses": "- Lipinski metric should also be reported in Table 1. This is a commonly used metric for measure drug-likeness in previous work. \n- Fragment-based methods, such as FLAG and DrugGPS, is missing in Table 1. For comprehensive comparison, I recommend add those two baselines. \n- Generated molecules have inferior drug-likeness properties. Isn't it possible to also align the pre-trained diffusion model with high drug-likeness region?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and questions! The replies to your questions are listed below:\n \n> **[W1] Lipinski metric should also be reported in Table 1.**\n\nThank you for your suggestions. To evaluate drug-likeness, we have compared QED across all comparison methods. Lipinski's Rule of Five[1] is another measurement for assessing drug-likeness, and we would like to incorporate this metric to validate our performance in generating drug-like molecules. The results of the Lipinski's scores are as follows:\n\n| Reference | AR | Pocket2Mol | targetDiff | IP-Diff | Ali-Diff |\n| - | - | - | - | - | - |\n| 4.27 | 4.75 | 4.88 | 4.51 | 4.52 | 4.48 |\n\nThe results are consistent with our evaluation using QED score, as all diffusion-based models are not achieving high drug-likeness. We maintain similar drug likeness as our backbone models targetDiff and IP-Diff. We will incorporate the results into our final version.\n \n> **[W2] Fragment-based methods.**\n\nThank you for pointing out the related works that are fragment-based. We appreciate your feedback and will incorporate DrugGPS[2] and FLAG[3] into our related work section in final version. Regarding the experiment, we did not compare our approach with fragment-based methods because they are not directly comparable since they rely on heavy prior knowledge of fragments, and also utilize a different experimental setting than the benchmark proposed by targetDiff. We have tested our performance using their train/test split, and the results are as follows:\n\n| Methods  | Vina  (↓)  | High Affinity (↑)  | QED (↑)   | SA (↑) | Lip  (↑) |\n|----------|-------|-------|-------|-----------|-------|\n| FLAG \t| -6.96 | 44.5% |  0.55 \t| 0.74 | 4.90 |\n| DrugGPS  | -7.28 | 56.5% | **0.61** | **0.74** |  **4.92** |\n| targetDiff  | -6.92 | 48.2% | 0.47 | 0.58 |  4.42 |\n| IP-Diff  | -7.33 | 64.7% | 0.51 | 0.60 |  4.48 |\n| AliDiff  | **-7.87** | **68.0%** | 0.50 | 0.56 |  4.43 |\n\nWe observe that our proposed method consistently outperforms fragment-based methods on binding energy-related metrics, which is consistent with our previous conclusion. The inferior performance over drug likeness and synthetic accessibility is mainly due to the nature of atom-based diffusion models, as our baseline model (targetDiff, IP-Diff) also achieves similar performance. We leave combining fragment-based methods with our preference optimization framework as a promising future direction, as it will take advantage of high binding affinity thanks to optimization, while maintaining QED and SA with prior knowledge of fragments.\n\n> **[W3] Align the pre-trained diffusion model with high drug-likeness region.**\n\nYes, it is indeed possible to align the pre-trained diffusion model with regions of high drug-likeness. We conduct additional experiments and report the results of combining multiple reward objectives as follows:\n\n| Choice of Reward | Vina Score (↓) | Vina Min (↓) | Vina Dock (↓) | High Affinity (↑) | QED (↑) | SA (↑) | Diversity (↑) |\n|-|-|-|-|-|-|-|-|\n| Affinity Avg.   | -7.07   | **-8.09**   | -8.90    | 73.4%  | 0.50    | 0.57  | 0.73    |\n| Med.   | -7.95  | **-8.17**   | **-8.81**    | 81.4%       | 0.50    | 0.56  | 0.71 |\n| Affinity + SA Avg. | -6.87 | -8.00   |  -8.81  | 72.7%  | **0.52**|**0.60**| **0.74**  |\n| Med.  | -7.76  |  -8.08   | -8.72    | 80.8%  | **0.55**|**0.59**| **0.73**  |\n| Affinity + QED  Avg.   | **-7.11**  |  -8.02   | **-8.91** | **73.7%** | 0.51|0.57| 0.73  |\n|Med. | **-7.99** | **-8.17**   | -8.72    | 82.0%  | 0.52|0.57| **0.73**  |\n\n\nBy incorporating metrics like QED and SA into the reward system, the model generates molecules that balance strong binding affinities with enhanced drug-likeness and synthetic accessibility. Aligning molecules with both binding affinity and QED will improve both QED and affinity, although the improvement in QED is relatively marginal. Therefore, by fine-tuning the pre-trained diffusion model using a multi-objective reward function, we can enhance the drug-like properties of generated molecules while preserving their binding efficacy. This approach allows for more robust and practical molecule generation in drug discovery applications. We will incorporate the results into our final version.\n\n------\n\nWe hope our response could address your questions!\n\n[1]: Experimental and computational approaches to estimate solubility and permeability in drug discovery and development settings. Advanced drug delivery reviews,64:4–17, 2012.\n\n[2]: Learning subpocket prototypes for generalizable structure-based drug design. International Conference on Machine Learning. PMLR, 2023.\n\n[3]: Molecule generation for target protein binding with structural motifs. The Eleventh International Conference on Learning Representations. 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "evnohgdreo", "reviewer": "Reviewer_fWJ4", "summary": "This paper proposes a novel and general alignment framework to align pretrained target diffusion models with preferred functional properties, named AliDiff. AliDiff adjusts the target-conditioned chemical distribution toward regions characterized by lower binding energy and structural rationality. AliDiff can generate molecules with lower binding energies.", "questions": "See weaknesses", "limitations": "None", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Structure-based drug design is an important scientific problem. In practice, it is more important to control the generation process, instead of learning the distribution of all drug candidates.\n- The authors propose a novel preference optimization framework (E2PO) to fine-tune the pre-trained diffusion model with RL.\n- The authors address the over-optimization issue with regularization on preference maximization, and propose an analytical guarantee.\n- AliDiff achieves strong performance on the CrossDocked2020 benchmark.", "weaknesses": "- The experiments are only conducted on the CrossDocked2020 dataset. Are there any alternative datasets?\n- In addition to well-established benchmarks, is it possible to show several practical case studies to verify the effectiveness of the proposed algorithms. For example, generating ligands for the target of SARS-CoV-2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and questions! The replies to your questions are listed below:\n\n> **[W1] Alternative dataset**\n\nAliDiff’s training requires the dataset to have multiple ligands with one single protein. CrossDocked2020 is one of the largest synthetic datasets where it has multiple liagnds bind to the same protein. We also follow your suggestion and test AliDiff on another high-quality protein-ligand pair dataset Binding MOAD [1]. We train the models by adopting DiffSBDD as the baseline and follow the same implementation details as our previous setting. The results are as follows:\n\n| Metrics         | GraphBP  | DiffSBDD | AliDiff-D|\n|---------|--------|----------|------------------|\n| Vina      | -4.84   | -7.31          | -7.62 |\n| QED             | 0.51    | 0.54            | 0.53 |\n| SA              | 0.31    | 0.62            | 0.60 |\n| Lipinski        | 4.95    | 4.78            | 4.72 |\n| Diversity       | 0.83    | 0.74            | 0.71 |\n\nWe observe that our proposed method consistently outperforms baseline methods on binding energy-related metrics, and slightly inferior performance over drug likeness and synthetic accessibility compared with our backbone model DiffSBDD. This is consistent with our previous conclusion on CrossDocked2020. We significantly improved all binding affinity related metrics without significant sacrifice on molecular properties. We will add results on this alternative dataset to the final version.\n\n\n\n> **[W2] Pratical case studies**\n\nIn addition to the benchmarks presented in the paper, we have explored several practical case studies to further demonstrate the effectiveness of the proposed method for generating ligands with desired properties. In Fig 4, we showcased visualizations of reference molecules and generated ligands for protein pockets (1l3l, 2e24) Protein 1L3L[2] is a transcription factor involved in quorum sensing, a process by which bacteria communicate and coordinate behavior based on their population density. Designing drugs targeting the 1L3Lprotein can prevent the expression of virulence genes. and suppress unwanted bacterial activity. Protein 2E24 [3] specifically targets and cleaves the pyruvated side chains of xanthan, a complex bacterial heteropolysaccharide.  Drug design targeting this enzyme could potentially enhance industrial applications of xanthan by enabling more precise modifications of its structure, thereby improving the rheological properties of xanthan-based products. These case studies highlight the method's potential to address real-world challenges in drug design and discovery. \nRegarding generating ligands for SARS-CoV-2, our proposed method will first preprocess the protein with a given pocket to extract the binding site. Since SARS-CoV-2 is a relatively new target and few empirical studies have been conducted, we plan to leave SARS-CoV-2 as future test case and incorporate more practical scenarios as our future directions.\n\n------\n\nWe hope our response could address your questions!\n\n\n[1] Binding MOAD, a high-quality protein–ligand database. Nucleic acids research 36.suppl_1 (2007): D674-D678.\n\n[2]: Structure of a bacterial quorum-sensing transcription factor complexed with pheromone and DNA. Nature 417.6892 (2002): 971-974.\n\n[3]: A structural factor responsible for substrate recognition by Bacillus sp. GL1 xanthan lyase that acts specifically on pyruvated side chains of xanthan. Biochemistry 46.3 (2007): 781-791.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ej2Get7RSh", "reviewer": "Reviewer_bNcB", "summary": "This paper presents ALIDIFF, a framework that aligns pre-trained target-aware molecule diffusion models with desired functional properties using preference optimization. The key contribution of ALIDIFF is the Exact Energy Preference Optimization method, which precisely aligns diffusion models to regions with lower binding energy and better structural rationality based on user-defined reward functions. Extensive experiments on the CrossDocked2020 benchmark showcase ALIDIFF's strong performance, generating molecules with state-of-the-art binding energies and maintaining competitive molecular properties. By incorporating user-defined reward functions and improving the Exact Energy Preference Optimization method, ALIDIFF achieves significant advancements in binding affinity.", "questions": "* Train/test split is 65K/100. It seems that the test set is too small compared to the train set. Is there any reason for that?\n* How much time does it take to pre-train, fine-tune and binding energy computations via AutoDock Vina?", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* ALIDIFF utilizes a preference optimization approach to effectively steer the model towards generating chemically and structurally relevant molecules.\n* It Demonstrates the effectiveness and robustness of the method through extensive experiments, showing superior binding energies while maintaining strong molecular properties.\n* The paper employs AutoDock Vina for accurate binding energy evaluations, ensuring practical outcomes in drug discovery applications.", "weaknesses": "* The method leverages IPDiff and further optimizes it using RL, which makes performance improvements easier. Consequently, the methodological novelty appears marginal.\n* The paper lacks specific information on the time required for pre-training, fine-tuning, and binding energy computations via AutoDock Vina. This absence makes it difficult to fully assess the advantages and limitations of the proposed method.\n  - AutoDock Vina is known to be time-consuming, and RL also requires significant computational time. Including these time details would help illustrate both the strengths and constraints of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and questions! The replies to your questions are listed below:\n\n> **[W1] Methodological novelty**\n\nWe argue our novelty for not only model design but also method formulation:\n1. The DPO framework is originally proposed for optimizing language models. In this paper, we propose to optimize the diffusion model with user-preference data to further facilitate drug design with practical pharmaceutical needs, which is already a new perspective in this field.\n2. Furthermore, we solve key technical challenges for designing optimization frameworks and further introduce exact energy preference optimization, where we further integrate numerical values of reward instead of directly taking a pair of preferred and dispreferred data.\nWe believe AliDiff offers considerable contributions to the community.\n\n> **[W2+Q2] Training time**\n\nOur proposed method is not time-consuming. First we want to highlight that training time is reported in the Appendix. The advantage of Ali-Diff is that it can take any pre-trained diffusion model and finetune with preference data. The finetuning process takes 30000 steps with approximately 1916  minutes. During training, we do not need to compute AutoDock Vina because it’s pre-computed in the training dataset. During evaluation, we sampled 100 ligands for each protein, where the evaluation takes 1 hour for each target protein.  We do notice that binding energy computations via AutoDock Vina are time-consuming, and we can also choose to compute binding energy with QVina [1], which will be more time-efficient yet less accurate. All metrics calculate the binding energy between protein and ligands and we can construct alternative datasets using other metrics that are not time-consuming. We will add this discussion to the final version. \n\n\n> **[Q1] Training test split**\n\nWe would like to point out that the train/test split we adopt is the benchmark setting from targetDiff and we want to keep this consistent with all other comparison methods. We do notice that 100 test data could be relatively too small, therefore, we also adopt another train/test split from fragment-based methods FLAG[2] and DrugGPS[3] and verify our performance in another setting. \n\n| Methods  | Vina  (↓)  | High Affinity (↑)  | QED (↑)   | SA (↑) | Lip  (↑) |\n|----------|-------|-------|-------|-----------|-------|\n| targetDiff  | -6.92 | 48.2% | 0.47 | 0.58 |  4.42 |\n| IP-Diff  | -7.33 | 64.7% | **0.51** | **0.60** |  **4.48** |\n| AliDiff  | **-7.87** | **68.0%** | 0.50 | 0.56 |  4.43 |\n\nWe observe that our proposed method consistently outperforms baseline methods on binding energy-related metrics by a notable margin, which is consistent with our previous conclusion. The inferior performance over drug likeness and synthetic accessibility is mainly due to the nature of atom-based diffusion models, as our baseline models (targetDiff, IP-Diff) also achieve similar performance. We leave combining fragment-based methods with our preference optimization framework as a promising future direction, as it will take advantage of high binding affinity thanks to optimization, while maintaining QED and SA with prior knowledge of fragments.\n\n> **[Q2] Training time for each step.**\n\nWe provide our response together with response to weakness2 above.\n\n------\n\nWe hope our response could address your questions!\n\n\n\n[1] Fast, Accurate, and Reliable Molecular Docking with QuickVina 2. Bioinformatics (2015) 31 (13): 2214-2216\n\n[2]: Learning subpocket prototypes for generalizable structure-based drug design. International Conference on Machine Learning. PMLR, 2023.\n\n[3]: Molecule generation for target protein binding with structural motifs. The Eleventh International Conference on Learning Representations. 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H5xt2qiQe8", "reviewer": "Reviewer_FKJR", "summary": "This paper focuses on generating ligands with desired properties, such as high binding affinity, for the protein-conditioned ligand generation task. The authors propose a Preference Optimization (PO)-based fine-tuning method for pre-trained generative models. They extend DPO (Data-driven Preference Optimization) to propose exact energy optimization that directly utilizes reward values in the loss function. Experiments demonstrate that the proposed method can generate ligand molecules with the desired properties.", "questions": "* The proposed objective function seems to be heavily influenced by the scale of the reward value r. Was any adjustment necessary? In particular, was simple addition sufficient for the Affinity + SA experiment that combined multiple rewards? Additionally, is it easy to incorporate other rewards, such as diversity?\n* The paper states that if $r^w \\gg r^l$, then $\\sigma(r^w - r^l) \\approx 1$. However, wouldn't it be more accurate to say that if $r^w - r^l \\ge 1$, then $\\sigma(r^w - r^l) \\approx 1$? What is the actual distribution of $r^w - r^l$ in the training data?", "limitations": "The proposed objective function cannot be used when reward values are not explicitly available, making it unsuitable for cases involving manually binary-labeled results.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Comprehensive ablation studies address readers' concerns.\n* An efficient objective function is designed for tasks where reward values are directly available, such as binding energy.\n* The proposed method's effectiveness is validated using multiple pre-trained models, both IPDiff and TargetDiff, demonstrating its generalizability.\n* The investigation into preference pair selection is interesting and shows room for future research.\n* The proposed method is well-designed.\n* The paper is well-structured and easy to read.", "weaknesses": "* The practical impact of the achieved improvement needs to be clearly quantified.\n* There is no consideration or results for comparison or combination with methods other than DPO, such as IPO or SLiC.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and questions! The replies to your questions are listed below:\n\n> **[Q1] Practical impact**\n\nOur benchmark results showcase the practical impact of optimizing molecules with user-defined reward functions. A lower binding energy between a protein and a ligand indicates a stronger and more stable interaction between the two molecules. In drug design, binding energy is a critical factor that influences the efficacy of a ligand in binding to its target protein. Our benchmark results suggest that while we maintain similar molecular properties as baseline models, we are able to improve binding affinity by a notable margin.\n\nIn addition to the benchmarks presented in the paper, we have explored several practical case studies to demonstrate the effectiveness of the proposed method for generating ligands with desired properties. In Fig 4, we showcased visualizations of reference molecules and generated ligands for protein pockets (1l3l, 2e24). For instance, protein 1l3l is a transcription factor involved in quorum sensing, a process by which bacteria communicate and coordinate behavior based on their population density. Designing drugs targeting the 1l3l protein can prevent the expression of virulence genes. and suppress unwanted bacterial activity. These case studies highlight the method's potential to address real-world challenges in drug design and discovery.  We will incorporate this discussion into final version.\n\n> **[Q2] Comparison with other RLHF methods**\n\nThank you for your insightful comment regarding exploring more RLHF methods. We appreciate your suggestion and added additional experiments incorporating the IPO method [1]. Below, we provide an updated comparison table that includes results for the DPO, E²PO, and IPO:\n\n| Method   | Vina Score (↓) | Vina Min (↓) | Vina Dock (↓) | High Affinity (↑) | QED (↑) | SA (↑) | Diversity (↑) |\n|-|-|-|-|-|-|-|-|\n| **AliDiff-DPO** Avg.  | -6.81 | -7.75  | -8.58  | 69.7%  | **0.50**| 0.56  | **0.74**  |\n|  Med.  | -7.62  | -7.79  |  -8.55  | 71.1% | **0.51**| **0.57** | **0.72**     |\n| **AliDiff-E²PO** Avg.  | **-7.07**  | **-8.09** |  **-8.90**  |  **73.4%** | **0.50**    | **0.57** | 0.73  |\n|   Med. | **-7.95**  |  **-8.17** |  **-8.81** |  **81.4%** | 0.50 | 0.56  | 0.71|\n| **AliDiff-IPO** Avg.  | -6.93 |  -7.80 | -8.68 |  71.7% | **0.50**| **0.57**  | 0.73  |\n|  Med.  | -7.82  | -7.92  | -8.62  | 78.9%| **0.51**| 0.56  | **0.72** |\n\nAliDiff-E²PO remains the top performer in terms of binding affinity and high-affinity metrics, which further supports the effectiveness of incorporating exact preference optimization. We will add this study to the final version.\n\n> **[W1] Scale of the reward value r**\n\n1. Yes, the proposed objective function is influenced by the scale of the reward value r. Therefore, we conducted a sensitivity analysis to determine the ideal r, as shown in Fig. 5. When combining multiple reward objectives Vina and SA, we applied a simple weighted sum approach. Since the numerical scale of binding affinity (Vina score) and molecular property (SA) are different, we evaluated different weighing factors and the results are shown as follows:\n\n| Weight |Avg. Vina Score (↓) |Avg. Vina Min (↓) | Avg.Vina Dock (↓) | Avg. High Affinity (↑) | Avg. QED (↑) | Avg. SA (↑) | Avg. Diversity (↑) |\n|-|-|-|-|-|-|-|-|\n| 1   | **-6.99** |  **-8.02**   | **-8.89**  | **73.3%** | 0.50  | 0.57  | 0.73  |\n| 10   | -6.87  | -8.00   | -8.81 | 72.7% | 0.52|**0.60**| **0.74** |\n| 100   |  -6.78 | -7.90   | -8.71  | 71.7% | **0.53**|**0.60**| 0.73 |\n\nWe leave finding the optimal approach for multi-objective optimization as a promising future direction, as it will take advantage of multiple objectives and allow for better optimization from each objective.\n\n2. Yes, it is indeed possible to align the pre-trained diffusion model with other reward functions. The results are shown below, where we combine affinity with QED and SA. By incorporating metrics like QED and SA into the reward system, the model generates molecules that balance strong binding affinities with enhanced drug-likeness and synthetic accessibility. Aligning molecules with both binding affinity and QED will improve both metrics, although the improvement in QED is relatively marginal. Therefore, by fine-tuning the pre-trained diffusion model using a multi-objective reward function, we can enhance binding efficacy while preserving their molecular properties. This approach allows for more robust and practical molecule generation in drug discovery applications. \n\n| Choice of Reward | Vina Score (↓) | Vina Min (↓) | Vina Dock (↓) | High Affinity (↑) | QED (↑) | SA (↑) | Diversity (↑) |\n|-|-|-|-|-|-|-|-|\n| Affinity Avg.   | -7.07   | **-8.09**   | -8.90    | 73.4%  | 0.50    | 0.57  | 0.73    |\n| Med.   | -7.95  | **-8.17**   | **-8.81**    | 81.4%       | 0.50    | 0.56  | 0.71 |\n| Affinity + SA Avg. | -6.87 | -8.00   |  -8.81  | 72.7%  | **0.52**|**0.60**| **0.74**  |\n| Med.  | -7.76  |  -8.08   | -8.72    | 80.8%  | **0.55**|**0.59**| **0.73**  |\n| Affinity + QED  Avg.   | **-7.11**  |  -8.02   | **-8.91** | **73.7%** | 0.51|0.57| 0.73  |\n|Med. | **-7.99** | **-8.17**   | -8.72    | 82.0%  | 0.52|0.57| **0.73**  |\n\n> **[W2] Is it be more accurate to say that if $r^w - r^l \\ge 1$, then $\\sigma(r^w - r^l) \\approx 1$? What’s the distribution of  $r^w - r^l$**\n\nMathematically, for the sigmoid function, we have that $\\sigma(1) = 0.7311$. Therefore we would keep the statement that “$\\sigma(r^w - r^l) \\approx 1$ when $r^w \\gg r^l$”. Please let us know if we misunderstand your point. \n\nFor the distribution of $r^w - r^l$, the statistics are listed: mean=3.35, median=2.37 and standard deviation=6.22, with boundary values max=120.35 and min=0.00. We will provide a figure for the distribution in our final version.\n\n------\n\nWe hope our response could address your questions!\n\n[1]A general theoretical paradigm to understand learning from human preferences. AISTATS, 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper contributes a novel diffusion-based model, focusing on generating ligand molecules with high affinity to their protein targets. To this end, Authors proposed a improved Exact Energy Preference Optimization method based on reinforcement raining. 

The reviewers have pointed out that the proposed approach has limited computational novelty, but still is of high practical utility. The rebuttal also discussed that the optimization of the affinity is associated with lower drug likeliness scores, which however seems a property of the diffusion models, and was traded for higher drug likeliness by fragment-based methods.

The paper seems of interest to the NeurIPS community.
Decision: Accept (poster)