TFG-Flow: Training-free Guidance in Multimodal Generative Flow
Authors: Haowei Lin
Shanda Li
Haotian Ye
Yiming Yang
Stefano Ermon
Yitao Liang
Jianzhu Ma
Venue: ICLR 2025 Poster (2025)
Keywords: flow matching
molecular design
training-free guidance
Primary area: generative models

Abstract: Given an unconditional generative model and a predictor for a target property (e.g., a classifier), the goal of training-free guidance is to generate samples with desirable target properties without additional training. As a highly efficient technique for steering generative models toward flexible outcomes, training-free guidance has gained increasing attention in diffusion models. However, existing methods only handle data in continuous spaces, while many scientific applications involve both continuous and discrete data (referred to as multimodality). Another emerging trend is the growing use of the simple and general flow matching framework in building generative foundation models, where guided generation remains under-explored. To address this, we introduce TFG-Flow, a novel training-free guidance method for multimodal generative flow. TFG-Flow addresses the curse-of-dimensionality while maintaining the property of unbiased sampling in guiding discrete variables. We validate TFG-Flow on four molecular design tasks and show that TFG-Flow has great potential in drug design by generating molecules with desired properties.

Published as a conference paper at ICLR 2025
TFG-FLOW: TRAINING-FREE GUIDANCE IN MULTI- MODAL GENERATIVE FLOW
Haowei Lin Shanda L Haotian Ye
Viming Yang", Stefano Ermon Yitao Liang', and Jianzhu Ma
Pekinng University. "Carnegie Mellon University, Staanford University, TTsinghua University
ABSTRACT
Given an unconditional generative model and a predictor for a target property (e.g., a classifier), the goal of training-free guidance is to generate samples with desirable target properties without additional training. As a highly efficient technique for steering generative models toward flexible outcomes, training-five guidance has gained increasing attention in diffusion models. However, existing methods only handle data in continuous spaces, while many scientific applications involve both continuous and discrete data (referred to multimodality). Another emerging trend is the growing use the simple and general flow matching framework in building generative foundation models, where guided generation remains under-explored. To address this, we introduce TFG-Flow, a novel training-free guidance method for multimodal generative flow. TFG-Flow addresses the tatee-ofo-mmisooaaaliii while maintaining the property of unbiased sampling in guiding discrete variables. We validate TFG-Flow on four molecular design tasks and show that TFG-Flow has great potential in drug design by generating molecules with desired properties.
INTRODUCTION
Recent advancements in generative foundation models have demonstrated their increasing power across a wide range of domains (Reid et al., 2024; Achiam et al., 2023; Abramson et al., 2024). In particular, siffusion-based foundation models, such as Stable Diffusion (Esser al., 2024) and SORA (Brooks et al., 2024) have achieved significant success, catalyzing a new wave of applications in areas such as art and science. As these models become more prevalent, a critical question arises: how can we steer these foundation models to achieve specific properties during inference time?
One promising direction is using classifier-based guidance (Dhariwal & Nichol, 2021) or classifier- free guidance (Ho & Salimans, 2022), which typically necessitate training specialized model for each conditioning signal (c.g., a noise-conditional classifier or a text-conditional denoiser). This resource-intensive and time-consuming process greatly limits their applicability. Recently, there has been growing interest in training free guidance for diffusion models, which allows users to steer the generation process using
aff-the-shell differentiable target predictor without requiring additional model training (Ye et al., 2024). A target predictor can be any classifier, loss, or energy function used to score the quality of the generated samples. Training-free guidance offers a flexible and efficient means of customizing generation, holding the potential to transform the field of generative AI.
Despite significant advances in generative models, most existing training guidance techniques are tailored to diffusion models that operate on continuous data, such as images. However, extending generative models to jointly address both discrete and continuous data- referred data-referred as multimodal data (Campbell et al., 2024)-remee fields (Wang et al., 2023). One key reason this expansion essential is that many real-world problems involve multimodal data, such as molecular design, where both discrete elements (e.g., atom types) and continuous attributes (e.g., 3D coordinates) must be modeled together. To address this, recent generative foundation models have increasingly adopted the flow matching framework (Esser et al.,
a critical challenge for broader applications in scientific
'These two authors contributed equally to the paper. Correspondence to Haowei (linhaove 0pku, edu. cn) and Jianzhu Ma (maj ianzhu@tsinghua. edu. edu.cn)
Code available https: ttttps///gtthii com.//inhooviii//FO---

---

Published as a conference paper at ICLR 2025
(a)
Go
Go.a
Go.4
G
Gas
G
40
N0, F)
Multimodal guided flow:
of eaff gail ei S Caeteee (44) 4 a 0F"D0,.f" (backward)
Hiloo ot wytel lolda, ollor Canifjal" al Relo!" i (ajj))) (forward)
(b)
leraatioss
(c)
+PV log Sog.(G.)) (G(y)
simplex I
ligand
Target structure (fingerprine)
G₁
Monte-Carto Carlo samples
Grode
plrua
Digste Momectt
model
sample
Pocket
Polorieahiiiy (Behe))
Dinding affinity (vina score)
tempenture T
Quantum property
(d)
Polarizabilliy (Bohr')
Figure I: (a) Multimodal guided flow. The backward flow (blue arrow) is constructed with linear interpolation between observed data Gi and sampled noise Go: forward flow (green arrows) is simulated by conditional velucity (ax (9) 'c)and conditional rate matrix R& Ra a (.44).).. (b) The illustration of TFG-Flow. TFG-Flow guides forward flow on each step G., which consists a gradient-based guidance and an importance sampling based guidance for discrete part a,; (c) Some examples of guidance targets. (d) Samples guided by TFG-Flow targeted at polarizability. The targeted value is at bottom of the samples.
continuous part X,
2024), prized for its simplicity and general applicability to both data types. Multiflow, recently introduced by Camphell et al. (2024) on protein co-design problem, presents promising foundation for tackling multimodal generation via Continuous Time Markov Chains (Anderson, 2012).
Unfortunately, guided generation within the flow matching framework remains relatively underex- plored, due the inherent differences between guiding continuous and discrete data. This paper investigates the problem of training-free guidance in multimodal flow matching, with specific focus on its application to inverse molecular design (Zunger, 2018). Inverse molecular design is a challeng- ing task that involves generating molecules that meet specific target properties, such a desired level of polarizability. Our method, TFG-Flow, construct guided flow that aligns with target predictor while preserving the marginals of unguided flow, effectively enabling plug-and-play guidance (Theo- rems 3.1 and 3.2). Unlike continuous variables, where gradient information is inherently informative, discrete variables cannot be adjusted continuously. Naive guidance methods that estimate transition probabilities between discrete states suffer from the curse of dimensionality, long-lasting problem in machine learning that is computationally intractable. To address this, TFG-Flow devises a consistent Monte- Carlo sampling approach that reduces the complexity from exponential to logarithmic for discrete guidance (Theorem 3.4), while leveraging partial derivative-based guidance mechanism for continuous variables that preserves geometric invariance (Theorem 3.5).
We apply TFG-Flow to various inverse molecular design tasks. When targeted quantum properties, TFG-Flow is able to generate more accurate molecules than existing training- free guidance methods for continuous diffusion (with an average relative improvement of +20.3% over the best baseline). When targeted to specific molecular structures, TFG-Flow improves the similarity to target structures of unconditional generation by more than 20%. When targeted at multiple properties, TFG-Flow outperforms conditional multimodal flow significantly. We also apply TFG-Flow to baeket-based drug design tasks, where TFG-Flow can guide the flow to generate molecules with more realistic 3D) structures and better binding energies towards the protein binding sites compared to the baselines.
Our main contributions are summarized as follows:
We present TFG-Flow, a novel approach for guiding multimodal flow models toward target properties in training-free manner (via off-the-shelf time-independent property functions).
2

---

Published as a conference paper at ICLR 2025
TFG-Flow sets solid theoretical foundation for multimodal guided flow, addresses the curse of dimensionality, and maintains geometric invariance via theorems 3.15
Experiments reveal that TFG-Flow is effective and efficient for designing 3D molecules with desired properties, opening up new opportunities to explore the chemical space.
2
PRELIMINARY
In this section, we provide readers with the essential knowledge required for 3D molecule generation using a multimodal flow model.
2.1 SO(3) INVARIANT 3D MOLECULE MODELING AND GENERATION
3D molecular representations. Let A denote the set of all atom types plus "mask" state [M] which indicates an undetermined atom type and will be useful in our generative model. molecule with atoms can be represented as G (X, a.) G, where = IR3x x A", X denotes the atomic coordinates, and a represents the atom types. We also use (+) and ( denote the coordinates and type of the i-th atom, respectively.
Invariant probabilistic modeling for 3D molecules. Invariance is crucial inductive bias in mod- cling 3D geometry of molecules (Gilmer et al., 2017). Molecular systems exist in direeddimessional Euclidean space, where the group associated with rotations is known SO(3). In this work, we are interested in the distribution p( X, a) whose marginal of X satisfies the following property:
SO(3)- -invariance: p(X) = PSS) for any E SO(3).
(1)
Here, (3) denotes the set of all the rotation matrices in 3D space. Intuitively, the definition requires the likelihood a molecular structure to be invariant with respect to any transformation in SO(3).
Additionally, molecular representations need to be invariant to translations. To ensure this, always project the atomic coordinates to r {X] (0)) +... x (n) 0} via mean subtraction: Projecty (X) = X
1 a n2 Σ (1) 1T, where 1, denotes the n-dimensional all-one row vector. n
Equivariant graph neural networks. In this work, we follow existing research and employ equiv- ariant neural networks to model the SO(3)-invariant distribution of molecule structures (Hoogeboom al., 2022; Xu al., 2022). Specifically, we apply Equivariant Graph Neural Networks (EGNNs) to process molecular representations (Satorras et al., 2021). EGNN takes molecular representations G (X, a) as its input and translate the atom type f into d-dimensional embedding h E Rd for each atom. In each layer, EGNN updates atomic coordinates and embeddings as follows:
(i5) - @ (1( 1,,)
-260112 0m):
(2)
n m (1,5); On
60
(1)
x <- x
x (1) x(1)) &y (m)( (x3); 0x),
(4)
In the above, (i.i) denotes intermediate message from atom to j, and om. o ox are learnable modules parameterized by em. fn, 0x respectively. We will formally prove that this architecture. along with the design of our algorithm, leads to SO(3) -nvvarian distributions.
2.2 MULTIMODAL FLOW MODEL
Multiflow a multimodal flow model originally developed for protein co-design (Campbell et al., 2024). In this work, we adapt for small molecule design. Multiflow constructs probability flow (G.) for t E [0, 1]. where Pn (Go) = Pnoise (G)) and pi (Gi) Pauza During inference, one first samples Go Po and then generates sequence of G; values by simulating the flow.
3

---

Published as a conference paper at ICLR 2025
Conditional flow, velocity, and rate matrix. At the core of the sampling process is the multimodal conditional flow² Ppp (G, By design, this flow can be factorized over both the number of atoms and their modalities:
Pej [u. = II Puja (a)) (0) |a) Prp (a) (a
(5)
In the above, the flows Pata (m) (1)
(z) (continuous) and ptja (a; (6) ) (discrete) are defined such
that they transport the noise distribution to the data distribution following straight-line paths, which
is known rectified flow (Liu et al., 2022): Dt (1) (6) ((aa (0) (1 1)²1) and a; [q] Cat(té(a;
(8) (9)
(6) at ( + (1 (1-18 t) [M] ( }). Here, 8(b, b' is the Kronecker delta, which is when b= 6' and 0 otherwise, and Cat denotes a Categorical distribution over A.
With the definition, we can build in closed form the conditional velocity Maj2 (-1xz)) and conditional rate matrix Raji (laa (a(")) which characterize the dynamics of the above conditional distributions:
Ra bla o(b, af 35(a,) [M]
B (a. (6) (1) 21
Ft
(6)
1-t
Deriving the unconditional flow. With Eq. (7), the conditional velocity and rate matrix can then be used to derive their unconditional counterparts by sampling from P1te (G1 [..) which the flow model learns during training. In Multiflow and our implementation, the flow model takes G3 as the input and predict the expectation of Ej for the continuous part well as the distribution Pija (a1 (.)) for the discrete part. Note that directly modeling the expectation E [ Gt) instead of the distribution Pije(ann (11jj) Gt)) is not problem since 2//1 is linear w.r.t. æı, which will be discussed in App- C.4.
ve (zt (6) = E (G. dlaa (αt (1) [x]")]
Re(ot (8) b) = (p. Rap (at" (1) bla()) (7)
The unconditional continuous and discrete flows satisfy the Fokker- Planck and Kolmogorov Equations: O.p: = -V. (V:p.) and o.p: = R. Therefore, during inference, starting from an initial sample Go Po one can iteratively estimate the unconditional velocity and rate matrix at the current time step t by sampling from the learned distribution Pije ()I[G]]),) The Fokker- Planck and Kolmogorov Equations can then be simulated generate Gz+Ar for the next time step, ultimately resulting in
G ~ Pi which approximates the data distribution.
METHODOLOGY
Problem setup. Our objective is develop effective, training-free method to guide uncondi- tional multimodal flow model to generate samples with desired properties. Formally, let e denote our target property, and assume we have time-independent target predictor Sa(G,) = p(c|G.) that quantifies how well a given molecule G2 satisfies the target property c. We also have access to a model g( (G.), which represents Pije (ije(Gi G.) of a flow model {p: (G.)))))ee (G)))re(o.). i.c., it takes G2 as input and returns the sample/iikclhhodd of G₁, as defined in Sec. 2.
In this section, we present our construction of multimodal guided flow satisfies pi (Gi Paama (Gi c). Then derive an algorithm which simulates the constructed flow using the target predictor (((G)), the flow model g( (G.), and a suitable velocity and rate matrix.
that
3.1 CONSTRUCTING THE GUIDED FLOW
Recall that the critical factor in the inference with flow models is to derive the velocity and rate matrix in the Fokker-Planck and Kolmogorov Equations so that we can simulate the sample over time. For guided generation, we need to address the following two challenges:
CConditional flow refers the flow Pap (G ( (G()) conditioned on the clean data G This contrasts with the desired unconditional flow for sampling, i.e., pe (G(")) which can derived using Eq. (7) from the conditional flow.
In the Kolmogorov Equation, the discrete flow should be as vector p. (a))))) In practice, we sample z6 ~ (N Ia) and ag [M] independently for each i. As discussed in Sec. 2.1, we additionally project X, the simplex
4

---

Published as a conference paper at ICLR 2025
Can construct the guided flow pr and derive the corresponding guided velocity and rate matrix in the Fokker-Planck and Kolmogorov Equations of it?
Can we efficiently estimate the guided velocity and rate matrix and simulate the Fokker- Planck and Kolmogorov Equations?
Constructing appropriate guided flows. While infinitely many flows (e(Gloo)e (o.1 satisfy (G)](c) = Pauaa P (G]c),, we seek to specify a joint distribution of ((G))e((.1)) c) which can be simulated via minimal adaptation of the original flow when incorporating the guidance fo(Gi). An ideal guided flow should maintain three key characteristics: (1) preserve the overall structure and dynamics of the original flow model by maintaining the flow marginals {Gr)eejo.1j- (2) align with the conditional distribution Je(G,) p(c| G ); and (3) ensure independence between {G, (e (0.1) and the condition when the clean data G1 is observed. These desiderata allows us to minimally modify the original flow while effectively steering the generation process towards the desired molecular properties. We formulate these intuitive criteria mathematically and prove the existence of a guided flow satisfying these conditions in the following theorem:
Theorem 3.1 (Existence of the guided flow (informal). Let G be the space of molecular rep- resentations and C be finite set which includes all the values of our target property. Given G-valued process
modeled by flow {pt ( (Gi))e and a function fel (G) which defines a valid distribution over C for any G1 G, there exists joint distribution p of random variables (G.)) (E)))(o..1) c) that satisfies the following:
Preservation of flow marginals: For any t E P. -f (G.).
Alignment with target predictor: For any E C and Gi E, p(c( Sc(G)).
Conditional independence of trajectory and target: For any t E 10. Gt and are = Ptp (T)
conditionally independent given G1, i.e.,
The formal version of Theorem 3.1 based on measure theory, along with its proof, can be found in App. B.1. We note that the conditional independence of trajectory and target is particularly appealing because it ensures that the central tool in flow model the conditional flow remains unchanged under guidance. Specifically, the conditional independence of trajectory and target ensures that for Multiflow model (defined in Sec. 2.2) under guidance, the guided conditional flow Pajz (G: G still satisfies the factorization property as defined in Eq. (5):
Piji GG. : II Pau (2) Ia (a)
(8)
Furthermore, in the guided flow, the flow marginals are preserved, hence the conditional velocity and rate matrix for the continuous and discrete conditional flows are the same as those defined in Eq. (6).
Finding the guided velocity and rate matrix. Parallel to Sec. 2.2, it now suffices to derive the unconditional guided velocity and rate matrix for the continuous and discrete flows based on the conditional counterpart. We present our result in the following theorem:
Theorem 3.2 (Guided velocity and rate matrix (informal)). Consider a continuous flow pr(at;) with conditional velocity deg (ag (a)))))) and discrete flow model pt pe(at) with conditional rate matrix R2 1 (a,, blØ1). Then the guided flows (c)) and pe(a, (e),e) defined via the construction in Theo- rem 3. 3, can be generated by vr (xx((c and R2 Eal,,, jlc) via Fokker- Planck Equation θ.pt -V-(pppp))
and Kolmogorov Equation d.p. = RrR, where
Guided velocity:
vx(xx(c) Eppje (m).e) Dep (x<|<=))
(9)
Guided rate matrix: Ra(a,, blc) = Epp (arlaae)
(ay blar)]
(10)
The formal statement and proof are in App. B.2. The conditional velocity dejs and conditional rate matrix R4 for Multiflow are defined in Eq. (6). Thus to simulate the guided flow pe (G, (c). it suffices to sample from Pije ( c) so that we can estimate the velocity and rate matrix. In the subsequent subsections, we present sampling methods for the discrete and continuous parts of the flow.
5

---

Published as a conference paper at ICLR 2025
3.2 DISCRETE GUIDANCE: ADDRESSING CURSE OF DIMENSIONALITY
Recall that the learned Multiflow model can output Pije(ai G+) (G.) for ai E AA. Furthermore, one can show via Theorem 3.1 that Pije(ar(r,, c) x Pije (ie(aa)Gi) (o) where fc(.) is the target predictor. Thus, (a (") -jlc) can be exactly calculated based on Eq. (10). However, we note that this is computationally intractable because of the curse of dimensionality: The complexity of exactly calculating the expectation is O(IA growing exponentially with the number of atoms n.
Therefore, it is natural to resort to Monte-Carlo approaches to estimate Ra ble). One straight- forward idea is to consider importance sampling. We show in Proposition B.4 that the guided rate matrix defined in Theorem 3.2 satisfies
(6)
(9) .lc) = (.. Re
p(c|Gr) p(c]G.)
(a (aaaa//lla))) bla{))
(11)
However, this approach requires access to p(c] i.e., time-dependent target predictor. This requires additional classifier training and prohibit the use of off-the-shelf predictors. Thus, importance sampling is not directly applicable and we build our method based on the following observation: Proposition 3.3. For & E [0, 1), the guided rate matrix defined in Theorem 3.2 satisfies
Re(a," (6) blc) (Pn(), [e( "blaa"
(12)
Epie(Ca (G.) [fc(Gr)]
Proposition 3.3 indicates that we can generate K samples from Pıje Pije (G)G., and obtain a good estimation of the guided rate matrix based on the known conditional rate matrix Raji (a{), (a,),)aa bba{() and target predictor fc(G1) We conduct theoretical analysis on the approximation error of this method in Theorem 3.4. We show that K O (log(n).A)) samples suffice to accurately approximate
Re Ra(a) ble) for any E [1,... n) and b E AA with high probability. This is in sharp contrast to the O("A complexity of exact expectation calculation, avoiding the offe----iiiiinaiittyy issue. Furthermore, our method does not require access to unknown quantities such as p(c(G,) in Eq. (11).
We show in the experiments that this approach can effectively simulate the discrete component of the multimodal guided flow.
Theorem 3.4. Let G "I GIIKKK ~ iid. P.))l-(G.). Define the estimation of Re Ra(a blc)
(1)
K
Re(a (1) Mc) Σ JalGai
(1)
(1le,h) Σ So(GuHe.)
((8)
(13)
Assume -== inf [c(G) O. Given any e (o/ ó (0,1), i/K θ GEg
nzz log -(al) then
sup iE(L,mamh m BEA
[Ra(aa") blc) Re(a, blc) <e 21-
(14)
3.3 CONTINUOUS GUIDANCE: BUILDING EQUIVARIANT GUIDED ODE
For the continuous component, recall that the learned Multiflow model predicts the expectation E, E [X, G/] given G; = (X,, a,) as the input, while we need [x to estimate the velocity for the guided flow demonstrated by Theorem 3.2. To bridge the gap, our method aims to generate G, (X,, , a.) such that P.pe( G.) Pije(: C) so that we can generate Kle from the given flow model and estimate the velocity.
Intuitively, X, needs to contain more information on the target e to impose the guidance on X1Ce- Motivated by training-frre guidance methods developed for continuous diffusion models (Bansal et al., 2023; He et al., 2024; Ye et al., 2024), we employ a gradient ascent strategy to align the modeled expectation of X,, with the target e. The gradient then propagated to X,, enabling us to derive the desired X, Specifically, we simulate the following iterative process:
X, X, Pt. Vx, x, log [E,X) X,, a:));
(15)

---

Published as a conference paper at ICLR 2025
where E[X||X,, ar] is approximated by the flow model, and ps is a hyperparameter to control the strength of guidance. We run the update for Nittr steps to obtain G, = (X,, az). Then we can estimate the guided velocity based on Theorem 3.2 and simulate the Blakerr-Paack Equation of the guided flow.
Additionally, we note that the continuous variables in our application represents coordinates in the 3D space. Translations and rotations in the 3D space should not change the likelihood of the variable. To ensure this property, we adopt two techniques: (a) As discussed in Sec. 2.1, we ensure that coordinates after translations are mapped to the same molecular representations by projecting the coordinates to the simplex I = {X xx1) + ....... + (n)) = 0}. When generating X, via Eq. (15) in our continuous
guidance algorithm, we implement the following update:
X, f Projectr (X; + P.Vx, x, / X,, (x,
(16)
(b) For rotation transformations, the flow {Px a [ needs to be S0(3)-invrrian w.E.t. X.,
as defined Sec. 2.1. We use Equivariant Graph Neural Networks (EGNN) as the backbone for the model go so that any SO(3) transformation on X; will lead to the same transformation on IlI P-)r(-|G,), i.e., the model ensures equivariant sampling (Satorras et al., 2021). We mathematically show that our techniques guarantees SO(3) invariincce of the simulated flow:
Theorem 3.5 (S0(3)-invariane) (proof in App. B.1)). Assume the target predictor fo(G) is SO(3)- invariant, the model go (G) is -qquiivaaaa and the distribution of Xu is S0(3) -invarann,,, flow then for any E [0,1], 1]. pa Po (Xilc) in the simulated multimodal guided flow is (0(3)-invarinl
3.4 TFG-FLOW: PUTTING THEM ALL TOGETHER
We combine the techniques of guiding the discrete component from Sec. 3.2 and the continuous component from Sec. 3.3 into our proposed TFG-Flow, which is illustrated in Figure 1(b) and presented as pseudo code in Algorithm 1. Specifically, TFG-Flow guides the molecules G. at each time step. For the discrete component a,, samples K Monte Carlo samples estimate the guided rate matrix R(at
(t) 4 Aelc) using Eq. (13). For the continuous component Ør, it simulates the ODE in the simplex T for Nier iterations with Eq. (15) and obtain the guided velocity v, (xx (c).
From implementation standpoint, also introduce temperature coefficient to adjust the strength of discrete guidance by controlling the uncertainty of the target predictor fe(G) = softmax(] (G: (/)T), where [(; vb) is typically an EGNN parametrized by db trained to classify or regress the property e of a molecule G. In the experiments, we set K = 512 and Niter = 4 to balance computational efficiency, and tune p, T via grid search (see App. D for details).
EXPERIMENTS
In this paper, we explore the application of TFG-Flow across four types of guidance targets: single quantum property, combined quantum properties, structural similarity, and target-aware drug design quality. Quantum properties examined using QM9 dataset (Ramakrishnan et al., 2014), while structural similarity is assessed on both QM9 and the larger GEOM-Drug dataset (Axelrod & Kooee-Bombareed 2022). The target-aware drug design quality is tested using CrossDocked2020 dataset (Francoeur et al., 2020). The baseline implementation and datasets details are in App. D.
4.1 QUANTUM PROPERTY GUIDANCE
Dataset and models. We follow the inverse molecular design literature (Bao et al., 2022; Hooge- boom et al., 2022) to establish this benchmark. The QM9 dataset is split into training, validation, and test sets, comprising 100K, 18K, and 13K samples, respectively. The training set is further divided into two ovn-overlap halves. To prevent reward hacking, use the first half to train a property prediction network for guidance and an unconditional flow model, while the second half is used to train another property prediction network that serves the ground truth oracle, providing labels for MAE computation. All three networks share the same architecture as defined by EDM, with an EGNN the backbone. In inference, we use 100 Euler sampling steps for the flow model.
For simplicity, we did schedule p and T (e.g., by increasing decreasing them over time).
7

---

Published as a conference paper at ICLR 2025
Table 1: How generated molecules align with the target quantum property on QM9 dataset. The results for Upper bound, #Atoms, Lower bound, Cond-EDM, EEGSDE are copied from Bao et al. (2022), and the results for training-fie free methods for diffusion are copied from Ye et al. (2024). The results of Cond-Flow and TFG-Flow are averaged over 3 seeds, with detailed results reported in App. E. Among all traiiin-basse or training-free methods, the best MAE for the same target is in bold, while the second best MAE is underlined.
Baseline cafegory
Method
Model category
C,
"
A
EHOMO
ELUMO
Upper bound #Atoms
6.87
1.61
8,98
1464
645
1457
Reference
1.97
1.05
3.86
886
426
813
Lower bound
0.040
0.043
0.09
65
39
36
Cond-EDM EEGSDE
1.065
1.123
2.78
671
371
601
Continuous Diffusion
Training-
0,941
0.777
2.50
487
302
447
Cond-Flow
Multimodal Flow
1.52
0.962
3.10
805
435
693
DPS I.GD
5.26
63.2
51169
13800
744
NA
3.77
1,51
7.15
1190)
664
1200
FreeDoM MPGD
Continuous Diffusion
2.84
1.35
5.92
1170
623
1160
Training- free
2.86
1,51
4.26
1070
554
1060
UGD
3.02
1.56
5.45
1150
582
1270
TFG
2.77
1.33
3.90
893
568
984
TFG-Flow
Multimodal Flow
1.75
0.817
2.32
804
364
941
Table 2: How generated molecules aliga with the target quantum property. Our training-free guidance TFG-Flow
significantly outperforms the conditional flow (Cond-Flow) which requires condition-lahelled data for training.
Method
MAEL
MAE2|
Method
MAEIJ
MAE2|
Method
MAEL MAE2
Co cal K), (D)
Ae (m. a (D)
a (Bohr") ). a (D)
Cond-Floo
4.96 10.00
1..0
Cond-Flow
53.5
1.57
Cond-Flow
9.33 10.01
1.54 1.54-001
TFG-Flow
2.36 10.01
1.13 10.04
TFG-Flow
46.4 0016
0.853
0.05
TFG-Flow
4.62 10.01
1.24 000
Guidance target. We study guided generation of molecules on quantum mechanics properties, including polarizability a (Bohr"), dipole moment μ (D), heat capacity C, ( (cal K), highest orbital energy tHомо (meV), lowest orbital energy ÉLUMO (meV) and their gap Ae V). Denote the property prediction network as then our guidance target is given by an energy function f(G) := exp(-IlE(G) - <ll2), where c is the target property value. For combined properties, we combine the energy functions linearly with equal weights following Bao et al. (2022) and Ye et (2024).
mol
Evaluation metrics. We use the Mean Absolute Error (MAE) measure guidance performance. We generate 4,096 molecules for each property to perform the evaluation following Ye et al. (2024). For completeness, also report more metrics such validity, novelty and stability in App. D.2.
Baselines. As the training- -fren guidance for multimodal flow is not studied before, the most direct baselines are training-free guidance methods continuous diffusion. We compare TFG-Flow with DPS (Chung& & Ye, 2022), LGD (Song et al., 2023), FreeDoM (Yu et al., 2023), MPGD (He et al., 2024), UGD (Bansal et al., 2023), and TFG (Ye et al., 2024) which treat both atom types and coordinates as continuous variables and perform guidance using gradient information. We also compare TFG-Flow with training-baser baselines, such as the conditional diffusion model (Cond- EDM (Hoogeboom et al., 2022)) and conditional multimodal flow (Cond-Flow)) and EEGSDE (Bao et al., 2022), which applies both training- based guidance and conditional training. We also provide referential baselines following Hoogeboom et al. (2022) (see App. D.3 for details).
Results analysis. The experimental results for single and multiple quantum properties are shown in Tables I and 2, respectively. First of all, our TFG-Flow exhibits the best guidance performance among all training-fre guidance methods, with an average relative improvement of +20.3% over the best training-free guidance method TFG continuous diffusion. Also, we note that TFG-Flow is comparable with Cond-Flow on single property guidance and outperforms Cond-Flow significantly on multiple property guidance, while Cond-Flow requires conditional training. These results justify that the discrete guidance is more effective for the discrete variables in nature. It is also noteworthy that all the training-fre free guidance methods underperform EEGSDE by a large margin, suggesting a large room for improvement in training-free guidance.
8

---

Published as a conference paper at ICLR 2025
Table 3: How generated molecules align with the target structure. The Tanimoto similarity results are averaged over random seeds.
Table 4: Evaluation of generated molecules for 100 pro- tein pockets of CrossDocked20211 test set. The results of AR, Pocket2Mol, TargetDiff are from Guan et al. (2023).
Dataset
Baseline
Similarity?
Vina scorel
QED1
SA
Upper bound
0.164 0.14440.001
Test set
-6.87-2.30
0.48+0.20
0.73#0.11
QM9
TFG-Flow
0.290=0.000
AR
-6.20=1.25
0,50
0.67
Rel. improvement
+76.83%
Pocket2Mol
-7.23=2.04
0.57
0.75
Upper bound TFG-Flow 0,208+t0.000
0.17020.001
TafgetDiff
-7.32-2.08
0.48++0.15
0.61
GEOM-Drug
Multiflow
-7.01-1.81
0.45:t0.21
20.10
Rel. improvement
+22.35%
TFG-Flow
-7.65
0.47#0.19
0.64
4.2 STRUCTURE GUIDANCE
Guidance target. Following Gebauer et al. (2022), represent the structural information of molecule using its molecular fingerprint. This fingerprint, denoted as c (cI...,,c2, consists of sequence of bits that indicate the presence or absence of specific substructures within the molecule. Each substructure corresponds to particular position in the bit vector, with the bit o if the substructure is present in the molecule and 0 if it is absent. To guide the generation of molecules with desired structure (encoded by the fingerprint c), we define the guidance target as f(G) exp(-I[E(G) ell?). Here, & refers to a multi-label classifier trained using binary cross-entropy loss to predict the molecular fingerprint, as detailed in App. D.2.
Evaluation metrics. We use Tanimoto coefficient (Bajusz et al., 2015) TC 02) = Jes Nez]
Deg| to
measure the similarity between the fingerprint ej of generated molecule and the target fingerprint C2.
Results analysis. The results are shown in Table 3. TFG-Flow improves the similarity uncondi- tional generative model by 76.83% and 22.35% on QM9 and GEOM-Drug, respectively. But we still note that the Tanimoto similarity of 0.290 and 0.208 are not satisfactory for structure guidance. We will make efforts to improve training-frees guidance for better performance this task in the future.
4.3 POCKET-TARGETER DRUG DESIGN
We also introduce novel benchmark for training-free guidance. In practical drug design, the goal typically to create drugs that can bind to specific protein target (see related work discussion in App. C), making the inclusion of pocket targets more realistic setting for guided generation. To enhance the effectiveness of drug design, we aim for the generated molecules exhibit strong drug- like properties, demonstrate high binding affinity to the target pocket, and be easily synthesizable. We integrate these criteria into our TFG-Flow to guide the drug design.
Datasets and Models. We utilize CrossDocked2020 training set train both the unconditional flow model and the drug quality prediction network. Unlike QM9 and GEOM- the input graph G to the flow model includes not only the coordinates and atom types of molecules but also the protein pocket, which remains fixed during message passing in the EGNN. Also, there is need to train an oracle target predictor, as the relevant metrics can be derived from publicly available chemistry software. Further details regarding the network and dataset are provided in App. D.
Guidance target. We use Vina score, QED score, and SA score as the proxy of binding affinity between the molecules and the protein, the drug-likenee of molecule, and the synthetic accessibility of molecule, respectively. We combine the three scores as holistic evaluation of the drug quality via linear combination: -0.1 Vina score + QED SA, and train a quality prediction network E(G E(G) to approximate the value. The guidance target is given as fGG) E(G).
Evaluation metrics. We compute Vina Score by QVina (Alhossary et al., 2015), SA and QED by RDKit. The metrics are averaged over 100 molecules per pocket in CrossDocked test set.
Baselines. We compare TFG-Flow with different state-of-the-an target-aware generative models AR (Luo et al., 2021). Pocket2Mol (Peng et al., 2022), and TargetDiff (Guan et al., 2023). We implement Multiflow for target-aware molecular generation and apply TFG-Flow on it.

---

Published as a conference paper at ICLR 2025
Nyw
Validity
T
Validity
3.5
0.07
0.70
3.0
6.6
9.70
9.75
6.79
0.2
6.6
0.0
0.20
9.77
6.77
640
0.70
1.0
0.201
2012
0.22
Figure 2: Efffect of varying hyperparameters (Niters T. p. K). The line plot indicates the validity (left y-axis, higher values are better), while the bar plot shows the MAE (right y-axis, lower values are better).
Results Analysis. As binding affinity is the most important metric in target-aware drug design, we prioritize Vina score in the design of our quality score. The results show that TFG-Flow improves the drug quality over unguided Multiflow and achieves the best Vina score among all the drug design methods. We also note that the autoregressive methods (AR, Pocket2Mol) generally possess better QED and SA results than diffusion- methods (TargetDiff, Multiflow, TFG-Flow), but with weaker binding affinity. Therefore, TFG-Flow is quite effective for target-aware drug design.
4.4 ABLATION STUDY
To understand how different hyper- paremeteasetesss (Nitor P, K) affect the performance of TFG-Flow, we conduct ablation study on quantum property guidance for polarizability a. On this task, the searched (p: 7) is (0.02, 10) and recall that we set K = 512 and Niter constantly. In our ablation, we fixed all the other prram-paraetters and change one of them to grid of values except for the study of Miterr and plot the validity (the ratio of valid molecules) and guidance accuracy (MAE) in Figure 2. For Niter, we present the results with the best (p, r) for the corresponding Niter-
The study on Niter shows Niter 4 already achieves good performance while maintaining compu- tational efficiency, as Niter 8 don't 't bring significant improvement. For other experiments, we could see a positive correlation between validity and MAE as trade-off between the quality of unconditional generation and the desired property alignment. Importantly, We note that the number of samples K z 16 is sufficient in our discrete guidance (Eq. (13)), which demonstrates the efficiency of our estimation technique with fast convergence rate. We can also learn that too strong guidance strength and t may not improve the guidance (MAE) but will severely deteriorate the validity.
5 DISCUSSIONS AND LIMITATIONS
The related work is reviewed in App. C. Our TFG-Flow complements the trend of generative modeling through the straightforward and versatile flow matching framework (Esser et al., 2024). It also unlocks the guidance for multimodal flow (Campbell et al., 2024) and has been applied effectively to both target-agnostic and target-specific molecular design tasks. We notice that a concurrent research (Sun et al., 2024) explores training-free guidance on continuous flow in image generation, however, we have identified several theoretical concerns with this approach as detailed in App. C.2. Overall, our TFG-Flow proves to be both novel and effective, with solid theoretical foundations.
Though TFG-Flow boosts the state-of-the-art training- free guidance, performance gap persists between training-basee based and training-free methods (Table 1). But as training- free guidance allows for flexible target predictor, we replace the guidance network as a pre-trained foundation model UniMol (Zhou et al., 2023) for Table in App. D.6, where the performance gap with EEGSDE is further narrowed. We also notice that some literature indicates that training-free guidance tends to perform well in powerful foundation models, such as Stable Diffusion (Ye et al., 2024; Bansal et al., 2023; Yu et al., 2023). This suggests that more capable models which Iearn from diverse data could possibly offer better steerability. On top of that, the future trajectory for Al-Driven Drug Design might involve developing large generative foundation models and applying training- guidance seamlessly to achieve desired properties. Beyond molecular design, insights multimodal guided flow are broadly applicable to other fields such as material, protein, or antibody. Given that multimodality encompasses both discrete and continuous data types, TFG-Flow provides general framework that could handle all kinds of guidance problem. We hope that TFG-Flow will inspire further innovation within both the generative modeling and molecular design communities.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
Haowei Lin and Jianzhu Ma were supported by the National Key Plan for Scientific Research and Development of China (Grant No. 2023YFC3043300), China's Village Science and Technology City Key Technology funding, and Wuxi Research Institute of Applied Technologies, Tsinghua University (Grant No. 20242001 120). Additionally, Haowei Lin and Yitao Liang received partial funding from the National Key Plan for Scientific Research and Development of China (Grant No. 20227D0160301).
REFERENCES
Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneherger, Lindsay Willmore, Andrew Ballard, Joshua Bambrick, et Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pp. 1-3, 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.00005 087774, 2023. 1, 28
Michael S Albergo and Eric Vanden-Fijnden Building normalizing flows with stochastic interpolants. arXiv preprint arXiv:2209 2022. 28
Michael S Albergo, Nicholas M Boffi, and Eric Vanden-Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:2303. 087997 2023. 28
Amr Alhossary, Stephanus Daniel Handoko, Yuguang Mu, and Chee-Keong Kwoh. Fast, accurate, and reliable molecular docking with quickvina Bioinformatics, 31(13):211-2211, 2015.
William J Anderson.
Continuous-time Markov chains: An appliciaaiiooss-rreeete approach. Springer
Science & Business Media, 2012. 2
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusson models in discrete state-spaces. Advances in Neural Information Processing Systems,
2021. 28
Simon Axelrod and Rafael
Geom, energy-annotated molecular conformations
for property prediction and molecular generation. Scientific Data, 9(1):185, 2022. 7, 28
Dávid Bajusz, Anita Rácz, and Károly Héberger. Why tanimoto index an appropriate choice for fingerprint-based similarity calculations? Journal of cheminformatics, 7:1-13, 2015. 9
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 843-852, 2023, 6, 8, 10, 29,32
Fan Bao, Min Zhao, Zhongkai Hao, Peiyao Li, Chongxuan Li, and Jun Zhu. Equivariant energy- guided sde for inverse molecular design. In The eleventh international conference learning representations, 2022. 7, 8, 32
Heli Ben-Hamu, Samuel Cohen, Joey Bose, Brandon Amos, Aditya Grover, Maximilian Nickel, Ricky TQ Chen, and Yaron Lipman. Matching normalizing flows and probability paths on manifolds. arXiv preprint arXiv:2207.047111 2022. 28
Heli Ben- Ben-Hamu, Omri Puny, Itai Gat, Brian Karrer, Uriel Singer, and Yaron Lipman. D-flow: Differentiating through flows for controlled generation. arXiv preprint arXiv:2402.14011 2024.
29
Helen M Berman, John Westbrook. Zukang Feng, Gary Gilliland, Talapady N Bhat, Helge Weissig, Ilya N Shindyalov, and Philip Bourne. The protein data bank. Nucleic acids research, 28(1): 235-242, 2000. 28
11

---

Published as a conference paper at ICLR 2025
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models world simulators. 2024. URL httpss:/opeett
com/research/
tirio--gneraaiiiommddoooe www.ddeee---sooo.-ssss-mmaaaa..- world-simlatoor..
Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. arXiv preprini arXiv:2402.049997 2024. 1, 3, 10, 28, 30
Ricky TQ Chen and Yaron Lipman. Riemannian flow matching on general geometries. arXiv preprint arXiv:2302.03360, 2023. 28
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018. 28
Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self codditondiion,.g. arXiv preprint arXiv:2208.08 04200, 2022. 28
Zixiang Chen, Huizhuo Yuan, Yongqian Li, Yiwen Kou, Junkai Zhang, and Quanquan Gu. Fast sampling via de-randomization for discrete diffusion models. arXiv preprint arXiv:2312.09193, 2023. 28
Hyungjin Chung and Jong Chul Ye. Score-based diffusion models for accelerated mri. Medical image analysis, 80:102479, 2022. 8, 29, 32
Valentin De Bortoli, Emile Mathieu, Michael Hutchinson, James Thornton, Yee Whye Teh, and Arnaud Doucet. Riemannian score-based generative modelling. Advances in Neural Information Processing Systems, 35:2406-2422, 2022. 28
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 14:8780-8794, 2021. 1, 28, 29, 30
Kieran Didi, Francisco Vargas, Simon V Mathis, Vincent Dutordoir, Emile Mathieu, Urszula J Komorowska, and Pietro Lio. A framework for conditional diffusion modelling with applications in motif scaffolding for protein design. arXiv preprint
2023. 30
Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, et al. Continuous diffusion for categorical data. arXiv preprint arXiv:2211. 15089, 2022. 28
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning. 2024. 1, 10,28 28
Griffin Floto, Thorsteinn Jonsson, Mihai Nica, Scott Sanner, and Eric Zhengyu Zhu. Diffusion on the probability simplex. arXiv preprint 200.::230090253300 2023. 28
Paul G Francoeur, Tomohide Masuda, Jocelyn Sunseri, Andrew Jia, Richard B lovanisci. Ian Snyder, and David R Koes. Three-dimenniooal convolutional neural networks and a cross-docked data set for structure-based drug design. Journal of chemical information and modeling, 4215) 2020. 7, 28
Niklas WA Gebauer, Michael Gastegger, Stefaan SP Hessmann, Klaus-Robert Müller, and Kristof T Schütt. Inverse design of 3d molecular structures with conditional generative neural networks. Nature communications, 13(1):973, 2022. 9
Justin Gilmer, Samuel Schoenholz, Patrick Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp. 1273-1272. PMLRR, 2017. 3
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu. and LingPeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. arXiv preprint 1rXiv:2210.08933, 2022. 28
12

---

Published as a conference paper at ICLR 2025
Jiaqi Guan, Wesley Wei Qian, Xingang Peng, Yufeng Su, Jian Peng, and Jianzhu Ma. 3d equiv- ariant diffusion for target-aware molecule generation and affinity prediction. arXiv preprint arXiv:2303.03543, 2023. 9, 29, 32
Ishaan Gulrajani and Tatsunori B Hashimoto. Likelihood-based diffusion language models. Advances in Neural Information Processing Systems, 36, 2024. 28
Jürgen Haas, Alessandro Barbato, Dario Behringer, Gabriel Studer, Steven Roth, Martino Bertoni, Khaled Mostaguir, Rafal Gumienny, and Torsten Schwede. Continuous automated model evalua- tion (cameo) complementing the critical assessment of structure prediction in casp Proteins: Structure, Function, and Bioinformatics, 86:387-398, 2018. 28
Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. Ssd-Im: Semi-autoregrrssive simplex-based diffusion language model for text generation and modular control. arXiv preprint arXiv:2210. 2022a. 28
Xizewen Han, Huangjie Zheng, and Mingyuan Zhou. Card: Classification and regression diffusion models. Advances in Neural Information Processing Systems, 35:18100- 11000181111,, 2022b. 28
Yutong He, Naoki Murata, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei- Hsiang Liao, Yuki Mitsufuji, J preserving guided diffusion. The Twelfth International Conference on Learning Representations, 2024. URL https: /ttp:/://////eieeeeeeeeee wwtp/e/rwmm.dd00000lloa.o
Zico Kolter, Ruslan Salakhutdinov, and Stefano Ermon. Manifold
6, 8, 3.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207. 2022. 1, 30
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020. 28
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances Neural Information Processing Systems, 35:8633- 2022. 28
Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International conference on machine learning, pp. 8867-8887. PMLR, 2022. 3, 7, 8, 33
Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In International Conference on Machine Learning, pp. 13213-13232. PMLR, 2023. 28
Chin- Wei Huang, Milad Aghajohari, Joey Bose, Prakash Panangaden, and Aaron C Courville. Riemannian diffusion models. Advances in Neural Information Processing Systems, 35:2750-2761, 2022. 28
Han Huang, Leilei Sun, Bowen Du, and Weifeng Lv. Learning joint 2d & 3d diffusion models for complete molecule generation. arXiv preprint arXiv:2305.12344 2023. 28
John J Irwin, Khanh G Tang, Jennifer Young, Chinzorig Dandarchuluun, Benjamin R Wong, Munkhzul Khurelbaatar, Yurii S Moroz, John Mayfield, and Roger A Sayle. Zinc20 a free ultralarge-scale chemical database for ligand discovery. Journal of chemical information and modeling, 60(12):60556607, 2020. 28
Wengong Jin, Jeremy Wohlwend, Regina Barzilay, and Tommi Jaakkola. Iterative refinement graph neural network for antibody sequence sequence-structure structure co-design. arXiv preprint 1rXhv:2110.04624 2021. 28
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 04-26565-26577 2022. 28
13

---

Published as a conference paper at ICLR 2025
Korrawe Karunratanakul. Konpat Preechakul, Emre Aksan, Thabo Beeler, Supasom Suwajanakom, and Siyu Tang. Optimizing diffusion noise can serve as universal motion priors. In Proceedings of the IEEE/CVF Conference Computer Vision and Patter Recognition, pp. 1334-1345, 2024. 29
Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual pre-training of language models. arXiv preprini urXiv:2302.03244 2023. 28
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint M.XXx:2009.0976: 2020. 28
Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. Tabddpm: Modelling tabular data with diffusion models. In International Conference on Machine Learning, pp. 17564- 17579. PMLR, 2023. 28
Andriy Kryshtafovych, Torsten Schwede, Maya Topf, Krzysztof Fidelis, and John Moult. Critical assessment of methods of protein structure prediction (casp)-roun) xiv. Proteins: Structure, Function, and Bioinformatics, 89(12):1607-161,, 2021. 28
Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsumori B Hashimoto. Diffusion-lm improves controllable text generation. Advances in Neural Information Processing Systems, 35: 4328 4343, 2022. 28
Yuesen Li, Chengyi Gao, Xin Song, Xiangyu Wang, Yungang Xu, and Suxia Han. Druggpt: A gpt-based strategy for designing potential ligands targeting specific proteins. bioRxiv, pp. 2023-06, 2023. 29
Haowei Lin, Baizhou Huang, Haotian Ye, Qinyu Chen, Zihao Wang, Sujian Li, Jianzhu Ma, Xiaojun Wan, James Zou, and Yitao Liang. Selecting large language model to fine-tune via rectified scaling law, arXiv preprint arXiv:2402.02314, 2024. 28
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022. 28
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209. 220.00033000 2022. 4, 28
Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, et al. Instaflow: One step is enough for high-quality diffusion-based text-to-image generation. In The Twelfth International Conference Learning Representations, 2023. 28
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310.16834, 2023. 28
Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2837-2845, 202 la. 28
Shitong Luo and Wei Hu. Score-based point cloud denoising. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3583344597 2021b. 28
Shitong Luo, Jiaqi Guan, Jianzhu Ma, and Jian Peng. A 3d generative model for structure-based drug design. Advances in Neural Information Processing Systems, 34:6229-6233 2021. 9, 32
Shitong Luo, Yufeng Su, Xingang Peng, Sheng Wang. Jian Peng, and Jianzhu Ma. aatigen-specing antibody design and optimization with diffusion- generative models for protein structures.
Advances in Neural Information Processing Systems, 35:9754-9767, 2022. 28
Rabeeh Karimi Mahabadi, Hamish Ivison, Jaesung Tae, James Henderson, Iz Beltagy, Matthew E Peters, and Arman Cohan. Tess: Text-to-text self-conditioned simplex diffusion. arXiv preprint 00X.0:2305.08879 2023, 28
Tomohide Masuda. Matthew Ragoza, and David Ryan Koes. Generating 3d molecular structures con- ditional on a receptor binding site with deep generative models. arXiv preprint arXiv:2010.14442 2020. 29
14

---

Published as a conference paper at ICLR 2025
Alex Morehead and Jianlin Cheng. Geometry-compleet diffusion for 3d molecule generation and optimization. Communications Chemistry, 7(1):150, 2024. 28
Xingang Peng, Shitong Luo, Jiaqi Guan, Qi Xie, Jian Peng, and Jianzhu Ma. Pocket2mol: Efficient
molecular sampling based on 3d protein pockets. In International Conference on Machine Learning. pp. 17644-17655, PMLR, 2022. 9, 29, 32
Xingang Peng, Jiaqi Guan, Qiang Liu, and Jianzhu Ma. Moldiff: Addressing the atom-bond inconsistency problem in 3d molecule diffusion generation. arXiv preprint arXiv:2305.07500. 2023. 28, 32, 33
Yiming Qin, Clement Vignac, and Pascal Frossard. Sparse training of discrete diffusion models for graph generation. arXiv preprint arXiv:23/ 142 2023a. 28
Yiming Qin, Huangjie Zheng, Jiangchao Yao, Mingyuan Zhou, and Ya Zhang. Class-balancing diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18434-18443, 2023b. 28
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, (0)11-7,0014, 78
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac. Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055500 2024. 1
Pierre H Richemond, Sander Dieleman, and Arnaud Doucet. Categorical sdes with simplex diffusion. arXiv preprint arXiv:2210.14755 2022. 28
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-36444 2022. 28
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks. In International conference on machine learning, pp. 99332933.. PMLR, 2021. 3,
Arne Schneuing, Yuanqi Du, Charles Harris, Arian Jamasb, Ilia Igashov, Weitao Du, Tom Blundell, Pietro Lió, Carla Gomes, Max Welling, al. Structure-based drug design with equivariant diffusion models. arXiv preprint arXiv:2210.13695, 2022. 29
Yifei Shen, Xinyang Jiang, Yezhen Wang, Yifan Yang, Dongqi Han, and Dongsheng Li. Un- derstanding training- free diffusion guidance: Mechanisms and limitations. arXiv preprint urXiv:2403.12404 2024. 29
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International conference on machine learning, pp.
PMLR, 2015. 28
Jiaming Song, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming- Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. Loss-guided diffusion models for plug-and-play controllable generation. In International Conference on Machine Learning, pp. 32483-32498, PMLR, 2023. 8, 29, 32
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019. 28
Yang Song and Stefano Ermon. Improved techniques for training score- score-based generative models. Advances in neural information processing systems, 33:12438 12448, 2020. 28
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 1. 13466, 2020. 28
15

---

Published as a conference paper at ICLR 2025
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in neural information processing systems, 34:1415-1428, 2021. 28
Robin Strudel, Corentin Tallec, Florent Altché, Yilun Du Yaroslav Ganin, Arthur Mensch, Will Grathwohl, Nikolay Savinov, Sander Dieleman, Laurent Sifre, al. Self-conditioned embedding diffusion for text generation. arXiv preprint / 1.04236, 2022. 28
Zhicheng Sun, Zhenhao Yang, Yang Jin, Haozhe Chi, Kun Xu, Liwei Chen, Hao Jiang, Di Zhang, Yang Song, Kun Gai, et al. Rectifid: Personalizing rectified flow with anchored classifier guidance. arXiv preprint arXiv:2405. 0222405..46777 /4677, 2024. 10, 29
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023. 28
Bram Wallace, Akash Gokul, Stefano Ermon, and Nikhil Naik. End-to-end diffusion latent optimiza- tion improves classifier guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7280-7290 2023. 29
Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, al. Scientific discovery the age of artificial intelligence. Nature, 620(7972):47-60, 2023.
Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. arXiv preprint arXiv:2212.004900 2022. 29
Yongkang Wang, Xuan Liu, Feng Huang, Zhankun Xiong, and Wen Zhang. multi-modal contrastive diffusion model for therapeutic peptide generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 3-11, 2024. 28
Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L L'Tripp,, Jason Yim, Helen E. Eisenach, Woody Ahern, Andrew Borst, Robert J Ragotte, Lukas F Milles, et al. De novo design of protein structure and function with rídiffusion. Nature, (0099996):009-11000
2023. 30
Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. arXiv preprint 2XXv:2203.00923, 2022. 3, 28
Minkai Xu, Alexander S Powers, Ron O Dror, Stefano Ermon, and Jure Leskovec. Geometric latent diffusion models for molecule generation. International Conference on Machine Learning, pp. 38592-386111 PMLR, 2023. 28
Hanshu Yan, Xingchao Liu, Jiachun Pan, Jun Hao Liew, Qiang Liu, and Jiashi Feng. Perflow: Piecewise rectified flow universal plug-and-play accelerator. arXiv preprint arXiv:2405.07510 10, 2024. 28
Haotian Ye, Haowei Lin, Jiaqi Han, Minkai Xu, Sheng Liu, Yitao Liang, Jianzhu Ma, James Zou, and Stefano Ermon. Tfg: Unified training- free guidance for diffusion models. arXiv preprint, 2024. 1, 6, 8, 10, 28, 29, 30, 31, 32, 34
Jiasheng Ye, Zaixiang Zheng, Yu Bao, Lihua Qian, and Mingxuan Wang. Dinoiser: Diffused conditional sequence learning by manipulating noises. arXiv preprint arXiv:2302. 10025, 2023. 28
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprini arXiv:2206. 10789, 2(3):5, 2022. 29
Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training- free energy-guide guided conditional diffusion model. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 23174-2311 2023. 8, 10, 29, 32
16

---

Published as a conference paper at ICLR 2025
Shu Zhang, Xinyi Yang. Yihao Feng, Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang, Silvio Savarese, Stefano Ermon, et al. Hive: Harnessing human feedback for instructional visual editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9026-9036, 2024. 28
Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation. arXiv preprint arXiv:2302.05737, 2023. 28
Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molecular representation learning framework.
Theoretical and Computational Chemistry, 2023. 10,
Yuanzhi Zhu, Kai Zhang, Jingyun Liang, Jiezhang Cao, Bihan Wen, Radu Timofte, and Luc Van Gool. Denoising diffusion models for blug-nnd-play image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1219-1229, 2023. 29
Alex Zunger. Inverse design in search of materials with target functionalities. Nature Reviews Chemistry, 2(4))0121, 2018. 2
17

---

Published as a conference paper at ICLR 2025
Appendix
Table of Contents
A Pseudo code for TFG-Flow
19
B Omitted Mathematical Derivations
20
Formal Statement Proof of Theorem
20
B.2 Formal Statement and Proof of Theorem 3.2
22
B.3 Importance Sampling Fails for Sampling the Discrete Guided Flow
23
B.4 Proof of Theorem 3.4
24
B.S Formal Statement and Proof of Theorem 3.5
26
Discussion Related Work
C.I General Discussion
28
C.2 Theoretical Issues with RectifID.
29
C33 The Motivation of Studying Training- Guidance
29
C.4 Discussion of Continuous Part Modeling in Multiflow
30
Experimental Details
31
Target-agnostic Design Setting
................................
31
D.2 Implementation of TFG-Flo FFow
................................
31
D.3 Baselines
31
D.4 Dataset Preprocessing
32
D.5 Evaluation Metrics..
33
Leveraging Pre-trained Foundation Models as Target Predictor
34
D.7 Hardware and Software Specifications
34
E Detailed Experimental Results
35
E.1 Conditional Flow
35
E.2 TFG-Flow
36
18

---

Published as a conference paper at ICLR 2025
A PSEUDO CODE FOR TFG-FLOW
Algorithm Training-free Guidance for Multimodal Flow Inference
Input: Unconditional rectified flow model go. target predictor fe. guidance strength p. temperature number of steps Niter's temporal step size At.
1: Initialization
2: Sample " .... _00") i.d. NoO, 13),
Xo 3: Xo f- Projectr(XX)). ao - [DM).... DQJT: 400
2
(m)
4: while < do
Simulate Fokker- Blaker- and Kolmogorov Equations
5:
X(j1))((111)) + a.)
6:
7: Discrete guidance
8: Sample d1j1.1 aijeK ~ w.id p(an)))
9:
for = 1, n do
K
K ((.
affekk) Sample a afat ~ Cat(E(a at (3) + M(a,) (1) Att
R( (0) b) Σ
10:
1m
(6)
11:
12:
end for
13:
14:
Continuous guidance
15: Sample aI} ~ Cat(fe alj(.)))
b Sample aije from {aije, 1K based on fa
16: for 1,... Nier do
17:
X, < Projectr (X, + Pr Xx, log (oo ax)x, a11()
end for
19:
X11e - (x(X,, ax)x
for i 1,... do
6(x;") (1) t- Dzj1 ( (00) (1))
21:
a. attt at + a(rr (1)) A end for
(1)
(1)
23:
24:
Xt-A: t Projectr (Sojectt((x+aA) (X++11)
25:
26:
t=t+
27:
end while
28: Output: X1, a
19

---

Published as a conference paper at ICLR 2025
B OMITTED MATHEMATICAL DERIVATIONS
In this section, we present the omitted derivation proofs for our theoretical results. Note that we always we (re)state the theorem for ease of reading, even it has appeared in the main paper.
B.I FORMAL STATEMENT AND PROOF OF THEOREM 3.1
Theorem B.1 (Existence of the guided flow (formal version of Theorem 3.1)). Let be the space of molecular representutions and C be finite set which includes all the values of our target property. Given a G-valued process (E
probability space FG, PG), and a function Jc(G1) which defines valid distribution over C. for any G1 G. there exists joint probability measure P on the product space FD, = (Pg X F. & 2 and random variables ((Gr)ee(o.1)) c) on it that
satisfies the following:
Preservation of flow marginals: For any t1 < tm < I.
P(G Ct.)==P(GGu, G..)).
Alignment with target predictor: For any E C and Gi G.
Conditional independence of trajectory and target: For any 0 4 ... Im < (Gr,, ,G. and are independent conditioning on G
Proof. First, construct random variables ((G))
the product space (9, For Note that we overload the
(w).c) E Ω define Gi(wg, (u(we) (t E [0, 1]); c(wa, notations [Gr)tejo.1] and e and redefine them as random variables in the new product space for simplicity.
Second, we construction of the joint probability measure P on the product space (S2 F).
For each Gi₃ G. define probability measure G on Qg by
G (Ec) = Pa (Ea G = G for any Eg E Fg.
(17)
Again, for each G E G, define probability measure p on by (d) (c) fe (Gi) By definition of for PC is valid distribution.
Then, we can define pC; on 5 as
p (EG
Ec) = P (Ec) for any EG E Fa; Ec 2
(18)
We obtain the joint probability measure IP the product space (I,F) by integrating over G¹ G:
P(E) = [ Ja pol (E)FF..))()) for any F,
(19)
where (-) is the marginal distribution (law) of G1 in (Rc, Pc).
Now we are ready to verify that the above joint probability measure P satisfies the desired properties.
Preservation of flow marginals. For any EG Fc.
P(EG C) / pGi (Eg x C) | p Ce' (Ec)P..(dG))) Po(EG). (20)
Thus, the marginal distribution of (Gt) under P is PG- Specifically, for any 0 0.0 t .... < <m 1,
P(G G...)) S.PP PPGGG Gt..).
20

---

Published as a conference paper at ICLR 2025
Alignment with Target Predictor. Note that in our definition, pci(.) P(( G1 Gi). Therefore, for any G₁ E G and c' E C, we have
P(c= d' G1 G₁) = pCi (Ng x {d')) = P (L)) = P ((d')) = fc(G)). (21)
((('))
This establishes the specified alignment with the target predictor.
Conditional independence of trajectory and target. Let S < 12 < ... < fm < 1, and let SP : 9 + R and ycc C - R be any bounded measurable functions. We need to show that
Ep pelGGG
G1] Eg (po
,G,,) G Ep (pc(c) G1].
Note that under Pc. given G1, the random variables (Gi Gt...) depend only wa, and depends on wa only through G1. Therefore.
Ee (PGG
"G.......GGG) G1]
(22)
- PG (G, c), Gtn (w/g c')) po (Wa, c)) pCi (dw'
(23)
naxc
- Pc (G, c), Gt., (w'. d Pc (W' cf pg (dw'c) p (de')
(24)
(1r (na pa (w6)), (wG:)) P& (lwet) po(d))) pe (dc))
(25)
Bp (pa(G Gt..)) Eepe [(c(c))
(26)
-Ep (ya (GraGr
.t...) Gi] Ep ((c(c) Gi),
(27)
indicating that ".. and are independent conditioning on G1.
To sum up, we have constructed probability measure IP on FF) and verify that IP satisfies the desired properties, which concludes the proof.
Remark. It's easy to see that our proof also applics when C = R. this case, the joint probability measure IP needs to be defined on ($2, F) (St x R, FG @ (R)). where B(R) denotes the Borel
o-algebra of TR. This lays the foundation of our method in the setting where fe is a regression model.
21

---

Published as a conference paper at ICLR 2025
B.2 FORMAL STATEMENT AND PROOF OF THEOREM 3.2
We present the formal version of Theorem 3.2 as two separate theorems: Theorem B.2 for the continuous flow and Theorem B.3 for the discrete flow.
Theorem B.2 (Guided velocity). Let {px be the conditional velocity that generates Piji (2,(21) via Fokker-Planck Equation, i.e., dePij1 = -V (innPun). Then the guided flow (EE()))) hefo, defined via the construction of Theorem 3.1
be continuous flow on R Let daja (asp(a (
can be generated by the following guided velocity via Fokker-Planck Equation:
(.xx le) (p(e((rsls.ee (10l111))))
(28)
Proof. We begin with the Fokker-Planck Equation of Ptj1 (2)/<r1)
depeni (2)) - (zee (z, PP: (æ,)=r))
(29)
Taking expectation with respect to Pdata (x1lc) over both sides yields
E (ms (Oepan [z1] - (an le)
V
(x)
(30)
The left-hand size of Eq. (30) can be simplified as
E (mplee (DePej(az [x1]]
Pdata Ptau(z11le)@ ((x
(31)
Rs
(Je Pdata (21n((01/c)pp
= a
)
(32)
(/
0e
Piina (2: (cllc)ppp ( c)dxr
(33)
Ra
=x/p((
(34)
Note that in Eq. (33), we use the conditional independence property of trajectory and target and get
Pdata (x) (c) pe (xt, /c)c).
For the right-hand size of Eq. (30), we have
V (v )ppp) (ax, [x1))]
(35)
V (v (zr )aams (an (c)da1)
(36)
R3
enn le (p) (xr (e: )aaaa (x) (c)
(37)
Ra
(
win (2x ( (xr,, cc)dz1
(38)
Ra
/
= (c(2))c)
vip
c)dx
(39)
R3
= (w, (2)cc()))c))
(40)
Putting the above together leads a.p, (x:(c) - (00(0)/cc))/(0))l)))
Theorem B.3 (Guided rate matrix). Let (po(a)) heja, be discrete flow on A. Let R Ra bla.) be the conditional velocity that generates Pejt = (pel (ala1) (ala1))aed via Kolmogorov Equation, i.e.
O.P Pt Byj1. Then the guided flow {pr(as ic) (o defined via the construction of Theorem 3.1 can be generated by the following guided rate matrix via Kolmogorov Equation:
R2 (a ble) = Eppp (o) [Rej (at, bla1)]
(41)
Proof. The proof idea is exactly the same as that of Theorem B.2, i.e., taking expectation with respect to (asa(())c) over both sides of Kolmogorov Equation and simplifying them. Again, Paata (ai (c)Pen (azl (a)) pe(a,, ai ()) will be useful in the simplification.
22

---

Published as a conference paper at ICLR 2025
B.3 IMPORTANCE SAMPLING FAILS FOR SAMPLING THE DISCRETE GUIDED FLOW
Proposition B.4 (Naive importance sampling for the discrete guided flow). For L E [0, 1), the guided rate matrix defined in Theorem 3.2 satisfies
Pc((())) (a." p(c/G.
( .5cc (..I.. Ra
(42)
Proof. By definition of the guided rate matrix and importance sampling, we have
Pip P Rej Rap (9) _laf%) Pija(Gr G.)
Re (0) .flc) = (..
(43)
Note that for any [0, 1). G, and are conditionally independent given G1. Therefore,
Pip ( (G. Pr[(C G.) Pip (GIGGG
(44)
G()
p(c[.
p(c(.))
Gi C) (r,)
p(c(G)
(45)
( G.)
which concludes the proof.
Proposition B.5 (Restatement of Proposition 3.3). For & E [0, 1), the guided rate matrix defined in Theorem 3.2 satisfies
(. (a (211a1))))))))11))
[e (1) .flc)
(46)
E PpIn [fo(G])
Proof. Proposition B.4 indicates that
Re(a .5lc) Ep.nG.. Ren (at []a
(47)
p(c[G.)
By Conditional independence of trajectory and target in the guided flow, we have
G.) G1 Golnne G.)dG
(48)
(G
(49)
(G.I... (G.)
(50)
Combining Eq. (47) and Eq. (50) leads the conclusion.
11
23

---

Published as a conference paper at ICLR 2025
B.4 PROOF OF THEOREM 3.4
Theorem B.6 (Restatement of Theorem 3.4), Lel " GIJLK ~ ii.d Pije(-Gat). Define the
estimation of Rs (a
(6) blc) as
K Banla
k
Re(a
(1) blc) Σ
Σ fo(G()..).
(51)
h=1
Assume fc = inf fc(G) > 0. Given any E E (0. 8e (0, 1), if θ GEG
log m then
22
sup (e(l.mmm) mn BEA
[a blc)) - (a blc) <E 21-6
(52)
Proof. Consider some fixed i {1.... n). b A. To simplify exposition, denote
N1.6 = PE [o( jlaf
(53)
D,,6
B (p [fo(G:)]
(54)
K
No.
fe(G ( (a) ( affee)
(55)
K
k=1
K
D.D Σ
K 1()()))))))
1
(56)
We note that for any & E f1. KK.
<< ((( Rap (a, (0) b bl alle,t) 1-
(57)
Therefore, by Hoeffding's s inequality, we have
p
P(1 (INAO [N.. - -N,o =(1 - f)fe² c{1=
K=a -t)
I 2exp
(58)
4
8
Similarly, for DrD Dro/. we have
(1000 -Drol > e(1-1)Je" 1][e
KE2(1 - 11)
< 2exp
(59)
4
<(1 - t)fe² and Di.o DeN S
E(1 - e(1-
t)ff²
Suppose No.6 - -Neo 5
then we have
4
NiDDi.D Ne.
NoD Dib
New
(60)
DAM
Dr.DDoD
Dib Ni. Nob + No. Dr, - Dib
(61)
Di.D
Dib + Neb
E(1 - -)se²
(62)
DiDD Ef?
1-1
Note that 0 0 Dr,,, No.6 < 1-t)
(63)
2D.DDDD
24

---

Published as a conference paper at ICLR 2025
Recall the definition of fe and the fact that (0. fc/ we have D&, 2 fe and Deo fc/2.
Ni.
No.
Plugging these two lower bounds into Eq. (63) yields
<E
Dd,b Dib
Na
Noo
<(1 t)
5(1 a(1 t)
Thus
must imply No.6 No.0
Di.b Do.6
Dibb
Dib
Re(af (0)
blc)
Rola;"
1aais,,00 .b1c) -(0c)/26) 28)
(64)
Ni.b
Nub
28
(65)
Di,b
Di.
c(1 - 1)5e e(1-1)1e)
(IDLS Diol > e(1 - [i]e
Ni.6
Ncbl
(66)
4
4
KE²(1
<4exp
(67)
8
Applying union bound on i (1,...,m), b A yields
(1)) fc
fe
Re(a, Ralaf%.bi
.lc) (blc) R,(o, blc) Ze E 44|AA/es exp
-
sup ie(L, m p. BEA
Set
8
log 4n|AA|
K
(68)
8² - 1)
Then we have
sup iE(l.". n), beA
Ra(a, ()) blc) blc))
Re(a, (5) blo) 2 56.
2=.6
(69)
which concludes the proof.
25

---

Published as a conference paper at ICLR 2025
B.5 FORMAL STATEMENT AND PROOF OF THEOREM 3.5
Before stating and proving the formal version of Theorem 3.5, we present additional definitions and technical lemmas:
Definition (S0(3)-img)) -invariance). We say a mapping h R + R is (in)aiinariaa if
h(SX) = h() for any X E IR 3xx and S E SO(3)
(70)
Definition 2 (30 equivariiici. We say mapping h g + 3 is -0e)ivequivaai if
h(SX) = (h for any X E 2 and S E SO(3).
(71)
(0 -invariant mapping. Lemma B.7 (Gradient of invariant mappings), Let h 133x R be an Then the gradient Vh is an S0(3)-equvvarian mapping.
Proof. By chain rule and (0) iivariinne,, we have Vl(SX) = SV for any SO(3).
Lemma B.8 (Composition and sum of equivariant mappings). Let hı and h2 be -equivarraa mappings. Then their composition h2 o h1 and their sum h1 + h2 is also -3e)ieequivaaiant,
Proof. The lemma follows immediately from the definition of -quiivarrii mappings.
Lemma B.9 (Push-forward of equivariant mappings). Let h be an S3 - mapping and be an 50(3))invarinn distribution. Then h&p is also an -im3)-imarii distribution, where # denotes the push-forward operator.
Proof. The lemma follows immediately from the definition of -qquivvaui mappings and SO(3) invariant distributions.
Theorem B.10 -innaaraae (formal version of Theorem 3.5)). Consider Algorithm Assume the target predictor (G) is SO(3) 50(3)-invarian. invariant, the flow model go G)) is -0(3)--quivariant and the
distribution of Go (O())-innrriaan w.r.t. the atomic coordinates X. Let W = 1/At. Then for any E {0, .., W), the distribution of xwst is SO(33)-nnvarinn) -invariant.
Proof. We prove by induction on W.
If = 0, then the distribution of Xwsr is invariant according to the assumption.
Assume that the distribution of ( is SO(3)-invarian -inariiat for some w E {1, W.. We show that the distribution of war is (On3)-invrriant
We note that in Algorithm 1, ww is generated from X (w-1)A via series of mappings: X.st h2 o ..... h1 (X (w- (w-1))r), where each h4 is one of the following:
Projection: X + Projectr (X).
Guidance based on the target predictor: X ト X + V. (o, aa)x)ar)))
Euler step: [x)1]
I i= +
(z(()
+
(x() a )t
We show that all these mappings are (0uui--qquvariant.
Projection.
By definition of Projectr (in Sec. 2.)), for any S SO(3),
Project, (SX) SX
Sa 1 I
-
S
x
x 1
SProject (X). (72)
n
Guidance based on the target predictor. According to the assumption, X + log fa aaj)) (On3)-invriiant. Thus, by lemma B.7, its gradient is -qqiivraaa Also note that X + go (X, ax)xx x (0n3a-invrriann. Thus, by lemma B.8, X X + pe V log fgo (X, ax)x ant))
26

---

Published as a conference paper at ICLR 2025
Euler step. By definition of oij1 (a+
(
(in Eq. (9)), we easily check the SO(3))equivariance
of [x)0 (000 + Uaja (2 (8)
(+)
At
To sum up, h h all (3( eqqiivaraan By lemma B.8, h2 o... h2 is -qquiivrira Recall that the distribution (w- 1) is -inaaairrriia Therefore, by lemma B.9, the distribution of X wt is also SO(3) -invaaannii. We conclude the proof by induction.
27

---

Published as a conference paper at ICLR 2025
C DISCUSSION OF RELATED WORK
C.I. GENERAL DISCUSSION
Diffusion and Flow Matching Diffusion models (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Song et al., 2020; 2021; Dhariwal & Nichol, 2021; Song & Ermon, 2020; Karras et al., 2022;
Hoogeboom et al., 2023: Han et al., 2022b; Qin al., 2023b) have demonstrated exceptional perfor- mance across a range of generative modeling tasks, including image and video generation (Saharia et al., 2022; Ho et al., 2022; Zhang et al., 2024), audio generation (Kong et al., 2020), and 3D geom- etry generation (Luo & Hu, 202 la;;; Xu et al., 2022; Luo et al., 2022), among others. In contrast, flow-based generative methods (Liu et al., 2022: Albergo Vanden-Eijnden, 2022; Lipman et al., 2022) present more streamlined alternative to diffusion models (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Ho et al., 2020; Song et al., 2020), bypassing the need for forward and backward diffuusion processes. Instead, they focus noise-data interpolants (Albergo et al., 2023), simplifying the generative modeling process and potentially leading to more optimal probability paths with fewer sampling steps (Liu et al., 2022). Flow matching, which employs ODE- base- continuous normalizing flows (Chen et al., 2018). further refines this approach. Conditional flow matching (CFM) (Lipman et al., 2022; Liu et al., 2022: Albergo & Vanden- Bijden, 2022) learns the ODE that maps the probability path from the prior distribution to the target by regressing the push-forward vector field conditioned on individual data points. Riemannian flow matching (Chen & Lipman, 2023) extends CFM to operate on general manifolds, reducing the need for costly simulations [Ben-Hamu al., 2022: De Bortoli al., 2022; Huang et al., 2022). Recent advancements in the rectified flow framework (Liu et al., 2022), which focuses on learning linear interpolations between distributions, have demonstrated notable improvements in both efficiency (Liu et al., 2023) and quality (Esser et al., 2024; Yan et al., 2024), particularly in text-to-image generation. Our work aligns with these trends by exploring how to guide generation within this simple and general framework a training-free manner (Ye et al., 2024).
Multimodal generative models. Diffusion models have demonstrated great success in modeling continuous data; however, many real-world applications involve multimodal data, such as tabular data (Kotelnikov et al., 2023), graph data (Qin et al., 2023a), and scientific data (Peng et al., 2023). A key challenge in this domain is generating discrete data within the diffusion framework. Unlike large language models (LLMs) (Achiam et al., 2023; Lin et al., 2024; Team et al., 2023; Ke et al., 2023), which excel modeling discrete text, diffusion models have struggled in this area. Li et al. (2022) introduced continuous language diffusion models, which embed tokens in a latent space and use nearest-neighhor dequantization for generation. Subsequent research has enhanced performance through alternative loss functions (Han et al., 2022a; Mahabadi et al., 2023) and by incorporating conditional information, such as infilling masks (Gong et al., 2022; Dieleman et al., 2022). Gulrajani & Hashimoto (2024) further improved language diffusion models by making various refinements to the training process, allowing their performance to approach that of autoregressive LMs. While continuous approaches to discrete data modeling have seen advancements (Richemond et al., 2022; Han et al., 2022a; Chen et al., 2022; Strudel et al., 2022; Floto et al., 2023), discrete diffusion methods like D3PM (Austin al., 2021) and subsequent works (Zheng al., 2023; Chen et al., 2023; Ye et al., 2023) have demonstrated greater efficiency. Recent innovations, such as SEDD (Lou et al., 2023), extend score matching discrete spaces, improving language modeling autoregressive models. Additionally. DFM (Campbell et al., 2024) applies continuous-timg Markov chains to enable discrete flow matching, contributing to the Multiflow framework by integrating continuous flow matching. Overall, multimodal generation is still an important open problem, and our work lays a theoretical foundation for multimodal guided flow, which will be beneficial for the study of multimodal generation.
level competitive with
Generative Model for Molecular Generation. Generative models have been applied to design drugs such as small molecules (Ramakrishnan et al., 2014; Axelrod & Gomez-Bonbarelli, 2022; Francoeur et al., 2020; Irwin et al., 2020), proteins (Berman et al., 2000; Kryshtafovych et al., 2021; Haas et al., 2018), peptide (Wang et al., 2024), and antibodies (Jin et al., 2021). Our work focuses on designing small molecules, while the method can also be easily adapted to proteins, peptide and antibodies which requires multimodal generation. Molecular design can be categorized into target- agnostic design (Huang et al., 2023; Xu et al., 2022; 2023; Morehead & Cheng, 2024), where the goal is generate valid sets of molecules without consideration for any biological target, and target-aware
28

---

Published as a conference paper at ICLR 2025
design (Li et al., 2023; Masuda al., 2020; Guan et al., 2023; Peng et al., 2022; Schneuing et al., 2022). Our work considers both types of molecule design tasks.
Guidance for Diffusion and Flow Models. Since the introduction of classifier guidance by Dhariwal & Nichol (2021), which employs a specialized time-dependent classifier for diffusion models, significant progress has been made in applying guidance these models. Early research focused on straightforward objectives for linear inverse problems such image super-resolution, deblurring, and inpainting(Chung & Ye, 2022; Wang et al., 2022; Zhu al., 2023). These approaches were later extended by leveraging more flexible time-independent classifiers, achieved through approximations of the time-dependent score using methods like Tweedie's formula (Yu et al., 2022) and Monte Carlo sampling (Song et al., 2023). Recent methods, such FreeDoM (Yu et al., 2023), UGD (Bansal et al., 2023), and TFG (Ye et al., 2024). incorporate various techniques to enhance the performance of training-free classifier guidance, leading to more advanced forms of guidance. However, some of these techniques lack clear theoretical support. A more recent work by Shen al. (2024) aims to improve the performance of training-free guidance by drawing on ideas from adversarial robustness. Our work provides a rigorous theoretical framework for the design choices in training-free guidance, which we believe will serve as strong foundation for future empirical research. In addition to classifier guidance, there are related works focusing on guiding the generation of specific diffusion models. For instance, DOODL (Wallace al., 2023), DNO (Karunratanakul et al., 2024), and D-Flow (Ben-Hamu et al., 2024) utilize invertible models or flow models to backpropagate gradients to the latent noise. These methods emphasize noise optimization and often involve training process tailored to a specific target, potentially making them slower than training-fit frren classifier guidance. Additionally, RectifID (Sun et al., 2024) is a concurrent work exploring training-five ffree guidance rectified flow for personalized image generation. However, we have identified several theoretical issues this work, which will be discussed in the following paragraph. This method addresses the training- free guidance via fixed fixed-point formulation, which is quite different from us and can only apply to continuous data.
C.2 THEORETICAL ISSUES WITH RECTIFID.
Similar to many training-free guidance methods, RectifID aims to bypass the noise-aware classifier, which is typically trained according the noise schedule of flow or diffusion models (as discussed in Equation 7 of their paper). They approach this by employing fixed- poidtpoint formulation:
20 + v(21, 1) Vay logp(c/21),
(73)
which corresponds to Equation 8 their paper. This equation derived under the assumption that the rectified flow follows a linear path 2, 2n + v(2, t)t, which is strong. Since the velocity v(2, f) is parameterized by a flow network and log p(c|zı) can estimated using a time-independent classifier, this formulation implies that 2 can be obtained by solving fixed-point problem. However, we think that when solving this fixed-point problem, the estimate of log p(c)21) from a time- time-innepon -emeinddeenndntt classifier may be unreliable because some values of æ could be out-of-distribute for the classifier.
C.3 THE MOTIVATION OF STUDYING TRAINING- FRRE GUIDANCE
As training-free guidance an emerging in generative model research, this section highlights its significance as a critical and timely topic that warrants greater attention and effort, particularly in leveraging off-the-shelf models.
Comparison with Conditional Generative Models. Conditional generative models, such as Cond- EDM and Cond-Flow, require labeled data for training. This reliance on annotated datasets can limit training efficiency and scalability. In contrast, training-free guidance allows the use of any unconditional generative model, which can be trained in an unsupervised manner. This flexibility enables the scaling of generative models without the need for task-specific annotations. Furthermore, conditional generative models are tied to predefined tasks during training, making them unsuitable for new tasks post-training.
29

---

Published as a conference paper at ICLR 2025
Comparison with Classifier Guidance. Classifier guidance (Dhariwal & Nichol, 2021) involves training a time-dependent classifier aligned with the noise schedule of the generative model. While this approach can utilize pre- trained unconditional flow or diffusion models, it still requires labeled data for classifier training on per-task basis. This constraint makes it less adaptable for scenarios where the target is defined as loss or energy function without associated data, when leveraging pre- trained foundation models for guidance. In contrast, training-fiee free guidance offers greater flexibility, allowing for broader and more dynamic applications without the need for task-specific classifier training.
Comparison with Classifier-Free Guidance. Classifier-free guidance (Ho & Salimans, 2022) has become popular approach for building generative foundation models. However, it still requires task definitions to be determined prior to model training. A more versatile alternative is instruction tuning, where text instructions act interface for variety of user-defined tasks. As shown by Ye et al. (2024), training-free free guidance surpasses text-to-X models (e.g., text-to-image or text-to-video) in flexibility and robustness, as demonstrated by two failure cases of GPT-4. These models often struggle with interpreting complex targets in text form or constraining generated distributions using simple text prompts. Training-free guidance offers more powerful and adaptable mechanism for controlling the generation process, enabling a broader range of user-defined targets and more precise control over outputs.
Applications of training-free guidance. In motif scaffolding, training-free guidance can direct the generated samples to possess a given motif without requiring a trained classifier (Didi et al., 2023). Similarly, in the symmetric oligomer design and enzyme design with concave pockets tasks in RFDiffusion (Watson et al., 2023) a foundational model for protein training free guidance is employed (referred guidance also extends to tasks in other domains, such as image, audio, and motion. Examples include deblurring, super-resolution, style transfer, audio declipping, obstacle avoidance, name just few. Notably, this approach eliminates the need for label-conditioned data collection or task-specific training when an off-the-shee generator and target predictor are available. For further insights into its applications, we direct readers to TFG (Ye et al., 2024), which highlights 16 tasks across 40 targets leveraging training-fre guidance.
"inference with external potentials"). Beyond biology, wraining-free
C.4 DISCUSSION OF CONTINUOUS PART MODELING IN MULTIFLOW
In this section, we examine the implicit assumptions underlying Multiflow (Campbell et al., 2024), which serves as a foundational basis for our method. While the derivation in Multiflow adopts a unified approach modeling the continuous component 1jj and the discrete component ajjr= its practical implementation differs significantly. Specifically, the likelihood of the discrete component aije can be directly modeled using the output of a neural network, whereas the modeling of the continuous component X presents greater challenges.
Recall the key component of Multiflow is the computation of unconditional velocity Eq. (7). We copy it here for convenience.
(xx) = E (G. [een (1)/<1)))) Re b) = E (.) RRn (ar, bla1)]
To compute the continuous velocity (x()), one could sample multiple Øijt from Pije (-J(C)G+r) and estimate the expectation 2n (z, (x1) using Monte Carlo methods, direct computation of the expectation involves intractable integrals. However, in Multiflow's ' and our implementation. the model predicts the expectation Eyj ( Gt). Consequently, (x(,))) = ztj1 (a>, [ where E, [Z1jel G.)] is parameterized by the output of the flow model (a neural network).
This assumption does not compromise the theoretical validity of our theorems or those the Multiflow paper since dxji is linear w.r.t. æ1 defined in Eq. (6): dan (21(=1) z,==._ And computing the expectation of (G. (G.) (ar, (z)) is equivalent to meju (ax [Tjeee (..)). Therefore, the linearity allows to model the expectation only.
30

---

Published as a conference paper at ICLR 2025
D EXPERIMENTAL DETAILS
D.1 TARGET-AGNOSTIC DESIGN SETTING
We conduct experiments in this setting following EDM (Hoogeboom et al., 2022), where the genera- tive process by first sampling c, n p(c, n),, where c the target property and the number of atoms, and then samples the molecule G given these two information. We compute n p(c, n) on the training partition as parameterized two dimensional categorical distribution where we discretize the continuous variable e into small uniformly distributed intervals.
For structure similarity, we use all the structures in the unseen half of training set for generative model and guidance predictor. The Tanimoto coefficient results are averaged over all the structures.
D.2 IMPLEMENTATION OF TFG-FLOW
Network configurations. To train Multiflow on aarget-agnostic small molecular generation, we follow EDM (Hoogeboom et al., 2022) to use an EGNN with 9 layers, 256 features per hidden layer and SiLU activation functions. We use the Adam optimizer with learning rate 10-4 and batch size 256. Only atom types (discrete) and coordinates (continuous) have been modelled, which is different from unconditional EDM that includes atom charges. All models have been trained for 1200 epochs (for QM9) and 20 epochs (for GEOM-Drug and CrossDocked) while doing early stopping by evaluating the validation loss. For target-aware molecular design, we follow TargetDiff (Guan et al., 2023) to use EGNN with &NN graph, where k = 32 and reduces the batch size 16 and the hidden layer dimension to 128. For target predictors, we train a 6-layer discriminative EGNN by adding linear head the average pooling of output node feature. In our implementation, we only model heavy atoms.
Hyperparameter searching. As noted by TFG (Ye et al., 2024), hyperparameter searching is the key problem for training-free free guidance. TFG designs complicated hyperparameter space where three hyperparameters (which consist of a scalar and a scheduling strategy) and two iteration parameters should determined. Our TFG-Flow only has the four hyperparameters Niters P; T, K to tuned, and we show that logarithmic scale of K is sufficient, and Niter can be set according to the computational resources. So we fix Niter- = 4 and K 512 in our experiments, while grid search the a and for different applications. The search space is defined by first find the appropriate magnitude, as different applications have varied order of value magnitude. After determining the appropriate magnitude, we double p and r for 4 times construct the search space. For example, the appropriate search space for polarizability is (p, T) is {0.01, 0.02, 0.04, 0.08} x {10, 20, 40, 40,80). We then search the space via small scale generation (the number of 512 following Ye et al. (2024)). The search target is to minimize the guidance performance (c.g., MAE in quantum property guidance), while keeping a validity over 75%.
D.3 BASELINES
We introduce the baselines used in this paper here. For baselines in target-agnostic molecular design,
This baseline is from EDM (Hoogeboom al., 2022), which removes any relation between molecule and property by shuffling the property labels in the unseen half of QM9 training set for oracle target predictor, and evaluate MAE on it. If a baseline outper- forms upper bound, then it should be able to incorporate conditional property information in to the generated molccules.
Upper bound:
#Atoms:
"#Atoms" predicts the molecular properties by only using the number of atoms in the molecule. If a baseline outperforms "#Atoms", it should be able to incorporate conditional property information into the generated molecular structure beyond the number of atoms.
Lower bound: The MAE of directly predicting property using the oracle target predictor.
Cond-EDM: The conditional EDM is implemented by simply inputting a property c to the neural network of EDM (Hoogeboom al., 2022), without changing the diffusion loss.
31

---

Published as a conference paper at ICLR 2025
EEGSDE: EEGSDE (Bao et al., 2022) trains time-dependent classifier to guide the conditional EDM. This is the best method for guided generation in quantum guidance task.
DPS: DPS (Chung & Ye, 2022) estimate the time-dependent gradient Va, log p(æ,, t) with Va, log p(zo)) using Tweedie's formula (here Do means the clean data in diffusion formulation, which different from flow where 21 is the clean data).
LGD: LGD (Song al., 2023) replaces the point estimation znjz with Gaussian ker- nel N, LGD uses Monte-Carlo sampling to estimate the expectation of Vz, log ((
FreeDoM: FreeDoMM (Yu al., 2023) generalizes DPS by introducing "time- strategy" that iteratively denoises ht-1 from 2; and adds noise to It-1 to regenerate ze back and forth.
MPGD: MPGD (He et al., 2024) computes the gradient V Majs log f (Zujt) instead of Vx, log (Zaz)) to avoid back poc-prrpagattin through the diffusion model.
UGD: UGD (Bansal et al., 2023) is similar to FreeDoM, which additionally solves backward optimization problem and guides Foje and a, simultaneously.
TFG: TFG (Ye al., 2024) unifies all the techniques training-free guidance, and turn the problem into hyperparameter searching.
Cond-Flow: Conditional flow implemented by inputting the target value of property to Multiflow. The modification is the same Cond-EDM with respect EDM.
For baselines target-aware molecular design,
AR: AR (Luo al., 2021) is based on GNN and generates atoms into protein pocket in an autoregressive manner.
Pocket2Mol: Pocket2Mol (Peng et al., 2022) generates atoms sequentially but in a more fine-grained manner by predicting and sampling atoms from frontiers.
TargetDiff: TargetDiff (Guan et al., 2023) generates the molecules via diffusion, which is similar to EDM that incorporates pocket information as inputs.
Multiflow: We follow TafgetDiff to adapt target-agnostic Multiflow to target-aware Mul- tiflow. The simplex I projection becomes to substracting the center of gravity w.r.l. the protein pocket.
D.4 DATASET PREPROCESSING
We introduce the dataset information for expeimernts.
QM9. The QM9 dataset is a widely-used resource for benchmarking models in the field of molecular generation, particularly for 3D structures. This dataset comprises about 134,000 molecules, each containing up to 9 heavy atoms (not counting hydrogen atoms), derived from subset of the GDB-I7 17 database which itself is database of 166 billion synthetic molecules. Each molecule the QM9 dataset is characterized by its chemical properties and 3D coordinates of atoms. In the specific usage of QM9 as mentioned, the dataset was divided into training. validation, and test sets with sizes of 100,000, 18,000, and 13,000 molecules, respectively, following the setup used by the EDM. We use the QM9 dataset from Huggingface Hub
GEOM-Drug. The GEOM-Drug dataset is tailored for applications in drug discovery, providing more complex and realistic set of molecules for the development and benchmarking of molecular generation models. This dataset was specifically chosen train and evaluate models that are intended to simulate and understand real-world scenarios in drug design. GEOM-Don --rog differs from simpler datasets like QM9 by focusing on molecules that are more representative of actual drug compounds. These molecules often contain major elements found in drugs, such as carbon (C), nitrogen (N), oxygen (O), fluorine (F), phosphorus (P), sulfur (S), and chlorine (CI), We follow MolDiff (Peng et al., 2023) to exclude hydrogen atoms and minor element types such as boron (B), bromine (Br),
32

---

Published as a conference paper at ICLR 2025
iodine (I), silicon (Si), and bismuth (Bi). We use the opensourced version of GEOM-Drug processed by MolDiff codebase', which filters the molecules through several criterion. The selection criteria were follows:
molecules must be compatible with the RDKit package;
structures should be intact;
molecules should contain between 8 and 60 heavy atoms;
only elements such as H. N. O, F, P. S, and CI were allowed;
permissible chemical bonds included single, double, triple, and aromatic bonds.
Following these criteria, hydrogen atoms were excluded, and the molecules were divided into training, validation, and lesting datasets containing 231,523, 28,941, and 28,940 molecules, respectively.
Droke20ckd The Crossdocked dataset initially comprises 22.5 million poses of small molecules docked to proteins, featuring 2,922 unique protein pockets and 13,839 unique small molecules. We utilized non-redundant subsets that were filtered and processed from the TargetDiff codebase These subsets include 100,000 protein-molecule pairs for training and 100 pairs for testing.
D.5 EVALUATION METRICS
We introduce the evaluation metries used in this paper.
MAE (Mean Absolute Error): A measure of the average magnitude of the absolute errors between the predicted values and the actual values, without considering their direction. It's caleulated as the average of the absolute differences between the predictions and the actual outcomes, providing a straightforward metric for regression model accuracy.
Tanimoto coefficient: A metric used to compare the similarity and diversity of sample sets. It is defined as the ratio of the intersection of sets their union, often used in the context of comparing chemical fingerprints in computational chemistry.
Vina score: A scoring function specifically designed for predicting the docking of molecules, particularly useful in drug discovery. It estimates the binding affinity between two molecules, such as a drug and a receptor, helping the identification potential therapeutics.
SA score (Synthetic Accessibility score): heuristic measure estimate the ease of synthesis of given molecular structure. Lower scores indicate molecules that are easier to synthesize, useful in the design of new compounds medicinal chemistry.
QED (Quantitative Estimate of Drug-likenes): metric that quantifies the drug-likeness of molecular structure based on its physicochemical properties and structural features, aiming to identify compounds that have desirable attributes of potential drug.
We also test other metrics related to generation quality following EDM (Hoogeboom et al., 2022) and MolDiff (Peng et al., 2023) in App. E:
Validity: Measures the percentage of generated molecules that are chemically valid (can be parsed by RDKit).
Atom stability: Assesses whether the atoms have the right valency.
Molecular stability: Assesses whether all the atoms are stable in the molecule.
Uniqueness: Quantifies the uniqueness of generated molecules.
Novelty: Measures how novel the generated molecules are when compared to the training set.
Connectivity: Checks if all atoms are connected in the molecule, We use the lookup table from EDM infer the chemical bonds.
33

---

Published as a conference paper at ICLR 2025
D.6 LEVERAGING PRE-TRAINED FOUNDATION MODELS AS TARGET PREDICTOR
In our main experiments, the target predictors were trained specifically for guidance purposes to ensure direct and fair comparison with existing baselines such as TFG (Ye al., 2024). However, significant advantage of training-free guidance methods like TFG-Flow is their ability to leverage pre-trained foundation models target predictors without additional training. This capability not only enhances performance but also reduces computational resources and time.
To demonstrate this advantage, conducted additional experiments using UniMol (Zhou al., 2023), an open-source universal 3D molecular foundation model applicable to various downstream tasks. UniMol powerful pre-trained model that can predict a wide range of molecular properties. We used the fine-tuned UniMol model to predict molecular properties, serving our guidance network in the experiments corresponding to Table 5.
Table 5: Conditional generation results on the QM9 dataset using pre-trained UniMol as target predictor. Lower values are better for all metrics.
Method
C. |4
4
Ac 4
CHOMO +
CLUMO
Cond-Flow
1.52
0.962
3.10
805
435
693
TFG-Flow (Vanilla predictor)
1.48
0.880
3.52
917
364
998
TFG-Flow (UniMol)
1.12
0.802
2.89
585
312
587
EEGSDE
0.941
0.777
2.50
487
302
447
As shown in Table 5, incorporating a strong pre-trained target predictor like UniMol significantly enhances the performance of TFG-Flow across all evaluated molecular properties. Specifically, the Mean Absolute Error (MAE) decreases notably when using UniMol compared to the vanilla predictor trained specifically for guidance. The performance gap between training-free TFG-Flow (with UniMol) and the training-bass based EEGSDE, which requires training both a conditional generative model and time-dependent predictor, has notably decreased.
These findings highlight several key advantages:
Improved Performance. The integration of a strong pre-trained model like UniMol boosts the performance of TFG-Flow, achieving results closer to competitive with training- based methods.
Flexibility. Training-free guidance methods can seamlessly incorporate any off-the-shelf pre-trained models, including those not originally designed for the guidance task. This flexibility particularly beneficial when high-quality pre-trained models are available.
Reduced Computational Effort. By leveraging pre-trained models, we eliminate the need to train additional target predictors or time-dependent classifiers, saving both time and computational resources.
Our experiments demonstrate that TFG-Flow benefits significantly from incorporating pre- pre-trained foundation models as target predictors. This capability enhances performance, narrows the gap with training-based methods, and underscores the practical advantages of training-free guidance.
D.7 HARDWARE AND SOFTWARE SPECIFICATIONS
We run most of the experiments on clusters using NVIDIA A800s with 128 CPU cores and IT RAM. implemented our experiments using PyTorch, RDKit, and the HuggingFace library. Our operating system is based on Ubuntu 20.04 LTS.
34

---

Published as a conference paper at ICLR 2025
E DETAILED EXPERIMENTAL RESULTS
E.1
CONDITIONAL FLOW
Table 6: The full results for conditional flow (Cond-flow) QM9 quantum property guidance. The results averaged over 3 random seeds, where standard deviations reported Table
Validity
Uniqueness
Novelty
Mol. stability?
Atom stability?
Connectivityt
MAEJ
78.03%
98. 78.78%
87.75%
89.54%
98.49%
69.55%
3.10
83.60%
97.81%
86.71%
92.87%
99.12%
74.40%
0.96
C, 64.96%
98.78%
85. 48.88%
81.20%
96.89%
57.91%
1.52
As 75.95%
97.42%
87.26%
88.41%
98.52%
64.86%
804.03
CHMMO
87.93%
95.90%
89. 19.14%
94.51%
99.32%
73.72%
436.02
ELUMO
80.76%
98.46%
90.57%
91.58%
99.00%
67.42%
692.78
Table The standard deviations for conditional flow (Cond-flow) QM9 quantum property guidance.
Validity
Uniqueness
Novelty
Mol. stability
Atom stability
Connectivity
MAE
0.5921
0.2458
0.4712
0.4000
0,0551
0.6409
0.0231
μ
0.7703
0.1607
0.4443
0.6853
0.0808
0,4022
0.0034
C 0.6730
0.2427
0,5601
1.0701
0.2290
0,5462
0.0467
Aa 0.4349
0,2875
0.3831
0,2139
0.0404
0.6322
12.74
SHOMO
0.1553
0,4576
0.7246
0.2464
0.0252
0.6218
9.054
ELUMO
0.4306
0.1234
0.5103
0.3798
0.0351
0.2207
9.139
35

---

Published as a conference paper at ICLR 2025
E.2 TFG-FLOW
Table The full results for IFG-Flow on QM9 quantum property guidance. random seeds, where standard deviations are reported in Table
The results are averaged over
Validityt
Uniqueness
Noveltyt
Mol. stability?
Atom stability?
Connectivity!
MAE!
76.31%
99.43%
93. %
89.89%
98.75%
66.40%
3.52
75.34%
99.01%
89.89%
86.85%
98.33%
63.64%
0.88
C, 75.56%
98.995,
90.89%
87.78%
98.43%
68.55%
1.48
A6
75.95%
97.42%
87.26%
88.41% %
98.52%
64.86%
914
еномо
87.93%
95.90%
89.14%
94.51%
99.32%
73.72%
364
ÉLUMO
80.76%
98.46%
90.57%
91.58%
99.00%
67.42%
998
Table 9: The standard deviations for IFG-Flow on QM9 quantum property guidance.
Validity
Uniqueness
Novelty
Mol. stability
Atom stability
Connectivity
MAE
0.8051
0.1015
0.4419
0.7518
0.0900
1.0564
0.0523
0.0693
0.1361
0.4844
0,3800
0.0751
0,3166
0.0164
C, 0.6660
0.2127
0,5501
1.0601
0.2390
0,3462
0.0455
Ae 0.6673
0.1058
0.2498
0,4743
0.3528
0.3407
20.7743
0.4603
0.1501
0.2425
0.1997
0.0416
0.2761
7,4903
SLUMO
0.1000
0.1501
0.1102
0.2589
0.0173
0.2454
14.9410
36


Reviews:
[{"review_id": "r0iYeWJfea", "reviewer": "Reviewer_t7Ci", "summary": "The authors develop a training-free approach to guide a pre-trained multimodal flow model to generate molecules with specified properties.  They do so by introducing  the guided velocity and guided rate matrix.", "questions": "1. Could you explain why the conditional independence of trajectory and target holds? \n2. Could you elaborate on how the hyperparameter \\alpha_t is chosen or tuned in practice?\n3. Since the method depends on a predictor to guide property adherence, is there any evaluation of how predictor quality or accuracy impacts the overall performance of the guided generation?\n4. There is typo in Table 4, for Vina score, TFG-Flow was falsely bolded.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Training-free guidance makes it computationally advantageous over other flow-based molecular generation models.", "weaknesses": "1. The performance of this method may heavily depend on the quality and accuracy of the target predictor.\n2. Conditional generation performance does not seem to be improved over the baselines", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7l4j5HmQVp", "reviewer": "Reviewer_oDU6", "summary": "This paper studies the conditional generation problem in the context of molecular generation. Specifically, the authors propose a \"classifier-guidance\" approach for inherently multimodal molecular structures (atom token and coordinate) with flow matching/rectified flow-based generative models. Experiments are conducted on several molecular optimization problems.", "questions": "* Is the guidance exact? i.e. if you write down the conditional vector field of the target distribution, is it equal to Eq 15? It seems you only have the term for guidance without the original vector field. In addition, in the diffusion literature, it is known that evaluating the guidance gradient as $\\nabla E(\\mathbb{E}_{X_1|X_t}(X_1))$ is not exact, where $E$ is the energy here, e.g. see [1].\n* for Eq 15, should it be $X_{t+1} \\leftarrow X_{t} + \\cdots$?\n* The rate matrix R is not properly defined, i.e. what does $R(\\cdot, \\cdot)$ mean?\n* In Sec 3.2, why is the expectation calculation exponentially growing? Since you only need to model for each atom their rate matrix, is it in the order of $n|A|$?\n* Is the Monte Carlo estimator for Eq 12 biased?\n* In structured-based drug design, there exist methods for property optimization, e.g. see [2] using an evolutionary algorithm.\n\n[1] Lu, C., Chen, H., Chen, J., Su, H., Li, C. and Zhu, J., 2023, July. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In International Conference on Machine Learning (pp. 22825-22855). PMLR.\n\n[2] Schneuing, A., Harris, C., Du, Y., Didi, K., Jamasb, A., Igashov, I., Du, W., Gomes, C., Blundell, T., Lio, P. and Welling, M., 2022. Structure-based drug design with equivariant diffusion models. arXiv preprint arXiv:2210.13695.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* Conditional generation is an important and practical problem in molecular design. In fact, previous work in this space (recent trend of diffusion-based generative models) has under-studied how to do conditional generation and molecular optimization.\n* The proposed method is straightforward and effective based on validations from experiments.", "weaknesses": "* Even though conditional generation is under-studied in the small molecular generation domain, it is well-studied in other domains (especially on exact guidance; see questions). [1,2,3]\n* Molecular optimization has been a long-standing challenge in this field and unfortunately, almost none of them are mentioned or compared, see the review [4] for references.\n\n[1] Lu, C., Chen, H., Chen, J., Su, H., Li, C. and Zhu, J., 2023, July. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In International Conference on Machine Learning (pp. 22825-22855). PMLR.\n\n[2] Wu, L., Trippe, B., Naesseth, C., Blei, D. and Cunningham, J.P., 2024. Practical and asymptotically exact conditional sampling in diffusion models. Advances in Neural Information Processing Systems, 36.\n\n[3] Zhao, S., Brekelmans, R., Makhzani, A. and Grosse, R.B., Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo. In Forty-first International Conference on Machine Learning.\n\n[4] Du, Y., Jamasb, A.R., Guo, J., Fu, T., Harris, C., Wang, Y., Duan, C., Liò, P., Schwaller, P. and Blundell, T.L., 2024. Machine learning-aided generative molecular design. Nature Machine Intelligence, pp.1-16.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mj4GsyuEJi", "reviewer": "Reviewer_U4d6", "summary": "The authors tackle the problem of training-free guidance in multi-modal generative flow. Specifically, the work builds on the recent development of \"Multiflow\", i.e. a flow-based generative model for mixed modality data (i.e. discrete and continuous). This paper describes how one can perform guidance in such models without any additional training and only requiring access to time-independent property predictor. Emprically, superiority over other training-free guidance methods is demonstrated.", "questions": "* Line 317: \"Mathematically, we expect...\". Why do we expect this? Could the authors elaborate? Is this a heuristic with desirable properties (favouring larger values of $f$) or is this more principled?\n* Line 319: \"One can show...\" I am not sure that this is true. To me this looks more like gradient ascent of $x \\to \\log f (\\mathbb{E}(X_1|X_t=x, a_t))$ than generating samples according to some distribution. Could the authors elaborate?\n* Table 1: I think that highlighting the results for TFG-Flow in bold is misleading. I was under the impression that this is the best method overall and not only among training-free methods. I think that the authors should make the rank of methods more transparent. This also goes back to the point of Weakness (2): Perhaps the training-free methods are not even faster, since we do not know the runtime of the baselines and the proposed method, I do not think this is justified to evaluate them separately (i.e. TFG-Flow should not be in bold since it is significantly worse than EEGSDE). If, however, it is much faster due to being training-free than one can argue why they might be considered in different categories.\n* Lines 470, 473:  The authors say that they train the drug quality prediction network, but then they say that there is no need to train the oracle target predictor. So what is trained and what does not need to be trained?\n* Line 479 - Is this a reasonable definition of the target? Do we want to match the quality to the reference or do we want to maximize it? The current formulation looks like it wants to match (maximized at $\\mathcal{E}(G) = c$). Why not maximize?\n* Line 1499 - why is this assumption incorrect?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* I think that the paper is very well written. The problem setup and motivation are clearly explained; the method and contributions are described in a way that is easy to understand.\n* Theoretical contributions, which are precisely stated and proven.\n* Encouraging experimental results.\n* Submitted code for improved reproducibility.", "weaknesses": "* To me, the biggest weakness lies in the following missing piece of the explanation of the \"mutlimodal flow model\". My understanding of the base model is the following. We train the model $p_{1|t}(G_1 | G_t)$, we then use it to estimate marginal velocities described in Equations (7), (9) and (10). This allows us to take a small step to estimate $G_{t + dt}$ based on $G_t$. This implicitly implies that $p_{1|t}$ is efficient to sample from. Otherwise, this procedure would be very expensive. My question is the following: \"If we know how to efficiently sample from $p_{1|t}$, why bother with the flow model at all? Why not just sample $G_0 \\sim p_0$ and directly sample $G_1 \\sim p_{1|0}(G_1 | G_0)$ ? I read the paper that authors build on [1], but I do not think this is explained properly there either. Here are follow-up questions:\n  * How is $p_{1|t}$ parametrized? I do not mean what architecture of the model, but what kind of a model? Is this a generative model? If so, what kind? (GAN, NF, VAE, sth else?).\n  * The marginal flow is given by an expectation under $p_{1|t}$. I assume that this is approximated using Monte Carlo estimation? How many samples are used? What variance does this exhibit?\n  * In lines 196-197 you say that $g$ parametrizes this model and for input $G_t$ returns \"sample/likelihood of $G_1$\". What does the slash \"/\" mean? Does it return both the sample and the corresponding likelihood? I think this narrows down the choices for $p_{1|t}$, because not all generative models can return do both: generate samples and their corresponding likelihoods.\n  * If one can efficiently sample from $p_{1|t}$ why not just generate a collection of $G_1 \\sim p_{1|t}(G_1|G_t)$ and choose the one with the highest value of our property predictor $f$? Have the authors considered such a baseline? How would this compare to the proposed method?  think that this is very closely related to what authors are actually doing implicitly (see Question 2 is the questions section below)\n* Benchmark model size/runtime comparison in Tables 1, 2 and 4. I think that the model comparison would be much more meaningful if we had the following information:\n  * What are the model sizes?\n  * What is the runtime of each model, both training and inference?\n* Lack of comparison with optimization-based structure-based drug design methods. In table 4, I would strongly suggest comparing with optimization-based method like Autogrow4. E.g. in [3] it was shown to be a very strong baseline and it makes a fair comparison, because the method you are proposing is an optimization-based method and not merely generative one like the ones used as baselines.\n\n---\n\nReferences\n\n[1] Campbell et al \"Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Design\" (ICML 2024)\n\n[2] Spiegel et al. \"Autogrow4: an open-source genetic algorithm for de novo drug design and lead optimization\" (ChemInf 2020)\n\n[3] Karczewski et al. \"WHAT AILS GENERATIVE STRUCTURE-BASED DRUG DESIGN: TOO LITTLE OR TOO MUCH EXPRESSIVITY?\" (Arxiv)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sNcCC13Kcx", "reviewer": "Reviewer_Pmbj", "summary": "In this paper, the authors present a guidance approach multi-modal continuous normalizing flows trained using flow matching. This field is currently very active, but the authors present a new formal approach to this problem, which they claim does not need any further training: Using a \"foundation model,\" (modeling p(x)) and an off-the-shelf predictive model (modeling y=f(x)), they generate conditional samples p(x|y). \nMinor comment:\nThe transport/continuity equation is repeatedly referred to as the Fokker-Planck equation.", "questions": "- One of the key contributions is Theorem 3.4, which provides a sampling estimator for the conditional rate matrix (guided discrete flow). The scaling property is good compared to the baseline, but may still be prohibitive in many applications, e.g. experimental design. How does, the variance of the estimator grow if k<K samples are used? Are there some ways to further improve on this estimator to make it useful in other application domains?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Strong formal framework. Broad applicability beyond molecular examples shown in this work.\n\t- Good results, and transparency about limitations and cases where performance is less good.\n\t- Several experiments targeting different molecular problems.", "weaknesses": "- Main condition for the paper is training free guidance, all the benchmarks shown where the results stand out, use models which are trained for the purpose guidance. \n        - Presentation suffers somewhat from being overly compact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)